2021

[1]Ali Abedi

1]KITE, University Health Network, Canada

# Affect-driven Ordinal Engagement Measurement from Video

<EMAIL_ADDRESS>Shehroz S. Khan<EMAIL_ADDRESS>[

###### Abstract

In education and intervention programs, user engagement has been identified as
a major factor in successful program completion. Automatic measurement of user
engagement provides helpful information for instructors to meet program
objectives and individualize program delivery. In this paper, we present a
novel approach for video-based engagement measurement in virtual learning
programs. We propose to use affect states, continuous values of valence and
arousal extracted from consecutive video frames, along with a new latent
affective feature vector and behavioral features for engagement measurement.
Deep-learning sequential models are trained and validated on the extracted
frame-level features. In addition, due to the fact that engagement is an
ordinal variable, we develop the ordinal versions of the above models in order
to address the problem of engagement measurement as an ordinal classification
problem. We evaluated the performance of the proposed method on the only two
publicly available video engagement measurement datasets, DAiSEE and EmotiW-
EW, containing videos of students in online learning programs. Our experiments
show a state-of-the-art engagement level classification accuracy of 67.4% on
the DAiSEE dataset, and a regression mean squared error of 0.0508 on the
EmotiW-EW dataset. Our ablation study shows the effectiveness of incorporating
affect states and ordinality of engagement in engagement measurement.

###### keywords:

Engagement Measurement, Engagement Detection, Affect States, Temporal
Convolutional Network, Ordinal Classification

##  1 Introduction

Online services, such as virtual education, telemedicine, and
telerehabilitation, offer many advantages compared to their traditional in-
person counterparts; being more accessible, economical, and personalizable.
These online services make it possible for students to complete their courses
mukhtar2020advantages  and patients to receive the necessary care and health
support nuara2021telerehabilitation  remotely. However, they also bring other
types of challenges. For instance, in an online classroom setting, it becomes
very difficult for the tutor to assess students’ engagement in the material
being taught venton2021strategies . For a clinician, it is important to assess
the engagement of patients in virtual intervention programs, as the lack of
engagement is one of the most influential barriers to program completion
matamala2020role . Therefore, from the view of the instructor of an online
program, it is important to automatically measure the engagement level of the
participants to provide them with real-time feedback and take necessary
actions to engage them to maximize their program’s objectives.

Various modalities have been used for automatic engagement measurement in
virtual settings, including user’s image delgado2021student , video
abedi2021improving , audio fedotov2018multimodal , Electrocardiogram (ECG)
belle2012automated , and pressure sensors rivas2021multi . Video
cameras/webcams are mostly used in virtual programs; thus, they have been
extensively used in assessing engagement in these programs. Video
cameras/webcams offer a cheaper, ubiquitous and unobtrusive alternative to
other sensing modalities. Therefore, majority of the recent works on objective
engagement measurement in online programs and human-computer interaction are
based on the visual data of participants acquired by cameras and using
computer-vision techniques doherty2018engagement ; dewan2019engagement ;
salam2022automatic ; khan2022inconsistencies .

The computer-vision-based approaches for engagement measurement are
categorized into image-based and video-based approaches. The former approach
measures engagement based on single images delgado2021student  or single
frames extracted from videos whitehill2014faces . A major limitation of this
approach is that it only utilizes spatial information from single frames,
whereas engagement is a spatio-temporal state that takes place over time
d2017advanced . Another challenge with the frame-based approach is that
annotation is needed for each frame ringeval2013introducing , which is an
arduous task in practice. The latter approach is to measure engagement from
videos instead of using single frames. In this case, one label is needed after
each video segment. Fewer annotations are required in this case; however, the
measurement problem is more challenging due to the coarse labeling.

The video-based engagement measurement approaches can be categorized into end-
to-end and feature-based approaches. In the end-to-end approaches, consecutive
raw frames of video are fed to variants of Convolutional Neural Networks
(CNNs) (many times followed by temporal neural networks) to output the level
of engagement abedi2021improving ; liao2021deep ; hu2022optimized ;
gupta2016daisee ; zhang2019novel . In feature-based approaches, handcrafted
features, as indicators of engagement, are extracted from video frames and
analyzed by sequential neural networks or machine-learning methods to output
the engagement level copur2022engagement ; whitehill2014faces ; liao2021deep ;
niu2018automatic ; thomas2018predicting ; huang2019fine ;
fedotov2018multimodal ; chen2019faceengage ; booth2017toward ;
kaur2018prediction ; wu2020advanced ; ma2021automatic . Various features have
been used in previous feature-based approaches, including behavioral features
such as eye gaze, head pose, and body pose. Despite the psychological evidence
for the effectiveness of affect states in user engagement d2017advanced ;
sinatra2015challenges ; khan2022inconsistencies ; altuwairqi2021new , none of
the previous computer-vision-based approaches have utilized these important
features for engagement measurement.

In this paper, we propose a novel feature-based approach for engagement
measurement from videos using affect states, along with a new latent affective
feature vector and behavioral features. These features are analyzed using
different ordinal sequential neural networks to output user engagement in
videos. Our main contributions are as follows:

  * •

We use affect states, including continuous values of user’s valence and
arousal, and present a new latent affective feature using a pretrained model
toisoul2021estimation , along with behavioral features for video-based
engagement measurement. We jointly train different models on the above
features for engagement measurement.

  * •

In view of the fact that engagement is an ordinal variable
(whitehill2014faces, ; khan2022inconsistencies, ), we approach engagement
measurement from an ordinal classification perspective and train and evaluate
ordinal classification models for engagement measurement.

  * •

We conduct extensive experiments on two publicly available video datasets for
engagement level classification and regression, focusing on studying the
impact of affect states and ordinality of the classification models on
engagement measurement. In these experiments, we compare the proposed method
with several end-to-end and feature-based engagement measurement approaches.

The paper is structured as follows. Section 2 describes what engagement is and
which type of engagement will be explored in this paper. Section 3 describes
related works on video-based engagement measurement. In Section 4, the
proposed pathway for affect-driven video-based ordinal engagement measurement
is presented. Section 5 describes the experimental settings and results on the
proposed methodology. In the end, Section 6 presents our conclusions and
directions for future works.

##  2 What is Engagement?

In this section, different theoretical, methodological categorizations, and
conceptualizations of engagement are briefly described, followed by brief
discussion on the engagement in virtual learning settings and human-computer
interaction which will be explored in this paper.

Dobrian et al. dobrian2011understanding  describe engagement as a proxy for
involvement and interaction. Sinatra et al. sinatra2015challenges  defined
”grain size” for engagement, the level at which engagement is conceptualized,
observed, and measured. The grain size ranges from macro-level to micro-level.
The macro-level engagement is related to groups of people, such as the
engagement of employees of an organization in a particular project, the
engagement of a group of older adult patients in a rehabilitation program, and
the engagement of students in a classroom, school, or community. The micro-
level engagement links with individual’s engagement at the moment, task, or
activity sinatra2015challenges . Measurement at the micro-level may use
physiological and psychological indices of engagement such as blink rate
ranti2020blink , head pose chen2019faceengage , and heart rate
monkaresi2016automated .

The grain size of engagement analysis can also be considered as a continuum
comprising context-oriented, person-in-context, and person-oriented engagement
sinatra2015challenges . The macro-level and micro-level engagement are
equivalent to the context-oriented, and person-oriented engagement,
respectively, and the person-in-context lies in between. The person-in-context
engagement can be measured by observations of a person’s interactions with a
particular contextual environment, e.g., reading a web page or participating
in an online classroom. Except for the person-oriented engagement, for
measuring the two other parts of the continuum, knowledge about context is
required, e.g., the lesson being taught to the students in a virtual classroom
sumer2021multimodal , or the information about rehabilitation consulting
material being given to a patient and the severity of disease of the patient
in a virtual rehabilitation session rivas2021multi .

In person-oriented engagement, the focus is on the behavioral, affective, and
cognitive states of the person in the moment of interaction d2017advanced .

  * •

Behavioral engagement involves general on-task behavior and paid attention at
the surface level sinatra2015challenges ; d2017advanced ; aslan2017human . The
indicators of behavioral engagement, in the moment of interaction, include eye
contact, blink rate, body pose, and hand pose.

  * •

Affective engagement is defined as the affective and emotional reactions of
the person to the content pekrun2012academic . Its indicators are activating
versus deactivating and positive versus negative emotions broughton2010nature
. Activating emotions are associated with engagement sinatra2015challenges .
While both positive and negative emotions can facilitate engagement and
attention, research has shown an advantage for positive emotions over negative
ones in upholding engagement broughton2010nature ; altuwairqi2021new .

  * •

Cognitive engagement pertains to the psychological investment and effort
allocation of the person to deeply understand the context materials
sinatra2015challenges . To measure cognitive engagement, information such as
person’s speech should be processed to recognize the level of person’s
comprehension of the context. Contrary to behavioral and affective
engagements, measuring cognitive engagement requires knowledge about context
materials.

Understanding a person’s engagement in a specific context depends on the
knowledge about the person and the context. From a data analysis perspective,
it depends on the data modalities available to analyze. In this paper, the
focus is on video-based engagement measurement. The only data modality is
video, without audio, and with no knowledge about the context. Therefore, we
propose new methods for automatic person-oriented engagement measurement by
analyzing the behavioral and affective states of the person at the moment of
participation in an online program.

##  3 Literature Review

Over the recent years, extensive research efforts have been devoted to
automatic engagement measurement doherty2018engagement ; dewan2019engagement ;
salam2022automatic ; khan2022inconsistencies , most of which focused on video-
based engagement measurement. In these approaches, computer vision, machine
learning, and deep-learning algorithms are used to analyze videos and output
the engagement level of the person in the video. Engagement measurement
approaches based on video can be classified into end-to-end and feature-based.

In end-to-end engagement measurement approaches, no hand-crafted features are
extracted from videos. The consecutive raw frames of RGB videos are fed to a
deep CNN (or its variant) classification, or regression model to output a
class label, or a continuous value corresponding to the engagement level of
the person in the video. The upper half of Table 1 summarizes the literature
on the end-to-end video-based engagement measurement approaches focusing on
their deep-learning video-analysis models. Most of the existing end-to-end
approaches evaluated their method on the DAiSEE dataset gupta2016daisee
(described in Section 5) for engagement level classification. Various
techniques, such as using different loss functions or adding attention
mechanisms to the neural networks, have been employed by researchers to
enhance the results of their proposed methodologies abedi2021improving .

Different from the end-to-end approaches, in feature-based approaches, first,
multi-modal handcrafted features are extracted from videos, and then the
features are fed to a classification or regression model to output engagement
chen2019faceengage ; doherty2018engagement ; dewan2019engagement ;
whitehill2014faces ; niu2018automatic ; thomas2018predicting ; huang2019fine ;
fedotov2018multimodal ; dhall2020emotiw ; booth2017toward ; kaur2018prediction
; wu2020advanced . The lower half of Table 1 summarizes the literature on
feature-based engagement measurement approaches focusing on their features,
feature fusion techniques, machine-learning models, and datasets.

As can be seen in the lower section of Table 1, in some of the previous
methods, conventional computer-vision features such as box filter, Gabor
whitehill2014faces , or LBP-TOP kaur2018prediction  are used for engagement
measurement. In some other methods, such as chen2019faceengage , for
extracting facial embedding features, a CNN containing convolutional layers,
followed by fully-connected layers, is trained on a face recognition or facial
expression recognition dataset. Then, the output of the convolutional layers
in the pretrained model is used as the facial embedding features. In most of
the previous methods, facial action units (AUs), eye movement, gaze direction,
and head pose features are extracted using OpenFace baltrusaitis2018openface ,
or body pose features are extracted using OpenPose cao2017realtime . Various
features are extracted and concatenated using these feature extraction
techniques to construct a feature vector for each video frame or each video
clip.

Late fusion and early fusion techniques were used in the previous feature-
based engagement measurement approaches to handle different feature
modalities. Late fusion requires the construction of independent models each
being trained on one feature modality. For instance, Wu et al. wu2020advanced
trained four models independently, using face and gaze features, body pose
features, and C3D features. Then they used a weighted summation of the outputs
of four models to output the final engagement level. In the early fusion
approach, features of different modalities are simply concatenated to create a
single feature vector to be fed to a single model to output engagement. For
instance, Niu et al. niu2018automatic , concatenated gaze, AU, and head pose
features to create one feature vector for each video clip and fed the feature
vectors to a Gated Recurrent Unit (GRU) to output the engagement level.

While there is psychological evidence for the big influence of affect states
on engagement sinatra2015challenges ; d2017advanced ; woolf2009affect ;
aslan2017human ; altuwairqi2021new , none of the previous methods utilized
affect states as the features in engagement measurement. In the next section,
we will present a new approach for video engagement measurement using affect
states along with latent affective features, blink rate, eye gaze, head pose,
and hand pose features. Despite the fact that engagement is an ordinal
variable (whitehill2014faces, ; khan2022inconsistencies, ), none of the
previous methods developed ordinal models for engagement measurement. For the
first time in the field of affective computing, we use ordinal models for
engagement level measurement.

Table 1:  (Upper section) End-to-end video-based engagement measurement
approaches, their video-analysis classification (C) or regression (R) models,
and the datasets they used. (Lower section) Feature-based video engagement
measurement approaches, their features, feature fusion techniques,
classification (C) or regression (R) models, and the datasets they used.

Ref., Year |  Features |  Fusion |  Model |  C/R |  Dataset
---|---|---|---|---|---
gupta2016daisee , 2016 |  - |  - |  InceptionNet, C3D, and LRCN |  C |  DAiSEE gupta2016daisee
geng2019learning , 2019 |  - |  - |  C3D |  C |  DAiSEE gupta2016daisee
zhang2019novel , 2019 |  - |  - |  Inflated 3D-CNN (I3D) |  C |  DAiSEE gupta2016daisee
liao2021deep , 2021 |  - |  - |  ResNet-50 with attention-based LSTM |  C, R |  DAiSEE gupta2016daisee , EmotiW-EW kaur2018prediction
abedi2021improving , 2021 |  - |  - |  ResNet with TCN |  C |  DAiSEE gupta2016daisee
hu2022optimized , 2022 |  - |  - |  ShuffleNet v2 |  C |  DAiSEE gupta2016daisee
ai2022class , 2022 |  - |  - |  Video Transformer |  R |  DAiSEE gupta2016daisee , EmotiW-EW kaur2018prediction
mehta2022three , 2022 |  - |  - |  3D DenseNet |  C, R |  DAiSEE gupta2016daisee , EmotiW-EW kaur2018prediction
whitehill2014faces , 2014 |  box filter, Gabor, AU |  late |  GentleBoost, SVM, logistic regression |  C, R |  HBCU whitehill2014faces
booth2017toward , 2017 |  facial landmark, AU, optical flow, head pose |  late |  SVM, KNN, Random Forest |  C, R |  USC booth2017toward
kaur2018prediction , 2018 |  LBP-TOP |  - |  fully-connected neural network |  R |  EmotiW-EW kaur2018prediction
niu2018automatic , 2018 |  gaze direction, AU, head pose |  early |  GRU |  R |  EmotiW-EW kaur2018prediction
fedotov2018multimodal , 2018 |  body pose, facial embedding, eye features, speech features |  early |  logistic regression |  C |  EMDC fedotov2018multimodal
thomas2018predicting , 2018 |  gaze direction, head pose, AU |  early |  TCN |  R |  EmotiW-EW kaur2018prediction
huang2019fine , 2019 |  gaze direction, head pose, AU |  early |  LSTM |  C |  DAiSEE gupta2016daisee
chen2019faceengage , 2019 |  gaze direction, blink rate, head pose, facial embedding |  early |  AdaBoost, SVM, KNN, Random Forest, RNN |  C |  FaceEngage chen2019faceengage
wu2020advanced , 2020 |  gaze direction, head pose, body pose, C3D |  late |  LSTM, GRU |  R |  EmotiW-EW kaur2018prediction
ma2021automatic , 2021 |  gaze direction, head pose, AU, C3D |  late |  Neural Turing Machine |  C |  DAiSEE gupta2016daisee
copur2022engagement , 2022 |  eye gaze, head pose, head rotation, and AU |  early |  LSTM |  R |  EmotiW-EW kaur2018prediction
proposed |  valence, arousal, latent affective features, blink rate, gaze direction, head pose, and hand pose |  joint |  Ordinal LSTM and TCN |  C, R |  DAiSEE gupta2016daisee , EmotiW-EW kaur2018prediction

##  4 Affect-driven Ordinal Engagement Measurement from Video

As discussed in Section 2, person-oriented engagement focuses on the
behavioral, affective, and cognitive states of the person at the moment of
interaction, at fine-grained time scales, from seconds to minutes
d2017advanced . Activating versus deactivating and positive versus negative
emotions are indicators of affective engagement broughton2010nature . Although
research has shown that positive emotions are more effective than negative
emotions in promoting engagement, both positive and negative emotions have the
potential to activate engagement broughton2010nature . Therefore, engagement
cannot be directly derived from high levels of positive or activating
emotions. Furthermore, it has been shown that, due to their abrupt changes,
six basic emotions, anger, disgust, fear, joy, sadness, and surprise, cannot
directly be used as reliable indicators of affective engagement
gupta2016daisee .

Woolf et al. woolf2009affect  discretize the affective and behavioral states
of persons into eight categories, and defined persons’ desirability for
learning, based on positivity or negativity of valence and arousal in the
circumplex model of affect russell1980circumplex  (see Figure 1), and on-task
or off-task behavior. In the circumplex model of affect, the positive, and
negative values of valence correspond to positive, and negative emotions, and
the positive, and negative values of arousal correspond to activating, and
deactivating emotions. Persons were marked as being on-task when they are
physically involved with the context, watching instructional material, and
answering questions, while the off-task behavior represents if the persons are
not physically active and attentive to the learning activities. Aslan et al.
aslan2017human  directly defined binary engagement state,
engagement/disengagement, as different combinations of positive/negative
values of affective states in the circumplex model of affect and on-task/off-
task behavior.

Figure 1: The circumplex model of affect russell1980circumplex . The positive,
and negative values of valence correspond to positive, and negative emotions,
respectively, and the positive, and negative values of arousal correspond to
activating, and deactivating emotions, respectively.

In this paper, we assume the only available data modality is a person’s RGB
video and propose a video-based person-oriented engagement measurement
approach. We train neural network models on affect and behavioral features
extracted from videos to automatically infer the engagement level based on the
affect and behavioral states of the person in the video.

###  4.1 Feature extraction

Toisoul et al. toisoul2021estimation  proposed a deep neural network
architecture, EmoFAN, to analyze facial affect in naturalistic conditions
improving the state-of-the-art in emotion recognition, valence and arousal
estimation, and facial landmark detection. Their proposed architecture,
depicted in Figure 2, is comprised of one 2D convolution followed by two
hourglass networks newell2016stacked  with skip connections trailed by five 2D
convolutions and two fully-connected layers. The second hourglass outputs the
facial landmarks that are used in an attention mechanism for the following 2D
convolutions.

Figure 2: The pretrained EmoFAN toisoul2021estimation  on AffectNet dataset
mollahosseini2017affectnet  is used to extract affect (continuous values of
valence and arousal) and latent affective features (a 256-dimensional feature
vector). The first blue rectangle and the five green rectangles are 2D
convolutional layers. The red and light red rectangles, 2D convolutional
layers, constitute two hourglass networks with skip connections. The second
hourglass network outputs the facial landmarks that are used in an attention
mechanism, the gray line, for the following 2D convolutions. The output of the
last green 2D convolutional layer in the pretrained network is considered as
the latent affective feature vector. The final cyan fully-connected layer
output the affect states.

Affect features: The continuous values of valence and arousal obtained from
the output of the pretrained EmoFAN toisoul2021estimation  (Figure 2) on the
AffectNet dataset mollahosseini2017affectnet  are used as the affect features.

Latent affective features: The pretrained EmoFAN toisoul2021estimation  on
AffectNet mollahosseini2017affectnet  is used for latent affective feature
extraction. The output of the final 2D convolution (Figure 2), a
256-dimensional feature vector, is used as the latent affective features. This
feature vector contains latent information about facial landmarks and facial
emotions.

Behavioral features: The behavioral states of the person in the video, on-
task/off-task behavior, are characterized by features extracted from the
person’s eye, head, and hand movements.

Ranti et al. ranti2020blink  demonstrated that blink rate patterns provide a
reliable measure of engagement with visual content. They implied that eye
blinks are withdrawn at precise moments in time so as to minimize the loss of
visual information that occurs during a blink. Probabilistically, the more
important the visual information is to the person, the more likely he or she
will be to withdraw blinking. We consider blink rate as the first behavioral
feature. The intensity of facial Action AU45 indicates how closed the eyes are
baltrusaitis2018openface . Therefore, a single blink with the successive
states of opening-closing-opening appears as a peak in time-series AU45
intensity traces chen2019faceengage . The intensity of AU45 in consecutive
video frames is considered as the first behavioral feature.

It has been shown that a highly engaged person who is focused on visual
content tends to be more static in his/her eye, head, and hand movements and
eye gaze direction, and vice versa sumer2021multimodal ; chen2019faceengage .
In addition, in the case of high engagement, the person’s eye gaze direction
is towards visual content. Accordingly, inspired by previous research
chen2019faceengage ; doherty2018engagement ; dewan2019engagement ;
liao2021deep ; whitehill2014faces ; niu2018automatic ; rivas2021multi ;
thomas2018predicting ; huang2019fine ; fedotov2018multimodal ; dhall2020emotiw
; booth2017toward ; kaur2018prediction ; wu2020advanced , eye location, head
pose, eye gaze direction, and wrist location in consecutive video frames are
considered as the behavioral features.

The features described above are extracted from each video frame and
considered as the frame-level features, including:

  * •

2-element affect features (continuous values of valence and arousal),

  * •

256-element latent affective features, and

  * •

12-element behavioral features (eye-closure intensity; x and y components of
eye gaze direction w.r.t. the camera; x, y, and z components of head location
w.r.t. the camera; pitch, yaw, and roll as head pose baltrusaitis2018openface
; x, y, and z components of wrist location w.r.t. the camera).

###  4.2 Predictive Modeling

Figure 3 shows the structure of the proposed architecture for engagement
measurement from video. The latent affective features, followed by a fully-
connected neural network, affect features, and behavioral features, are
concatenated to construct one feature vector for each video frame. The feature
vector extracted from each video frame is considered as one timestamp of a
sequential model. The sequential model analyzes the sequences of feature
vectors, and the output of the final timestamp of the sequential model,
trailed by a fully-connected layer, outputs the engagement level of the person
in the video. The fully-connected neural network after latent affective
features is jointly trained along with the sequential model and the final
fully-connected layer. The fully-connected neural network after the latent
affective features is trained to perform joint fusion and to reduce the
dimensionality of the latent affective features before being concatenated with
the affect and behavioral features. Different sequential models, including
Long Short-Term Memory (LSTM) and Temporal convolutional network (TCN)
bai2018empirical , are investigated in Section 5.

Engagement Measurement through Ordinal Classification: In ordinal
classification, the order of classes is taken into consideration in designing
and training machine-learning models kook2022deep ; cardoso2011measuring . The
approach used in this study to incorporate the ordinality of engagement level
in developing machine-learning models is as follows.

Training–The original $(K+1)$-level ordinal labels, $y=0,1,…,K$ in the
training set are converted into $K$ $y_{i}$ binary labels as follows, if
$y>i:y_{i}=0$, else: $y_{i}=1$, $i=0,1,…,K–1$. Then, $K$ binary classifiers
($C_{i}$, $i=0,1,…,K–1$) are trained with the training set and $K$ $y_{i}$
binary labels.

Testing–For ordinal classification of test samples, each binary classifier
$C_{i}$ gives a probability estimate for each test sample $x_{t}$, the
probability of being in binary class $y_{i}$, $y_{t}=y_{i}$, $i=0,1,…,K–1$.
Then, $K$ binary probability estimates are converted into one multi-class
probability of being in class $y=0,1,…,K$ as follows,

$\displaystyle p(y_{t}=k)=\left\\{\begin{array}[]{ll}1-p(y_{t}>0),&\mbox{if
}k=0\\\ p(y_{t}>k-1)-p(y_{t}>k),&\mbox{if }0<k<K-1\\\ p(y_{t}>K-2),&\mbox{if
}k=K-1\end{array}\right..$ (4)

Figure 3: The proposed architecture for engagement measurement from video, see
Section 4.2.

##  5 Experimental Results

In this section, the performance of the proposed engagement measurement method
is evaluated compared to the previous feature-based and end-to-end methods.
The classification and regression results on two publicly available video-
based engagement datasets are reported. The ablation study of different
feature sets is studied, and the effectiveness of affect states in engagement
measurement is investigated. Finally, the impact of incorporating the
ordinality of engagement in model training is investigated through comparing
the results of ordinal and non-ordinal models.

###  5.1 Datasets

The performance of the proposed method is evaluated on the only two publicly
available video-only engagement datasets, DAiSEE gupta2016daisee  and EmotiW-
EW kaur2018prediction .

DAiSEE: The DAiSEE dataset gupta2016daisee  contains 9,068 videos captured
from 112 persons in online courses. The videos were annotated by four states
of persons while watching online courses, boredom, confusion, frustration, and
engagement. Each state is in one of the four levels (ordinal classes), level 0
(very low), 1 (low), 2 (high), and 3 (very high). In this paper, the focus is
only on the engagement level classification. The length, frame rate, and
resolution of the videos are 10 seconds, 30 frames per second (fps), and 640 ×
480 pixels. Table 2 shows the distribution of samples in different classes and
the number of persons in the train, validation, and test sets. These sets are
used in our experiments to fairly compare the proposed method with the
previous methods. The results are reported on 1784 test videos. As can be seen
in Table 2, the dataset is highly imbalanced.

EmotiW-EW: The EmotiW-EW dataset has been released in the student engagement
measurement sub-challenge of the Emotion Recognition in the Wild Challenge
kaur2018prediction . It contains videos of 78 persons in online classroom
settings. The total number of videos is 262, including 148 training, 48
validation, and 67 test videos. The videos are at a resolution of 640 × 480
pixels and 30 fps. The lengths of the videos are around 5 minutes. Each video
has been annotated as having one of four engagement levels, 0.00, 0.33, 0.66,
or 1.00, where 0.00, and 1.00 indicate the person is completely disengaged,
and highly engaged, respectively. In this sub-challenge, the engagement
measurement has been defined as a regression problem, and only training and
validation sets are publicly available. We use the training, and validation
sets for training, and validating the proposed method, respectively. The
distribution of samples in this dataset is also imbalanced, Table 3.

Table 2: The distribution of samples in different engagement level classes, and the number of persons, in train, validation, and test sets in the DAiSEE dataset gupta2016daisee . level |  train |  validation |  test
---|---|---|---
0 |  34 |  23 |  4
1 |  213 |  143 |  84
2 |  2617 |  813 |  882
3 |  2494 |  450 |  814
total |  2358 |  1429 |  1784
# of persons |  70 |  22 |  20
Table 3: The distribution of samples in different engagement level values in the train and validation sets of the EmotiW-EW dataset kaur2018prediction . level |  train |  validation
---|---|---
0.00 |  6 |  4
0.33 |  35 |  10
0.66 |  79 |  19
1.00 |  28 |  15
total |  148 |  48

###  5.2 Experimental Setting

The behavioral eye and head features, and hand features (described in Section
4.1) are extracted by the OpenFace baltrusaitis2018openface , and MediaPipe
lugaresi2019mediapipe , respectively. The OpenFace also outputs the extracted
face regions from video frames. The extracted face regions of size 256 × 256
are fed to the pretrained EmoFAN toisoul2021estimation  on AffectNet
mollahosseini2017affectnet  for extracting affect features and latent
affective features 4.1.

The fully-connected neural network after the latent affective features in
Figure 3 contains two fully-connected layers with 256 × 128, and 128 × 32
neurons. The output of this fully-connected neural network, a 32-element
reduced dimension latent affective feature vector, is concatenated with the
affect and behavioral features to generate a 46 (32 + 2 + 12)-element feature
vector for each video frame and correspondingly for each timestamp of the
sequential model. We have implemented two sequential models for analyzing the
extracted features and outputting the engagement level, including LSTM and
TCN. The LSTM has two unidirectional layers with 46 × 128 and 128 × 64 layers,
followed by a fully-connected layer at its final timestamp with 64 × 4 neurons
for 4-class classification. The parameters of the TCN, giving the best
results, are as follows, 8, 128, 16, and 0.25 for the number of levels, number
of hidden units, kernel size, and dropout bai2018empirical . At the final
timestamp of the TCN, a fully-connected layer with 4 output neurons is used
for 4-class classification (in the DAiSEE dataset).

The EmotiW-EW dataset contains videos of around 5-minute length. The videos
are divided into 10-second clips with 50% overlap thomas2018predicting , and
clip-level features are extracted from each clip as follows. First, affect
states and behavioral features, described in Section 4.1, are extracted from
consecutive video frames in each clip. Then, for each clip, a 49-element clip-
level feature vector is created as follows.

  * •

4 features: the mean and standard deviation of valence and arousal values over
consecutive clip frames,

  * •

1 feature: the blink rate, derived by counting the number of peaks above a
certain threshold divided by the number of frames in the AU45 intensity time
series extracted from the input clip,

  * •

8 features: the mean and standard deviation of the velocity and acceleration
of x and y components of eye gaze direction,

  * •

12 features: the mean and standard deviation of the velocity and acceleration
of x, y, and z components of head location,

  * •

12 features: the mean and standard deviation of the velocity and acceleration
of head’s pitch, yaw, and roll, and

  * •

12 features: the mean and standard deviation of the velocity and acceleration
of x, y, and z components of wrist location.

The sequence of clip-level features is analyzed by a two-layer unidirectional
LSTM with 49 × 128 and 128 × 64 layers, followed by a fully-connected layer at
its final timestamp with 64 × 1 neurons for regression in the EmotiW-EW
dataset. In addition, a TCN with 8, 128, 16, and 0.25 for the number of
levels, number of hidden units, kernel size, and dropout bai2018empirical  has
also been implemented. At the final timestamp of the TCN, a fully-connected
layer with one output neuron is used for regression in the EmotiW-EW dataset.

The evaluation metrics are classification accuracy and Mean Squared Error
(MSE) for the regression task. The experiments were implemented in PyTorch
paszke2019pytorch  and Scikit-learn pedregosa2011scikit  on a server with 64
GB of RAM and NVIDIA TeslaP100 PCIe 12 GB GPU.

###  5.3 Results

Table 4 shows the engagement level classification accuracy of different
settings of the proposed method on the test set of the DAiSEE dataset. As
ablation study, in the upper section of Table 4, the results of different
classification models are reported using different feature sets. In both LSTM
and TCN, adding behavioral and affect features to the latent affective
features improves the accuracy, showing the effectiveness of affect states in
engagement measurement. The TCN can achieve higher accuracy than the LSTM
because of its superiority in modeling sequences of longer lengths and its
ability to retain memory of history. The lower section of Table 4 shows the
results of ordinal LSTM and ordinal TCN using all the features. As described
in Section 4.2, three binary classifiers are trained and used for inference
for the 4-level ordinal engagement level classification problem in the DAiSEE
dataset. Comparing the non-ordinal and ordinal versions of the LSTM and TCN
(using all the features) in the upper and lower sections of Table 4,
respectively, shows the superiority of the ordinal classifiers. The highest
classification accuracy (67.4%) was achieved by the ordinal TCN with all the
latent affective, behavioral, and affect features. Table 5 shows the
engagement level classification accuracy of the previous methods on the test
set of the DAiSEE dataset. According to Table 4 and 5, the proposed method
(ordinal TCN with all the features) outperforms all the previous methods.

Table 4: Engagement level classification accuracy of different settings of the proposed method on the test set of the DAiSEE dataset. LSTM: long short-term memory, TCN: temporal convolutional network. feature set |  model |  accuracy
---|---|---
latent affective |  LSTM |  60.8
latent affective + behavioral |  LSTM |  62.3
latent affective + behavioral + affect |  LSTM |  63.9
latent affective |  TCN |  60.2
latent affective + behavioral |  TCN |  62.5
latent affective + behavioral + affect |  TCN |  64.7
latent affective + behavioral + affect |  Ordinal LSTM |  65.6
latent affective + behavioral + affect |  Ordinal TCN |  67.4
Table 5: Engagement level classification accuracy of the previous methods on the DAiSEE dataset (see Section 3). method |  accuracy
---|---
C3D gupta2016daisee  |  48.1
I3D zhang2019novel  |  52.4
C3D + LSTM abedi2021improving  |  56.6
C3D with transfer learning gupta2016daisee  |  57.8
LRCN gupta2016daisee  |  57.9
DFSTN liao2021deep  |  58.8
C3D + TCN abedi2021improving  |  59.9
DERN huang2019fine  |  60.0
ResNet + LSTM abedi2021improving  |  61.5
Neural Turing Machine ma2021automatic  |  61.3
3D DenseNet mehta2022three  |  63.6
ResNet + TCN abedi2021improving  |  63.9
ShuffleNet v2 hu2022optimized  |  63.9

According to Abedi and Khan abedi2021improving  and Liao et al. liao2021deep ,
due to highly imbalanced data distribution (see Table 2), none of the previous
approaches on the DAiSEE dataset (outlined in Table 5) are able to correctly
classify samples in the minority classes (classes 0 and 1). Table 6 shows the
confusion matrices of one of the previous end-to-end approaches (a) and three
different settings of the proposed feature-based approach (b)-(d) with
superior performance on the test set of the DAiSEE dataset. Comparing Tables 6
(c) and (d) with (b) shows the effectiveness of adding affect state features
and using ordinal classification settings in the correct classification of
samples in the minority classes 0 and 1. Specifically, the ordinal setting in
(d) compared to the regular setting in (c) significantly increases the number
of samples in the main diagonal of the confusion matrix, i.e., the number of
correctly classified samples.

Table 6: The confusion matrices of (a) an end-to-end approach, ResNet + TCN
abedi2021improving  and the proposed feature-based approaches on the test of
the DAiSEE dataset, (b) (latent affective + behavioral) features + TCN, (c)
(latent affective + behavioral + affect) features + TCN, and (d) (latent
affective + behavioral + affect) features + Ordinal TCN.

Figure 4 shows the importance of affect and behavioral features using a random
forest classifier by checking their out-of-bag error chen2019faceengage . One
set of clip-level features (described in Section 5.2) is extracted from each
video sample of the DAiSEE dataset and used as the input to the random forest
for feature importance ranking. The feature importance ranking in Figure 4 is
consistent with evidence from psychology and physiology. (i) The first, and
third indicative features are the mean values of arousal, and valence,
respectively. It aligns with the findings in woolf2009affect ; aslan2017human
indicating that affect states are highly correlated with the desirability of
content, concentration, satisfaction, excitement, and affective engagement.
(ii) The second important feature is the blink rate. It aligns with the
research by Ranti et al. ranti2020blink  showing that blink rate patterns
provide a reliable measure of individual behavioral and cognitive engagement
with visual content. (iii) The least important features are extracted from eye
gaze direction. Research conducted by Sinatra et al. sinatra2015challenges
shows that to have a high impact on engagement measurement, eye gaze direction
features should be complemented by contextual features that encode general
information about the changes in visual content.

Figure 4: Features importance ranking of clip-level features using Random
Forest (see Section 5.3).

Table 7 shows the regression MSE of applying different methods to the
validation set in the EmotiW-EW dataset. All the outlined methods in Table 7
are feature-based, described in Section 3. In the proposed method, the clip-
level features (see Section 3) are extracted from video samples and used as
the input to the consecutive timestamps of the sequential models for
regression. Adding the affect features to the behavioral features reduces MSE
(second and fourth rows of Table 7), showing the effectiveness of the affect
states in engagement level regression. The best setting of the proposed
method, including the behavioral and affect features and TCN, outperforms all
the previous methods except copur2022engagement .

Table 7: Engagement level regression MSE of applying different methods on the validation set of the EmotiW-EW dataset. method |  MSE
---|---
clip-level behavioral features + LSTM (proposed) |  0.06230
clip-level (behavioral + affect) features + LSTM (proposed) |  0.0597
clip-level behavioral features + TCN (proposed) |  0.0618
clip-level (behavioral + affect) features + TCN (proposed) |  0.0508
eye and head-pose features + LSTM dhall2020emotiw  |  0.1000
DFSTN liao2021deep  |  0.0736
eye, head-pose, and AU features + TCN thomas2018predicting  |  0.0655
eye, head-pose, and AU features + GRU niu2018automatic  |  0.0671
eye, head-pose, and AU features + LSTM niu2018automatic  |  0.0427

###  5.4 An Illustrative Example

Figure 5 shows 5 (of 300) frames of three different videos of one person in
the DAiSEE dataset. The videos in (a), (b), and (c) are in low (1), high (2),
and very high (3) levels of engagement. Figure 5 (d), and (e) depict the clip-
level eye and head behavioral features, and affect features of these three
videos, respectively. As can be observed in Figure 5 (d), the behavioral
features of the two videos in classes 2 and 3 are different from the video in
class 1. Therefore, the behavioral features can differentiate between classes
2 and 3, and class 1. While the values of the behavioral features for the two
videos in classes 2 and 3 are almost identical, according to Figure 5 (e), the
mean values of valence and arousal are different for these two videos.
According to this example, the confusion matrices, and the accuracy and MSE
results in Tables 4-7, affect features are necessary to differentiate between
different levels of engagement.

Figure 5: An illustrative example for showing the importance of affect states
in engagement measurement. 5 (of 300) frames of three different videos of one
person in the DAiSEE dataset in classes (a) low, (b) high, and (c) very high
levels of engagement, the clip-level (d) behavioral features and (e) affect
features of the three videos. In (a), the person does not look at the camera
for a moment and behavioral features in (d) are totally different for this
video compared to the videos in (b) and (c). However, the behavioral features
for (b) and (c) are almost identical. According to (e), the affect features
are different for (b) and (c) and are effective in differentiation between the
videos in (b) and (c).

###  5.5 Discussion

The proposed method achieved superior performance compared to the previous
feature-based and end-to-end methods on the DAiSEE dataset, particularly in
terms of classifying low levels of engagement. However, the overall
classification accuracy on this dataset needs to be improved. This can be due
to the following two reasons. The first reason is the imbalanced distribution
of data, with a relatively small number of samples in low levels of engagement
when compared to high levels of engagement, Table 2. During training, with a
conventional shuffled dataset, there will be many training batches that do not
include samples with low levels of engagement. We tried to mitigate this
problem with a sampling strategy in which the samples of all classes were
included in each batch. By doing so, the model will be trained on samples of
all classes in each training iteration. The second reason is the annotation
problems in the DAiSEE dataset. According to our study on the videos in the
DAiSEE dataset, there are annotation mistakes in this dataset. These mistakes
are more obvious when comparing videos and annotations of one person in
different classes, as it is discussed with examples by Liao et el.
liao2021deep  and Mehta et al. mehta2022three . Obviously, the default reason
for not achieving greater classification accuracy is the difficulty of this
classification task, the small differences between videos at different levels
of engagement, and the lack of any other type of data other than video.

##  6 Conclusion and Future Work

In this paper, we proposed a novel method for objective engagement measurement
from videos of a person watching an online course. The experiments were
performed on the only two publicly available engagement measurement datasets
(DAiSEE gupta2016daisee  and EmotiW-EW kaur2018prediction ), containing only
video data. As any complementary information about the context (e.g., the
degree level of the students, the lessons being taught, their progress in
learning, and the speech of students) was not available, the proposed method
is a person-oriented engagement measurement using only video data of persons
at the moment of interaction. Again, due to the lack of context (e.g.,
students’ and tutors’ speech), their cognitive engagement could not be
measured. Therefore, we were only able to measure affective and behavioral
engagements based on visual indicators (see Section 2). We used affect sates,
continuous values of valence and arousal extracted from consecutive video
frames, along with a new latent affective feature vector and behavioral
features for engagement measurement. For the purpose of predicting engagement
as an ordinal variable, we developed ordinal classification models. The
proposed affect-driven ordinal approach achieved superior performance in terms
of classification accuracy and correctly classifying disengaged videos in the
DAiSEE dataset, as well as very low regression MSE on the validation set of
the EmotiW-EW dataset.

In future work, we plan to collect an audio-video dataset from patients in
virtual rehabilitation sessions. We will extend the proposed video-based
engagement measurement method to work in tandem with audio. We will
incorporate audio data, and also context information (progress of patients in
the rehabilitation program and rehabilitation program completion barriers) to
measure engagement at a deeper level, e.g., cognitive and context-oriented
engagement (see Section 2). In addition, we will explore approaches to measure
affect states from audio and use them for engagement measurement. To avoid any
mistakes in engagement annotation, as it was discussed in Section 5.5, we will
use psychology-backed measures of engagement khan2022inconsistencies . Another
line of our future research will focus on detecting disengagement as an
anomaly using autoencoders and contrastive learning techniques
khosla2020supervised .

Data availability
The datasets analyzed during the current study are publicly available in the
following repositories:
https://people.iith.ac.in/vineethnb/resources/daisee/index.html
https://sites.google.com/view/emotiw2020/

Conflict of Interest
The authors declare that they have no conflict of interest.

## References

  *   * (1) Mukhtar, K., Javed, K., Arooj, M., Sethi, A.: Advantages, limitations and recommendations for online learning during covid-19 pandemic era. Pakistan journal of medical sciences 36(COVID19-S4), 27 (2020)
  * (2) Nuara, A., Fabbri-Destro, M., Scalona, E., Lenzi, S.E., Rizzolatti, G., Avanzini, P.: Telerehabilitation in response to constrained physical distance: an opportunity to rethink neurorehabilitative routines. Journal of neurology, 1–12 (2021)
  * (3) Venton, B.J., Pompano, R.R.: Strategies for enhancing remote student engagement through active learning. Springer (2021)
  * (4) Matamala-Gomez, M., Maisto, M., Montana, J.I., Mavrodiev, P.A., Baglio, F., Rossetto, F., Mantovani, F., Riva, G., Realdon, O.: The role of engagement in teleneurorehabilitation: A systematic review. Frontiers in Neurology, 354 (2020)
  * (5) Delgado, K., Origgi, J.M., Hasanpoor, T., Yu, H., Allessio, D., Arroyo, I., Lee, W., Betke, M., Woolf, B., Bargal, S.A.: Student engagement dataset. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3628–3636 (2021)
  * (6) Abedi, A., Khan, S.S.: Improving state-of-the-art in detecting student engagement with resnet and tcn hybrid network. In: 2021 18th Conference on Robots and Vision (CRV), pp. 151–157 (2021). IEEE
  * (7) Fedotov, D., Perepelkina, O., Kazimirova, E., Konstantinova, M., Minker, W.: Multimodal approach to engagement and disengagement detection with highly imbalanced in-the-wild data. In: Proceedings of the Workshop on Modeling Cognitive Processes from Multimodal Data, pp. 1–9 (2018)
  * (8) Belle, A., Hargraves, R.H., Najarian, K.: An automated optimal engagement and attention detection system using electrocardiogram. Computational and mathematical methods in medicine 2012 (2012)
  * (9) Rivas, J.J., del Carmen Lara, M., Castrejon, L., Hernandez-Franco, J., Orihuela-Espina, F., Palafox, L., Williams, A., Bianchi-Berthouze, N., Sucar, L.E.: Multi-label and multimodal classifier for affective states recognition in virtual rehabilitation. IEEE Transactions on Affective Computing 13(3), 1183–1194 (2021)
  * (10) Doherty, K., Doherty, G.: Engagement in hci: conception, theory and measurement. ACM Computing Surveys (CSUR) 51(5), 1–39 (2018)
  * (11) Dewan, M., Murshed, M., Lin, F.: Engagement detection in online learning: a review. Smart Learning Environments 6(1), 1–20 (2019)
  * (12) Salam, H., Celiktutan, O., Gunes, H., Chetouani, M.: Automatic context-driven inference of engagement in hmi: A survey. arXiv preprint arXiv:2209.15370 (2022)
  * (13) Khan, S.S., Abedi, A., Colella, T.: Inconsistencies in measuring student engagement in virtual learning–a critical review. arXiv preprint arXiv:2208.04548 (2022)
  * (14) Whitehill, J., Serpell, Z., Lin, Y.-C., Foster, A., Movellan, J.R.: The faces of engagement: Automatic recognition of student engagementfrom facial expressions. IEEE Transactions on Affective Computing 5(1), 86–98 (2014)
  * (15) D’Mello, S., Dieterle, E., Duckworth, A.: Advanced, analytic, automated (aaa) measurement of engagement during learning. Educational psychologist 52(2), 104–123 (2017)
  * (16) Ringeval, F., Sonderegger, A., Sauer, J., Lalanne, D.: Introducing the recola multimodal corpus of remote collaborative and affective interactions. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–8 (2013). IEEE
  * (17) Liao, J., Liang, Y., Pan, J.: Deep facial spatiotemporal network for engagement prediction in online learning. Applied Intelligence 51(10), 6609–6621 (2021)
  * (18) Hu, Y., Jiang, Z., Zhu, K.: An optimized cnn model for engagement recognition in an e-learning environment. Applied Sciences 12(16), 8007 (2022)
  * (19) Gupta, A., D’Cunha, A., Awasthi, K., Balasubramanian, V.: Daisee: Towards user engagement recognition in the wild. arXiv preprint arXiv:1609.01885 (2016)
  * (20) Zhang, H., Xiao, X., Huang, T., Liu, S., Xia, Y., Li, J.: An novel end-to-end network for automatic student engagement recognition. In: 2019 IEEE 9th International Conference on Electronics Information and Emergency Communication (ICEIEC), pp. 342–345 (2019). IEEE
  * (21) Copur, O., Nakıp, M., Scardapane, S., Slowack, J.: Engagement detection with multi-task training in e-learning environments. In: International Conference on Image Analysis and Processing, pp. 411–422 (2022). Springer
  * (22) Niu, X., Han, H., Zeng, J., Sun, X., Shan, S., Huang, Y., Yang, S., Chen, X.: Automatic engagement prediction with gap feature. In: Proceedings of the 20th ACM International Conference on Multimodal Interaction, pp. 599–603 (2018)
  * (23) Thomas, C., Nair, N., Jayagopi, D.B.: Predicting engagement intensity in the wild using temporal convolutional network. In: Proceedings of the 20th ACM International Conference on Multimodal Interaction, pp. 604–610 (2018)
  * (24) Huang, T., Mei, Y., Zhang, H., Liu, S., Yang, H.: Fine-grained engagement recognition in online learning environment. In: 2019 IEEE 9th International Conference on Electronics Information and Emergency Communication (ICEIEC), pp. 338–341 (2019). IEEE
  * (25) Chen, X., Niu, L., Veeraraghavan, A., Sabharwal, A.: Faceengage: robust estimation of gameplay engagement from user-contributed (youtube) videos. IEEE Transactions on Affective Computing (2019)
  * (26) Booth, B.M., Ali, A.M., Narayanan, S.S., Bennett, I., Farag, A.A.: Toward active and unobtrusive engagement assessment of distance learners. In: 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 470–476 (2017). IEEE
  * (27) Kaur, A., Mustafa, A., Mehta, L., Dhall, A.: Prediction and localization of student engagement in the wild. In: 2018 Digital Image Computing: Techniques and Applications (DICTA), pp. 1–8 (2018). IEEE
  * (28) Wu, J., Yang, B., Wang, Y., Hattori, G.: Advanced multi-instance learning method with multi-features engineering and conservative optimization for engagement intensity prediction. In: Proceedings of the 2020 International Conference on Multimodal Interaction, pp. 777–783 (2020)
  * (29) Ma, X., Xu, M., Dong, Y., Sun, Z.: Automatic student engagement in online learning environment based on neural turing machine. International Journal of Information and Education Technology 11(3), 107–111 (2021)
  * (30) Sinatra, G.M., Heddy, B.C., Lombardi, D.: The challenges of defining and measuring student engagement in science. Taylor & Francis (2015)
  * (31) Altuwairqi, K., Jarraya, S.K., Allinjawi, A., Hammami, M.: A new emotion–based affective model to detect student’s engagement. Journal of King Saud University-Computer and Information Sciences 33(1), 99–109 (2021)
  * (32) Toisoul, A., Kossaifi, J., Bulat, A., Tzimiropoulos, G., Pantic, M.: Estimation of continuous valence and arousal levels from faces in naturalistic conditions. Nature Machine Intelligence 3(1), 42–50 (2021)
  * (33) Dobrian, F., Sekar, V., Awan, A., Stoica, I., Joseph, D., Ganjam, A., Zhan, J., Zhang, H.: Understanding the impact of video quality on user engagement. ACM SIGCOMM computer communication review 41(4), 362–373 (2011)
  * (34) Ranti, C., Jones, W., Klin, A., Shultz, S.: Blink rate patterns provide a reliable measure of individual engagement with scene content. Scientific reports 10(1), 1–10 (2020)
  * (35) Monkaresi, H., Bosch, N., Calvo, R.A., D’Mello, S.K.: Automated detection of engagement using video-based estimation of facial expressions and heart rate. IEEE Transactions on Affective Computing 8(1), 15–28 (2016)
  * (36) Sümer, Ö., Goldberg, P., D’Mello, S., Gerjets, P., Trautwein, U., Kasneci, E.: Multimodal engagement analysis from facial videos in the classroom. IEEE Transactions on Affective Computing (2021)
  * (37) Aslan, S., Mete, S.E., Okur, E., Oktay, E., Alyuz, N., Genc, U.E., Stanhill, D., Esme, A.A.: Human expert labeling process (help): towards a reliable higher-order user state labeling process and tool to assess student engagement. Educational Technology, 53–59 (2017)
  * (38) Pekrun, R., Linnenbrink-Garcia, L.: Academic emotions and student engagement. In: Handbook of Research on Student Engagement, pp. 259–282. Springer, ??? (2012)
  * (39) Broughton, S.H., Sinatra, G.M., Reynolds, R.E.: The nature of the refutation text effect: An investigation of attention allocation. The Journal of Educational Research 103(6), 407–423 (2010)
  * (40) Dhall, A., Sharma, G., Goecke, R., Gedeon, T.: Emotiw 2020: Driver gaze, group emotion, student engagement and physiological signal based challenges. In: Proceedings of the 2020 International Conference on Multimodal Interaction, pp. 784–789 (2020)
  * (41) Baltrusaitis, T., Zadeh, A., Lim, Y.C., Morency, L.-P.: Openface 2.0: Facial behavior analysis toolkit. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 59–66 (2018). IEEE
  * (42) Cao, Z., Simon, T., Wei, S.-E., Sheikh, Y.: Realtime multi-person 2d pose estimation using part affinity fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7291–7299 (2017)
  * (43) Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., Picard, R.: Affect-aware tutors: recognising and responding to student affect. International Journal of Learning Technology 4(3/4), 129–164 (2009)
  * (44) Geng, L., Xu, M., Wei, Z., Zhou, X.: Learning deep spatiotemporal feature for engagement recognition of online courses. In: 2019 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 442–447 (2019). IEEE
  * (45) Ai, X., Sheng, V.S., Li, C.: Class-attention video transformer for engagement intensity prediction. arXiv preprint arXiv:2208.07216 (2022)
  * (46) Mehta, N.K., Prasad, S.S., Saurav, S., Saini, R., Singh, S.: Three-dimensional densenet self-attention neural network for automatic detection of student’s engagement. Applied Intelligence, 1–21 (2022)
  * (47) Russell, J.A.: A circumplex model of affect. Journal of personality and social psychology 39(6), 1161 (1980)
  * (48) Newell, A., Yang, K., Deng, J.: Stacked hourglass networks for human pose estimation. In: European Conference on Computer Vision, pp. 483–499 (2016). Springer
  * (49) Mollahosseini, A., Hasani, B., Mahoor, M.H.: Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing 10(1), 18–31 (2017)
  * (50) Bai, S., Kolter, J.Z., Koltun, V.: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 (2018)
  * (51) Kook, L., Herzog, L., Hothorn, T., Dürr, O., Sick, B.: Deep and interpretable regression models for ordinal outcomes. Pattern Recognition 122, 108263 (2022)
  * (52) Cardoso, J.S., Sousa, R.: Measuring the performance of ordinal classification. International Journal of Pattern Recognition and Artificial Intelligence 25(08), 1173–1195 (2011)
  * (53) Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.-L., Yong, M.G., Lee, J., et al.: Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172 (2019)
  * (54) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019)
  * (55) Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al.: Scikit-learn: Machine learning in python. the Journal of machine Learning research 12, 2825–2830 (2011)
  * (56) Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D.: Supervised contrastive learning. Advances in Neural Information Processing Systems 33, 18661–18673 (2020)