*Q1: Can we induce stress and workload in a controlled manner?* We validated stress and workload manipulation of our experimental protocol using participants' self-assessments, behavioral performance, and physiological indicators of sympathetic nervous system (SNS) activation (i.e., GSR, ECG). Stress is expected to increase perceived anxiety and SNS activity, while workload increase should be reflected in increased perceived arousal and mental effort, decreased performance, as well as increased SNS activity (Verwey and Veltman, 1984; Boucsein, 1992).

*Q2: Can we train a workload classifier based on the data collected via this protocol?* To ensure that we are using a state-of-the-art workload classifier, we trained the classifier on all data, irrespective of context, as done in conventional studies. We expect a performance of about 70% as shown by Grimes et al. (2008) and Brouwer et al. (2012) under similar conditions.

*Q3: Does the classifier generalize across affective contexts, and if so, how well?* To study the effect of different affective contexts on the classification performance, we compared the results from classifiers trained in either stressful or non-stressful context and applied it then to test data from the same ("within") or the other context ("across"). We expect a higher "within" compared to "across" performance to indicate the difficulty of the classifier to generalize.

*Q4: Does training based on multiple context render the classifier resistant against changes in affective context,and if so, how resistant?* To test if the training with combined data from both affective contexts is effecting the classifier's capability to generalize, we compare the performance depending on the training context ("single," that is training on only stress or non-stress context, or "combined," that is training over contexts) and expect higher performance for a classifier trained on data from the combined contexts.

# **MATERIALS AND METHODS**

As mentioned before, we designed a protocol in which subjects had to do cognitive tasks involving two levels of mental workload, manipulated via task difficulty, while being exposed to two levels of psychosocial stress. We used the EEG signals collected with this protocol to design and assess a workload classifier across different stress conditions. This section describes in details the subjects involved, the protocol and the method to validate it, the EEGbased workload classifier used and the evaluations performed with it.

#### **PARTICIPANTS**

Twelve female and twelve male participants were recruited for our experiment. The participants were between 18 and 54 years old, with a mean age of 24*.*7 ± 7*.*9, and except four all were righthanded. Educations varied between high school degree and Ph.D., with a mean education of 3*.*1 ± 2*.*4 years after high school. To be admitted, people had to be at least 18 years, to speak the local language and to sign an informed consent. Furthermore, non-inclusion criteria were applied: bad vision, heart condition, neurological or psychological diseases, and affective troubles. Moreover, people were asked to select a time for the experiment in which they would feel alert. Finally, we asked them not to drink coffee and tea less than 2 h before the experiment.

#### **MATERIAL**

For our recordings, we used the following sensors: ElectroEncephaloGram (EEG, 28 active electrodes in a 10/20 system without T7, T8, Fp1, and Fp2), ElectroCardioGram (ECG, two active electrodes), facial ElectroMyoGram (EMG, two active electrodes), ElectroOculoGram (EOG, four active electrodes), breath belt (SleepSense), pulse (g.PULSEsensor), and a galvanic skin response sensor (g.GSRsensor). All sensors were connected and amplified with three synchronized g.USBAmp amplifiers (g.tec, Austria). The workload task was designed in the Presentation software (Neurobehavioral Systems, www*.*neurobs*.* com/presentation) and EEG signals were recorded and visually inspected with Open ViBE (Renard et al., 2010). **Figure 1** shows a participant sitting fully-wired in the experimental environment.

Subjects were first asked to sign an informed consent and to fill out three questionnaires: one assessing personal characteristics (such as gender, age and education) and form Y-A (anxiety state) and Y-B (anxiety trait) of the State-Trait Anxiety Inventory (STAI) (Spielberger et al., 1970) (see below for details). Then, all the sensors were installed and a 3 min baseline recorded. To avoid order effects, we counterbalanced the order of stress and relax condition (affective context) and 0-back and 2-back task (workload blocks), resulting in four scenarios (see **Figure 2A**). Each scenario was composed of 12 workload blocks in the stressful and 12 workload blocks in the relaxed context. The scenarios therefore begin with either relaxation or stress induction, and the workload blocks either start with the low workload (0-back) or high workload (2 back) condition. In each affective context, the subject performs, in alternating order, six times each workload condition (low/high) (6 × 2 × 2 = 24 min per block), with a short break after six tasks (i.e., after about 12 min). After each context was absolved, that is

**FIGURE 1 | A fully wired participant in the experimental environment during the relaxation induction period.**

after the induction phase and the 12 workload blocks, the STAI form Y-A questionnaire was administered again to assess the anxiety state. Finally, the sensors were removed and the participant was debriefed about the aim of the experiment.

#### *Stress and relaxation inductions*

In order to manipulate stress, we used a stress-induction protocol based on the Trier Social Stress Task (TSST) (Kirschbaum et al., 1993) and a relaxation condition using a resting phase, music and/or videos. The stress-induction protocol is composed of three parts lasting together about 15 min and it requires the participation of three people, "the committee," who are presented as being body language experts. In the first part, a member of the committee asks the subject to prepare, during 5 min, a fake job interview for a position fitting the professional profile of the subject. During the second part, the committee asks the person to do this job interview and to speak about himself for 5 min. They tell the subject that he is filmed for a future behavioral analysis and take notes during the whole interview. The committee acts as being serious and neutral/unresponsive toward the subject. The third part is a 3 min long arithmetic task (the subject has to count from 2083 to 0 by steps of 13) and to begin again at any mistake or hesitation. At the end of this protocol, in order to keep the stress level high, the committee tells the subject he will be filmed during the workload tasks and that he will have to do another interview, which will be longer, and a self-evaluation based on the recorded film material after it. Furthermore, during the experiment, participants are receiving visual feedback about their performance in the workload tasks. During the stress condition, these feedbacks have been modified to display a performance 5–10% below their actual performance. Thereby, this protocol includes psychosocial stress and uncontrollability in order to maximize the chance to trigger a stress response for all the participants (Dickerson and Kemeny, 2004). On the other hand, the goal of the relaxation induction was to create a condition (referred to as "relax" condition) in which participants would be able to relax and thus execute the workload task without the influence of additional psychosocial and psychological stressors. To allow for an effective relaxation, participants were allowed to choose between resting in silence or select music/videos that would help them to feel calm (Krout, 2007). In order to measure the level of anxiety of the subjects and thereby to validate the stress/relax manipulation, the "State Trait Anxiety Inventory" (Spielberger et al., 1970) is used. It is composed of two scales of 20 propositions each: STAI form Y-A and STAI form Y-B. STAI form Y-A score measures anxiety state and is increased when the person currently experiences psychological stress. A college student (female/male) has a mean state anxiety index of 35/36, while values higher than 39/40 have been suggested to detect clinically significant symptoms (see Julian, 2011).

#### *Workload tasks*

We used the n-back task (Kirchner, 1958) as workload task (see **Figure 2B**), as it is easy to modify workload while keeping visual stimulation and behavioral motor requirements the same. Similar to Grimes et al. (2008) and Brouwer et al. (2012), we decided for a manipulation of task-difficulty to manipulate workload. Specifically, we used 0-back (low workload) and 2-back (high workload) varieties of the n-back task, which were presented in blocks of 2 min each. In both tasks, a stream of 60 white letters appears on a black background on the screen. Each letter is presented for 500 ms, followed by an inter-stimulus interval of 1500 ms. Among these letters, 25% are targets. In both tasks, when a letter appears, the subject is asked to perform a left mouse click if this is a target letter, and a right mouse click otherwise. For the 0 back task, the low workload condition, the target is the letter "X": each time an "X" appears, the subject has to do a left click, and in all the other cases he has to do a right click. For the 2-back task, the high workload condition, the subject has to do a left click if the letter that appears is the same as the one preceding the last letter. For example, if the sequence "C A C" appeared, the second "C" would be a target. At the end of each 2-min block, the subject has to report his level of arousal (on a scale from 1 to 9) (Bradley and Lang, 1994) and the perceived effort necessary to perform the task (Rating Scale of Mental Effort—RSME, Zijlstra, 1993). Finally, a screen with his performance during the block (see section 4.3.2) appears. As mentioned before, during the stressful condition, this displayed performance is lower than the actual performance to induce additional uncertainty.

#### **PROTOCOL VALIDATION METHODS**

#### *Self-assessment data*

To investigate the effect of the psychosocial stress induction on the STAI score, we computed an ANOVA with this score in the three factor-levels "baseline," "after relaxation," and "after stress induction." To assess the effect of both stress and workload manipulation, we conducted 2 (stress) × 2 (workload) ANOVAs for the averaged-over-blocks ratings on the arousal scale of the SAM and on the RMSE.

# *Behavioral data*

To investigate the effects of the experimental manipulations on behavior, we calculated the performance per block based on the number of true positive (*TP*), true negative (*TN*), false negative (*FN*), and false positive (*FP*) responses resulting from the button presses within the n-back task (left click for targets, right click for non-targets) using the following equation: *Per f* = (*TP*+*TN*) (*TP*+*TN*+*FP*+*FN*). As for ratings, we analyzed the data in a 2 (stress) × 2 (workload) ANOVA.

# *Physiological data*

Physiological responses were analyzed with respect to heart rate (HR) and galvanic skin response (GSR). Before applying statistical methods, the GSR data was pre-processed by extracting the mean GSR value (*μ*S) for each block and then averaging these values over blocks as described above. The ECG signal was band-pass filtered between 5 and 200 Hz, applying a notch-filter 48–52 Hz to reduce power line noise, before mean HR for each of the blocks was extracted. As for the former analyses, we analyzed the data with a 2 (stress) × 2 (workload) ANOVA. We are reporting data as significant if *p <* 0*.*05 and as trend if *p <* 0*.*1. For all ANOVAs partial eta squared values (*ηp* 2) are calculated as a measure of effect size.

# **EEG SIGNAL PROCESSING**

Our system aims at estimating the level of mental workload of the user from its EEG signals. To do so, we employed a machine learning approach based on state-of-the-art algorithms developed for Brain-Computer Interfaces (BCI) technologies (Lotte et al., 2007; Blankertz et al., 2008; Ang et al., 2012). This section describes the way EEG signals were preprocessed and segmented into trials, the machine learning algorithms used as well as the approach followed for the evaluating our method (see **Figure 3** for a schematic overview of these procedures).

# *EEG preprocessing and segmentation*

We first cleaned signals from eye movements (EOG) contamination using the automatic method proposed in Schlögl et al. (2007). The EEG signals from each 2 min n-back task were

**from EEG signals. Top:** training set, aiming at identifying the relevant frequency bands (i.e., spectral filters) and channels (i.e., spatial filters), using the Filter Bank CSP and REFSF approach. **Bottom:** testing set, using the

optimized spectral and spatial filter to estimate the workload level from an unknown EEG trial. (CSP, Common Spatial Patterns; REFSF, regularized Fisher spatial filter; mRMR, maximum Relevance Minimum Redundancy; LDA, Linear Discriminant Analysis).

then divided into 60 EEG trials, i.e., one EEG trial per letter appearance. More precisely, each EEG trial was defined as starting at a letter appearance onset and ending 2 s later, i.e., just before the next letter appearance. This resulted in 60 EEG trials per task, i.e., 720 trials per workload level (360 trials in the stressful condition, 360 in the non-stressful condition). Among them, trials corresponding to target letters were discarded in order to avoid confounding and interfering effects that may result from Event Related Potentials (ERP—notably a P300) likely to be triggered by target identification. This left 540 trials per workload levels (270 trials per psychosocial stress condition).

### *Machine Learning algorithms*

In order to estimate workload levels from EEG signals, we investigated two different types of neurophysiological information: (1) oscillatory activity and (2) Event Related Potentials (ERP), both of which having been shown to be useful for such a task (Brouwer et al., 2012). We set up state-of-the-art signal processing pipelines in order to estimate workload using these two types of information, both individually and in combination (see **Figure 3**). They are described below:

*Oscillatory activity.* To classify low mental workload vs. high mental workload in EEG signals based on oscillatory activity, we used a variant of the Filter Bank Common Spatial Patterns (FBCSP) algorithm (Ang et al., 2012) in order to learn optimal spatial and spectral features, i.e., EEG frequency bands and channels. The FBCSP is one of the most efficient algorithms to extract spatio-spectral features from EEG signals. It was indeed the algorithm used by the winners of the last BCI competition on all EEG data sets (Ang et al., 2012; Tangermann et al., 2012), showing the superiority of this method over other approaches. The FBCSPbased approach we employed works as follows. The first step—the training step—consists in identifying the most relevant frequency bands (i.e., spectral filters) and EEG channels (i.e., spatial filters), using examples of EEG signals from the high and low workload conditions (see below for details on the definition of the training sets). To do so, we first filter each training EEG trial into multiple frequency bands using a bank of band-pass filters. Here we used band-pass filters in the following frequency bands, which correspond to classical EEG rhythms: *δ* (1–4 Hz), *θ* (4–8 Hz), *α* (8–12 Hz), *β* (12–30 Hz), *γ* (30–47 Hz), and high *γ* (53–90 Hz). Then for each of these bands, the band-pass filtered EEG trials are used to optimize spatial filters, i.e., linear combinations of the original EEG channels. These spatial filters are optimized using the Common Spatial Pattern (CSP) algorithm (Blankertz et al., 2008), which finds the optimal channel combination such that the power of the resulting spatially filtered signals is maximally discriminant between the two conditions (here, low and high workload). We optimize 12 (6 pairs) such CSP filters for each frequency band. Then, the power of the spectrally and spatially filtered EEG signals is used as features, resulting in each EEG trial being described by 72 features (12 CSP filters × 6 frequency bands). From these 72 features, the 18 most relevant ones are selected using the maximum Relevance Minimum Redundancy (mRMR) feature selection algorithm (Peng et al., 2005). This amounts to selecting the 18 most relevant pairs of spectral and spatial filters. Finally, the 18 selected power features are used to train a shrinkage Linear Discriminant Analysis (LDA) classifier (Blankertz et al., 2010; Lotte and Guan, 2010) to discriminate low workload EEG trials from high workload ones. This concludes the training step. For testing, i.e., to predict the workload level of a given EEG trial, the EEG signals are first filtered using the 18 selected pairs of spectral and spatial filters, then the power of the resulting signals is computing and given as input to the previously trained LDA classifier whose output indicates the workload level (high or low).

*Event related potentials.* To classifiy low mental workload vs. high mental workload in EEG signals based on ERP, we first bandpass filtered the signals between 0.5 and 16 Hz, and downsampled them to 36 Hz, to reduce the signal dimensionality. We only used the first second of EEG signals from each trial (i.e., the first second after letter presentation in the N-back task) to analyse ERP, i.e., 36 samples per channels. Then, based on these 1-second of EEG signals from the training set, we learned optimal spatial filters for the discrimination of ERP based on EEG samples, by using the Fisher Spatial Filters (FSF) proposed by Hoffmann et al. (2006). We extracted 6 such spatial filters, which resulted in 216 features (6 filters × 36 EEG samples per filter), using a regularization parameter *λ* = 0*.*4 for optimizing the FSF for all subjects. We finally selected 18 features (i.e., 18 EEG samples) out of these 216 initial ones, using mRMR feature selection. These 18 selected features were used to train a shrinkage LDA. For testing, the EEG signals were preprocessed in the same way (i.e., band-pass filtered in 0.5–16 Hz and downsampled to 36 Hz), spatially filtered using the 6 Fisher Spatial Filters optimized during training, and the 18 resulting selected features were used as input to the previously trained LDA classifier whose output indicates the workload level (high or low).

*Combination of oscillatory activity and ERP.* In order to combine both oscillatory activity and ERP information, we extracted 18 FBCSP features as described above and 18 ERP features, as described above as well, from each trial. These 36 features were concatenated into a single feature vector, which was used as input to a shrinkage LDA classifier.

# *Evaluation scheme*

The performance of our workload-level estimator was assessed using sixfold stratified Cross-Validation (CV), separately for each subject. This means the data from each subject was divided into six parts, each part containing the same number of trials from each class (high/low workload). Five of these parts were used for training, i.e., to identify the relevant spectral and spatial filters, as well as to train the LDA classifier. The 6th part was used for testing the resulting workload-level estimator for that subject. This process was repeated six times, with each part used exactly once as the testing set. For three subjects we used only three- and fourfold CV due to missing blocks in the end of the recording. The performance, here the classification accuracy (i.e., rate of trials with correctly estimated workload-level), hence obtained on each testing part are then averaged to give a final performance of the workload-level estimator for that subject.

The goal of our work is to design a generic workload-level estimator, usable in practice, i.e., that can work across different affective contexts (here, different psychosocial stress levels). To do so, we performed different evaluations to estimate (1) the general performance of our system, independently of the affective context; (2) how it behaves *within* a given affective context; (3) how it behaves *across* different affective contexts, i.e., can a workloadlevel estimator calibrated on data from a given affective context (e.g., a relaxed condition) be used to estimate workload in another affective context (e.g., a stressful condition), (4) if effects of time can explain across-context classification performance loss, and (5) whether calibrating our system with data from different affective contexts makes the system better or worse, even if used in a single affective context. Different sub-parts of the data were thus used for training and testing within our CV scheme, in particular:

affective contexts, we devised an analysis similar to the above two analyses, but with first and second half of each context instead of relax and stress context. Therefore, we trained our classifiers on the data of 4 blocks and tested them on 2 blocks from either the same or the other half of the context. This was done in a threefold cross-validation scheme and resulted in two within and two across classification performance values (one from 1st half to second half, and one backwards) for each affective context. These were averaged over the affective contexts and yielded one value for the workload classification accuracy for within- and across-context (i.e., "half") per participant per half2 . For a genuine effect of affective context instead of an effect of simply the time passing between both contexts, the "within vs. across halfs" performance loss for a classifier that was only trained on one half should be smaller compared to the loss between "within vs. across affective context" performance loss for a classifier that was only trained on one affective context.

5. **Calibration across affective context performance estimation:** When considering different affective contexts, an interesting question is whether using data from different contexts to calibrate the workload-level estimator will make it better or worse, notably as compared to the within affective context evaluation. Indeed, on the one hand, using data from different contexts can force the machine learning approach to identify workload indices that are invariant to the affective context, thus improving the system, but on the other hand it adds more noise and variability to the data, which can impede the machine learning process. Therefore, with this evaluation, in each fold of the cross-validation, 20 blocks were available for training, coming from both the relaxed and stressful condition, and 2 blocks were available for testing, coming from either the stressful or the relaxed condition (but not both). To ensure that the comparison of this approach with the withincontext approach is fair, we had to use the same number of training trials for each approach. Indeed, using all the trials available in the 20 training blocks would mean using more training trials than in the within-context evaluation, which could result in higher performance simply due to a larger number of training trials. Therefore, for this last evaluation, we randomly selected 6 blocks from each context for training, from 4 of which all trials were used, while we selected every other trial from the remaining 2 blocks to keep the workload classes balanced within context. Further two blocks were selected from each context for testing. This procedure was repeated six times for a cross-validation comparable to the within-/across context evaluation.

# **RESULTS**

In this section, we first present the validation analysis, suggesting that our protocol indeed induced different levels of workload and stress (Q1). Then the results of the EEG-based workload classification over, within, and across affective contexts are presented, showing that a state-of-the-art subject-specific workload

2For three subjects, the averaging only contained data from the stress context due to missing blocks in the 2nd half of the relax context.

classifier (Q2) has difficulty generalizing over affective contexts (Q3), but can be rendered less context-sensitive by calibration across affective contexts (Q4).

#### **VALIDATION OF THE PROTOCOL**

#### *Subjective indicators*

Each subject filled in three "STAI form Y-A" (state) questionnaires: one at the beginning (*STAIBL*) of the experiment and one in the end of each affective context, that is after performing the n-back tasks under stress or relax condition (stress: *STAIS*; relax: *STAIR*) (see **Figure 4A**). Three data sets were excluded due to incompleteness. A repeated-measures ANOVA (*N* = 21) with the factor levels "baseline," "stress," and "relax" showed a significant difference of perceived anxiety between the conditions [*F*(2*,* 20) = 3*.*6225, *p <* 0*.*05, *ηp* 2 = 0*.*108]. We conducted a *post hoc* analyses using paired *t*-tests with the hypothesis that subjectively perceived anxiety increases due to the stress induction procedure relative to baseline and relaxation condition. The results suggest that the stress-induction protocol indeed increases anxiety compared to baseline and relaxation condition, and keeps it significantly higher until measured in the end of the affective context (see **Figure 4A**): *STAIS* scores (mean = 37*.*5 ± 12*.*6) are significantly higher [*t*(20) = 2*.*87, *p* = 0*.*01] than *STAIBL* scores (mean = 30*.*1 ± 4*.*6) and they are significantly higher [*t*(20) = 2*.*37, *p* = 0*.*028] than *STAIR* scores (mean = 32*.*2 ± 8*.*6). This increased anxiety seems mainly due to the interview and the apprehension of a final evaluation, rather than due to the n-back task as such: we found no difference between *STAIR* and *STAIBL* [*t*(20) = 1*.*27, *p* = 0*.*22], that is when they performed the n-back tasks knowing that there would be no evaluation.

We furthermore asked the subjects after each block to rate their arousal on the respective scale of the Self-Assessment Maneken (see **Figure 4B**) and to rate the mental effort on the Rating Scale Mental Effort (see **Figure 4C**). Two data sets were excluded due to incompleteness. We submitted the data of each scale to a 2 (stress) × 2 (workload) repeated-measures ANOVA. Regarding the subjectively perceived arousal, we only found a main effect of the workload manipulation [*F*(1*,* 21) = 4*.*444, *p* = 0*.*047, *ηp* 2 = 0*.*175] with higher perceived arousal for the 2-back task (mean = 4*.*7 ± 1*.*4) compared to the 0-back task (mean = 4*.*3 ± 1*.*7). Regarding the subjectively perceived workload, we only found a main effect of the workload manipulation [*F*(1*,* 21) = 63*.*216, *p <* 0*.*0001, *ηp* 2 = 0*.*751] with higher perceived effort for the 2 back task (mean = 48*.*1 ± 11*.*5) compared to the 0-back task (mean = 28*.*6 ± 12*.*9).

#### *Objective indicators*

For the analysis of the objective indicator of behavioral performance, we logged all responses and computed the task accuracy for each task block (see **Figure 5A**). Two data sets were excluded due to incompleteness. We submitted the accuracy to a 2 (stress) × 2 (workload) repeated-measures ANOVA. As for the subjective indicators of perceived arousal and effort, we found a main effect of the workload manipulation [*F*(1*,* 21) = 65*.*251, *p <* 0*.*0001, *ηp* 2 = 0*.*757] with higher accuracy for the simple 0-back task (mean = 97*.*3 ± 2*.*0) compared to the hard 2-back task (mean = 91*.*1 ± 4*.*8).

As a further objective indicator, we computed skin conductance level and heart rate. Four data sets were excluded due to incompleteness. For heart rate analysis a further data set was excluded due to malfunctioning sensors. We submitted the data of the physiological signals to a 2 (stress) × 2 (workload) repeated-measures ANOVA. For GSR (see **Figure 5B**), we found an increase of the skin conductance level [*F*(1*,* 19) = 4*.*4806, *p* = 0*.*048, *ηp* 2 = 0*.*191], indicating higher sympathetic arousal during the stress condition (mean = 3*.*83 ± 2*.*05) compared to the relax condition (mean = 3*.*52 ± 2*.*07). Skin conductance level increased for high compared to low workload condition as well, however, not significantly. For HR (see **Figure 5C**), we found a trend toward an increase of the heart rate [*F*(1*,* 18) = 3*.*2123, *p* = 0*.*089, *ηp* 2 = 0*.*151], indicating higher sympathetic arousal during the stress condition (mean = 79*.*41 ± 10*.*23) compared to the relax condition (mean = 78*.*30 ± 10*.*08). More importantly, we found a highly significant effect of the workload manipulation on

HR [*F*(1*,* 18) = 36*.*1431, *p <* 0*.*0001, *ηp* 2 = 0*.*667], with a higher HR for the more challenging 2-back task (mean = 80*.*4 ± 9*.*89) compared with the relatively easy 0-back task (mean = 77*.*27 ± 10*.*19).

In summary, we found evidence for the validity of the stress and workload induction (Q1) in both, the subjective (questionnaires) and objective (performance and physiological sensors) measures. This ensures that calibrating and evaluating a workload classifier on the EEG recorded with this protocol is meaningful.

#### **CLASSIFICATION OF EEG**

#### *General performance estimation*

In this section we report the general classification performance for a training on the whole data set, showing that our setup is stateof-the-art compared to similar studies hence positively answering question Q2. Specifically, we obtained performances similar to the best performances that were presented more recently with the n-back task paradigm and with 2 s short trials by Grimes et al. (2008) and Brouwer et al. (2012). The data of two participants was excluded due to incompleteness and of another one due to malfunctioning EEG sensors.

For the training and testing on the basis of all available data, those trials recorded during stress *and* relax context, we achieved an average classification accuracy of 76.1% when using only frequency-domain features, with performances between 58.7% and 95.4% (see **Figure 6**). According to Müller-Putz et al. (2008), we determined the above chance-level performance via a binomial test. For a two-class problem and given the number of 1080 trials used in our sixfold cross-validation scheme, the chance-level is at 53.1% for *p* = 0*.*05. Consequently, the classification performance was above chance for each subject, with a highly significant better-than-random performance for the average result over all subjects (*p* -0*.*0001).

Subsequently, we tested the previously observed increase of performance for increasing decision intervals, that is when more data is available for testing (Grimes et al., 2008; Brouwer et al., 2012). A majority vote over the classifier decisions for all 45 relevant trials of a given block, using only frequency-domain features, leads to an accuracy of 96%, well over the 71% chance-level resulting from a binomial test on the basis of 24 decisions (one per block). For time-domain features, we observed an average accuracy of 74% for 2 s trials (of which only the first was used), and 96% for the judgement after 45 trials. For both feature varieties in combination, the 2-second accuracy was the highest with 80.4%, though the block-wise accuracy was only 94.4%. Since all accuracies are well over chance level the used classification schemes enable for a solid classification performance for all feature varieties with the combined frequency- and time-domain features performing best for short estimation intervals and separate feature varieties performing best for the long decision intervals.

From a scientific point of view it is necessary to know about the source of the classification performance: is the information of neural origin or is it derived from muscular activity that is known to contaminate higher frequency bands of the EEG (Goncharova et al., 2003)? Although this question is often eluded in previous works (Grimes et al., 2008), we tried to answer it by first computing the percentage of the features selected from each frequency band in the FBCSP algorithm. As **Figure 6** indicates, the majority (about 65%) of features selected with the mRMR feature selection algorithm employed came from lower frequency bands (i.e., delta, theta, alpha). However, the remaining 35% originated in high frequency bands, those over 12 Hz (beta, gamma, gamma2). To ensure that the classifier performance does rely on neuronal sources and not on muscle activity, we repeated the workload classifier evaluation excluding these potentially contaminated high frequency bands, both for training and testing. We achieved a somewhat lower, but again much better-than-random (*p* - 0*.*0001) classifier performance of 74.2%, with accuracies between 53.9% and 88.2%. This suggests that our workload classifier does rely mostly on neural information from low frequency bands.

#### *Within- vs. across-context estimation*

In this section we tested the generalization of the classifier to a different affective context (question Q3). To evaluate the effects of testing in dependence of training context, we conducted a 2 (training context: relax, stress) × 2 (testing context: same-astraining, different-from-training) repeated-measures ANOVA for each feature type. **Figures 7**, **8** depict the average classifier performance when tested within and across affective context and the average loss of performance for the three used feature varieties (and the loss for the specific frequency bands), respectively.

The main effect found for the testing context when using frequency-domain features alone [*F*(1*,* 20) = 5*.*610, *p* = 0*.*028, *ηp* 2 = 0*.*219] shows that the transfer from one context to another is problematic and results in a decrease of classifier performance (mean = 69*.*4 ± 9*.*7%) compared to testing on the same context as for the training (mean = 72*.*4 ± 9*.*4%). An exploratory analysis of the effect of context change on classifiers using only specific frequency bands revealed a significant contribution of the low frequency bands to the performance decline, while the less relevant high frequency bands were not or only minimally contributing (see **Figure 8**).

For time-domain features alone, the decrease of classifier performance for across context is as well significant, though stronger [*F*(1*,* 20) = 21*.*002, *p <* 0*.*001, *ηp* 2 = 0*.*512], with a lower across-context classification performance (mean = 69*.*1 ± 5*.*5%) compared to within-context classification performance (mean = 73*.*3 ± 5*.*1%).

For frequency- and time-domain features combined, the decrease of classifier performance across-context (mean = 73*.*2 ± 8*.*8%) compared within-context (mean = 77*.*3 ± 7*.*9%) is as well marked [*F*(1*,* 20) = 12*.*104, *p* = 0*.*002, *ηp* 2 = 0*.*377].

To rule out that the differences between within-context and across-context training were caused by the time passing between affective contexts, we divided each context into two parts (1st half,

**performance of a classifier trained in different training contexts (relax, stress, combined) and tested on data from relax and stress context.** The differences between the testing performance for stress and relax context show generalize to another context. The higher performance for the combined training set relative to the training on data from a single context indicates a gain of the classifier in invariance and hence a protection against over-fitting.

2nd half) and trained and tested the classifiers in the same manner as done for the within (e.g., training and test on 1st half) and across affective context (e.g., training on 1st half and test on 2nd half) tests. With the data averaged over affective contexts, we conducted a 2 (training context: 1st half, 2nd half) × 2 (testing context: same-as-training, different-from-training) repeated-measures ANOVA for each feature type. We did not find the pattern of performance loss that we observed for within vs. across affective context testing. Surprisingly, the only effect we found was a increase of performance for across vs. within context (half) testing for the frequency-domain only feature variety [*F*(1*,* 20) = 5*.*142, *p <* 0*.*04, *ηp* 2 = 0*.*204] from 61.1% to 63.7%.

Summarizing, all feature varieties have been found susceptible to changes in affective context. For the frequency-domain features, only classifiers using the low frequency bands of delta, theta and alpha are significantly declining in performance when tested in an affective context different from the training context (see **Figure 8**). However, as we showed, these frequency bands are the most informative regarding the workload level. An additional test of the within vs. across effects between the 1st and 2nd half of the affective contexts on classifier performance showed that the time effect alone does not lead to a consistent decrease of performance.

#### *Across-context calibration*

To evaluate the use of a combined training context to increase the capability of the classifier to generalize over affective contexts (question Q4), we conducted a 2 (training context: average single, combined) × 2 (testing context: stress, relax) repeatedmeasures ANOVA for each feature type. The specific effects of across-context calibration in comparison to single context (stress and relax) calibration are depicted in **Figure 7**.

The main effect of the training context for frequency-domain features alone [*F*(1*,* 20) = 6*.*816, *p* = 0*.*017, *ηp* 2 = 0*.*254] indicates a higher performance for training with combined (mean =

72*.*4 ± 9*.*5%) vs. with single affective context (mean = 70*.*9 ± 9*.*3%). There is no significant difference between testing on the (optimal) same context vs. combined testing.

For time-domain features the increase of classifier performance between single (mean = 71*.*2 ± 5*.*2%) and combined context (mean = 72*.*1 ± 4*.*9%) training is as well significant [*F*(1*,* 20) = 6*.*703, *p* = 0*.*017, *ηp* 2 = 0*.*251]. Despite the observed increase due to training with combined data from both contexts, there is still a significant decrease of performance of about 1.2% relative to training and testing on the same context [*t*(20) = −3*.*526, *p <* 0*.*01].

For frequency- and time-domain features combined, we observed an increase of classifier performance between single (mean = 76*.*7 ± 7*.*6%) and combined context training (mean = 75*.*2 ± 8*.*1%) with [*F*(1*,* 20) = 6*.*306, *p* = 0*.*021, *ηp* 2 = 0*.*240]. There is no difference between testing on the (optimal) same context vs. combined testing.

Summarizing, for those classifiers trained with frequencydomain and combined frequency- and time-domain features, training on combined contexts leads to an increase of performance comparable with (optimal) same context training and testing. For classifiers trained with time-domain features only, we observe a significant increase of classification performance when training on combined context, but there is still a loss of performance compared to the (optimal) same context training and testing. Since the number of trials for both conditions are kept equal, this is evidence for a gain in resilience of the workload classifier against contextual changes, especially for classifiers based on frequency-domain features.

#### **DISCUSSION**

If we want to create passive brain-computer interfaces that work in the wild, we need to take the variability of such environments into account. To test how well a workload classifier would be able to cope with variability due to changes in affective context, we trained it on the data from a subject performing a task under the evaluative pressure of an impending interview, the same subject in a non-stressful setting, and from both contexts.

We validated the experimental protocol using subjective and objective indicators of the psychophysiological activation expected due to stress/relaxation induction and different workload levels. Though we did not see a significant difference in the perceived arousal measure (SAM), higher values for the STAI and increased sympathetic nervous system activity (as indicated by significant differences for GSR and a trend for HR) support a successful induction of anxiety in the stressful compared to the non-stressful condition. Higher perceived arousal and mental demand, higher sympathetic nervous system activity (as indexed by HR) as well as lower behavioral performance for high compared to low workload levels support the efficacy of the workload induction paradigm.

We showed that workload can be classified on the basis of 2 s of neurophysiological signals with an accuracy of 76.1%. This is comparable to previously reported results for such short intervals of data (Grimes et al., 2008; Brouwer et al., 2012). It was shown that the accuracy can be increased using decision-level fusion over the results of several trials (Brouwer et al., 2012) or simply by using longer signal epochs (Grimes et al., 2008), however, with the tradeoff of a less fine-grained, more discrete, and lagging measure of workload. We observed a similar increase of classifier performance to between 94.4% and 96% using a majority vote based on the classifier outcome of the relevant 45 trials of a given block.

While the source of information measured via EEG, neuronal or myographical, might seem of no immediate significance for an application on able-bodied users, it seems relevant to us to ensure that we indeed measure the neural activity implied by pBCI. In this regard, it is noteworthy that the distribution of relevant frequencies vary between subjects. While in general the majority of features (65%) is selected from low frequency bands (delta, theta, alpha), some subjects have a strong contribution of high frequencies (beta, gamma, gamma2) up to 50%. Since these higher frequency bands are notorious for their response to muscle activity in addition to neuronal information (Goncharova et al., 2003), we tested if the workload classification would suffer considerably when excluding them from the feature pool. The average performance did indeed decrease slightly to 74.2%. However, the highly significant above-chance performance over all subjects indicates an only marginal role of muscular activity in workload estimation3 . This is in line with other studies that suggest a relevance of low frequency bands for workload (Jensen et al., 2002; Jensen and Tesche, 2002) and its estimation (Zarjam et al., 2013). Consequently, we showed that the trained classifier uses the neural correlates of workload to discern two workload levels with a performance equaling that reported in similar studies.

Regarding the classifier generalization to different affective contexts, we show that a classifier created in a non-stressful context can generalize to a stressful context and vice versa. However, the training context has a significant influence on the classification performance, with decreasing performance for cross-context classification (i.e., from 72.4% to 69.4% for frequency-domain features, from 73.3% to 69.1% for timedomain features, and from 77.3% to 73.2% for features from both domains). Interestingly, we found that a training which takes several relevant contexts into account enables the generalization of the classifier to a certain degree. Classifiers based on frequencydomain and on combined frequency- and time-domain features perform comparably well after training with data from both affective context (72.4% and 76.7%, respectively) as after being trained and tested within a specific context. Classifiers based on timedomain features profit as well from a training with data from both affective contexts (72.1%), but still show a declined performance relative to optimal, within-context training and testing.

The current study is limited in its generality by the use of a stress induction paradigm which manipulates affective context only once. We chose the TSST because it is a recognized standard of social stress induction and a powerful elicitor that allows to keep stimuli and task comparable during the workload session of stressful and non-stressful condition. However, since we have only two stress conditions and not several interleaved stress conditions, the stress manipulation is synonymous with a change in time, though with a counter-balanced order. Both affective contexts are separated by at least 10 min and we can not exclude that signal changes with time played a role for classifier performance. The analysis of effects of time within the affective contexts, however, did not reveal general performance decreases due to time passing and thus adds to the evidence of context-related performance loss. Similarly, the spread of training blocks over a larger time in combined compared to single testing contexts limits comparability of both performance measures. To ensure that our results hold for stress in specific, interleaved stress induction methods can be used, though a viable experiment length, reliability of stress induction, and comparability of stimuli and task need to be guaranteed.

Another limitation of the paradigm can result from a potential interaction of (psychosocial) stress and workload. For example, impaired cognitive processes or increased engagement in the face of evaluative pressure, could lead to differences in participant performance between affective contexts (Eysenck and Derakshan, 2011). Despite the lack of such interaction effects in our analysis, the possibility of participant's performance-related differences being reflected in brain activity is a general issue that needs to be considered, since such changes in brain activity would be only indirectly related to stress. Therefore, future research needs to identify the processes that are responsible for the signal variability in the face of psychosocial stress. On a related note, other stressors could be manipulated to identify the source of the performance decrease, for example in terms of impaired cognitive processes.

The result of our study suggests that classification performance for passive BCIs can be increased using not only a larger quantity of training data, but by introducing qualitative variations. Here, we varied the stress level of our participants during the task performance. This manipulation is comparable to the variation of the affective context of a task in real-world scenarios, for example task performance under pressure vs. normal task performance. Consequently, to create more reliable BCIs for

3Alternatively, the decrease might be due to the removal of relevant neural information represented in beta or gamma bands.

workload detection, robust against alterations in contextual conditions, such as affective factors (emotions, moods), the training data should include data collected under the relevant contextual conditions.

Zander and Jatzev (2012) found that certain metrics might enable the identification of phases of changed contexts and therefore identify phases were additional calibration might be necessary. One could then use transfer learning (Pan and Yang, 2010) or other re-calibration strategies to enable an adaptation of the transfer algorithm to the new context. However, the suggested metric specifically enables the detection of LOC, which is useful for the detection of perceived LOC and subsequent reliability decrease of active BCIs when environmental and internal factors of the user change. Passive BCIs are not directly related to a feeling of control since they do not enable nor aim at the intentional control of machines. Therefore, for passive BCI one needs other indicators of reliability.

Currently, several groups are investigating the cognitive, affective, and demographic factors that influence active BCI performance (see Lotte et al., 2013). We argue that a similar research program would allow to build more robust passive BCIs by (1) taking into account changes in relevant contextual factors (e.g., stress), (2) by exploring indicators of such changes or the subsequent loss of reliability, and (3) by the exploration of strategies to update the classifier in face of the loss of reliability due to contextual changes.

#### **CONCLUSION**

The current work has relevance for the development of passive brain-computer interfaces that are able to specifically classify one psychophysiological construct (e.g., workload), while being invariant to others (e.g., stress). We devised and validated a protocol to test the effect of stress on pBCI approaches. We showed that a classifier has trouble transfering from stressful training data to non-stressful test data and vice versa, indicating an influence of affective task context on the performance of a workload classifier. Moreover, we found that the classification profits from the training on a mix of the varied affective task contexts. Such classifiers perform comparably well to those trained and tested on the same affective context. More generally spoken, the results suggest that the classification performance is not only dependent on quantitative factors, such as the numbers of channels, amount of training data, or length of trials, but also on qualitative factors, such as the affective context. This underlines the need for studies that identify such contextual factors and that elucidate ways to deal with detrimental effects related to their influence. Future research and development of workload classification systems using physiological sensors needs to take the contextual factors into account to increase the generality and ecological validity of the system.

#### **REFERENCES**

Kirchner, W. K. (1958). Age differences in short-term retention of rapidly changing information. *J. Exp. Psychol.* 55, 352–358. doi: 10.1037/h0043688

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 31 January 2014; accepted: 30 April 2014; published online: 12 June 2014. Citation: Mühl C, Jeunet C and Lotte F (2014) EEG-based workload estimation across affective contexts. Front. Neurosci. 8:114. doi: 10.3389/fnins.2014.00114*

*This article was submitted to Neuroprosthetics, a section of the journal Frontiers in Neuroscience.*

*Copyright © 2014 Mühl, Jeunet and Lotte. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Modeling temporal sequences of cognitive state changes based on a combination of EEG-engagement, EEG-workload, and heart rate metrics

*Maja Stikic 1 \*, Chris Berka1, Daniel J. Levendowski 1, Roberto F. Rubio1, Veasna Tan1, Stephanie Korszen1, Douglas Barba2 and David Wurzer 2*

*1 Advanced Brain Monitoring Inc., Carlsbad, CA, USA*

*2 Center for Performance Psychology, National University, Carlsbad, CA, USA*

#### *Edited by:*

*Anne-Marie Brouwer, Netherlands Organisation for Applied Scientific Research, Netherlands*

#### *Reviewed by:*

*Lauren E. Reinerman-Jones, University of Central Florida, USA Ali Bahramisharif, Radboud University Nijmegen, Netherlands*

#### *\*Correspondence:*

*Maja Stikic, Advanced Brain Monitoring Inc., 2237 Faraday Avenue, Suite 100, Carlsbad, CA 92008, USA e-mail<EMAIL_ADDRESS>
The objective of this study was to investigate the feasibility of physiological metrics such as ECG-derived heart rate and EEG-derived cognitive workload and engagement as potential predictors of performance on different training tasks. An unsupervised approach based on self-organizing neural network (NN) was utilized to model cognitive state changes over time. The feature vector comprised EEG-engagement, EEG-workload, and heart rate metrics, all self-normalized to account for individual differences. During the competitive training process, a linear topology was developed where the feature vectors similar to each other activated the same NN nodes. The NN model was trained and auto-validated on combat marksmanship training data from 51 participants that were required to make "deadly force decisions" in challenging combat scenarios. The trained NN model was cross validated using 10-fold cross-validation. It was also validated on a golf study in which additional 22 participants were asked to complete 10 sessions of 10 putts each. Temporal sequences of the activated nodes for both studies followed the same pattern of changes, demonstrating the generalization capabilities of the approach. Most node transition changes were local, but important events typically caused significant changes in the physiological metrics, as evidenced by larger state changes. This was investigated by calculating a transition score as the sum of subsequent state transitions between the activated NN nodes. Correlation analysis demonstrated statistically significant correlations between the transition scores and subjects' performances in both studies. This paper explored the hypothesis that temporal sequences of physiological changes comprise the discriminative patterns for performance prediction. These physiological markers could be utilized in future training improvement systems (e.g., through neurofeedback), and applied across a variety of training environments.

**Keywords: unsupervised learning, self-organizing map, cognitive state, electroencephalography (EEG), electrocardiography (ECG)**

#### **INTRODUCTION**

Passive Brain Computer Interface (BCI) utilizes spontaneously occurring brain signals to implicitly infer information about the cognitive or affective state of the user. Zander and Kothe (2011) presented a wide range of scenarios for the general population, i.e., healthy users that do not have difficulties to move or speak. This opens a number of new challenges, as it is difficult to compete with the standard communication channels such as speech and interaction devices (e.g., a mouse or keyboard) available to healthy users (Brouwer et al., 2013). However, passive BCIs provide an implicit input modality that carries information about the actual user cognitive or affective state that is not intentionally sent by the user. This offers numerous opportunities for adaptive human-computer interaction techniques that would adjust to the ongoing user's mental state (Cutrell and Tan, 2008; Zander et al., 2008; George and Lecuyer, 2010). Novel applications for passive BCI technology in domains such as entertainment, gaming, marketing, and industry continue to emerge (van Erp et al., 2012), e.g., implicit tagging of multimedia content, different avatar's appearances reflecting the gamers' mental state, adaptive automation in driving environments depending on the driver's workload, etc. Insight into the cognition and internal mental state of the user provides valuable information for detection of emergency due to the lack of concentration, drowsiness, or too high mental workload in safety-critical applications, such as driving or security surveillance (Zander and Kothe, 2011). Furthermore, training and education could also benefit from a more in-depth understanding of cognitive and affective states during learning with technology designed for cognitive enhancement and neurofeedback (Berka et al., 2010; van Erp et al., 2012).

A large number of previous studies in the context of training and education focused on error detection and performance prediction based on brain activity (Gehring et al., 1993; Falkenstein et al., 2000; Ferrez and Millan, 2008; Lehne et al., 2009; Stikic et al., 2011). This line of research has shown great promise in providing the means to increase the pace of skill learning, which is typically characterized as a three stage process (Fitts and Posner, 1967). In the initial cognitive stage, new knowledge is assembled to understand the skill to be learned (i.e., what to do) by effortful processing of perceptual cues. In the next, the associative stage, sensory processing of relevant stimuli is refined through practicing (i.e., how to do it) until reaching the third, the autonomous stage, which is characterized by automated execution of the task with minimal conscious mental effort. A growing body of research has been investigating physiological patterns associated with skill acquisition, mostly in sport environments that require complex visuomotor coordination, ranging from archery (Salazar et al., 1990; Landers et al., 1994), over marksmanship (Haufler et al., 2000; Hillman et al., 2000; Kerick et al., 2001; Deeny et al., 2003), to golf (Crews and Lander, 1993; Babiloni et al., 2008). These tasks are also characterized by a motionless preparatory period ("pre shot routine") before the skilled movement occurs. Since there are minimal movement artifacts during this preparatory period, EEG recordings in such settings are feasible. Gevins et al. (1999) suggested that EEG is more effective than other neuroimaging techniques in providing dynamic measures of skill acquisition due to the high temporal resolution, its online monitoring ability, and the potential for real-time feedback based on cognition, attention, and arousal.

All above mentioned skill acquisition studies focused on the analysis of low-level EEG features such as Power Spectral Densities (PSDs) grouped into the standard bandwidths, or EEG coherence analysis. Salazar et al. (1990) found alpha power in the left hemisphere to be dominant during the aiming period in archery. Haufler et al. (2000) compared the EEG of novices and expert shooters, finding that the experts exhibited less activation than the novice shooters during the aiming period, with a pronounced difference in the left central-temporal-parietal area. Crews and Lander (1993) suggested a decrease in left hemisphere, motor cortex activity as golf players prepared to putt. Babiloni et al. (2008) found that stronger event-related desynchronization (ERD) in high frequency alpha power (10–12 Hz) over the frontal midline (Fz and Cz) and right primary sensorimotor cortex (C4) is correlated with successful golf putts. The results of a study conducted by Deeny et al. (2003) revealed that, compared to lesser skilled shooters, experts engaged in less cortico-cortical communication, particularly between left temporal association and motor control regions, which implies decreased involvement of cognition with motor processes.

Changes in heart rate (i.e., the number of heartbeats per unit of time) have also been analyzed in relation to expert performance, and they are believed to reflect the focusing of attention and the skill related aspects of sensory-motor preparation for performance. The pre-shot period is typically characterized by heart rate deceleration (Landers et al., 1994; Kontinnen et al., 1998; Tremayne and Barry, 2001).

Building upon these findings, Berka et al. (2010) went one step further by developing a device called the Adaptive Peak Performance Trainer (APPT) that provides continuous psychophysiological monitoring and feedback (i.e., visual, auditory, and haptic) to the trainee in real-time. The device was based on a unique identified Pre-Shot Peak Performance (PSPP) expert profile in three different tasks (i.e., marksmanship, archery, and golf), which was characterized by an increase in midline theta and left temporal-parietal alpha EEG frequency bands and a local deceleration of heart rate. Skill learning was accelerated significantly when novices were trained with the APPT compared to controls, suggesting its feasibility as a training aid.

None of these past studies incorporated higher level cognitive state metrics such as EEG-engagement and EEG-workload that could potentially also relate to an individual's skill development. These metrics could contribute to better generalizability across different tasks compared to the low level EEG based features. Engagement reflects the allocation of processing resources (Anderson, 2004) including the cognitive processes associated with decision making, such as information gathering, visual scanning, audio processing, and selectively sustaining concentrated attention on one aspect of the environment while ignoring other distractions. Workload is typically defined as the amount of mental or physical resources required to perform a particular task (Son and Park, 2011). Seidler et al. (2012) have shown that individual differences in spatial working memory capacity predict the rate of motor skill learning. Bainbridge (1989) argued that workload is reduced with the development of a certain skill. However, Brouwer et al. (2014) did not find physiological variables to indicate decreasing effort associated with learning. EEG-based neural indices of cognition that reflect levels of alertness, selective attention, and working memory have been identified in previous studies (Hillyard et al., 1973; Gevins et al., 1998; Mikulka et al., 2002). However, these indices have only started to be explored in skill acquisition and performance prediction studies. Moreover, the neurophysiological data are typically analyzed during the pre-shooting period only, neglecting the changes over the entire process that include, for example, observation of the dynamic environment, evaluating possible threats, and eliminating enemies in the marksmanship scenarios. Lastly, the previous studies did not account for the individual variability in the EEG data, and many of them lacked evaluation of the generalizability of the proposed approaches across different users and tasks.

Performance prediction algorithms are based on either supervised or unsupervised machine learning methods. The main difference between these two types of methods is whether they utilize labeled data (supervised learning) or unlabeled data (unsupervised learning). Many of the supervised approaches are not based on EEG data, but on kinematical models, e.g., using computer vision. Couceiro et al. (2013) compared different classifiers (linear and quadratic discriminant analysis, Support Vector Machines— SVM, and naive Bayes) in their golf study. The experimental results showed the superiority of the SVM classifier, however, a large individual variability in the data was present due to the different game styles of each player. Jensen et al. (2012) evaluated AdaBoost, Fisher LDA, k-Nearest Neighbor (kNN), naive Bayes, and SVM classifiers in a study of kinematic golf putt data classification based on the inertial sensor. Nagashima et al. (2009) assessed marksmanship performance by sensor-based breath and trigger control measures that were fitted with a logistic regression model. Muangjaroen and Wongsawat (2012) developed an EEGbased real-time index for predicting putt success based on high alpha power in the C4 sensor site, theta power in Fz, and both high alpha and theta power in Pz. An impeding bottleneck of the supervised learning methods is the substantial amounts of labeled training data required to train a classifier. Labeling of data is typically a labor-intensive, time-consuming, and error-prone process that scales poorly to the large number of users and skill learning sessions necessary to achieve the required level of expertise. This limits the applicability of the supervised learning approaches in challenging skill acquisition settings where fine grained labels of all relevant session's periods would need to be precisely obtained. Another line of research avoids the labeling efforts by unsupervised discovery of structure in the data. However, there are only a few studies where unsupervised approaches were applied for performance prediction, such as data fusion of EEG and ECG signals through Self-Organizing Maps (SOM) (Bandeira et al., 1999). In this approach, shooting data were classified by majority voting of the labeled neurons of the trained SOM. However, this study focused on the data fusion and analyzed only low-level EEG features. The study included a single constrained shooting scenario with only seven subjects. Labeling of the neurons consisted of finding the "winning" neuron in a SOM for each feature vector in the dataset, and assigning the feature vector's label to the neuron's label. Thus, this approach also required labeled training data. The activated NN nodes represented the single subject's data, and did not include similar performance data. Data fusion slightly improved performance of the classifier, but they did not conduct cross-validation of the algorithm on an independent dataset or made any effort to determine whether the model would generalize across tasks.

In this paper, we present an alternative approach toward performance estimation in dynamic environments requiring rapid decisions. We focus on the two tasks that were often analyzed in the past: combat marksmanship (e.g., Haufler et al., 2000; Deeny et al., 2003) and golf training (e.g., Crews and Lander, 1993; Babiloni et al., 2008). These two tasks both require complex visuomotor coordination, mental and sensory-motor preparation, focused attention, and concentration. None of the previous studies in the literature explored the potential similarities of the neurophysiological performance predictors on these tasks. This novel approach is based on a combination of the two types of higher level EEG-based cognitive state measures associated with decision-making, alertness, attention, and working memory: engagement (Johnson et al., 2011) and workload (Berka et al., 2007). Furthermore, heart rate is also included in the analysis, as it adds the dimensions of stress, anxiety, frustration, and arousal experienced by a shooter (Berka et al., 2010). EEG-engagement reflects information-gathering, visual processing, and allocation of attention, while EEG-workload is associated with increased working memory load during problem solving, integration of information, and analytical reasoning (Berka et al., 2007). The EEG-based metrics are used as an indicator of how well a shooter is processing information, as well as an indicator of the focused and relaxed mental state that is a prerequisite for good shooting performance (Raphael et al., 2009). We tested the following hypotheses: (1) temporal sequences of EEG-based engagement and workload and ECG-based heart rate comprise discriminative patterns related to performance on the combat marksmanship task, (2) the discovered patterns are good indicators of performance on the golf training task, and (3) individual differences in the neurophysiological metrics could be overcome by data normalization. In order to overcome the main drawback of the supervised approaches that require large amounts of labeled training data, we explored the potential of the unlabeled data by applying an unsupervised approach. The approach is based on the one dimensional self-organizing NN inspired by work of Stevens et al. (2009, 2013). However, we extended their algorithm by combining all three metrics together (i.e., EEGengagement, EEG-workload, and heart rate) into a single NN. Multiple neurophysiological metrics could facilitate developing a deeper understanding of skill acquisition, as they have different functional properties reflecting various aspects of the learning process. Furthermore, we incorporated the temporal modeling of these metrics on a fine-grained scale by including 3 consecutive seconds into a feature vector. All metrics were self-normalized to account for the individual differences inherent in the physiological data. The NN model was trained and auto-validated on the combat marksmanship training data. The trained model was then cross-validated in two different ways. First, we utilized 10-fold cross-validation (Kohavi, 1995) to evaluate generalization capabilities of the approach across different subjects. Second, we also evaluated the generalization capabilities of the model by crossvalidating it on the golf-training data. Correlation analysis of the temporal cognitive state changes was performed to investigate if they comprise the discriminative patterns for performance prediction that could ultimately allow for faster learning of different skills through neurofeedback. Lastly, the model was compared to the linear regression model on the same set of variables to investigate whether the utilized variables or their temporal modeling with the NN approach contributed to the results.

# **MATERIALS AND METHODS**

Two studies were conducted: combat marksmanship training on the military simulator, and golf putting training at the indoor golf facility. In this section, we describe the protocols of both studies, and we detail data recording setup, signal processing, and algorithm development procedures.

#### **PARTICIPANTS**

For the combat marksmanship study, a total of 51 participants (22 females and 29 males, mean age 27.4 ± 6.7 years) were recruited from local colleges and newspaper/online advertisements. All participants had normal or corrected-to-normal vision. No participants that had undergone formal marksmanship training were admitted to the study. Informed consent was obtained from all participants in accordance with the guidelines and approval of the Biomedical Research of America IRB.

A total of 22 subjects were recruited for the golf training study using the National University Golf Academy's (NUGA) IRB approved protocol. The participants were split into two groups: experienced players and novices, based on their previous experience. Each group had 11 participants and in each group 45% of the population was male. Average overall age was 33.9 ± 11.1.

No study procedures were conducted until the consent form was fully explained and the consent form was signed. There was no overlap between the participants in these two studies. All participants in both studies were right-handed.

#### **STUDY PROTOCOL**

The study protocols were comprised of screening, baseline, and the experimental session.

#### *Screening procedure*

The eligibility screening criteria aimed to cover anything known to effectively alter the EEG signals, so those subjects with general health problems such as psychiatric, neurological, behavioral, attention, or sleep disorders, as well as any pulmonary or eating disorder, diabetes, high blood pressure, or history of stroke were excluded from the studies. Those who reported using pain medications regularly, stimulants such as amphetamine, illicit drugs, or those who consumed excessive alcohol or tobacco on a daily basis were also excluded. Exclusion criteria also included head injuries within the past 5 years and pregnancy. Eight subjects failed this initial screener. Participants who passed the screener were scheduled for the baseline session. Participants were asked to ensure they got a full night of sleep (between 7.5 and 9 h) in the night leading up to their study appointments. Participants were also requested to refrain from drinking alcohol at least 24 h before each study visit, and abstain from caffeine on the day of the study and nicotine 1 h before the start of the study visit.

#### *Baseline session*

In both studies, the participants completed a 15 min baseline (BL) using the Alertness Memory Profiler (AMP). The AMP was developed by Advanced Brain Monitoring Inc. (ABM) to integrate EEG, ECG, and performance measures in an easy-to-administer platform designed for quantitative assessment of neurocognitive functions, including alertness, attention, learning, and memory. The AMP uses a multivariate approach that allows simultaneous acquisition and analysis of data.

The baseline session comprised a Visual Passive Vigilance Task (VPVT), an Auditory Passive Vigilance Task (APVT), and a 3-Choice Active Vigilance Task (3CVT). During the 5 min of VPVT and APVT, subjects were required to press the space bar on the keyboard every 2 s. Subjects were prompted to maintain the 2 s time intervals by a 10 cm diameter red circle that appeared in the center of the monitor during the VPVT, or by an audio tone during the APVT. The 3CVT (**Figure 1**) is a sustained attention task that requires subjects to discriminate one primary target (presented 70% of the time) from two secondary non-target geometric shapes that were presented for 0.2 s and randomly interspersed over the test period. It challenges the ability to sustain attention by increasing the inter-stimulus interval (ISI) from 1.5 s up to 10 s at the end of the task. Participants were instructed to respond as quickly and as accurately as possible to each stimulus presented by pressing the left arrow to indicate target stimuli, and the right arrow to indicate nontarget stimuli. A brief training period was provided prior to the

**FIGURE 1 | A subject wearing the wireless EEG sensor headset while performing the 3CVT task.**

start of the testing period to minimize practice effects for the 3CVT. The training period lasted until a certain number of correct responses to both targets and non-targets was reached (2 targets and 2 non-targets). The applied criterion was the same for all participants in both studies. On average, training period lasted 30 s.

## *Combat marksmanship training experiment*

The military training platform Virtual Battle Space 2 (VBS2), Tactical Warfare Simulation running on the Real Virtuality 2 simulation engine and developed by Bohemia Interactive, was used to create the combat marksmanship training scenarios. The VBS2 platform was designed in close cooperation with the United States Marine Corps and Australian Defence Force, and it has been used as a training battlefield simulation system by federal, state, and local government agencies worldwide for tactical training and mission rehearsal in military organizations. It is an interactive, 3D training system that allows users to construct specific missions based on their individual needs. In collaboration with Laser Shot Inc. and a subject-matter expert (SME) from Washington State University, we developed five custom combat scenarios using VBS2. The scenarios (**Figure 2**) had realistic settings and contexts in which participants (acting as soldiers) were required to make deadly force decisions. In order to mimic reality as closely as possible, the room was equipped with life-size projection of threats, stereo sound delivered via earbuds, and other paraphernalia found in the battlefield environment. The participant used a demilitarized "airsoft" replica of an M4 rifle that interacted with the platform using a wireless laser-based training system from Laser Shot Inc. The M4 was mounted with a holographic weapon sight, or red dot scope, commonly used in a combat environment for quick target acquisition. Sandbags were provided to support the weight of the weapon, to both simulate combat firing procedures and investigate the effect of muscle fatigue on performance. The weapon also had a CO2 gas recoil system that simulated the kickback of the weapon.

**FIGURE 2 | Combat marksmanship scenario scenes.**

Participants were initially given marksmanship instructions and requested to undergo a set of training tasks. Training addressed the fundamentals of marksmanship (aiming, breath control, trigger control, etc.) and the rules of engagement applicable in the testing scenarios. During the three training scenarios, the participants practiced firing the M4: (1) at close-range static targets, (2) at targets of various distances, and (3) at targets that were randomly displayed among non-targets. Five testing scenarios were administered in a randomized order for each subject. Each scenario was set in a unique environment that replicated typical fire-fighting situations for soldiers (e.g., Afghanistan compound, checkpoint, and market; Improvised Explosive Device (IED) compound, or urban alley). In order to avoid excessive fatigue in participants, the scenarios were designed from a fixed point of view and each scenario lasted only 3–4 min, on average. Throughout the scenarios, a mixture of enemy and friendly units (both stationary and moving) appeared at varying distances. Two scenarios were enemy-heavy in which 90 and 76% of the units were enemies, two were friendly-heavy in which 76 and 60% of the units were friendly, and one had about equal mix of enemy and friendly units (47% enemy units and 53% friendly units). Participants were instructed to evaluate threats and eliminate all enemy units as quickly as possible.

To enable detailed event tracking, an event logging and synchronization platform was utilized as an essential piece for the study testbed. An External Sync Unit (ESU) was used to synchronize the physiological signals with the relevant events in the scenarios (such as friend/enemy became visible, enemy fired, etc.), as well as user responses (e.g., rifle shots, hit friend/enemy, etc.). Synchronization in the Windows environment is dependant on the windows task scheduler and cannot guarantee an upper bound for user level tasks, while the ESU is a general purpose data integration platform that can synchronize multi-source digital data (serial and/or parallel port protocols) with physiological signals to millisecond-level precision.

#### *Golf training experiment*

The golf training experiment was designed in close cooperation with SMEs from NUGA. The participants were asked to come to the indoor golf facility at the NUGA's premises. Upon arrival, they were asked to complete 10 sessions of 10 putts each wearing the wireless EEG headset (**Figure 3**). The average session duration was 5.5 min. It was decided that 100 putts from each subject would be required for the EEG data analysis. To avoid possible fatigue, those 100 putts were broken down into 10

**FIGURE 3 | NUGA's golf facility.**

sessions. All sessions were administered within the same day, with 3 min breaks between the sessions. The putting distance was far enough (10 feet) to be challenging for both experienced players and novices. Each putt comprised a series of steps including: (1) preparation period—the participants were asked to stand a few feet behind the ball before stepping up to the ball; (2) step up to putt—when the subjects were ready, they would walk up to the ball and get into their putting stance; (3) start putt—when the putter made contact with the ball; and (4) end putt—motion of the ball was stopped or off the green. All relevant events were manually marked and logged during the real-time EEG acquisition and for each putt the three signal sequences of interest were analyzed: preparation period, pre-putt period, and post-putt period (shown in **Figure 4**).

#### **DATA RECORDING AND SIGNAL PROCESSING**

EEG and ECG data were collected using the wireless EEG sensor headset (**Figure 1**) developed by ABM (Berka et al., 2004). For the combat marksmanship study, nine Ag/AgCl EEG electrodes were located at F3, Fz, F4, C3, Cz, C4, P3, POz, P4, according to the international 10–20 system. For the golf study, 20 EEG channels were utilized: Fp1, Fp2, Fz, F3, F4, F7, F8, T3, T4, T5, T6, Cz, C3, C4, Pz, P3, P4, O1, O2, and POz sites. The EEG channels were referenced to linked reference electrodes located behind each ear on the mastoid bone. Even though different montage was used in these two studies, only the data from the EEG channels that are necessary for calculating EEG-based engagement and workload metrics were analyzed: Fz, F3, Cz, C3, C4, and POz. The most discriminative variables for both models (i.e., EEG-engagement and EEG-workload) were selected in the previous study (Berka et al., 2007) using stepwise regression. ECG was recorded with electrodes placed on the clavicle and opposite lower rib. All data was sampled at 256 Hz and transferred in real-time via Bluetooth link to a nearby host computer where the proprietary data acquisition software stored the data onto the disk.

The EEG signals were filtered with a band-pass filter (0.5–65 Hz) before the analog-to-digital conversion. To remove environmental artifacts from the power network, notch filters at 50, 60, 100, and 120 Hz were applied. The decontamination algorithm (Berka et al., 2004) detected and removed any sudden changes in amplitude, i.e., artifacts in the time-domain EEG signal, including spikes caused by tapping or bumping of the sensors, amplifier saturation, and excursions. Eye blinks were identified and decontaminated by wavelet transform based upon an algorithm presented in Berka et al. (2007). The wavelet decomposition coefficients for differential Fz-POz and Cz-POz EEG channel derivations in the exponential 0–2, 2–4, 4–8, 8–16, 16– 32, 32–64, and 64–128 Hz bands were calculated by applying the Coiflet order-1 wavelet filter (Wei, 1998). These coefficients were further utilized in the discriminant function analysis to detect the eye blink regions. Decontamination of eye blinks was accomplished by computing mean wavelet coefficients from the nearby non-contaminated regions and replacing the contaminated data points. The EEG signal was then reconstructed by wavelet composition. From the filtered and decontaminated EEG signal, the log-transformed absolute power spectral densities (PSD) were calculated for each 1 s epoch of data by applying Fast Fourier Transformation (FFT). Relative PSD values were derived by subtracting the PSD for each 1 Hz bin from the total PSD power in the range of 1–40 Hz. Lastly, excessive muscle activity (EMG) was detected by identifying epochs in which (a) PSD bins from 35 to 40 Hz were above a certain threshold and (b) the square root of the PSD bins' sum from 70 to 128 Hz was above a defined cut-off value. The threshold values were customized empirically by visual inspection of the large EEG data cohort contaminated with EMG during the EEG sensor development. Epochs with detected EMG artifacts were discarded from further analysis. On average, 21.9 and 23.4% of the epochs per session were contaminated with artifacts in the combat marksmanship and golf study, respectively. There were no significant differences in the number of detected artifacts for different combat marksmanship scenarios (19.9, 20.6, 21.2, 23.4, and 24.2%).

The ECG signal was filtered with the band-pass 4th order IIR filter (15–35 Hz). This improved the contrast between the QRS complex and the T wave, and minimized double peak detection, leading to the more robust detection of peaks. The filtered signal was processed by a real-time algorithm that computed sample-by-sample running mean and standard deviation to derive a threshold for peak detection. The inter-beat (R-R) interval was determined as the number of seconds between the current and the previous peak. Heart rate was estimated as a number of beats per minute, i.e., 60/(R-R interval), and logged in real-time. The algorithm assessed the quality of detected beats by monitoring the standard deviation of up to 6 consecutive beats.

#### **EEG-BASED ENGAGEMENT AND WORKLOAD METRICS**

In order to explore the applicability of alertness quantification in performance estimation, we incorporated the B-Alert® model (Johnson et al., 2011) into the analysis. It is an individualized model that classifies a subject's cognitive state into different levels of alertness: distraction/relaxed wakefulness, low engagement, or high engagement. It utilizes the absolute and relative PSD values from the midline Fz-POz and Cz-POz derivations during VPVT, APVT, and 3CVT BL data to derive coefficients for a discriminant function that generates classification probabilities for each 1 s epoch. During the model training procedure, APVT, VPVT, and 3CVT represent distraction/relaxed wakefulness, low engagement, and high engagement, respectively. The quality of the individual models were assessed by auto-validation on these three tasks (APVT, VPVT, and 3CVT), and for all subjects the majority of the epochs were classifed into the expected class. This model has been validated in a number of previous studies across a range of applications, such as sleep deprivation (Westbrook et al., 2004), team collaboration (Stevens et al., 2009), and emergent leadership (Waldman et al., 2013). The high engagement output posterior probabilities of the model were added to the feature vector for further analysis.

The EEG-based workload model (Berka et al., 2007) was utilized to extract the subject's cognitive workload levels on a second-by-second basis. This is a general model that was trained on the EEG data from a large population performing the Forward Digit Span (FDS) and Backward Digit Span (BDS) tasks. During these two tasks, the subjects sit still in front of a computer screen and memorize sequences of 2 up to 9 digits that are shown on the computer screen as a series of single digits of increasing lengths, followed by an empty box prompting the participant to reproduce the sequence by typing in the memorized digits in the presented order (FDS) or the reverse order (BDS). For both FDS and BDS, the task difficulty was manipulated by increasing the number of digits at each level. The model utilizes the absolute and relative PSDs from the differential EEG derivations (C3-C4, Cz-POz, F3-Cz, Fz-C3, and Fz-POz) into a 2-class linear Discriminant Function Analysis classifier (i.e., low workload and high workload). The present study employed the posterior probability of the high workload class in order to identify a continuous measure of EEG-workload and analyze its temporal changes during skill training.

It has been shown in the previous studies (Stevens et al., 2013) that EEG-engagement and EEG-workload metrics have different functional properties in response to different tasks and they are poorly correlated with one another. Correlation between EEGengagement and EEG-workload was *R* = −0*.*24 ± 0*.*21 with an *R*2 of 0.11 ± 0.11 across the marksmanship training sessions. Correlation between EEG-engagement and EEG-workload across the golf training sessions was *R* = −0*.*11 ± 0*.*22 with an *R*2 of 0.06 ± 0.11. Correlation between these two metrics in Stevens et al. (2013) was in the same range *R* = −0*.*19 ± 0*.*24 (*R*2 = 0*.*09 ± 0*.*05).

#### **PERFORMANCE METRICS**

In both the combat marksmanship and golf training studies, the subjects' performance scores were calculated. For the combat marksmanship data, the following metrics were included in the analysis: percent of enemy hits, enemy deaths, and misses. The enemy was hit if they were shot in a non-vital area such as hand, foot, or shoulder, while they were killed when shot in a vital area like head or chest. For the golf study, we calculated the percent of putts that ended up in the hole as the main performance metric.

#### **ALGORITHM DEVELOPMENT**

EEG-derived engagement and workload, and ECG-derived heart rate metrics were combined together in order to analyze their common changes over time. To normalize the data for individual variability, all three metrics were self-normalized (i.e., z-scored) with respect to the training task in question. This accommodated for individual differences inherent in the physiological data. The feature vector was derived by applying a 3 s sliding window to each second of data (i.e., the window was shifted in 1 s increments). Thus, the feature vector comprised the three analyzed metrics (i.e., EEG-engagement, EEG-workload, and heart rate) over three consecutive seconds, summing up to 9 features.

These vectors were used for training an unsupervised selforganizing artificial NN (Kohonen, 1990) consisting of 20 nodes placed on a 1D grid (20 × 1), which is a discrete representation of the continuous input space with the preserved spatial properties of that space (i.e., topological relationships within the training set are maintained). In the NN, the nodes are represented with a weight vector (i.e., code vector) whose dimension is equal to the dimension of the input vectors and a neighborhood function that decays with distance and dictates the topology of the map. Training consists of the competitive learning process which develops the topology in the following manner. The feature vectors most similar to each other (based on the Euclidean distance) activate the same nodes, i.e., in the end similar nodes are closer to each other on the grid than the more dissimilar ones. For each input pattern, the node with the closest weight vector is declared the Best Matching Unit (BMU). The weights of the BMU and its neighbors are then adjusted toward the input vector. The neighborhood size is at first large during the ordering phase, and afterwards it is reduced in each iteration until only the BMU's weight starts to be updated during convergence phase when fine tuning of the NN weights is performed. After training, mapping is carried out by assigning each new feature vector to the NN node it activated. In our experiments, both training and mapping were performed in MATLAB® using Neural Nework Toolbox. The batch version of the learning algorithm was applied: instead of presenting only a single feature vector per iteration, all vectors are presented to the NN in each iteration and the weights change so that each node's new weight vector is the weighted average of the input vectors that activated that node. The weight vectors were initialized with the most significant principal components of the input space to start with a reasonable ordering. By distributing the weights in this manner and using the batch learning algorithm, the number of iterations needed to reach convergence was reduced, and we ran 200 iterations in our experiments.

The goal of the analysis was to find relevant patterns in the trained NN nodes' activations related to the skill training process. In our experiments, the NN model was trained on the combat marksmanship data that were also used for auto-validation as a proof of concept. The NN model was validated by utilizing 10 fold cross-validation. In the first nine cross-validation iterations, the data from five subjects were used for training and the rest of the data were used for testing. In the last round, the data from six subjects were employed for training and the rest of the data was employed for testing. By choosing the training folds in this manner, we evaluated person-independent modeling (i.e., the test data did not comprise the data from the subjects that were used for training) and every subject in the study was used only once in the training phase. The trained NN model was also tested on the golf study data to evaluate generalization capabilities of the model across different tasks. An extensive set of analyses was performed. First, we characterized the NN nodes by aggregating the feature vectors that activated the same node and analyzed the size of each node, i.e., how many times each node was activated. Second, the distributions of activated NN nodes during different relevant events and periods in both datasets were analyzed. The distributions were aggregated by counting all node activations whenever the event of interest appeared in the combat marksmanship dataset or during a certain period of interest in the golf dataset. The middle point of the 3 s long feature vector window was taken as a reference point, i.e., we analyzed EEG-engagement, EEG-workload, and heart rate metrics of the 1 s epoch when the event occurred, and their changes 1 s before and 1 s after the event. One-Way Multivariate Analysis of Variance (MANOVA) was applied to the estimated distributions to investigate if they are statistically different, i.e., if different neurophysiological patterns are associated with these events and periods of interest. Third, time series analysis of the adjacent NN node activations was performed by deriving a transition matrix that shows the overall number of the subsequent NN node activations for all possible pairs of the NN nodes. Fourth, based on the observed patterns in the transition matrix, we introduce a transition score that measures the size of temporal state changes as a sum of transitions between the subsequent activations of the NN nodes in the following manner. Let *t* be an overall number of activated nodes during the subject's skill training session (i.e., an overall number of feature vectors presented to the NN model), and let *i(j)* be the NN node activated during epoch *j* (*i* ∈ [1*, N*]*, N* = 20*, j* ∈ [1*, t*]). We define transition score as:

$$Transition\, score = \sum\_{j=2}^{t} abs(i\left(j\right) - i(j-1))$$

Correlation analysis was performed to investigate if these transition scores were related to the subject's performance scores in both studies. Lastly, these performance prediction results were compared to the linear regression model on the same set of variables used for training the NN model.

#### **RESULTS**

In this section, the following results are presented: the frequency of activated NN nodes and their patterns during the important events and periods of interest in both the combat marksmanship and golf training datasets, time series analysis of activated NN nodes throughout the entire training sessions obtained by calculating transition matrices, analysis of performance scores, correlations between transition scores and performance scores, and comparison with the linear regression model.

#### **ANALYSIS OF THE NN NODES**

After training on the combat marksmanship dataset was completed, we grouped together and analyzed the feature vectors that activated the same NN nodes. Average EEG-engagement, EEGworkload, and heart rate values of the grouped feature vectors for each node and node sizes (i.e., a number of feature vectors that activated the nodes) are shown in **Figure 5**. The nodes 8, 6, and 11 were activated the most often (2071, 1955, and 1827, i.e., 7.9, 7.5, and 6.9% of the time), and they represent three distinct neurophysiological states in which there are no large changes in any of the three analyzed metrics. Node 8 represents a state of low EEG-engagement, as its z-scored value is negative (i.e., below the average value over the subject's entire training session) that is increasing over the observed 3 s. Heart rate is also slightly increasing and it is above the subject's average value, while EEG-workload is slightly decreased during the third second, but still above the average subject's value. Node 6 represents EEG-engagement and heart rate below the subject's average value, while EEG-workload is above the average value. Lastly, EEG-engagement and EEG-workload of node 11 are positive, while heart rate is around the subject's average value with a slight increase tendency. The least activated nodes were 2 and 13 — they were activated 518 and 589 times, i.e., 1.9 and 2.2%

**nodes (i.e., the number of feature vectors that activated the corresponding node).**

of the time, respectively. The main characteristic of these nodes is that they have a large decrease (node 2) or increase (node 13) in EEG-engagement and, at the same time, relatively large opposite change of EEG-workload. These two nodes were the least often activated nodes during the validation of the trained NN on the golf training data as well (0.8% node 2, and 0.9% node 13). The most often activated nodes for the golf dataset were also nodes 11 (10.4%) and 6 (7%), in addition to node 5 (7.5%) that has around average EEG-engagement, above average EEG-workload, and below average heart rate values.

For the combat marksmanship dataset, we analyzed the changes in the subjects' EEG-engagement, EEG-workload, and heart rate when an enemy appeared and compared them to the typical subject's reactions to the friendly unit's appearance by examining distributions of activated NN nodes for these two events. In **Figure 6** are shown distributions of the activated NN nodes for the two analyzed events (i.e., "Became visible friend" and "Became visible enemy"). MANOVA analysis showed statistically significant difference between these two events' NN node activations [*F*(20*,* 427) = 1*.*716, *p* = 0*.*028, effect size η2 partial = 0*.*07, observed power = 0.966]. When looking at differences in the individual node's activations, the largest difference was for nodes 15 and 11. Node 15 was activated more often as a response to the appearance of friendly units and node 11 was a frequent response to the enemy's appearance. Node 15 represents the relatively relaxed state when all three metrics (i.e., EEG-engagement, EEGworkload, and heart rate) are below their average values, while node 11 is a representative of above average EEG-engagement, EEG-workload, and heart rate values. Furthermore, there is a slight decrease of EEG-engagement and heart rate in node 15, during the third second, when friendly units appeared, while in node 11 an increase of EEG-engagement and heart rate is encountered during the third second, i.e., after enemies appeared.

We also characterized the three periods of interest in the golf study: preparation period, pre-putt period, and post-putt period by their distribution of activated NN nodes. These results are shown in **Figure 7**. From the plots one can observe that the most frequent node during the pre-putt period was node 11, while node 5 was the most often activated during the post-putt period. Comparison of those 2 nodes' EEG-engagement, EEGworkload, and heart rate values (shown in **Figure 5**) indicated above average EEG-engagement, EEG-workload, and heart rate levels during the pre-putt period, while during the post-putt period EEG-engagement was back to its average value, heart rate decreased below average value, and EEG-workload was still above average but slightly decreased compared to the pre-putt period. Furthermore, we compared pre-putt and post-putt periods' node activation distributions for the hits and the misses, and MANOVA showed statistically significant differences between pre-putt and post-putt periods for these two types of putts [*F*(20*,* 2178) = 4*.*018, *p <* 0*.*001, effect size η2 partial = 0*.*036, observed power

= 1 for pre-put period and *F*(20*,* 2128) = 1*.*743, *p* = 0*.*02, effect size η2 partial = 0*.*016, observed power = 0.974 for post-putt period].

#### **TRANSITION MATRICES**

Time series analysis of the activated NN nodes was performed and transition matrices on the second-by-second basis were constructed. Transition matrices for both combat marksmanship and golf study are shown in **Figures 8**, **9**, respectively. From the plots it can be observed that both transition matrices followed the same pattern of changes, i.e., most of the transitions were local, which was reflected in the transition matrix as movement around a diagonal. This is confirmed in **Figure 10**, which shows a number of transitions aggregated across distance between the nodes (based on their sole position on the NN grid, and not on Euclidean distances between them). From the plots it can be clearly seen that the number of transitions decreased almost exponentially with distance between the nodes in both datasets. This indicates that the analyzed physiological metrics were not changing dramatically over time. On the other hand, larger state changes were less dominant, however, after investigating their timestamps it turned out that they typically represented some significant events in the datasets that caused large increase or decrease of EEG-engagement, EEG-workload, or heart rate metrics. Consequently, more distant nodes of the

**FIGURE 8 | Transition matrix during auto-validation on the combat marksmanship dataset: a number of subsequent transitions between each pair on the NN nodes.**

trained NN model were activated, and in such cases an average nodes' distance on the NN grid was 4. In order to quantify this phenomenon, we calculated transition scores for each subject's skill training session and correlation analysis was performed to investigate if they were associated with the subject's performance.

#### **ANALYSIS OF PERFORMANCE SCORES**

In the combat marksmanship dataset, even though the scenarios had varying number of enemies, there were no statistically significant differences in performance across different scenarios. The order of the scenarios was randomized across subjects, and there was a slight trend of decrease (about 5%) in performance in the later scenarios as shown in **Figure 11**, but it was not statistically significant (*p* = 0*.*45).

In the golf dataset, experienced players performed better on average (by 16%), but after ranking all subjects by the achieved performance, the set of the best performers comprised an equal split of novices and experienced players. Averaged subjects' performance for each session is shown in **Figure 11**, and there was an increase of 12% in performance during the first half of the sessions. Afterwards, performance started to decrease, but when comparing the first and the last session, there was an overall slight improvement in performance (by 4.8%).

#### **CORRELATION ANALYSIS**

Correlations during auto-validation between transition scores and performance scores for combat marksmanship are shown in **Table 1**. **Table 2** shows correlations during 10-fold crossvalidation between transition scores and performance scores for combat marksmanship. In **Table 3** are shown correlations between transition scores and performance scores for the golf study. In all three cases, the correlations were statistically significant based on the two-tailed *t*-test. Statistically significant

**marksmanship and golf dataset.**

**Table 1 | The Pearson's correlation coefficients during auto-validation between transition scores and performance scores for the combat marksmanship dataset.**

**Table 2 | The Pearson's correlation coefficients during 10-fold cross-validation between transition scores and performance scores for the combat marksmanship dataset.**

positive correlations between transition scores and percent of enemy hits/enemy deaths for the combat marksmanship dataset, as well as between transition scores and percent of "in" putts for the golf study, indicate that the larger the transition score the better the performance. Negative correlations between transition scores and percent misses in the combat marksmanship dataset during both auto-validation (**Table 1**) and 10-fold crossvalidation (**Table 2**) show a trend of lower transition scores for the worst performing subjects.

#### **COMPARISON WITH THE LINEAR REGRESSION MODEL**

We built the linear regression model using the same set of variables as in the NN model to predict the performance scores in the combat marksmanship study. We tested the variables for **Table 3 | The Pearson's correlation coefficients between transition scores and performance scores for the golf dataset.**

#### **Table 4 | Multicollinearity test: VIF and tolerance of the EEG-engagement, EEG-workload, and heart rate variables.**

**Table 5 | The linear regression model performance prediction results: coefficient of determination (***r***2),** *p***-value for the** *F***-test of the overall linear regression model, and the** *p***-value for the** *t***-test of the predictor variables.**

multicollinearity by calculating tolerance and the corresponding variance inflation factor (VIF). The results are shown in **Table 4**. As tolerance *>* 0.2 (i.e., VIF *<* 5), one can conclude that multicollinearity was not present. Thus, the linear regression model was built without regularization. The regression results are shown in **Table 5**. One can observe that only the model that predicted the "percent enemy deaths" performance score was statistically significant (*F*-Test, *p*-value = 0.018). In that particular model, only 3 predictor variables were statistically significant: EEG-Engagement—sec 2 (*t*-test, *p*-value = 0.004), EEG-Workload—sec 1 (*t*-test, *p*-value = 0.05), and EEG-Workload—sec 3 (*t*-test, *p*-value = 0.05). When comparing the *r*2 scores in **Table 1** (i.e., based on the NN model) and **Table 5** (i.e., based on the linear model), one can conclude that the NN model outperformed the linear regression model, as it was able to predict the performance scores more accurately.

# **DISCUSSION**

In this paper, we explored a novel way of combining EEG-based metrics of engagement and workload, and an ECG-based heart rate metric to capture temporal cognitive state changes related to performance on two different skill training tasks, namely combat marksmanship and golf training. The main benefit of the algorithm is that it does not require prior labeling of the data as it employs the unsupervised self-organizing NN approach. The supervised learning approaches require large amounts of labeled data to train a classifier, but obtaining that large amount of labeled training data is a time-consuming and error-prone process that has limited the real-world applicability of many previous algorithms. Instead of utilizing low-level EEG features, we investigated the higher level EEG-based metrics of engagement and workload that contributed to the better generalizability of the approach across different tasks. The richness of the combat marksmanship dataset in the different EEG-engagement and EEG-workload levels allowed for successful training of the NN model. The model was able to cover all important patterns in cognitive state changes that were relevant for the golf task as well. In order to overcome individual differences inherent in the physiologically induced metrics, we normalized the data. Our algorithm was cross-validated in two different ways. First, person-independent training was performed by utilizing 10-fold cross-validation. Second, the NN model trained on the combat marksmanship data was tested on the golf dataset to evaluate generalization capabilities of the approach across different tasks. The temporal sequences of physiologically estimated cognitive state changes detected in the combat marksmanship dataset occurred in the golf dataset as well. The model was able to capture differences in the subject's psychophysiological states in response to different events such as appearance of the friendly and enemy units in the combat marksmanship scenarios or during different stages of golf putting. This could be also attributed to a difference in motor activity. We aimed to minimize that effect by excluding the epochs with detected EMG artifacts from the analysis. This demonstrated the richness of a model combining EEG-engagement, EEG-workload, and heart rate metrics that successfully provide a window on the internal mental state of the user. Analysis of the dynamic changes over time of these three metrics combined together contributes to a more in-depth understanding of different aspects and stages of skill training. For example, EEG-engagement plays an important role during gathering information necessary for performing a task, EEGworkload is essential in processing the gathered information and comparing it to internal mental models, while heart rate is an indicator of the subject's preparation to receive sensory inputs and reaction to the processed information.

Statistically significant correlations between the introduced NN nodes' transition scores and performance were found in both datasets, which supports our first two hypotheses that temporal sequences of EEG-based engagement and workload and ECG-based heart rate comprise patterns related to performance on the combat marksmanship dataset, and that the discovered patterns are good indicators of performance on the golf training task as well. This is in line with the findings of Stevens et al. (2013) that showed relationship between the fluctuations in the entropy of the activated NN nodes and the relevant events in the task that require significant cognitive re-organization. These large cognitive state transformations might reflect adaptation to changes in the task. In their study, decreases in entropy were also associated with periods of poorer task performance. Though preliminary, our results also suggest physiological signatures may distinguish elements of perception indicative of good performance. Unlike previous studies that analyzed the relationship between the performance scores on the combat marksmanship and golf training tasks (e.g., Crews and Lander, 1993; Haufler et al., 2000; Deeny et al., 2003; Babiloni et al., 2008), and low-level EEG features such as PSD bandwidths over different scalp regions, we focused on the higher level metrics such as EEG-engagement, EEG-workload, and heart rate. The discriminative patterns in these three metrics were general across the two analyzed tasks and could be included in future neurofeedback-based or other intelligent training systems to accelerate learning across different tasks. Beyond performance estimation and skill acquisition scenarios, the proposed unsupervised approach based on time series analysis holds potential to also enrich information systems with additional data on cognitive state changes of the user that could be employed, for example, for testing the effectiveness of different marketing advertisements, user's reactions to different movie storyline endings, or even to political speeches. Due to the general nature of the developed algorithm, different NN nodes would be more active under diverse application scenarios, and the speed of the feature vector changes specific to different applications could be captured.

Even though the average ages of the subjects in these two datasets are different (27.4 ± 6.7 in the combat marksmanship dataset vs. 33.9 ± 11.1 in the golf dataset) and physiological signals are age dependent, the transfer of learning over these two datasets was successful, presumably due to the normalization of data. Furthermore, the EEG-based engagement metric is derived from the model individualized for each subject based on the EEG data during the baseline VPVT, APVT, and 3CVT tasks. Utilization of individualized metrics such as EEG-based engagement further facilitates knowledge transfer between different tasks. Although the algorithm proposed in our work is built upon the approach of Stevens et al. (2009, 2013) we extended it in a number of ways. First, we did not analyze the three metrics separately, but we combined EEG-engagement, EEG-workload, and heart rate into a single NN to capture synchronous changes of the metrics. Second, we did not apply a quantization into the upper, middle, and lower quartile, rather, we analyzed the metrics on a fine-grained continuous scale to be able to capture subtle cognitive changes during the tasks. Third, we incorporated temporal modeling in the NN itself by including 3 consecutive seconds of data into a feature vector. Thus, already the NN nodes were, to some extent, able to capture relevant temporal patterns in the data. Further investigations are needed to estimate the optimal window size, but increasing it from 1 to 3 s allowed for better temporal modeling and interpretation of the NN nodes. Fourth, we reduced the number of the NN nodes from 25 [that was used in Stevens et al. (2009, 2013)] to 20, which proved to be sufficient for our model and at the same time, reduced the time required to train the model, resulting in a more efficient algorithm. Fifth, we introduced a simple, yet effective NN state transition score that was correlated with performance scores in both analyzed datasets. Lastly, Stevens et al. (2009, 2013) focused on the analysis of team settings, which present a number of confounding factors, but the goal of this study was to analyze the individual changes in cognitive states and to eventually reveal the patterns relevant for characterizing performance on different tasks. We will also investigate the potential of better feature interactions coverage in the two-dimensional NN grid.

The current study sought to develop a method for capturing cognitive state changes that could be integrated in a number of applications, but this is only the first step in developing a performance estimation system that could be implemented in real-time and utilized in real-world settings. In order to apply the proposed approach in real-time, after the NN model is trained, one would need to collect the appropriate baseline data for new users to allow for individual data normalization (i.e., z-scoring utilized in our algorithm). Another option would be to compute mean and standard deviation of the subject's EEG-engagement, EEG-workload, and heart rate metrics (that are necessary for z-scoring) adaptively online during the task itself. Thus, the algorithm could be relatively easily implemented in real-time. The main limitation is the 1 s delay required to capture the third second of data in our feature vector. In most of the envisioned potential applications of the proposed approach, this delay would be acceptable.

Furthermore, we analyzed different performance scores, such as percent of enemy hits and enemy deaths in the combat marksmanship study, to evaluate if we can estimate subtle differences in performance by capturing neurophysiological differences in the subjects' reactions to relatively similar events in this very rich dataset. However, larger studies are necessary in order to further refine the NN model to be able to also capture neurophysiological correlates of performance improvement. In the combat marksmanship study there was no clear overall improvement in performance throughout the scenarios, while in the golf study, performance was slightly improved at the end of the study compared to the initial performance, but there was a drop in performance seen in the second half of the golf sessions. As the golf study sessions lasted longer than the combat marksmanship sessions, the decrease in performance in the golf study was presumably due to a loss in concentration as the study progressed. Thus, a larger number of skill training sessions across different days is needed. We utilized simulated training scenarios in these initial studies to minimize the noise that could be introduced by environmental variables in the real-world settings, however, the training tasks were made as realistic as possible by working closely together with established military and golf training domain experts. We plan to extend the combat marksmanship study and include professional soldiers in order to obtain more realistic data, and compare their physiological profiles with the novices' profiles. Further validation on the additional tasks, such as driving, is also planned. Nonetheless, the statistically significant correlations between the NN-based transition scores and performance are a promising first step toward the final goal of performance prediction that could be further used across a wide range of real-world application scenarios to improve, accelerate, and increase efficiency of the skill learning process by identifying an adaptive, intelligent, and multimodal neurofeedback based on the relevant neurophysiological patterns discovered in our datasets.

# **ACKNOWLEDGMENTS**

This work was supported by the Defense Advanced Research Projects Agency contract NBCH090054. The views, opinions, and/or findings contained in this article are those of the authors and should not be interpreted as representing the official views or policies, either expressed or implied, of the Defense Advanced Research Projects Agency.

# **REFERENCES**

marksmanship training. *Found. Augmented Cogn.* 5638, 630–639. doi: 10.1007/ 978-3-642-02812-0\_72

**Conflict of Interest Statement:** Authors Stikic, Berka, Levendowski, Rubio, Tan, and Korszen are paid salaries and/or shareholders of Advanced Brain Monitoring, Inc. The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 15 February 2014; accepted: 08 October 2014; published online: 05 November 2014.*

*Citation: Stikic M, Berka C, Levendowski DJ, Rubio RF, Tan V, Korszen S, Barba D and Wurzer D (2014) Modeling temporal sequences of cognitive state changes based on a combination of EEG-engagement, EEG-workload, and heart rate metrics. Front. Neurosci. 8:342. doi: 10.3389/fnins.2014.00342*

*This article was submitted to Neuroprosthetics, a section of the journal Frontiers in Neuroscience.*

*Copyright © 2014 Stikic, Berka, Levendowski, Rubio, Tan, Korszen, Barba and Wurzer. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Cardiovascular state changes in simulated work environments

#### *Arjan Stuiver 1 \* and Ben Mulder 2*

*1 Neuropsychology, Behavioural and Social Sciences, University of Groningen, Groningen, Netherlands*

*2 Experimental Psychology, Behavioural and Social Sciences, University of Groningen, Groningen, Netherlands*

#### *Edited by:*

*Thorsten O. Zander, Technical University of Berlin, Germany*

#### *Reviewed by:*

*Michal Lavidor, Bar Ilan University, Israel Johanna Wagner, Graz University of Technology, Austria*

#### *\*Correspondence:*

*Arjan Stuiver, Neuropsychology, University of Groningen, Grote Kruisstraat 2/1, 9712 TS Groningen, Netherlands e-mail<EMAIL_ADDRESS>
The usefulness of cardiovascular measures as indicators of changes in cognitive workload has been addressed in several studies. In this paper the question is explored whether cardiovascular patterns in heart rate, blood pressure, baroreflex sensitivity and HRV that are found are consistent within and between two simulated working environments. Two studies, were performed, both with 21 participants: one in an ambulance dispatch simulation and one in a driving simulator. In the ambulance dispatcher task an initial strong increase in blood pressure is followed by a moderate on-going increase in blood pressure during the next hour of task performance. This pattern is accompanied by a strong increase in baroreflex sensitivity while heart rate decreases. In the driving simulator study, blood pressure initially increases but decreases almost to baseline level in the next hour. This pattern is accompanied by a decrease in baroreflex sensitivity, while heart rate decreases. Results of both studies are interpreted in terms of autonomic control (related to both sympathetic and para-sympathetic effects), using a simplified simulation of a baroreflex regulation model. Interpretation of the results leads to the conclusion that the cardiovascular response patterns in both tasks are a combination of an initial defensive reaction, in combination with compensatory blood pressure control. The level of compensatory blood pressure control, however, is quite different for the two tasks. This helps to understand the differences in response patterns between the two studies in this paper and may be helpful as well for understanding differences in cardiovascular response patterns in general. A substantial part of the effects observed during task performance are regulatory effects and are not always directly related to workload manipulations. Making this distinction may also contribute to the understanding of differences in cardiovascular response patterns during cognitive workload.

**Keywords: cardiovascular reactivity, mental workload, state assessment, simulated work, baroreflex**

# **INTRODUCTION**

Cardiovascular measures are extensively studied in both laboratory and applied environments to gain insight in either responsiveness or in operator state changes during continuing mental work. In recent years, research on operator state assessment has frequently focused on developing applications for adaptive automation. Cardiovascular measures are mentioned by different authors as good candidates for the assessment of operator state because they can be measured relatively easily and continuously (Hockey et al., 2003; Mulder et al., 2003, 2004).

In several studies it has been shown how psychophysiological measures can be used in adaptive automation (Pope et al., 1995; Prinzel et al., 2000; Fairclough and Venables, 2006; Ting et al., 2010). Knowledge about a person's current state, in combination with information about task load and task performance, can be applied to adapt the working environment to fit the user's demands or needs (Haas and Hettinger, 2001). In this context, it is important to take into account the demand a task is placing on the user. The basic idea of adaptive automation or adaptive support is to design a system that will give valuable help, being a companion for the operator during periods of expected over- or underload (Hoogeboom and Mulder, 2004), for example by controlling task demands.

Physiological information that can be used in adaptive automation may consist of response patterns to momentary changes in task load or could consist of state changes related to continuous work or both. In this context, a reaction to changes in workload that last only a short period of time, e.g., 30 s to 1 min, is (part of) a response pattern to momentary changes. State changes indicate the long-term effects up to a couple of hours. One of the prerequisites for using either short-term response patterns or state changes is that consistent cardiovascular patterns have to be available at an individual level for the task under focus. Another requirement is that a distinction between periods of low and high workload can be made on the basis of changes found in these patterns. Knowing that this is a multidimensional problem, the focus of the present paper will be on the consistency of cardiovascular state changes in two task environments: an ambulance dispatching task and a driving simulator. Based on the results found suggestions will be made on how to apply such measures and which issues have to be addressed for applications in the world of adaptive automation.

Effects of mental effort on heart rate (HR) and heart rate variability (HRV) have been extensively studied during laboratory tasks (Mulder and Mulder, 1987; Backs and Seljos, 1994), simulated work (Brookings et al., 1996; Veltman and Gaillard, 1998; De Rivecourt et al., 2008; Dijksterhuis et al., 2011) and during real work (Roscoe, 1992; Wilson, 1993; De Waard et al., 1995; Hankins and Wilson, 1998). In general, during effortful working periods in these tasks, a pattern is found of increased HR in combination with decreased HRV, compared to resting baselines or compared to conditions of lower workload.

However, the sensitivity of these effects depends strongly on the task load differences between conditions, the type of task and time on task (Mulder and Mulder, 1987; Althaus et al., 1998; Backs and Boucsein, 2000). Effects of increased task demand correspond with elevated (finger) blood pressure (BP), lowered baroreflex sensitivity (Mulder and Mulder, 1981; Reyes del Paso et al., 2004) and higher respiration rate (Wientjes, 1992). Some authors have characterized this response as a defense reaction or a preparatory fight-or-flight response (Mulder, 1980; Jordan, 1990; Berntson et al., 1991). Generally speaking, the effects in laboratory tasks and simulated work are more transparent than those in real life work environments, due to varying task demands, unknown task characteristics, time on task effects and adaptivity of the short term blood pressure control system (baroreflex, e.g., Mulder, 1992) to these changing task demands. In laboratory conditions HRV clearly showed more consistent effort effects than HR and is therefore proposed as a good indicator of invested mental effort in cognitively demanding tasks (Mulder and Mulder, 1981).

A well-known fatigue and monotony effect on HR, i.e., a gradual decrease during the working day, is reported in many studies (Myrtek et al., 1994; Raggatt and Morrissey, 1997). A similar phenomenon is found during much shorter sessions, e.g., an hour of continuous task load. Mulder et al. (1993) showed that initial HR(V) effects resembling a defense reaction disappeared after 10–20 min in a memory search task that lasted 45 min, while BP and baroreflex sensitivity (BRS) remained the same after the initial effects, i.e., BP remained high and BRS remained low. The authors concluded that these effects were directly related to short-term blood pressure control (baroreflex, Van Roon et al., 2004; Mulder et al., 2009), a control system in the body that tries to maintain stable blood pressure. Baroreceptors monitor changes in blood pressure and influence sympathetic and parasympathetic activation. A continuing elevation of blood pressure increases sensitivity of this system and increases parasympathetic and decreases sympathetic activation, lowering heart rate and indirectly blood pressure. The question arises how generalizable and how consistent the patterns described above (an initial defense response followed and possibly overshadowed by effects of the short-term blood pressure regulation mechanism) actually are and how to interpret such changes in terms of autonomic control (Berntson et al., 1991; Backs, 1998).

The effects of the typical adaptive baroreflex response described above may obscure the direct effects of task demand manipulations, because these adaptive effects might overshadow or at least diminish the effects of changes in task demand. It has to be kept in mind that both types of effects occur simultaneously and are in principle present at any moment, which makes interpretation of effects of task demand more difficult. This may be a reason to split the research regarding these effects in two directions: the first perspective is to look at specific cardiovascular state changes over longer time periods. The second is to look at smaller time segments connected to specific short lasting changes in task demand. In this paper we will focus on the first approach.