Experiments of this study were conducted in an 8-screen and 8-projector immersive virtual-reality (VR) environment that simulates the 1st person view scene of highway driving. This study adapted an event-related lane-departure driving paradigm originally proposed by Huang et al. (2006, 2007a,b, 2009) that allowed objective and quantitative measures of momentary event-related brain dynamics following lane-departure events and drivingperformance fluctuations over longer periods. The VR scenes simulated driving at a constant speed (at ∼100 km/h) on a highway with the simulated car randomly drifting away from the center of the cruising lane to simulate driving on non-ideal road surfaces or with poor alignment (Huang et al., 2007a,b, 2009; Lin et al., 2008b, 2009). The scene was updated according to the land-departure events and the subject's manipulation. The vehicle trajectory, user's input, and lane-departure events could be accurately logged and time-synchronized to the EEG recordings (Huang et al., 2007a,b, 2009; Lin et al., 2008b). There were no traffic or distractive objects other than 4-lane roads and dark sky appeared in the VR while the simulated car was cruising on the highway.

Thirty-two channel EEG data were collected from participants by the NuAmp system (32-channels Quick-Cap, Compumedics Ltd., VIC, Australia). The electrodes were placed according to a modified international 10–20 system with a unipolar reference at the right earlobe. The EEG activities were recorded with 500 Hz sampling rate and 16-bit quantization level.

# **EXPERIMENTAL PARADIGM**

**Figure 1A** shows the experimental paradigm of this study. The simulated car starts cruising at a fixed speed (∼100 km/h) on the 3rd lane and drifting to either right or left with equal probability within 8–10 s. Subjects were instructed to steer the simulated car back to the 3rd lane as soon as they noticed the lane drift. The simulated car keeps cruising on the right (or left) most lane if the subjects failed to respond to lane drift. The baseline period of each lane-departure epoch is defined as the 3 s before the onset of a lane-drifting event. The empty circle in **Figure 1A** represents the unexpected lane-departure events marked as the "deviation

represents the deviation onset. The double circle represents the response onset. The circle with the cross represents the response offset. The baseline is defined as the 3 s period prior to deviation onset. The response time (RT) of a driver is the interval from the deviation (empty circle) to the response onset (the double circle). A trial starts at deviation onset and ends at

tasks. The height of an arrow represents the response time in a single trial. The warning was delivered to the subject when the RT in the trial exceeded three times the mean RT of trials in the first 5 min of the task, when the subject was presumably alert and fully attended to lane-departure events. In this figure is adapted with permission from Figure 1 of Lin et al. (2010a).

onset." After the deviation onset, subjects were instructed to steer the simulated car back to the center of the cruising lane immediately (double circle), and the time when the subjects started steering was marked as the "response onset." The moment that the simulated car reached the center of the cruising lane (circle with cross) was marked as the "response offset." A subject's response time (RT) was defined as the time between the deviation onset and the response onset. At the first 5 min of the experiment, subjects were asked to be fully alert, verified by the vehicle trajectory and the video from a surveillance camera, to obtain an averaged alert RT (aRT) for each subject (1.51∼2.54 s), which is a threshold for the entire experiment. The entire experiment consisted of 5 min training and 85 min driving periods.

**Figure 1B** shows the criterion of delivering auditory warnings in the experiment. When a subject failed to respond within three times the aRT, the system treated the trials as a behavioral lapse and triggered a 1750 Hz tone-burst to arouse the subject from fatigue-related lapse in half (50%) of these drowsy trials (marked as the "current trial (CT)" in **Figure 1A**). The very next trial is defined as CT + 1, and so on. The lapse trials that were randomly selected to receive arousing warning were referred to as CT with warning, whereas the remaining half of trials that did not receive auditory warning were referred to as CT without warning. Note that our previous studies showed that in some trials subjects remained non-responsive following the arousing warning, which was analogous to sleeping through an alarm clock (Jung et al., 2010; Lin et al., 2013). If the RT of the following trial (CT + 1) was shorter than the double of the averaged aRT, the warning signal delivered in the CT trial was defined as an "effective warning." On the other hand, if the RT of the CT + 1 trial was longer than triple of the averaged aRT, the warning was defined as an "ineffective warning." This study did not include the trials with RTs between 2 and 3 aRT to define the alert vs. fatigue spectral thresholds because the cognitive states of the subjects during those trials were unclear. Note that, subjects didn't know about the warning before the experiments.

#### **DATA ANALYSIS**

The 32-channel EEG data were first down-sampled to 250 Hz, and a low-pass filter of 50 Hz and a high-pass filter of 0.5 Hz were applied. Channels or trials with severe artifacts (such as body movements or muscle activities) were manually removed (less than three channels and 20% trials per subject in general). The remaining EEG data were segmented into several 115 s trials, each of them consisting of 15 s before and 100 s after the lane-deviation onsets. Independent Component Analysis (ICA, Bell and Sejnowski, 1995; Makeig et al., 1997) implemented in EEGLAB (Delorme and Makeig, 2004) was then applied to decompose the ∼32-channel EEG into ∼32 independent components (ICs), based on the assumption that the collected EEG data from the scalp were a weighted linear mixture of electrical potentials projected instantaneously from temporally ICs accounting for distinct brain sources. The comparable ICs across subjects were grouped into component clusters based on their scalp maps, equivalent dipole locations and baseline power spectra of component activations (Jung et al., 2001; Delorme and Makeig, 2004). Across 11 subjects, there were 155 trials with warning (30 trials were ineffective and 125 trials were effective) and 192 trials without warning.

Since the RT and EEG power were not normally distributed, non-parametric statistic tests were performed for the data analysis (Delorme and Makeig, 2004). The Wilcoxon rank-sum test (Matlab statistical toolbox, Mathworks) was used to assess the effects of warning on RTs. Bootstrapping (EEGLAB toolbox, University of California, San Diego) was used to test the statistical significance of EEG power changes at specific frequency bins from 2 to 30 Hz with a 0.25 Hz resolution. To test group statistics, the intrinsic inter-subject RT differences were reduced by dividing RTs by the mean RT. The EEG spectra were normalized by dividing the spectral power by the standard deviation of the spectral distribution.

# **RESULTS: NEUROPHYSIOLOGICAL CORRELATES OF BEHAVIORAL LAPSES**

**EFFICACY OF AROUSING AUDITORY SIGNALS FOR RECTIFYING LAPSES** This study first explored the efficacy of the delivery of arousing auditory signals by measuring the change in subjects' reaction time. **Figure 2A** shows the boxplots of RTs of three trial groups:

**FIGURE 2 | Experiment results. (A)** The boxplot for the RT distribution of trials with effective warning, ineffective warning, and without warning among CTs and CTs + 1. Note that middle horizontal line is the median of the distribution, and the top and bottom of the rectangle are the third and first quartile, and the dash line ends are the maximum and minimum after outlier removal. **(B)** The component spectra of the alert CTs (black curve), with an effective warning (red curve), with an ineffective warning (light blue curve) and without warning (dark blue curve). The red, light blue and blue horizontal lines mark the spectral differences between the alert trials and trials with an effective warning, with an ineffective warning, and without warning, respectively. All the spectral plots were calculated from the activity of the bilateral occipital components separated by ICA.

Alertness, CT, and CT + 1 (left to right). The averaged aRT of trials within the Alertness group across 11 subjects was ∼676 ms. The RTs of the CT group with arousing warning (red and light blue) were statistically significantly shorter than those of trials without receiving arousing warning (dark blue). The RTs of the CT + 1 group with effective vs. ineffective warning differed while the RTs of the preceding group (CT) were comparable. Even though the subjects responded to the arousing warning by immediately steering the simulated car back to the cruising position, they could well be totally non-responsive to the very next lanedeparture event (∼10 s later). In other words, the arousing signals reliably rectified human behavioral lapses, but did not guarantee that subjects were fully awake, alert, or attentive. This suggests an analogous regime of snooze after an alarm is turned off.

#### **EEG DYNAMICS PRECEDING BEHAVIORAL LAPSES**

**Figure 2B** shows the mean scalp map of the bilateral occipital cluster (upper-right corner) and its component baseline power of drowsy trials without auditory warning (dark blue), with either effective (red) or ineffective warning (light blue). First, among the resultant ICA clusters, bilateral occipital components exhibited statistically significant spectral differences between trials with and without auditory warning. Second, the component power spectra exhibited tonic increases in theta (4–7 Hz), alpha (8–12 Hz), and beta (13–30 Hz) bands in drowsy trials (red, dark blue, and light blue), compared to the alert trials (black). Horizontal lines mark the frequency bins under which the spectral differences between alert trials and drowsy trials with either (in)effective warning, or without warning were statistically significant (alpha = 0.05, Bonferroni adjusted *p*-value of 0.05/(112 frequency bins) = 0.0004 for multiple comparisons). Note that the spectra shown here were calculated from the component activities prior to the lane-deviation onset. The nearly identical pre-lapse spectra of these three groups of non-responsive trials demonstrate the robustness of the broadband spectral augmentation preceding the behavioral lapses, suggesting the feasibility of using theta and alpha power from the lateral occipital areas to predict behavioral lapses in this sustained-attention driving task.

#### **EFFECTS OF AROUSING AUDITORY SIGNALS ON THE EEG**

Next, this study explored temporal spectral dynamics preceding, during and following fatigue-related behavioral lapses and following arousing warning. **Figure 3** shows time courses of spectral changes in the bilateral occipital area following ineffective warning (light blue trace), effective warning (red trace), and without warning (dark blue trace), compared to those of the alert trials (black trace). **Figure 3** shows that both theta- an alpha-band power steadily increased prior to the lane-departure onset (at time 0 s). Again, the trends of steady increasing theta- and alpha-band power leading to behavioral lapses in the three groups of drowsy trials were nearly identical, indicating the robustness of the theta and alpha augmentation preceding the behavioral lapses.

**Figure 3** also shows that after the lane-departure onset (at time 0 s), the alpha (top panel), and theta (bottom panel) power abruptly decreased by over 10 and 5 dB to nearly the alert (black trace) baseline, respectively. More importantly, following the subjects' responses, the spectra of trials with ineffective warning (light blue trace) and without warning (dark blue trace) rapidly rose from the alert baseline to the drowsy level in 5–15 s. The theta and alpha power of trials with effective warning, however, remained low for ∼40 s. The green horizontal lines mark the time points when the difference between the spectra of trials with

ineffective feedback, without feedback, and in alertness, respectively. The green horizontal line indicates the statistically significant differences (*p <* 0*.*01) between trials with effective feedback and without feedback. The brown indicates the statistically significant differences (*p <* 0*.*01) between trials with effective feedback and ineffective feedback.

effective warning and without warning were statistically significant (*p <* 0*.*01). The spectral difference between the trials with effective warning and without warning was significant from 7 to 18 s in alpha band and from 7 to 21 s in the theta band (*p <* 0*.*01). Furthermore, the spectral difference between the trials with effective and ineffective warning was significant from 7 to 16 s in both alpha and theta bands (brown horizontal lines).

In sum, these results provided invaluable insights into the optimal electrode locations (lateral occipital region) and EEG features (theta- and alpha-band power) for a practical OCLDM system detailed below. The EEG and behavioral data collected from this experiment were used to assess the EEG correlates of fatiguerelated lapses and build a lapse prediction model for the second experiment.

### **DEVELOPING A OCLDM SYSTEM**

Our previous study (Wang et al., 2012) proposed a cellphone based drowsiness monitoring and management system to continuously and wirelessly monitor brain dynamics using a lightweight, portable, and low-density EEG acquisition headgear. The system was designed to assess brain activities over the forehead, detect drowsiness, and deliver arousing warning to users experiencing momentary cognitive lapses, and assess the efficacy of the warning in near real-time. However, the system was not fully implemented nor experimentally validated in humans. Furthermore, according to the neurophysiological results in section Results: Neurophysiological Correlates of Behavioral Lapses, EEG signals collected over the lateral occipital regions were more informative for lapse detection. This study extends the previous work to design, develop, and test an OCLDM System.

#### **SYSTEM ARCHITECTURE**

**Figure 4A** shows the system diagram of the proposed OCLDM System. The system consists of two major components: (1) a mobile platform featuring the OCLDM algorithm, and (2) a mobile and wireless 4-channel headgear measuring EEG signals over the hair-bearing occipital regions with dry EEG sensors (Liao et al., 2011). The OCLDM System was implemented as an App on an Android-based platform (e.g., Samsung Galaxy S3). The smartphone has a Bluetooth module, 16 GB RAM, an ARM Cortex-A9 processor, Android (Ice Cream Sandwich) OS, and other components. When the App is launched, it can automatically search and connect to a nearby EEG headgear to receive data from the EEG acquisition headgear. In the mean time, the App opened an USB port to receive the events from a four-lane highway scene to synchronize the EEG data and scene events. The build-in speaker (or plug-in a ear set) of the smartphone delivers auditory warning signal once the OCLDM System detects that the subject is experiencing a cognitive lapse. Both the EEG data and scene-generated events could be logged onto either smartphone's build-in memory or an external microSD card for further analysis.

The mobile and wireless EEG acquisition headgear features a 4-channel lightweight portable bio-signal acquisition device powered by a 3.7 v Li-ion battery (Lin et al., 2010b). It consists of a TI MSP430 microprocessor, a pre-amplifier, a battery-charging circuit, a 24 bit ADC, a Bluetooth module, and dry spring-loaded

The EEG headgear collected 4-channel brain activities from the lateral occipital area while a subject was performing the lane-drifting experiment. The mobile signal-processing platform received the acquired EEG raw data through Bluetooth, and the event markers generated from the lane-departure scene through an USB interface. Finally, the auditory feedback was delivered to the subject when the averaged EEG power across four channels was 3 dB over the alert baseline. **(B)** A photo of a subject performing the on-line driving experiment while wearing a 4-channel EEG headgear (the white small box attached on a flexible band) over the lateral occipital area.

EEG sensors (Liao et al., 2011). The spring-loaded probes of the sensor can penetrate the hair to provide good electrical conductivity with the scalp. The microprocessor controls all the components including the amplifiers, digitizers, and transmits the digitalized EEG data to the Bluetooth module. The 4-channel EEG data are then transmitted to the authorized receiver of the OCLDM System. Depending on the applications, the system's sampling rate can be programmed at 128, 256, or 512 Hz. An experienced subject can easily put on this EEG acquisition device within 1–3 min without any help from a technician. **Figure 4B** shows a photo of a subject wearing a 4-channel EEG headgear and performing the simulated driving experiment.

### **SYSTEM SOFTWARE DESIGN**

**Figure 5** shows the program's state diagram of the proposed OCLDM System. Three major states, including Baseline Collecting (BC), Driving Performance Monitoring (DPM), and Warning Efficacy Assessment (WEA), were implemented in the program. When the program is launched, one can modify the parameters in the SETTING page, shown as a square box in the figure. For instance, the parameters can be the duration of baseline data collection, or the threshold of auditory warning delivering for the other two states. Depending on the applications, the lapse threshold in the DPM state can be calculated accordingly. For example, one can use a combination of power of alpha, beta, theta, and delta bands to detect cognitive lapse. The program then enters the next (DPM) state after the Baseline (calibration) data collection has completed. The DPM module continuously monitors the driver's neurophysiological data. The program stays in the DPM state until the lapse threshold is met, which depends on the neurophysiological results as shown in section Results: Neurophysiological Correlates of Behavioral Lapses. For instance, when the subject's power spectrum in alpha band is 3 dB higher than the threshold (alert baseline collected in the BC state), the program delivers an auditory warning to arouse the subject and enters the FEA state. The current value is stored as the lapse reference in the FEA module. The system repeatedly delivers auditory warning until the EEG power decreases to another threshold.

### **ON-LINE EXPERIMENTAL PARADIGM**

Three new male subjects (who did not participated in the first experiment) with normal hearing and aged 25–30 years old

over the alert baseline. Lapse2 represents that the averaged EEG power across four channels has not yet dropped 3 dB from the lapse power.

participated in the on-line closed-loop lane-departure driving experiments to evaluate the OCLDM System in a more naturalistic setting (in a regular office without any electromagnetic shielding). All of them were asked to read and sign the informed consent form before participating in the studies.

The entire experiment consisted of a 1 min training and a ∼60 min driving periods. During the training session, subjects were asked to stay fully alert. The averaged alpha power collected in the BC session was used as an alert baseline to determine whether a subject is experiencing cognitive lapses in the driving task. The subject performed the lane-departure driving experiments following the protocol below:

The cognitive lapses were detected when the subject's alpha-band power (Jung et al., 2010; Lin et al., 2010a, 2013), calculated by a moving-averaged STFT with a 256-point sliding window advanced at 1 s step running on the smartphone, was 3 dB over the alert baseline power (Lin et al., 2010a, 2013 and Results in section Results: Neurophysiological Correlates of Behavioral Lapses). This study used the alpha power fluctuations to monitor cognitive lapses because (1) a recent study showed that the alpha augmentation was sensitive to the transition from full alertness to mediate drowsiness, while the theta augmentation was more sensitive to the transition from mediate to deep drowsiness (Chuang et al., 2012); (2) the empirical results of this study showed that the augmentation of alpha-band power changes was greater than that of the theta-band power (**Figure 2**). The system would repeatedly deliver auditory warning until the alpha-band power amplitude has dropped to 3 dB below the power level when the cognitive lapse was identified.

# **RESULTS FROM THE OCLDM SYSTEM**

The numbers of detected cognitive lapses varied across subjects. **Table 1** lists the numbers of trials with effective, ineffective warning and without warning, respectively. Here, the way we defined the effective trials was based on the RT in response to the lane-departure event immediately following the arousing signal (CT + 1 whose RT was shorter than two times aRT); while the ineffective trials had RT longer than three times aRT.

**Figure 6** shows the boxplot of behavioral performance (RTs) of trials with effective trials (red), ineffective trials (light blue), and without warning (dark blue), compared to the averaged aRT (black) during the on-line experiments. The effective trials had RTs comparable to the averaged aRT (less than 1 s) in both CT and CT + 1. Note that the RTs of CT + 1 with effective vs. ineffective warning differed largely because that was how the effective and ineffective trials were defined. However, the RTs of CT trials (red and light blue) of these two groups of trials were very comparable. That is, even though the subjects responded to the arousing warning by steering the simulated car immediately back to the cruising position, they could well be totally non-responsive to the very next lane-departure event. This finding is consistent to our off-line study reported in section Results: Neurophysiological Correlates of Behavioral Lapses in which the arousing warning was delivered to the subjects who just had a behavioral lapse.

**Figure 7** showed the averaged alpha-band spectral time courses across subjects and trials with effective warning (red trace), with ineffective warning (light blue trace), and without warning (dark blue trace), compared to averaged aRT (black trace). All spectral time courses were aligned to the user response onset (thick vertical black line at time 0 s), and the auditory warning for effective- and ineffective-trials were delivered ∼5 s before

(light blue), and without feedback (blue trace), compared to trial with aRT (black). The time course of power was estimated by short time Fourier transform with 256 points of time window and 224 points overlapping.

the user response. In the trials following effective auditory warning, the alpha power decreased steadily and reached the averaged aRT in ∼7 s. The power spectra remained as low as that of the alert baseline from 7 to 20 s after response onset. In the trials with ineffective auditory warning, the spectral time series fluctuated fierously due to the small number of trials. In the trials without warning, the alpha power fluctuated before response onset and steadily dereased until ∼7 s. Thereafter, the alpha power increased again from ∼7 to 13 s, suggesting the subjects might be partially arouse by the lane-departure event and their own bebavioral reposense temportally but returned to the fatigue state rapidly thereafter.

#### **DISCUSSIONS**

Many studies have shown that the brain dynamics correlated with behavioral lapses can be assessed from EEG data. Recent studies have also shown auditory signals can arouse drowsy subjects and affect EEG activities (Lin et al., 2010a, 2013). However, in these studies, the arousing warning was delivered to subjects after they displayed behavioral lapses, which in reality may be too late because the behavioral lapse might have already had catastrophic consequences. Therefore, a system that features real-time lapse detection and delivers warnings to the drowsy subjects is desirable in preventing catastrophic incidents while driving.

The first experiment of this study showed that EEG power changes in either alpha or theta band can be used as an indicator for assessing the subjects' fatigue (cf. **Figure 3**), and auditory warning temporarily reduces the alpha and theta band power and mitigates the behavioral lapses (cf. **Figure 2A**). In addition, EEG changes after delivery of auditory warning are a good indicator of the efficacy of arousing warning. More importantly and interestingly, empirical results of the first study showed that arousing auditory signals could always reliably mitigate human behavioral lapses, but these immediate behavioral responses could not guarantee the subjects were fully awake, alert, or attentive, similar to snooze after an alarm is turned off. This finding may open a new research direction of how to accurately confirm a subject's cognitive level for some sustained-attention tasks, such as an aircraft navigator or a long-haul truck driver. In other words, further studies to explore the brain changes in this sleep inertia period may provide valuable insights of brain dynamics during a transitional state of lowered arousal occurring immediately after awakening from sleep. Based on previous studies (Lin et al., 2010a, 2013) and the results of the first experiment, this study further developed a truly OCLDM system to detect/predict cognitive lapse based on the EEG spectra, deliver arousing warning on the occurrence of cognitive lapse, and assess the efficacy of the arousing warning, again, based on the EEG spectra. Most importantly, the EEG spectra changes within ∼10 s after delivering arousing warning were closely monitored, such that any false-awake situations could be decreased. This study then documented the design, development, and on-line evaluation of the proposed OCLDM System that featured a lightweight wireless EEG acquisition headgear and a smartphone-based signalprocessing platform. Experimental results showed that subjects' EEG power could almost remain at the alert state without bouncing back to the drowsy level (cf. **Figure 7**). These results suggest that the proposed system could prevent potential behavioral lapses based solely on the EEG signals, and this demonstration could lead to a real-life application of the dry and wireless EEG technology and smartphone-based signal-processing platform. An interesting question is if the neural correlates of fatigue could be generalized across different sustained-attention tasks and different recording conditions. In the past few years, we have conducted several sustained-attention tasks, including auditory target detection tasks (Makeig and Jung, 1995, 1996; Jung et al., 1997), visual compensatory tracking tasks (Huang et al., 2008), and simulated driving tasks (Lin et al., 2005, 2006, 2008b) and found that performance-related EEG dynamics were comparable across tasks (Huang et al., 2007b). Results of these studies also showed the fatigue-related brain dynamics were quite consistent across different recording environments (within a well-controlled EEG laboratory vs. a 6-degree-of-freedom motion platform) and responding methods (using a button press or a steering wheel). Therefore, it is reasonable to believe the methods developed under this study could be translated from laboratory settings to real-world environments.

In sum, this study demonstrated the feasibility of translating a laboratory-based passive BCI system to a neuroergonomic device that is capable of continuously monitoring and mitigating operator neurocognitive fatigue using a pervasive smartphone in realworld environments. The passive BCI technologies might also be applicable to other real-world cognitive-state monitoring, such as attention, distraction, comprehension, confusion, and emotion. We thus believe more real-world passive BCI implementations will emerge in the foreseeable future.

#### **ACKNOWLEDGMENT**

This work was supported in part by US Office of Naval Research, Army Research Office (W911NF-09-1-0510), Army Research Laboratory (W911NF-10-2-0022), and DARPA (DARPA/USDI D11PC20183). This work was also supported in part by the UST-UCSD International Center of Excellence in Advanced Bioengineering sponsored by the Taiwan National Science Council I-RiCE Program under Grant Number: NSC-102-2911-I-009- 101, and the Aiming for the Top University Plan of National Chiao Tung University sponsored by the Ministry of Education of Taiwan, under Grant Number 103W963. The authors also appreciate Melody Jung's editorial assistance.

### **REFERENCES**

Zander, T. O., Kothe, C., Welke, S., and Roetting, M. (2009). "Utilizing secondary input from passive brain-computer interfaces for enhancing human-machine interaction," in *Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience*, eds D. Schomorrow, I. Estabrooke, and M. Grootjen (Berlin; Heidelberg: Springer), 759–771.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 15 February 2014; accepted: 24 September 2014; published online: 13 October 2014.*

*Citation: Wang Y-T, Huang K-C, Wei C-S, Huang T-Y, Ko L-W, Lin C-T, Cheng C-K and Jung T-P (2014) Developing an EEG-based on-line closed-loop lapse detection and mitigation system. Front. Neurosci. 8:321. doi: 10.3389/fnins.2014.00321*

*This article was submitted to Neuroprosthetics, a section of the journal Frontiers in Neuroscience.*

*Copyright © 2014 Wang, Huang, Wei, Huang, Ko, Lin, Cheng and Jung. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Knowledge-based identification of sleep stages based on two forehead electroencephalogram channels

# *Chih-Sheng Huang1,2, Chun-Ling Lin1,3, Li-Wei Ko1,4,5\*, Shen-Yi Liu1,6, Tung-Ping Su1,6 and Chin-Teng Lin1,2\**

*<sup>1</sup> Brain Research Center, National Chiao-Tung University, Hsinchu, Taiwan*

*<sup>2</sup> Institute of Electrical Control Engineering, National Chiao-Tung University, Hsinchu, Taiwan*

*<sup>3</sup> Department of Electrical Engineering, Ming Chi University of Technology, New Taipei City, Taiwan*

*<sup>4</sup> Institute of Bioinformatics and Systems Biology, National Chiao-Tung University, Hsinchu, Taiwan*

*<sup>5</sup> Department of Biological Science and Technology, National Chiao-Tung University, Hsinchu, Taiwan*

*<sup>6</sup> Department of Psychiatry, Taipei Veterans General Hospital, Taipei, Taiwan*

#### *Edited by:*

*Thorsten O. Zander, Technical University of Berlin, Germany*

#### *Reviewed by:*

*Josef Faller, Graz University of Technology, Austria Sarah Christin Freytag, Berlin Institute of Technology, Germany*

#### *\*Correspondence:*

*Li-Wei Ko, Brain Research Center, Institute of Bioinformatics and Systems Biology, and Department of Biological Science and Technology, National Chiao-Tung University, 416 MIRC., 1001 University Rd., Hsinchu 300, Taiwan e-mail<EMAIL_ADDRESS>Chin-Teng Lin, Brain Research Center and Institute of Electrical Control Engineering, National Chiao-Tung University, 416 MIRC., 1001 University Rd., Hsinchu 300, Taiwan*

*e-mail<EMAIL_ADDRESS>
Sleep quality is important, especially given the considerable number of sleep-related pathologies. The distribution of sleep stages is a highly effective and objective way of quantifying sleep quality. As a standard multi-channel recording used in the study of sleep, polysomnography (PSG) is a widely used diagnostic scheme in sleep medicine. However, the standard process of sleep clinical test, including PSG recording and manual scoring, is complex, uncomfortable, and time-consuming. This process is difficult to implement when taking the whole PSG measurements at home for general healthcare purposes. This work presents a novel sleep stage classification system, based on features from the two forehead EEG channels FP1 and FP2. By recording EEG from forehead, where there is no hair, the proposed system can monitor physiological changes during sleep in a more practical way than previous systems. Through a headband or self-adhesive technology, the necessary sensors can be applied easily by users at home. Analysis results demonstrate that classification performance of the proposed system overcomes the individual differences between different participants in terms of automatically classifying sleep stages. Additionally, the proposed sleep stage classification system can identify kernel sleep features extracted from forehead EEG, which are closely related with sleep clinician's expert knowledge. Moreover, forehead EEG features are classified into five sleep stages by using the relevance vector machine. In a leave-one-subject-out cross validation analysis, we found our system to correctly classify five sleep stages at an average accuracy of 76.7 ± 4.0 (*SD*) % [average kappa 0.68 ± 0.06 (*SD*)]. Importantly, the proposed sleep stage classification system using forehead EEG features is a viable alternative for measuring EEG signals at home easily and conveniently to evaluate sleep quality reliably, ultimately improving public healthcare.

**Keywords: sleep quality, sleep stages, polysomnography (PSG), electroencephalogram (EEG), sleep stage classification system**

# **INTRODUCTION**

Monitoring human physiology during sleep is essential for individual health. Sleep is increasingly viewed as playing an important role in restitution (Akerstedt et al., 2007). As an important aspect of well-being, sleep quality is closely related to overall quality of life, life satisfaction, secretion of the stress hormone, cortisol, and inadequate immunity (Gallagher et al., 2010). Evaluating of sleep quality is especially relevant, owing to a considerable number of pathologies linked to the sleep. Sleep stages are also recorded for clinical diagnosis and the treatment of sleep disorders. Sleep quality is most closely related to the distribution of depth of sleep; indeed, sufficient sleep quality must reach adequate deep sleep. The depth of sleep is characterized by different cortical electrical activities. Several sleep stages can be defined by variations of cortical electrical activities and other physiological signals, i.e., muscle activity and eye movement. According to Rechtschaffen and Kales rules (R&K rules), sleep stage can be segmented into wakefulness, movement time (MT), REM and sleep stages S1, S2, S3, and S4 based on features of EEG, EOG, and EMG (Kales and Rechtschaffen, 1968). In addition to modifying the standard guidelines for sleep classification by R&K, the American Academy of Sleep Medicine (AASM) developed guidelines for terminology, recording method, and scoring rules for sleep-related phenomena (Iber et al., 2007). In the AASM guidelines, sleep stages S1 to S4 are referred to as NREM stage 1 (N1), NREM stage2 (N2), and NREM stage3 (N3). N3 reflects slow wave sleep (SWS, R&K stages S3 + S4).

As the reference standard clinical multi-parametric system, polysomnography (PSG) (Holland et al., 1974) is used in sleep studies to define the different physiological sleep stages and diagnose many sleep disorders, including narcolepsy, restless legs syndrome, rapid eye movement (REM) behavior disorder, parasomnias, and sleep apnea. The PSG system requires a minimum of 11 channels, including electroencephalogram (EEG), electromyogram (EMG), electrooculogram (EOG), oxygen saturation (SpO2), and electrocardiogram (ECG). However, assessing a complete PSG has several limitations. First, PSG is not a portable device and typically placed in a sleep center, which is unfamiliar environment for patients. Second, PSG requires many physiological electrodes and wires placed on the scalp and body, possibly affecting their sleep further. Third, a standard sleep diagnosis in clinical practice is time-consuming and expensive (Zoubek et al., 2007). These processes are monotonous, and time consuming and unproductive. A simpler EEG acquisition and analysis system must be operable by patients for home use, as well as solve current PSG problems.

Recent studies have adopted bioelectrical signals (i.e., EEG, ECG, EMG, and EOG signals), which allow subjects to operate at home in order to develop sleep stage scoring methods, while attempting to obtain results similar to those of experts involved in visual scoring (Park et al., 2000; Anderer et al., 2005; Tian and Liu, 2005; Berthomier et al., 2007; Doroshenkov et al., 2007; Virkkala et al., 2007; Wang et al., 2009; Güne¸s et al., 2010; Jo et al., 2010; Yιlmaz et al., 2010; Eiseman et al., 2011). The classification structure of most of sleep stage classifications consists of feature extraction and classification schemes. The references differ from each other not only in the presented feature extractions and the corresponding classification schemes, but also in different bioelectrical signals used, such as EEG, EOG, or ECG. Feature extraction is a highly efficient means of achieving a satisfactory classification performance in order to develop a sleep stage classification approach. Certainly, if extracted features can achieve a high separability in distinguishing between different classes, classifiers can perform satisfactorily. In recent studies (Berthomier et al., 2007; Doroshenkov et al., 2007; Güne¸s et al., 2010; Jo et al., 2010), the signal process procedure regards an entire 30-s epoch as a processing unit to extract spectral and temporal information directly. The specific characteristics of sleep stages are smoothened easily within an entire 30-s signal process. For instance, the k-complex and sleep spindle only appear suddenly in a short period with 0.5–1.5 s in a 30-s epoch. Therefore, a short-term signals process should be incorporated when a developing feature extraction approach. Compared with EEG measurement, despite overcoming the hair problem, EOG and ECG still have certain limitations. For instance, EOG and ECG requires adhesive electrode pads, and the locations of EOG (or ECG) and corresponding amplifier are divergent from each other. A subject's sleep position may interfere with the wire, thereby degrading the EOG and ECG signal quality. Despite the persistent hair and conductive gel problems associated with EEG measurement, recent developments to resolve these problems include a headband and portable EEG recording device, as well as a dry polymer foam electrode for long-term EEG measurement (Lin et al., 2008, 2011). Additionally, physiological characteristics during sleep are more easily identified by EEG than by EOG and ECG, explaining why the former is preferred when classifying sleep stages.

Berthomier et al. (2007) assessed an automatic sleep scoring software (ASSEEGA). The system adopts a 3-step procedure for automatic sleep scoring, based on a single EEG channel. In classifying five sleep stages, the agreements between ASSEEGA and two expert manual scorings are 76.0% (kappa = 0.67) and 78.2% (kappa = 0.69). Although highly promising for diagnostic and automatic ambulant scoring. This system still requires 2 bipolar channels (Cz-Pz, international 10–20 standard system), which are located at the back of the skull and hair site, conductive gel, and a laboratory EEG recording device to achieve a high resolution. The connection between hair and conductive gel, and subject's sleep position may also worsen the EEG quality, further lowering the estimation accuracy of the sleep stage.

This work develops a sleep stage classification system via two forehead EEGs, i.e., FP1 and FP2. FP1 and FP2 EEG measurements have the following advantages: non-hairy site EEG recording is performed; the two signals also contain eye movement information; and the system is easily self-adhesive and self-applicable for homecare users and long-term monitoring. The proposed classification system incorporates a novel feature extraction approach, capable of extracting spectral information while considering manual scoring rules. The proposed system further incorporates the relevance vector machine (RVM) as the basic classifier. Importantly, the proposed system provides preliminary results for diagnostic assistance and automatic ambulant scoring to determine whether a patient requires detailed testing with the PSG system in a sleep laboratory. Furthermore, the headband and portable EEG device as well as dry EEG electrodes (Lin et al., 2008, 2011) greatly facilitate the implementation of the proposed system in homecare setting for long-term monitoring of sleep quality, as well as for large-scale population studies.

# **MATERIALS AND METHODS SUBJECTS AND DATA ACQUISITION**

Ten right-handed adults participated in our study (ten males; mean age 24 ± 6 years). None of the participants reported having a history of psychological disorders. Following a detailed explanation of the experimental procedure, all participants completed a consent form before the experiment. To avoid influences from other external factors, all subjects were instructed not to consume alcoholic or caffeinated drinks or sleeping pills beforehand. The experiments were performed at night (10:00 p.m.–08:00 a.m.). All experimental procedures received approval from the local ethics committee (Institutional Review Board of Taipei Veterans General Hospital, Taiwan).

The sleep PSG signals were recorded with a sampling rate of 128 Hz using Sandman Elite (Sandman Elite, Nellcor Puritan Bennett [Melville] Ltd., Kanata, Ontario, Canada) (**Figure 1D**). All subjects were required to sleep for a single night in the sleep laboratory of National Chiao-Tung University (**Figure 1A**) and wore all PSG electrodes during sleep. The complete PSG recording contains six channels EEG (F3, F4, C3, C4, O1, O2), two channels EOG, chin EMG, leg EMG, airflow signals, lead-II ECG, oximetry, nasal pressure, snoring sounding, and body position (**Figure 1B**). Forehead EEG signals from FP1 and FP2 were also recorded by the same PSG system simultaneously (**Figure 1C**). All of the EEG signals were re-referenced to the opposite lateral mastoids (A1

**FIGURE 1 | Experiment environment and the polysomnography recording device (Holland et al., 1974). (A)** The experiment environment in the sleep laboratory of National Chiao-Tung University. **(B)** The real situation with

**Table 1 | Distribution of sleep stages for each subject.**

*The number in each lattice represents the number of corresponding sleep stages. SC refers to the subject code.*

and A2). The contact impedance between all of the electrodes and scalp was controlled to be lower than 5 k*-*. No adjustment or artificial removal techniques were applied to the data. Each data set contained 5–8 h of forehead EEG signals and complete PSG signals.

#### **SLEEP STAGE MANUAL SCORING**

Sleep data for each subject were scored visually based on the manual scoring rules of AASM to five sleep stages by an experienced sleep expert. The five sleep stages are W, N1, N2, N3, and REM, and each 30 s sequential epochs was assigned to a sleep stage. **Table 1** summarizes the distribution of sleep stage belonging to subjects.

#### **PROPOSED AUTOMATIC SLEEP STAGES CLASSIFICATION SYSTEM**

Visual manual scoring often identifies the different sleep stages based on EEG activities. While attempting to incorporate the advantages of EEG activities of manual scoring rule, this work presents a novel sleep stage classification system embedded with a feature extraction approach, which is inspired by the sleep complete PSG recording. **(C)** The FP1 and FP2 EEG channels are also recorded by the same PSG system simultaneously. **(D)** The PSG amplifier (Sandman Elite, Nellcor Puritan Bennett [Melville] Ltd., Kanata, Ontario, Canada).

clinician's expert knowledge in translating two forehead EEG signals to the relevant features, and relevance vector machine (RVM) in order to classify the sleep stages automatically. **Figure 2** displays the flowchart of the proposed sleep stage classification system. As per AASM recommendations, a 30-s sequential EEG recording should function as a unit to assign a sleep stage. In the preprocessing step, all of the 30-s EEG signals are filtered by a band pass filter within 0.5–50 Hz. The following sections described the proposed feature extraction, normalization, and RVM procedures in detail. After RVM, the input 30-s EEG recording assigns a sleep stage. When the recording procedure stops, the final sleep stage results for the whole recording can be estimated.

#### *Feature extraction*

Previous studies extracted frequency-domain features by fast Fourier transform (FFT) within the entire30-s signals. However, the previous studies regarded the entire 30-s signals as a processing unit to directly extract frequency-domain features by FFT. Under this circumstance, the specific characteristics of power spectrum density of 30 s signals are easily smoothened, and the corresponding sleep spectral activities are lost, when the characteristics of sleep appear only at a short period in the time signals. The entire 30 s signals contain a significant amount of information, and the spectral information obtained from FFT directly cannot accurately reflect the advantages of the manual scoring rules. To resolve this problem, this work presents a novel feature extraction approach to extract spectral features by short-time Fourier transform and manual scoring knowledge, which retain the properties of temporal manual scoring rules and represent the spectral response in power spectral density. According to the manual scoring rules of AASM (Iber et al., 2007), the EEG activities include alpha rhythm, theta rhythm, K complex, sleep spindle and slow waves. For instance, the epoch is scored the wakefulness when more than 50% of the epoch has alpha (8–12 Hz) rhythm. The epoch is scored as the N1 when alpha rhythm is attenuated and replaced by low amplitude, predominantly theta (5–7 Hz) rhythm for more than 50% of the epoch. The epoch is scored as N2 when the K complex or sleep spindle (12–14 Hz) occurs

with the theta background rhythm. The epoch is scored as N3 when more than 20% of the epoch has high amplitude slow wave activity. If a single epoch contains 2 or more stages, the stage that contains the greatest portion of the epoch is assigned.

**Figure 3** describes the proposed feature extraction for each 30-s EEG signal. Following the expert knowledge of manual scoring rules, the short-time Fourier transform (STFT) with a 1 s Hamming window overlapped with a 0.5 s window is used rather than using FFT within the entire 30-s signals (**Figure 3A**). Power spectrum densities (PSD) of fifty-nine segments are calculated after using STFT in entire 30-s EEG signals. Following STFT, the PSD of each segment is normalized to avoid individual differences. PSD of each frequency bin of each segment is divided by the total PSD of each segment (**Figure 3B**). Proposed features are identified in the following definition.

The slow wave activity (0.5–2 Hz and amplitude of greater than 75 microvolt) refers to an important index to score N3, and the ratio of slow wave activity has to be greater than 20% visually. Therefore, PSD in lower delta (0.5–2 Hz, denoted as ↓Delta, **Figure 3C**) is chosen to represent the feature of the slow wave activity. Moreover, the average PSDs of the lower delta of upper and lower 80% of 59 segments are viewed as the features of slow wave activity.

As the most important characteristics in visually identifying N2, and K complex and sleep spindle occur spontaneously and roughly every two epochs. K complex is a well-delineated negative sharp wave in EEG immediately followed by a positive component with total duration at least 0.5 s; in addition, sleep spindles are oscillations of sigma (12–14 Hz) with duration of 0.5–1.5 s. Hence, the maximum PSD of sigma band among 59 segments and the average PSD of sigma band of the remaining 58 segments is represented as features of sleep spindle. Moreover, the maximum PSD of delta band (1–4 Hz) among 59 segments and the average PSD of delta band of the remaining 58 segments are represented as features of K complex (**Figure 3C**).

While tending to appear during drowsy, meditative, and sleep onset, theta rhythm scores the epoch as N1, N2, and REM. The average PSDs of theta of upper and lower 50% of 59 segments represent the features of light sleep (N1+N2) (**Figure 3C**).

Movement time, normal resting waking consciousness and wakeful relaxation with eyes closed are accompanied by gamma rhythm (30–50 Hz), beta rhythm (15–30 Hz) and alpha rhythm, respectively. Experts score the epoch as stage wakefulness, when beta and alpha rhythm appear more than 50% of epoch. Thus, the average PSDs of beta and alpha band of upper and lower 50% of 59 segments are represented as the features of stage wakefulness. Movement time stage is mainly accompanied by muscle artifacts obscuring the EEG for more half an epoch. Hence, the average PSD of gamma of 59 segments is represented as the feature of movement time stage (**Figure 3C**).

As mentioned earlier, for FP1 and FP2 EEG channels, sixteen features are extracted, respectively. Two features are extracted as ↓delta activity; two features are extracted as delta activity; two features are extracted as theta activity; two features are extracted as alpha activity; two features are extracted as sigma activity; two features are extracted as beta activity; and one feature is extracted as gamma activity.

For investigating the influence of the proposed feature extraction approach, the conventional PSD feature extraction approach is compared with the proposed one. The conventional PSD feature extraction approach is calculated by the fast Fourier transformation directly for each entire 30 s EEG signal. Notably, this work does not further consider the feasibility of integrating the frequency bins to the specific frequency bands such as delta and theta. The PSD activity ranging from 1 to 50 Hz is used here as input features. Also, for FP1 and FP2 EEG channel, fifty features are extracted, respectively.

#### *Relevance vector machine*

Relevance vector machine (RVM) is a learning algorithm based on Bayesian framework and support vector machine (SVM) (Tipping, 2001). RVM has a form similar to that of SVM; they differ in the measurement between binary classes. SVM learns the maximal distance of margins between binary classes, while RVM learns the maximal probability of margins between binary classes. In contrast with RVM, SVM has the following disadvantages: the number of support vectors (SVs) grows with an increasing number of training patterns; the overfitting problem may occur if SVM selects too many SVs; the decision value is derived from the hyperplane function of SVM in the feature space, making its formation as the probability degree impossible; and the penalty parameter of SVM must be set; this penalty parameter significantly influences the classification results. This parameter is generally determined by the cross-validation approach. Further details of RVM and SVM can be found in Tipping (2001).

#### **SYSTEM PERFORMANCE VALIDATION**

To illustrate the efficiency of the proposed feature extraction approach, this work evaluates the separability of different feature extraction approaches by using the Fisher criteria (Fukunaga, 1990). Two Fisher criteria are expressed as follow:

$$J\_1 = \frac{\operatorname{tr}(\mathbf{S}\_b)}{\operatorname{tr}(\mathbf{S}\_\mathbf{w})}$$

$$J\_2 = \operatorname{tr}(\mathbf{S}\_\mathbf{w}^{-1}\mathbf{S}\_{b)}$$

where *Sb* and *Sw* denote the between-class and within-class scatter matrix, and *tr (A)* refers to the trace of square matrix *A*. A larger *J***<sup>1</sup>** and *J***<sup>2</sup>** imply a larger separability of the presented features in feature space.

Under the extracted feature approaches, this work compares the classification performances of linear discriminate analysis (LDA) (Fukunaga, 1990), k-nearest neighbor classifier (k-NN) (Fukunaga, 1990), SVM (Chang and Lin, 2011; Li et al., 2012), and RVM (Tipping, 2001). Several trials are performed for k-NN, in which the value of k is varied from 1 to 20, to determine the value that maximizes the accuracy. The selected k in k-NN is 13. For simplicity, this work only adopts the linear kernel for SVM and RVM to evaluate how the proposed feature approach influences. From the perspective of SVM and RVM, the advantage of SVM and RVM is the extension of feature space by the kernel function. The feature space of SVM and RVM is implicitly defined by the kernel function. Hence, two popular kernel functions, i.e., linear and radial basis function (RBF) kernel of SVM and RVM, are more closely examined. As for the SVM, a penalty parameter (also called slack variable) C of SVM, in which the trade-off between the margin and the size of the slack variables in this experiment is controlled, is determined by a grid search within given set {0*.*1*,* 0*.*5*,* 1*,* 10*,* 50*,* 100*,* 500*,* 1000*,* 1500}. Here, the C selected from the grid search is 50. A grid search is also performed to derive the proper parameter of RBF kernel within a set {0*.*1*,* 0*.*25*,* 0*.*5*,* 0*.*75*,* 1*,* 2*,* 5*,* 10*,* 50*,* 100*,* 1000}. Here, the selected parameter of RBF for both SVM and RVM is 0.5.The multiclass strategy in SVM and RVM adopted in this work is a one-against-all strategy (Bottou et al., 1994; Li et al., 2012).

In this work, the sleep PSG data of ten subjects are collected. To evaluate the performance of the proposed classification system, the classification performance is evaluated using leaveone-subject-out cross validation (LOSO). Implementing LOSO involves taking the data from one subject as the testing set and the data from other remaining subjects as the training set; the same procedure is repeated until all subjects are including in the testing set. As is well known, the training data and the testing data should be independent of each other. Restated, the testing information should not be used in the training step. The k-fold cross validation approach is the conventional means of evaluating the classification performance. However, this approach cannot ensure that the training data and testing data are independent. Because the training data and testing data are from different subjects, the training data and the testing data in LOSO are independent. Hence, LOSO is less subjective than the normally adopted k-fold cross validation within the single subject.

This system performance is evaluated using three valid indices, i.e., overall accuracy, sensitivity, and Cohen's kappa coefficient. Overall accuracy refers to proximity of measurement results to the actual value and precision to the repeatability or reproducibility of the measurement. Sensitivity is performed to reflect the ability to identify positive results for each class. Cohen's kappa is a statistical measure of inter-rater agreement or inter-annotator agreement for qualitative (categorical) items (Cohen, 1960). As is generally assumed this measurement is more robust than simple percent agreement calculation since kappa takes into account the agreement occurring coincidentally.

# **EXPERIMENTAL RESULTS**

Performance of the proposed feature extraction approach is evaluated by using Fisher criteria, i.e., *J***<sup>1</sup>** and *J***2**, to demonstrate the separability. **Table 2** lists the values of *J***<sup>1</sup>** and *J***2**. Both of *J***<sup>1</sup>** and *J***<sup>2</sup>** in the proposed feature extraction approach are larger than **Table 2 | Separability measurements by using Fisher criterion for the conventional PSD feature extraction and the proposed feature extraction approach.**

the conventional PSD feature extraction approach, implying that the proposed feature extraction approach has a better separability than conventional PSD feature extraction approach. For illustration, principal component analysis (PCA, Fukunaga, 1990) is performed to decompose the proposed extracted features and conventional frequency PSD features to first two principal components (PCs), respectively. **Figures 4A,B** show the scatter plot of the first two PCs from the conventional PSD feature extraction approach and the proposed feature extraction approach, respectively. These figures clearly reveal that the spatial distribution of the proposed feature extraction in PC space has the better scatter distribution than that of the conventional PSD feature extraction one. The scatter plot of the proposed feature extraction approach presents the scatter points from different groups, i.e., wakefulness, N2, N3, and REM, which are leading ones in their own industries. However, the conventional PSD feature extraction approach can not verify this observation. Most data points from the conventional PSD feature extraction approach are mixing in PC space. Additionally, regardless of in which feature extraction approaches, most of the data points of N1overlap with N2, and REM, because the EEG characteristics of N1 in manual scoring rules closely resemble that of N2, REM.

**Figure 5** and **Table 3** display the average classification performances from two feature extraction approaches and four classifiers. The performances, both in terms of overall accuracy and kappa coefficient, of the proposed feature extraction more significantly improves (*p <* 0*.*05, paired *t*-test) than the conventional frequency PSD features. The proposed feature extraction approach has an approximately 20% greater increase in overall accuracy and kappa coefficient than the conventional frequency PSD extraction approach. The overall accuracy and kappa coefficient can reach as high as 76.7 and 68.2% by RVM, respectively.

**Figure 4** shows scatter plots of the sleep pattern, while **Table 2** lists the separability values of Fisher criteria, which compare the

**FIGURE 5 | The classification performance comparison between conventional frequency PSD feature extraction and the proposed feature extraction with LDA, k-NN, SVM, and RVM.** Acc. and kappa represents

overall accuracy and Cohen's kappa coefficient, respectively. Error bars indicate standard deviations. For better visualization, the kappa values have been scaled with the factor 100. ∗*p <* 0*.*05, ∗∗*p <* 0*.*01, ∗∗∗*p <* 0*.*001.

**Table 3 | Comparison of feature extraction approaches in terms of classification performance.**

*Each lattice represents the mean value of the validation index from ten subjects, and the corresponding bracket is the standard deviation. Acc. and kappa represents overall accuracy and Cohen's kappa coefficient, respectively. Highlighted parts display the optimum performance of all comparison results.*

features of the conventional extracted PSD with those of the proposed approach. According to the above results and Fisher criteria, the proposed feature extraction approach has a better separability than the conventional PSD feature extraction approach, as also verified by the classification performance in **Table 3**. The rise in classification performance depends on the proposed feature extraction approach while considering the manual scoring criteria. In particular, the kappa coefficient in the proposed feature extraction approach is increasing significant implying that the proposed method improves both the overall performance of classification and its accuracy for each class with a balance trade-off. For instance, the sensitivity for wakefulness in LDA (conventional PSD feature extraction approach) is 95.7%, i.e., the highest sensitivity of a single class; however, the sensitivity of the other classes is extremely low.

**Table 4** and **Figure 6** summarize the results of SVM and RVM with RBF kernel and liner kernel. The optimum result is RVM with linear kernel, in which the accuracy and kappa can reach 76.7% and 0.68, respectively. Next, RVM and SVM are compared, revealing a significant increase in both linear kernel function (over accuracy, *p* = 0*.*012, paired *t*-test; kappa coefficient, *p* = 0*.*024, paired *t*-test) and RBF kernel function (over accuracy, *p* = 0*.*033, paired *t*-test; kappa coefficient, *p* = 0*.*069, paired *t*-test). Two kernel functions in SVM and RVM are also compared. Applying the RBF kernel in SVM has a ∼1.5% improvement in overall accuracy and kappa coefficient. However, it does not reach a statistically significant level (over accuracy, *p* = 0*.*071, paired *t*test; kappa coefficient, *p* = 0*.*089, paired *t*-test). In terms of RVM, applying the RBF kernel in RVM also does not reach a statistically significant level (over accuracy, *p* = 0*.*582, paired *t*-test; kappa coefficient, *p* = 0*.*658, paired *t*-test).

In terms of SVM, although the number of SVs is normally less than the training patterns, the number of SVs grows larger with an increasing number of training patterns. However, the overfitting problem occasionally occurs in SVM with a large number of SVs. The number of SVs of SVM is generally larger than that of RVs

**Table 4 | Performance comparison of different kernel functions in**

*Each lattice represents the mean value of the validation index from ten subjects, and the corresponding bracket is the standard deviation. Acc. and kappa represents the overall accuracy and Cohen's kappa coefficient, respectively.*

N3 91.3% 89.8% 87.9% 86.7% REM 63.8% 76.0% 77.6% 77.6%

in RVM. The number of support vectors (SVs) is also compared with that of relevance vectors (RVs). RVM is characterized by the fewer number of RVs than that of SVs in SVM.

Fewer RVs can avert the overfitting problem. For each subject, the training patterns come from the remained nine subjects, explaining the variation in the number of training patterns. This work adopts the one-against-all multiclass strategy, which is a "one class vs. all others" method, for SVM and RVM. Therefore, the train population is considered from all classes. In the oneagainst-all multiclass strategy, SVM and RVM train individual decision hyperplane (formula of SVs or RVs and corresponding coefficients) for each class. From ten subjects, the mean of number of train pattern is 7425.9, and standard deviation is 131.4. **Table 5** shows the mean of number of SVs and RVs from linear kernel and RBF kernel. Regardless of whether linear kernel or RBF kernel is adopted, the number of RVs is significantly less than the number of SVs (all *p*-values are less than 0.0001 with Student's *t*-test). Although the number of trained SVs is in the thousands, the number of trained RVs only ranges less than several hundred, thus representing a significant difference between RVs and SVs.

To investigating how class imbalance prior influences the classification performance, this work describes the confusion matrix between the proposed sleep classification approach, in which RVM is used with linear kernel function and expert manual scoring (**Table 6**). The overall accuracy and kappa coefficient, as computed from this confusion matrix, are 76.7% and 0.69, respectively. **Table 6** and the performance in **Table 4** demonstrate that the reported validation indices are not biased.

**Figure 7** shows the estimation results, based on the proposed classification approach and the manual scoring results for one subject (S07). The top plot is the distribution of the estimated sleep stages from proposed classification, and the below plot is the distribution of the sleep stages from the sleep expert. For this subject, the accuracy between the proposed classification approach and the sleep expert's scoring is 82.5% (kappa = 0.77). The proposed classification approach with only forehead EEG can reach a quite similar performance with the sleep expert.

**FIGURE 6 | The classification performance comparison between SVM and RVM with linear and RBF kernel functions.** Acc. and kappa represents overall accuracy and Cohen's kappa coefficient, respectively. Error bars indicate standard deviations. For better visualization, the kappa values have been scaled with the factor 100. ∗*p <* 0*.*05.

**Table 5 | Number of support vectors in SVM and number of relevance vectors in RVM.**

*Each block is the mean from all ten subjects, and the corresponding bracket is the standard deviation.*

#### **DISCUSSION**

Based on the scatter plot and Fisher criteria, this work demonstrates that the proposed feature extraction approach can achieve a better separability than the conventional PSD feature extraction approach. The classification performance also demonstrates that the proposed feature extraction approach is more effective

than the conventional PSD feature extraction approach in distinguishing between different sleep stages. The proposed feature extraction approach is considered with the sleep stage manual scoring knowledge. Additionally, time-frequency analysis is performed to extract the spectral activities in a short segment window signal not whole 30-s signals. The proposed feature extraction approach characterizes not only by its temporal information of manual scoring, but also by its spectral response. Sleep experts distinguish between the different sleep stages by counting the rhythm and the amplitude of EEG visually by per-second EEG signals from whole 30-s signals. The proposed feature extraction approach applies the short-time Fourier transform with a one second window and overlap 0.5 s. Also, the PSD activity is transferred from STFT to several specific frequency bands, e.g., low delta (1–2 Hz), delta (1–4 Hz), theta (5–7 Hz), alpha (8–12 Hz), sigma (12–14 Hz), beta (15–30 Hz), and gamma (30–50 Hz). With the entire 30-s data, 59 segments are extracted. To overcome the individual differences, the PSD of each segment is divided by the total sum of the PSD of each segment. Hence, the PSD activity in the proposed feature extraction approach is the proportion response of PSD. Based on the expert's manual scoring knowledge, several features are extracted from these specific frequency bands of 59 segments. STFT can achieve the spectral activity of specific frequency response which may be more or less resonant in the spectral space. For instance, the sleep spindle is a sigma rhythm lasting from 0.5 to 1 s, and occurring suddenly. If the fast Fourier transfer is applied to a N2 epoch with one spindle of whole 30 s EEG signals, the specific sigma frequency response cannot be identified clearly. The principal spectral response might be a background spectral response, i.e., theta spectral response, and the amplitude of PSD within the sigma band might be extremely low. However, STFT can separate the signals into several segments in order to calculate the PSD individually. If the spindle occurs, STFT can enhance the corresponding spectral activity more than that of FFT with whole signals. Moreover, in addition, to using STFT to transfer the signals to PSD, this work also proposes sorted power activities to extract the manual rules' properties for the specific frequency spectral activity as the sleep features. The corresponding specific frequency spectral activities are extracted by following the manual scoring rules. For instance, the maximum power of sorted sigma power activities can represent the feature of the sleep spindle. If the current epoch is N2, the value of the maximum power of sorted sigma power activities is higher than that of the maximum power of sorted sigma power activities from the other sleep stages.

The value extracted by average PSDs of alpha and beta of upper 50% of 59 segments from wake stage is higher than that of sleep. The value extracted by the average PSDs of the slow wave of upper 80% of 59 segments from N3 stage is higher than that of other stages. Similarly, the other features from different frequency bands can represent the other sleep stages. Berthomier et al. (2007) characterized several contrast functions, which are feature extraction approaches, as defined by the EEG power activity calculating from whole epoch directly. Although Berthomier et al. (2007) considered the baseline resting EEG frequency of each individual to adjust the spectral criteria, the extracted features were still calculated from the whole 30-s EEG signals. The specific frequency activity still diminishes, when the frequency activity is calculated within the whole 30-s signals. Hence, the proposed feature extraction approach is a more effective means of extracting the sleep characteristics.

SVM has recently achieved higher empirical accuracy and better generalization capabilities than other standard supervised classifiers (Fatma Guler and Ubeyli, 2007; Lotte et al., 2007; Xu et al., 2009; Li et al., 2012). However, as mentioned earlier, SVM is limited in the number of SVs and the selection of penalty parameter. The penalty parameter in SVM is adjusts the generalization capability. RVM is an extension algorithm that eliminates the disadvantages of SVM. SVM learns the maximal distance of margins between binary classes while, in contrast, RVM learns the maximal probability of margins by exploiting a probabilistic Bayesian learning framework between binary classes. The penalty parameter in SVM is usually determined by the cross-validation approach. The chosen penalty parameter depends on the setting candidate set. Too much training time is expended when selecting the penalty parameter in SVM, if the setting candidate set has a wide range. The chosen penalty parameter is a local optimum parameter, depending on the setting candidate set, not the global optimum parameter. The chosen penalty parameter, which affects the number of SVs, incurs the overfitting problem in training phase. RVM can estimate penalty parameters automatically. Additionally, RVM can improve the problem in the number of SVs. Accounting to our results, the number of SVs can significantly decrease in the number of RVs; the classification performance can also be improved by RVM. Hence, the RVM is applied as the basic classifier in the proposed sleep stage classification system.

The detection of N1 is always the most problematic aspect of the sleep stages (Virkkala et al., 2007). Identifying a significant feature in EEG that could separate N1 from wakefulness, N2, and REM, is rather difficult because N1 is a transition phase in the changes of wakefulness and other sleep stages (Virkkala et al., 2007). The sleep EEG characteristics of N1 closely resemble those of N2, REM, and resting wakefulness. Our results demonstrate that the extracted features from PSDs in N1 resemble to N2, REM and resting wakefulness (**Figure 4**). Moreover, many epochs of N1 are misclassified to Wake, N2, and REM (**Table 6**), explaining the difficulty in automatically identifying N1 by a computer. Efforts are underway in our laboratory to address this problem.

Recent efforts have attempted to develop a more reliable sleep system with few bioelectrical channels, i.e., one EEG, one ECG, or two EOGs, in order to simplify the complex PSG inspection (Berthomier et al., 2007; Virkkala et al., 2007; Yιlmaz et al., 2010). Virkkala et al. (2007) devised an automatic sleep stage classification via two EOGs. The performances (Virkkala et al., 2007) with 5 sleep stages are 72.5% epoch-to-epoch agreement and 0.63 Cohen's kappa, and the sensitivity of Wake, REM, N1, N2, and SWS are 74.10, 72.7, 39.2, 79.1, and 73%, respectively. Although this work applies two channel signals, the FP1 and FP2 EEGs, which can reflect the eye movement, have more information in classifying sleep stages. Hence, both the characteristics of sleep EEG and eye movement are captured. Additionally, the proposed classification system accurately estimates sleep stages. Yιlmaz et al. (2010) presented a sleep stage and obstructive apneic epoch classification via single-lead ECG. The performances (Yιlmaz et al., 2010) with 6 sleep stages are 73.1% epoch-to-epoch agreement, and the sensitivities of Wake, REM, NREM1, NREM 2,NREM 3, and NREM 4 are 95.6, 84.9, 98.5, 61.8, 94.3, and 87.4%, respectively.

Performance of the classification (Yιlmaz et al., 2010) is satisfactory, even the sensitivity of NREM 1 (98.5%), which is the most difficult sleep stage to be identified automatically (Berthomier et al., 2007; Virkkala et al., 2007). Yιlmaz et al. (2010) applied the 10-fold cross validation within a single subject data, which totally separates a subject's self-data as the training data and also as the testing data. For instance, for a subject with total 800 epochs, partitioning produces 10 subsets with 80 epochs each. Therefore, the training set (720 epochs) and testing set (80 epochs) include totally separate sets of data. The training set and testing set originate from a specific subject. Moreover, the properties of training and testing data resemble each other, the corresponding with the over-fitting problem in training phase. Notably, attempting to use a subject-dependence model by a specific subject in order to classify another independent dataset may cause worst results. LOSO cross-validation is a more objective evaluation approach for machine learning experiment involving human subjects to allow for subject-to-subject variation. The testing data are subject-independent to the training data. Hence, the performance evaluation by LOSO is more effective and reliable than k-fold cross validation in developing a general model involving human subjects. As mentioned earlier, the right and left EOG signals are recorded by placing two electrodes at the nasal and temporal canthal regions of the eye, in which one electrode is attached to the middle of the forehead as ground electrode while another electrode is placed on the left mastoid M1 as reference electrode. ECG signals are acquired by two electrodes in a modified leads II configuration (Malmivuo and Plonsey, 1995). The positive and negative leads are placed on the fourth inter costal space and the left of the sternum. Also, both of the EOGs and ECG still require adhesive electrode pads, and the locations of EOGs (or ECG) and corresponding amplifier diverge from each other. The sleep position may affect the recording quality of physiological signals. Berthomier et al. (2007) presented an ASSEEGA based on an EEG channel. In terms of performance, the proposed sleep stage classification system is nearly equivalent to the ASSEEGA. However, ASSEEGA still requires 2 bipolar channels (Cz-Pz, international 10–20 standard system), making it infeasible for homecare applications. First, the position of electrodes is not easily identified, self-adhesive, and self-applicable for a self-operating user. Second, the conductive gel and a laboratory EEG recording device are still required in the recording signals. Although several portable EEG devices (e.g., Mindo-4S (Mindo, Hsinchu, Taiwan), MindWave Mobile (NeuroSky, CA, USA), and Emotiv epoc headset (Emotiv, Eveleigh NSW, Australia), as well as dry sensors) can overcome this problem, the comfort of dry electrodes of Cz and Pz is still a major challenge during sleep.

Anderer et al. (2005) recently developed and optimized an automatic classification system based on a central EEG channel, two EOG channels and a chin EMG channel; in addition, the final validation of overall epoch-by-epoch agreement is 80% (Cohen's kappa is 0.72) between the proposed automatic classification system and human expert scoring. Obviously, the data-rich recordings have more information, e.g., sleep brain electrical activity from EEG, muscle activity from EMG, and eye movement from EOGs. Such information-rich physiological data provide more important indices to classify the difference between REM and light sleep, i.e., the rapid eye movement and the lowest mandible muscle activity. Hence, the data-rich recording can achieve an excellent performance. Although the performance in this work is not equivalent to that of the classification system (Anderer et al., 2005), the proposed system attempts to reduce the number of full PSG signals to fewer physiological channels, as well as further classify the sleep stages effectively. Therefore, the proposed system uses forehead EEGs, i.e., FP1 and FP2. FP1 and FP2 have the following advantages: the physiological data include information from sleep brain electrical activity and eye movement; and adopting the forehead EEGs makes it feasible for self-application for a self-operating user. With the portable EEG device (Lin et al., 2008) and dry sensors (Lin et al., 2011), home-based users can easily to wear the EEG headband to record the sleep forehead EEG signals by self-applicable. Furthermore, we can further analyze with the collected data can be analyzed further, even leading to the development of on-line sleep stage classification software.

Despite its contributions, this work has certain limitations. The collected data are limited to young and healthy study participants. The sleep stages in normal person are expected to diverge from the norm and to be more heterogeneous than those of older or younger, healthy individuals or patients. The sleep stage manual scoring rules are based on counting the rhythm of different frequency activities. The proposed system attempts to comply with this criterion in order to extract the sleep features. The system also uses STFT as temporal and visual rules, i.e., processing the signal within one-second window, and further extracts features by PSDs representing different rhythms of different frequency activities. This system is created by the healthy subjects, and does not have obvious evidence to verify that the proposed sleep stage classification is reliable in older individuals or patients. Therefore, efforts are underway in our laboratory to study the relation between patients and the proposed sleep stage classification system.

# **CONCLUSION**

This work presents a novel sleep stage classification system, consisting of a novel feature extraction method and RVM classifier, based on only two forehead EEG channels. Also, the classification performance is consistent with the sleep clinician's expert knowledge. Experimental results demonstrate the feasibility of using the proposed system as the preliminary screening results for a preclinical diagnosis to assist clinicians in making a diagnosis (rather having a depth testing with PSG system in a sleep laboratory) to reduce time for the procedure. Moreover, the proposed system only uses two forehead EEG signals, allowing us to apply the wearable and wireless EEG recording device (Lin et al., 2008; Liao et al., 2012) [e.g., Mindo-4S (Mindo, Hsinchu, Taiwan) and MindWave Mobile (NeuroSky, CA, USA)] in order to record the patient's EEG signals at home. Importantly, the proposed system provides an easier way for large population studies, long-term sleep monitoring, and home-based daily care. Efforts are underway in our laboratory to integrate the wearable and wireless EEG recording device and the proposed sleep stage classification system. As an important aspect of performance, the automatic artifact detection might be a possible way to improve the efficacy of the proposed system. Hence, the efficacy of the automatic artifact detection should be considered in the proposed system in the future work.

### **ACKNOWLEDGMENTS**

This work was supported in part by the Aiming for the Top University Plan of National Chiao Tung University, the Ministry of Education, Taiwan, under Contract 102W9633, and in part by UST-UCSD International Center of Excellence in Advanced Bioengineering sponsored by the Ministry of Science and Technology I-RiCE Program under Grant Number: NSC-102- 2911-I-009-101. Research was sponsored by the Army Research Laboratory and was accomplished under Cooperative Agreement Number W911NF-10-2-0022. The views and the conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S Government. The U.S Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

# **REFERENCES**

interface device with novel dry foam-based sensors. *J. Neuroeng. Rehabil.* 9:5. doi: 10.1186/1743-0003-9-5

making method. *IEEJ Trans. Electron. Inf. Syst.* 129, 614–619. doi: 10.1541/ieejeiss.129.614

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 17 February 2014; accepted: 05 August 2014; published online: 04 September 2014.*

*Citation: Huang C-S, Lin C-L, Ko L-W, Liu S-Y, Su T-P and Lin C-T (2014) Knowledge-based identification of sleep stages based on two forehead electroencephalogram channels. Front. Neurosci. 8:263. doi: 10.3389/fnins.2014.00263*

*This article was submitted to Neuroprosthetics, a section of the journal Frontiers in Neuroscience.*

*Copyright © 2014 Huang, Lin, Ko, Liu, Su and Lin. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Brain fingerprinting classification concealed information test detects US Navy military medical information with P300

#### *Lawrence A. Farwell <sup>1</sup> \*, Drew C. Richardson2, Graham M. Richardson3 and John J. Furedy4*

*<sup>1</sup> Brain Fingerprinting Laboratories, Inc./Brain Fingerprinting, LLC, Seattle, WA, USA*

*<sup>2</sup> Federal Bureau of Investigation, FBI Laboratory, Quantico, VA, USA (at the time of the research)*

*<sup>4</sup> Department of Psychology, University of Toronto, Toronto, ON, Canada*

#### *Edited by:*

*Anne-Marie Brouwer, Netherlands Organisation for Applied Scientific Research, Netherlands*

#### *Reviewed by:*

*Maja Stikic, Advanced Brain Monitoring, Inc., USA Maarten Andreas Hogervorst, Netherlands Organisation for Applied Scientific Research, Netherlands*

#### *\*Correspondence:*

*Lawrence A. Farwell, Brain Fingerprinting Laboratories, Inc., 14220 37th Ave. NE, Seattle, WA 98125, USA e-mail<EMAIL_ADDRESS>A classification concealed information test (CIT) used the "brain fingerprinting" method of applying P300 event-related potential (ERP) in detecting information that is (1) acquired in real life and (2) unique to US Navy experts in military medicine. Military medicine experts and non-experts were asked to push buttons in response to three types of text stimuli. Targets contain known information relevant to military medicine, are identified to subjects as relevant, and require pushing one button. Subjects are told to push another button to all other stimuli. Probes contain concealed information relevant to military medicine, and are not identified to subjects. Irrelevants contain equally plausible, but incorrect/irrelevant information. Error rate was 0%. Median and mean statistical confidences for individual determinations were 99.9% with no indeterminates (results lacking sufficiently high statistical confidence to be classified). We compared error rate and statistical confidence for determinations of both information present and information absent produced by classification CIT (Is a probe ERP more similar to a target or to an irrelevant ERP?) vs. comparison CIT (Does a probe produce a larger ERP than an irrelevant?) using P300 plus the late negative component (LNP; together, P300-MERMER). Comparison CIT produced a significantly higher error rate (20%) and lower statistical confidences: mean 67%; information-absent mean was 28.9%, less than chance (50%). We compared analysis using P300 alone with the P300 + LNP. P300 alone produced the same 0% error rate but significantly lower statistical confidences. These findings add to the evidence that the brain fingerprinting methods as described here provide sufficient conditions to produce less than 1% error rate and greater than 95% median statistical confidence in a CIT on information obtained in the course of real life that is characteristic of individuals with specific training, expertise, or organizational affiliation.

**Keywords: P300, concealed information test, brain fingerprinting, P300-MERMER, ERP, LNP, event-related potential, detection of concealed information**

# **INTRODUCTION**

#### **THE CLASSIFICATION CIT**

The concealed information test (CIT) or guilty knowledge test (GKT) has been used to detect concealed information since Lykken (1959). Until the 1980s, the dependent measures were autonomic nervous system (ANS) responses. The ANS-based CIT is a *comparison* CIT (Lykken, 1959). The comparison CIT compares the responses to crime- or situation-relevant and irrelevant items. If the responses to the relevant items are larger, then the determination is made that the subject knows the relevant information. ("Larger" is variously defined.) Otherwise, the determination is made that the subject does not know the information.

Farwell and Donchin (1991) introduced three innovations in the CIT (Farwell, 2013). They (1) applied a classification CIT, rather than the conventional comparison CIT; (2) used event-related brain potentials (ERPs) as the dependent measure; and (3) computed a statistical confidence for each individual determination using the technique of bootstrapping. (Farwell and Donchin, 1991 was preceded by abstracts on the same studies, Farwell and Donchin, 1986, 1988a). Several researchers subsequently applied ERPs and bootstrapping in a comparison CIT (e.g., Johnson and Rosenfeld, 1992; Rosenfeld et al., 2004, 2008; Meixner and Rosenfeld, 2014). This is a fundamentally different paradigm (see Discussion and Appendix 2).

In the classification CIT, three types of stimuli are presented. (1) "Probes" are relevant to the investigated situation. Probes contain information that the subject has no way of knowing other than participation in the investigated situation (and, in field cases, that the subject denies knowing or recognizing as being crime-relevant). (2) "Targets" also may be relevant to the investigated situation. (In all our recent applications including the study

*<sup>3</sup> Department of Cell and Developmental Biology, Vanderbilt University, Nashville, TN, USA*

reported here, they are.) Experimental protocols ensure that the subject knows the targets, for reasons other than participation in the investigated situation. The response to targets provides a template for the subject's response to known, situation-relevant information. (3) "Irrelevants" contain irrelevant information. The response to irrelevants provides a template for the subject's response to irrelevant information. If the ERP response to the probes is mathematically classified as being more similar to the ERP response to the known, relevant target information than to the irrelevants, the subject is determined to be "information present" with respect to the information contained in the probes. If the ERP response to the probes is more similar to the ERP response to the irrelevant information than to the targets, the subject is determined to be "information absent." If the probe ERP response cannot be classified with a high statistical confidence as being more similar to either the target or the irrelevant response, no determination is made; the outcome is "indeterminate."

The classification CIT with ERPs can be applied in two different types of tests. Specific issue tests detect knowledge of a specific event such as a crime. Specific screening or focused screening tests detect knowledge relevant to specific training or expertise, or inside knowledge of a particular organization or group. This study is a specific screening test conducted in collaboration with the US Central Intelligence Agency (CIA) and the US Navy. The information detected is military medical knowledge in US Navy military medical experts.

We compared the classification CIT with the comparison CIT by analyzing our data with both methods. Farwell and Donchin (1991) used the classification CIT with the P300, an electrically positive event-related potential (ERP) maximal at the parietal midline area of the head that is elicited when a subject recognizes and takes note of a stimulus that is significant in the current context. In this research we compared the results of using the P300 alone vs. the P300 plus the late negative component (LNP; together, P300-MERMER, memory and encoding related multifaceted electroencephalographic response)1 . We compared error rate and statistical confidences produced by the classification CIT with the results of analysis applying the comparison CIT on the same data. We investigated whether the classification CIT provides significantly lower error rate and higher statistical confidences than the comparison CIT.

#### **PREVIOUS RESEARCH ON SUFFICIENT AND NECESSARY CONDITIONS FOR A VIABLE CIT FOR REAL-WORLD FIELD USE**

In our view, in order to be considered reliable, an ERP-based CIT must reliably produce less than 1% error rate and median statistical confidences of greater than 95% for individual determinations, including median statistical confidences greater than 90% for both information-present and information-absent determinations, across different field and laboratory conditions. These same criteria, in our view, are the minimum criteria required to effectively and ethically apply a technique in criminal investigations or any application with non-trivial consequences.

Farwell and Donchin's (1991) method provided the sufficient conditions to meet these requirements. They only established sufficient conditions, and did not investigate which conditions were necessary. Since then, research has progressed substantially in two parallel, largely non-overlapping series of studies. One series of studies has investigated the sufficient conditions to meet these criteria under varying field and laboratory conditions. Another series of studies has investigated the necessary conditions.

Eight previous peer-reviewed studies conducted by six researchers in four laboratories have applied a specific set of methods in the ERP-based CIT [Farwell and Donchin, 1991 (two studies); Allen and Iacono, 1997; Farwell and Smith, 2001; Farwell et al., 2013 (four studies); see also Iacono, 2008]. These specific methods are the only methods that have reliably produced less than 1% error rate and median 95% statistical confidences for individual determinations, including over 90% for both information-present and information-absent determinations, in the laboratory and the field. These same error rates and statistical confidences have been achieved with countermeasures, without countermeasures, and in field conditions where it is unknown whether countermeasures are being used or not (Farwell et al., 2013). (Countermeasures are physical or mental procedures that a subject may practice in an attempt to influence the outcome of a test. They were not studied in this research).

The methods applied in these studies are the same as in the original Farwell and Donchin studies, with several improvements based on the more demanding requirements of field applications, as described below. Farwell (2012) documented these methods (or rules or recommendations) as 20 brain fingerprinting scientific standards (Appendix 1). Farwell (1994, 2012) defined "brain fingerprinting" as the classification-CIT technique incorporating the 20 standards. These methods are applied here2 . We have focused our previous research primarily on establishing the sufficient conditions because this provides a technique we can use, and have successfully used, in the field. Previous studies by others investigated the necessary conditions to obtain low error rates and high statistical confidences (Farwell, 2012, 2014; Appendix 2).

# **P300 AND LNP**

Farwell and Donchin (1991) used the classification CIT with the P300. In this research we compared the results of using the P300 alone vs. the P300 plus the LNP. Our rationale for this is as follows.

Early P300 research (e.g., Sutton et al., 1965) used very simple stimuli, such as auditory clicks and tones. As the sophistication of experimental designs progressed, more complex stimuli were used, including simple words and phrases presented visually. The latency of P300 was found to increase with stimulus complexity

<sup>1</sup>To obtain statistical confidences for each individual determination, Farwell, Donchin, Wasserman, and Bockenholt collaborated to introduce the statistical technique of bootstrapping in the field of psychophysiology. This first was published in conference abstracts by Farwell and Donchin (1986, 1988a), and full papers by Wasserman and Bockenholt (1989) and Farwell and Donchin (1991), which reported the same studies as the abstracts. Wasserman and Bockenholt used Farwell and Donchin's application of bootstrapping in the classification CIT as an example of the correct use of the technique.

<sup>2</sup>The study reported here appeared first as a conference abstract, Farwell and Richardson (2006).

and the concomitant stimulus evaluation time (Magliero et al., 1984). With simple words and phrases, an inter-stimulus interval (ISI) of 1000–1500 ms or less was adequate for the subject to process the stimuli and to capture the entire ERP response (Farwell and Donchin, 1988b). Farwell and Donchin (1991), for example, used phrases consisting of two, one-syllable words, and an ISI of 1500 ms.

In conducting research at the FBI in 1993, however, Farwell et al. (2013; Farwell, 1994, 1995a,b) had the task of developing text stimuli that accurately represented knowledge unique to FBI agents. This required some stimuli to be several long words. To give subjects time to fully process the stimuli, we extended the ISI to 3000 ms. Under these conditions, we found that the positive P300 peak was followed by a negative peak with a peak latency of up to 1200 ms, which we termed the late negative potential (LNP).

The stimuli we used in this study and in previous research (Farwell and Smith, 2001; Farwell et al., 2013) were more personally significant than the stimuli presented in most previous P300 research. LNP may be driven at least in part by this personal significance. Compared to many previous P300 studies, our stimuli may also be more salient, be more related to previous memories, require more complex processing, and involve a task more important to the subject. They are also presented with a longer ISI than that applied in most previous P300 studies. Further research is necessary to identify the antecedent conditions and delineate the functional significance of the LNP.

We called the overall pattern of the P300 followed by the LNP in the time domain, along with concomitant changes in the frequency domain and other putative changes measurable by other mathematical methods, a P300-MERMER. Although the P300-MERMER—and for that matter, the P300—may be comprised of additional features that are not visible in the time domain (Farwell, 2012; Farwell et al., 2013), the time-domain pattern is sufficient to define and to detect the response. This pattern consists of a positive peak followed by a negative peak (or a negative-positive-negative pattern if the N200, a wellknown negative component that generally precedes the P300, is included).

We compared results obtained using the P300 alone with results obtained by including the P300 plus the LNP. Our computations consider only the conventional, time-domain characteristics of the signals. The difference between our two epoch-related analyses is the length of the epoch analyzed, and therefore the inclusion or exclusion of the LNP and its amplitude, morphology, and latency.

In the early 1990s, when Farwell et al. (2013) first encountered the LNP that follows the P300, we initially hypothesized that LNP was an artifact, perhaps generated by the analog filters and the return of the P300 to baseline. The data contradicted this hypothesis, however. If the LNP were an artifact produced by the filter's effect on the P300, then similar P300s with identical filters would produce similar LNPs. We found that the latency, amplitude, and morphology of the LNP varied independently of the P300. Also, the scalp distribution of the LNP was more frontal than that of the P300. Moreover, the negative peak persisted when we varied our filter settings (Farwell, 1994, 2012). Even recording without analog filters did not eliminate the LNP, or substantially change its characteristics. This definitively disproved the filter-generated-artifact hypothesis (Farwell et al., 2013).

The data we recorded with filters also contradict the hypothesis that the LNP is an artifact. We used the same recording equipment for all subjects and all scalp sites. If the LNP were an artifact produced by the equipment, the same equipment would produce identical effects in different scalp sites and different subjects. The features of the LNP would be a function of the features of the P300. This was not observed. For different scalp sites in one subject, and for different subjects, the relative amplitude, latency, and morphology of the LNP and the P300 were very different. Sometimes there was a difference of hundreds of milliseconds in the latency, and amplitude differences of a factor of two or more, in LNPs that followed virtually identical P300s recorded from different subjects (Farwell et al., 2013). In some cases the LNP was considerably larger than the P300 at one scalp site (Fz) and considerably smaller than the P300 at another (Pz) for the same subject. In short, the data contradict the hypothesis that the LNP (or the latter part of the P300-MERMER) is an artifact produced by some combination of the P300, the return to baseline after the P300, and the filters and other equipment.

In the current paradigm, a negative peak (the N200) precedes the P300 positive peak, and another negative peak (the LNP) follows the P300. Our first observation of this tri-phasic negative-positive-negative morphology in the ERP response was in the early 1990s (Farwell, 1994, 2012; Farwell and Smith, 2001; Farwell et al., 2013). Others applying intracranial recordings have observed this same negative-positive-negative pattern in a number of brain structures (Halgren et al., 1998; Linden, 2005). These include dorsolateral and orbital frontal cortices, anterior cingulate (Baudena et al., 1995), amygdala and hippocampus (Halgren et al., 1986; Stapleton and Halgren, 1987), superior temporal sulcus (Halgren et al., 1995), and inferior parietal lobe/supramarginal gyrus (Smith et al., 1990).

Others investigating the ERP-based CIT, including Meijer et al. (2007), have also reported the LNP. Brouwer et al. (2010) observed the LNP and investigated its utility in brain-computer interfaces. Several other studies (Matsuda et al., 2009; Gamer and Berti, 2010, 2012) reported a difference in the N200 in responses to relevant stimuli in ERP-based CITs. Virtually all researchers conducting research on ERP-based CITs now include in their data-analysis algorithms both the P300 and the LNP (for reviews, see Farwell, 2012, 2014), although some refer to the entire response including both positive and negative peaks as "P300" (e.g., Rosenfeld et al., 2008) and some refer to the positive peak as "P300" and the entire response as "P300-MERMER" or "P300 + LNP" (e.g., Sutton et al., 1965; Farwell, 2012, 2014; Farwell et al., 2013).

Changes in the frequency domain and other changes in the dimensionality and other characteristics of the signal may be included in the term "P300-MERMER." The positive and negative time-domain changes constituting the P300 and the LNP are sufficient to detect and characterize the response, and are all that are measured in this research, although they undoubtedly do not constitute a complete and comprehensive description of all the patterns of electrophysiological activity that manifest the underlying information-processing brain activity (Farwell, 1994, 2012; Farwell and Smith, 2001).

We compared the error rate and statistical confidences produced by data analysis including the P300 plus the LNP with the results of analysis using the P300 alone. We investigated whether the classification-CIT analysis with the P300 plus the LNP provides significantly lower error rate and/or higher statistical confidences than the analysis with the P300 alone.

# **SUMMARY OF RESEARCH QUESTIONS**

Our primary research questions are as follows:

# **MATERIALS AND METHODS**

# **SUBJECTS**

We tested 16 experts (information present) and 14 non-experts (information absent) in military medicine. Experts were students and faculty at Uniformed Services University of the Health Sciences (USUHS) possessing professional knowledge of military medicine. Non-experts lacked this specific expertise and training. Mean age of 30 subjects was 26; standard deviation was 2.9. Mean ages of information-present and information-absent subjects were 27 and 25, respectively; standard deviations were 3.2 and 2.6, respectively. 15 subjects (8 information present) were female.

Experimental procedures were approved by the Brain Fingerprinting Laboratories, Inc., ethics committee and performed in accordance with the ethical standards of the 1964 Declaration of Helsinki, including written informed consent prior to participation.

#### **STIMULI**

Three types of stimuli consisting of words or phrases were presented on a computer screen: probes, targets, and irrelevants. Probes contain specific information relevant to the investigated situation. The test is designed to detect the subject's knowledge or lack of knowledge of the information contained in the probes as relevant in the context of the investigated situation.

In this specific screening study, the relevant information detected was known only to experts in military medicine. Information was obtained from interviews with USUHS military medical experts. Individuals interviewed were not tested. Probe stimuli contained the relevant information to be detected. We presented two additional types of stimuli. Responses to target stimuli provide a template for the subject's brain response to known information relevant to the investigated situation. Responses to irrelevant stimuli provide a template for the subject's brain response to irrelevant information. Target stimuli present information relevant to the investigated situation that is known to be known to the subject. There are significant, proven advantages to using targets that are relevant to the investigated situation rather than inherently irrelevant targets that are made relevant only by task instructions (Farwell, 2012; Farwell et al., 2013), although we and others have successfully used the latter (Farwell and Donchin, 1991). Target stimuli, unlike probes, were identified as such to the subject in experimental instructions. Subject instructions also conveyed the significance of each target in the context of the investigated situation, and required a different behavioral response to targets than to probes and irrelevants, as described in the next section.

For each probe (and each target) comparable irrelevants were structured that contained similar, plausible, but incorrect information about the investigated situation. For a subject lacking the relevant knowledge contained in the probes, the irrelevants and probes were equally plausible as correct, relevant details. Each probe and its comparable irrelevants were indistinguishable for a subject lacking the information that the test was structured to reveal. Each probe contained correct, relevant information fitting the description of that probe. The two irrelevants comparable to each probe contained incorrect information that would be plausible as fitting that same description for an individual lacking the information contained in the probes. For example, a probe stimulus could be the technical name of a military medical procedure in which experts are trained. Corresponding irrelevants could be technical terms that do not name any real procedure. For security reasons, the exact stimuli cannot be given. Subjects were provided with a description of each probe that specified the significance of the probe in the context of the investigated situation, but were not informed which was the correct, situation-relevant probe and which were the corresponding irrelevants.

Similarly, each target stimulus contained correct, situationrelevant information, and the two irrelevant stimuli comparable to each target contained comparable, incorrect but plausible information. Unlike probes, targets were identified as such in instructions to the subjects.

Stimuli were constructed in groups of six: one probe, one target, and four irrelevants. For each probe there were two comparable irrelevants. For each target there were two comparable irrelevants. We used a ratio of 1/6 targets, 1/6 probes, and 2/3 irrelevants so targets and probes were relatively rare, which is known to enhance P300 amplitude (Farwell and Donchin, 1991).

Our prediction was that targets would elicit a large P300 + LNP (or P300-MERMER) in all subjects, irrelevants would not elicit a large P300 + LNP, and probes would elicit a large P300 + LNP only in information-present subjects. Thus, for informationpresent subjects, ERP responses to probes would be similar to ERPs for targets. For information-absent subjects, ERP responses to probes would be similar to ERPs for irrelevants.

There were 32 unique probes, 32 unique targets, and 128 unique irrelevants, a total of 192 unique stimuli. These comprised 32 groups of stimuli, each consisting of one probe, one target, and four irrelevants. 20 probes were words or phrases embodying the relevant knowledge; 12 were acronyms. The same stimuli were presented to all subjects. Each unique stimulus was presented more than once, so the total number of stimulus presentations was greater than the total number of unique stimuli.

### **PROCEDURE**

Before the test, we made certain that the subject understood the significance of the probes. We described the significance of each probe to the subject. We then showed the subject the probe and the corresponding irrelevants, in the context of the description of the significance of the probe, without revealing which was the probe. Thus, subjects were informed of the significance of each probe stimulus, but were not told which stimulus was the probe and which were corresponding irrelevants. For example, subjects were told, "One of these three items is the term for a medical technique applied to burn victims in battlefield situations" followed by a list of one probe and two irrelevants (in alphabetical order). Although the descriptions of the probes were made known to subjects, the probe stimuli themselves were never identified as probes.

Targets were explicitly identified to the subjects. Experimental instructions ensured that the subject knew the targets and their significance in the context of the investigated situation. We described the significance of each target to the subject. We showed the subject each target and the corresponding irrelevants, in the context of the description of the significance of the target. We also showed subjects a list of the targets, and noted that subjects would be required to recognize the targets during the test. We instructed subjects to press a button with one thumb in response to targets, and another button with the other thumb in response to "all other stimuli." The subject's task was to read and comprehend each stimulus, and then to indicate by a button press whether the stimulus was a target stimulus or not.

For a subject possessing the knowledge embodied in the probes, "all other stimuli" consisted of two types of stimuli: probes containing the known situation-relevant information, and irrelevant stimuli. For a subject lacking the tested knowledge, "all other stimuli" appeared equally irrelevant. Probes were indistinguishable from irrelevants. For "all other stimuli" (that is, everything except targets), the subject was instructed to push the opposite button from the one pushed in response to targets. This instruction applied whether the subject perceived these as a single category (all equally irrelevant, if the subject was information absent) or as two categories (irrelevant, and relevant to the concealed information being tested, if the subject was information present).

The differential button-press task in response to every stimulus presentation ensured that the subject was required to read and comprehend every stimulus, including the probe stimuli, and to prove behaviorally that he had done so on every trial. This allowed us to avoid depending on detecting brain responses to assigned tasks that the subject could covertly avoid doing, while performing the necessary overt responses (see Appendix 2).

Testing was divided into separate blocks. In each block the computer display presented 72 stimulus presentations or trials. In blocks 1–3, four stimulus groups were presented in each block, that is, in each block there were four unique probes, four unique targets, and 16 unique irrelevants. Each stimulus was presented three times in a block to make the total of 72 stimulus presentations per block. Stimuli were presented in random order. In blocks 4–7, five stimulus groups were presented in each block in random order until 72 trials had been presented. (Since the total of 72 trials is not divisible by 5, some randomly selected stimuli were presented 3 times and some 4.) In blocks 1–3, stimuli were acronyms. In blocks 4–7, stimuli were words and phrases.

Immediately before each block, we repeated the description of the significance of each of the probes and targets that were to appear in each block (but not the actual stimuli). For example, "In this test you will see the term for a medical technique applied to burn victims in battlefield situations, a medical instrument applied in field wound treatments, a type of injury sustained from exposure to chemical weapons, and the name of the individual who developed the preferred treatment for exposure to sarin gas."

Stimuli were presented for 300 ms at an ISI of 3000 ms. A fixation point ("X") was presented for 1000 ms prior to each stimulus. For each trial, the sequence was a fixation point for 1000 ms, the stimulus (target, probe, or irrelevant) for 300 ms, a blank screen for 1700 ms, and then the next fixation point.

Trials contaminated by artifacts generated by eye movements or muscle-generated noise were rejected on-line, and additional trials were presented until 72 artifact-free trials were obtained. Trials with a range of greater than 97.7 microvolts in the EOG channel were rejected. Data for "rejected" trials were collected and recorded, but rejected trials did not contribute to the count of trials presented, so each rejection resulted in an additional stimulus presentation. In 7 blocks, a total of 84 probe, 84 target, and 336 irrelevant artifact-free trials were collected, for a grand total of 504 trials. (Previous research, e.g., Fabiani et al., 1987 has shown that repeating the stimuli does not substantially affect the relevant brain response.)

Brain responses were recorded from the midline frontal, central, and parietal scalp locations (Fz, Cz, and Pz, International 10–20 system) referenced to linked mastoids, and from a location on the forehead to track eye movements. Med Associates silver-silver chloride disposable electrodes were held in place by a custom headband.

Data were digitized at 333 Hz, and resampled at 100 Hz off-line for analysis. Electroencephalograph (EEG) data were amplified at a gain of 50,000 using custom amplifiers. Electro-oculograph (EOG/eye movement) data were amplified at a gain of 10,000. Impedance did not exceed 10 kilohm. Analog filters passed signals between 0.1 and 30 Hz. Data were stored on disk for off-line analysis.

### **DATA ANALYSIS**

We analyzed ERP data from the Pz scalp site. Data were digitally filtered using a 49-point, equal-ripple, zero-phase-shift, optimal, finite impulse response, low-pass filter with a passband cutoff frequency of 6 Hz and a stopband cutoff frequency of 8 Hz (Farwell et al., 1993). Trials with a range of greater than 97.7 microvolts in the EOG channel were excluded from analysis. We decided on this threshold based on our previous experience (Farwell and Donchin, 1991; Farwell et al., 2013). In exploratory data analysis, we have varied this threshold considerably, and the results are robust even if we change this parameter within quite a wide range.

For each subject's data we conducted two separate classification-CIT analyses applying bootstrapping as described below. One analysis used the positive P300 peak followed by the LNP, a later negative peak (together also known as the P300-MERMER). A second analysis included only the positive P300. The P300 + LNP epoch was defined as 300–1800 ms after stimulus onset. The P300 epoch was 300–900 ms after stimulus onset. The two analyses were identical except for the epoch analyzed. A third analysis applied bootstrapping with the comparison CIT on the full P300 + LNP epoch, as in previous ERP studies with the comparison CIT.

The data analysis produced three sets of results for each subject: (1) a determination of information present or information absent along with a statistical confidence for the determination using the classification CIT and the full P300 + LNP epoch; (2) a comparable determination and statistical confidence using the P300 alone with the classification CIT; and (3) a comparable determination and statistical confidence using the comparison CIT on the full epoch. This allowed us to compare the error rate/accuracy and statistical confidence provided by (a) the P300 + LNP vs. the P300 alone in a classification CIT, and (b) the classification CIT vs. the comparison CIT.

#### **BOOTSTRAPPING**

#### *Classification-CIT bootstrapping method*

The primary data-analysis task was to determine whether the ERP responses to the probe stimuli contained a large P300 and LNP similar to that elicited by the targets, or whether the probe responses lacked a large P300 and LNP, like the irrelevants.

We used bootstrapping (Wasserman and Bockenholt, 1989; Farwell and Donchin, 1991; Farwell et al., 2013) to determine whether the probe responses were more similar to the target responses or to the irrelevant responses, and to compute a statistical confidence for this determination for each individual subject. The metric for similarity was double-centered correlation.

The bootstrapping procedure accomplished two goals: (1) to take into account the variability across single trials, while also maintaining the smooth and relatively noise-free shape provided by signal averaging; (2) to isolate the critical variable—knowledge of the information embodied in the probes—by classifying the responses to the probe stimuli as being either more similar to the target responses or to the irrelevant responses. We conducted two classification-CIT analyses, one using only the P300 and one using the P300 plus the LNP (together also known as the P300-MERMER).

Briefly, the bootstrapping procedure for the classification CIT is as follows. We repeat the following procedure 1000 times. Randomly sample P probes, T targets, and I irrelevants, with P, T, and I equal to the total number of probe, target, and irrelevant trials in the data set, respectively. In each iteration, compare the probe-target correlation with the probe-irrelevant correlation. Count the number of times that the probe-target correlation is greater than the probe-irrelevant correlation, and convert this to a percentage. This is the probability that the probe response is more similar to the target response than to the irrelevant response, or the probability that information present is the correct determination. 100% minus this is the probability that the probe response is more similar to the irrelevant response, or the probability that information absent is the correct determination.

We set an *a priori* bootstrapping probability criterion of 90% for an information-present determination and 70% (in the opposite direction) for an information-absent determination. If the probability was greater than 90% that the probe response was more similar to the target response than to the irrelevant response, we classified the subject as information present. The bootstrap probability is the statistical confidence for this determination.

The probability that information absent is the correct determination is 100% minus the probability that information present is the correct determination. For example, if there is a 90% probability that the probe response is more similar to the target than to the irrelevant response (information present is correct), then there is a 10% probability that the probe response is more similar to the irrelevant response (information absent is correct). If the probability was greater than 70% that the probe response was more similar to the irrelevant response than to the target response (equivalent to a 30% probability that the probe response was more similar to the target response), we classified the subject as information absent. The bootstrap probability is the statistical confidence for this determination.

If the results did not meet either criterion, we did not classify the subject in either category. The outcome would then be indeterminate (although there were no indeterminates).

For each subject, each data analysis method produced a determination and a statistical confidence, e.g., information present, 99.9% statistical confidence. The statistical confidence is the probability that the determination is correct, based on the withinsubjects statistical computation taking into account the size of the effect and the variability in the data.

**Figure 1** illustrates example stimuli, ERP responses, bootstrapping probabilities, and determinations for a hypothetical classification CIT to determine if an individual has information regarding US political history.

Error rate is the percentage of incorrect information-present (false positive) and information-absent (false negative) determinations. Accuracy is 100% minus the error rate. In reporting error rates and/or accuracy, indeterminates must be reported as such. In reporting "accuracy," some authors have confounded indeterminates with false positives and/or false negatives, reporting "accuracy" as the percentage of tests that result in a correct determination, and hiding the number of indeterminates. This irretrievably hides the true error rate if there are indeterminates, and makes it impossible to make a meaningful comparison with studies that report the true error rate. In any meaningful reporting, indeterminates if any must be identified as such, and not confounded with false positive or false negative errors. (Some legitimate techniques such as Bayesian analysis do not allow indeterminates, in which case this must also be reported).

We restricted our conclusions to a determination as to whether or not a subject knew the specific situation-relevant knowledge embodied in the probes at the time of the test. Our procedures recognize the fact that the ERP-based classification CIT detects only presence or absence of information—not guilt, innocence, honesty, lying, deception, or any past action or non-action.

#### *Comparison-CIT bootstrapping method*

The comparison CIT uses bootstrapping in an entirely different way. The comparison CIT ignores the target responses and applies

**FIGURE 1 | Classification CIT: stimuli, responses, and determinations.** Example of a classification CIT to determine whether an individual has information about US political history. Subject instructions: "You will view names of former US presidents. Some of these are on this list: George Washington, John Adams, Ronald Reagan. Press button T [target] for any name on this list. Otherwise press button O [other].

bootstrapping to compute the probability that the amplitude of the probe ERP is larger than the amplitude of the irrelevant ERP. The amplitude of the ERP response is defined as the difference between the highest voltage in the P300 window (300–900 ms) and the lowest voltage in the LNP window (900–1800 ms). (This is essentially the sum of the peak amplitudes of the P300 and the LNP.) This is in accord with the metric used in previous applications of the comparison CIT (e.g., Rosenfeld et al., 2008).

Trials are randomly sampled with replacement and averaged as described above for the classification CIT, except that only probe and irrelevant trials are sampled and averaged. In each of 1000 iterations, the amplitude of the ERP in the sampled probe average is compared with the amplitude of the ERP in the sampled irrelevant average. The percentage of times that the sampled probe ERP is larger than the sampled irrelevant ERP provides an estimate of the probability that the probe ERP is larger than the irrelevant ERP. If the probability that the probe ERP is larger than the irrelevant ERP is greater than 90%, then the subject is determined to be information present. If the probability that the probe ERP is larger than the irrelevant ERP is less than 90%, then the subject is determined to be information absent. (The comparison CIT does not have an indeterminate category.) A probability of 90% that information present is correct (that is, probe ERP is larger than irrelevant ERP) is equivalent to a probability of 10% (that is, 100%–90%) that information absent is correct. Therefore, any subject with a probability of over 10% that information absent is correct is determined to be information absent. This results in subjects being determined to be information *absent* when the computed bootstrap probability is as high as 89.9% that information *present* would be the correct determination, that is, as low as a 10.1% statistically computed probability that the selected information-absent determination is correct. Information-absent George Washington was a US president on the list, so then you should press T." Since Bill Clinton was not on the list and was not identified to the subject as a US president, the subject will press button O for Bill Clinton, whether he recognizes him as a US president (information present) or not (information absent). IP, Information Present; IA, Information Absent.

statistical confidences range from 10.1% to 99.9% and average 50% (chance) (see Appendix 2).

Correct information-absent determinations are of two types, valid and invalid. Valid determinations are those that have a greater than 50% (chance) statistical confidence, i.e., a greater than chance computed probability of being correct. An invalid determination is a (correct) determination where the statistical confidence is less than chance (50%); that is, the computed probability that the determination is correct is less than 50%. Such a result is invalid because clearly one cannot validly report that "Our statistical procedure determined that the individual is information absent; the statistics computed a probability of [15%] that the determination is correct." (This also applies to any other percentage lower than 50%.) Such a statement is not statistically meaningful or logically tenable. To be valid, the computed statistical confidence for a result must at least be better than chance (see **Figure 2** and Appendix 2). To be scientifically meaningful and practically useful, it must be considerably better than chance.

There are serious scientific, mathematical, logical, and statistical flaws with the ERP-based comparison-CIT data-analysis procedure, as described in Farwell (2012, 2014), Farwell et al. (2013), and Appendix 2. These flaws cannot be corrected by simply changing the criterion for information present/absent determinations. We have implemented this procedure, however, because this is the way that the bootstrapping statistical confidence has been computed in all or virtually all of the comparison-CIT studies that have previously applied bootstrapping (e.g., Rosenfeld et al., 2008).

**Figure 2** illustrates example stimuli, ERP responses, bootstrapping probabilities, and determinations for a hypothetical comparison CIT to determine if an individual has information regarding US political history.

**FIGURE 2 | Comparison CIT: stimuli, responses, and determinations.** Example of a comparison CIT to determine whether an individual has information about US political history. Subject instructions and button presses are the same as in **Figure 1**. IP, Information Present; IA, Information Absent.

# **RESULTS**

The results are delineated in **Tables 1**–**4** and illustrated in **Figures 3** and **4**. **Table 1** presents the error rate/accuracy of the results of the classification CIT, for both P300 + LNP and P300 analysis methods. Both P300 + LNP and P300 analysis methods produced 0% error rate, 100% accuracy. Both also produced no indeterminates.

**Table 2** presents the error rate/accuracy of the comparison CIT. (Only correct determinations and errors are tabulated: the comparison CIT does not have an indeterminate category).

**Table 3** presents the individual determinations and the statistical confidences for each subject whose true state was information present. It compares the results obtained with the classification CIT with P300 + LNP with the other two methods: classification CIT with P300 and comparison CIT.

**Figure 3** presents the brain responses to probe, target, and irrelevant stimuli for each of the information-present subjects, averaged across all trials for each subject.

**Table 4** presents the individual determinations and the statistical confidences for each subject whose true state was information absent. It compares the results obtained with the classification CIT with P300 + LNP with the other two methods: classification CIT with P300 and comparison CIT.

**Figure 4** presents the brain responses to probe, target, and irrelevant stimuli for each of the information-absent subjects, averaged across all trials for each subject.

# **RESULTS OF THE CLASSIFICATION CIT WITH P300 + LNP ANALYSIS**

All classification-CIT determinations with the P300 + LNP analysis were correct. Error rate was 0%: there were no false positives and no false negatives. Accuracy was 100%. Also, there were no indeterminates. Grier's A' (Grier, 1971) was 1.0.