#### *Federated linked data repository and semantic search*

In order to allow BCI users to maintain recorded data in their own servers as well as conducting semantic data search among multiple servers, our BCI system must be equipped with a distributed Linked Data repository and a federated semantic data querying scheme. Both of these facilities are safeguarded by Internet communication security and multi-domain attribute-based access control mechanisms.

The distributed Linked Data repository consists of two functional components: (1) the individual Fog/Cloud Servers that maintain the actual BCI data sets and (2) the RDF repository spread across the Cloud Servers that manage the meta-data of the Linked Big Data superstructure. In order to protect user privacy, all personal information and raw BCI data shall be stored in either the Fog Server(s) on users' premise or the trusted Cloud Server(s) authorized by the users. All sensitive data are protected by strong communication and information security measures. Only the anonymous subject identifiers, the universal resource identifiers (URI) and the meta-data tags of the data sets may be disseminated among the Cloud Servers. Together, the Cloud Servers maintain a distributed *RDF repository* that can be queried under anonymity protection using the *SPARQL Protocol and RDF Query Language* (SPARQL) v.1.1 (W3C, 2014c).

SPARQL 1.1 query language supports the *federation* of multiple SPARQL endpoints. As shown in **Figure 3**, a client can issue a SPARQL 1.1 query to a *query mediator*, which will convert the query into several *sub-queries* and forward them to different SPARQL endpoints. Each endpoint then processes the sub-query it received and sends back the query results. Finally, the mediator joins the query results from different endpoints to produce the final result.

Currently, we use *Virtuoso Universal Server* (VUS) v6.01 (OpenLink Software, 2014) to host the distributed RDF repository. Offered freely as a key component of (LOD2 Technology Stack, 2013), VUS is the most popular open-source semantic search engine for Linked Data applications. VUS can perform *distributed RDF link traversals* as a rudimentary mechanism to support federated SPARQL. To use this mechanism, we developed a Federated Query Mediator that can run on any Fog Server. This mediator can accept semantic data queries expressed in the RESTful/JSON web service format; transform them into

ontology design pattern.

SPARQL 1.1 sub-queries and then issue these sub-queries to the VUS installed in multiple Cloud Servers. This RESTful/JSONcompatible Federated Query Mediator not merely implements the federated semantic search; it also provides a standard web service interface for any authorized mobile applications to issue SPARQL queries and thus access our linked BCI repository.

(tan colored nodes with dbp prefix). The sub-graph with red outlines

#### **RESULTS**

#### **PILOT SYSTEM**

In the past two years, the Pervasive Embedded Technology (PET) Laboratory at NCTU and the SCCN at UCSD have been working together closely to develop a proof-of-concept prototype of the proposed pervasive EEG-based BCI system. In this endeavor, we chose to use wireless dry-electrode EEG headsets and MEMS motion sensors as the *front-end devices*, Android mobile phones as the personal user interfaces, compact personal computers as the *near-end Fog Servers* and a supercluster of computers hosted by the Taiwan NCHC as the *far-end Cloud Servers*. **Table 1** provides a detail list of hardware and software components that are used to build this proof-of-concept pilot system.

This pilot system is currently deployed on two application/fogcomputing sites: (1) NCTU PET Lab, (2) UCSD SCCN, and two cloud-computing sites: (1) NCHC supercluster and (2) UCSD SCCN virtual machine server. **Figure 4** illustrates the system configuration at these sites. Both NCTU and UCSD fog-computing sites have participated in all pilot experiments and demonstrations. Currently, the NCHC cloud-computing site is hosting the BCI data repository and the BCI web portal while the SCCN server is maintaining an archive of legacy BCI data sets.

In the past year, both PET and SCCN teams have used this pilot system to perform different experiments demonstrating the capability and the potential of pervasive real-world BCI operations. Following subsections describe the two multi-site experiments we have performed.

#### **SYNCHRONOUS BCI DATA STREAMING OVER INTERNET**

The NCTU-UCSD team performed a successful live demonstration of real-time synchronous multi-modal BCI data streaming at a project review meeting of the Cognition and Neuroergonomics Collaborative Technology Alliance (Can-CTA) Program on March 13, 2013. In that intercontinental demonstration, Prof. John Zao was wearing a four-channel wireless *MINDO-4S* EEG headset and a 9-DOF *BodyDyn* motion sensor at NCTU PET Lab in Hsinchu, Taiwan. Sampled data from both sensors were transmitted simultaneously via Bluetooth to a Samsung Galaxy Note 1 smart phone. The data streams were then sent to a Fog Server at the PET Lab and multicasted over the Internet to a Cloud Server at the NCHC also in Hsinchu, Taiwan and a desktop computer at UCSD SCCN in San Diego, California. Four-channel EEG data as well as 3D linear acceleration and 3D angular velocity—with a total of 10 channels—were displayed at SCCN in synchrony with the live image of Prof. Zao's movements that was beaming through a Google Hangout session. Almost no perceptible delay can be seen between the video images and the EEG/motion waveforms appeared on the display at SCCN. A video clip attached to this paper shows an excerpt of that demonstration session.

Detail timing measurements of the end-to-end synchronous transports were made later in August during several replay of the demonstration and analyzed off time. **Figure 5** shows the time traces of standalone and concurrent transport of the two data streams. **Table 2** lists the formats and sizes of individual messages as well as the statistics of timing measurements of the transports. The significant differences in the mean values of transport latency were due to the offsets existing between the system clocks in the mobile phone at NCTU and the desktop computer at UCSD.

These time traces show that no message was lost because the transport was conducted using MQTT messaging over TCP sessions. Small standard deviations of transport latency imply that few retransmissions were needed to provide reliable delivery. Latency of the EEG sessions fluctuates slightly more than that of the motion sessions; this suggests that a few more retransmissions were needed to deliver the longer EEG messages. The average transmission intervals (237–243 ms) in both standalone and concurrent transport sessions match closely with the expected quarter-second (250 ms) emission interval of the data messages. Besides, the average reception intervals also match closely with the average transmission intervals. These matching figures hinted smooth transmissions that were free of hop-by-hop traffic congestion and end-to-end message queuing. This superb performance may be partially due to the fact that the experiment was carried out between two university campuses equipped with gigabit Ethernets. Larger fluctuations in transmission/reception intervals as well as transport latency shall be expected when the data streaming is conducted over home networks.

Both the live demonstration and the performance statistics indicate that it is entirely possible to send BCI data streams reliably in real time to multiple destinations over the Internet. Thus, this experiment affirms the feasibility of Internet-based on-line EEG-BCI operation. Nonetheless, we must point out a potential *scalability* issue that may arise during multicasting of multi-channel EEG data streams. As the EEG channel numbers and sampling rates increase, the data rates of the multicasting sessions may quickly exceed the up-link bandwidth (approximately 1 Mbps) of home networks. In order to avoid causing network congestion in these cases, data compression techniques such as *compressive sampling* (Candes and Wakin, 2008) must be employed to reduce the message size. In fact, as a general principle, we should avoid sending raw data over the Internet in real time because such a practice will not only consume more network bandwidth but also incur longer transport latency. With the presence of ubiquitous Fog Servers, we should perform most real-time signal processing and brain state prediction on the Fog Servers and send only the extracted signal features, the brain states

#### **Table 1 | Hardware and software components for the pervasive on-line EEG-BCI pilot system.**

#### **HARDWARE COMPONENTS**

and the meta-data over the Internet in real time. This operation principle was demonstrated in the following experiment.

#### **MULTI-PLAYER ON-LINE INTERACTIVE BCI GAME**

In order to optimize the communication and computation efficiency, users of our pervasive EEG-BCI system should always use a Fog Server nearby to perform real-time signal processing and brain state prediction rather than performing the computation at the frontend sensors / mobile phones or sending the raw data over the Internet to the Cloud Servers. To demonstrate this operation principle, we developed the *EEG Tractor Beam*, a multiplayer on-line EEG-BCI game, and launched its first game session on September, 2013. Since then, this game has been played in several public occasions with players from both US and Taiwan.

**Figure 6** illustrates the system architecture for this game, which is also a typical configuration for multi-site interactive BCI operation. Each user has a typical BCI frontend (shown as a sky blue box) consisting of an EEG headset and a mobile phone that are connected to a local Fog Server (a navy blue box). The Fog Servers associated with different users may exchange information with one another and a Cloud Server (the green box). The game was running as a mobile application on each user's mobile phone, which serves mainly as a graphic user interface (GUI). Raw EEG data streams were sent directly to the Fog Server or through the mobile phones. Real-time signal processing and prediction were performed on the Fog Servers, each of which ran a BCI signal processing pipeline. The brain states of individual users were published by the Fog Servers and sent to the game running on each mobile phone, which subscribed for the brain state information.

On its display, the multiplayer game shows all the players on a ring surrounding a target object. Each player can exert an attractive force onto the target in proportion to her level of concentration, which was estimated using the following formula (Eoh et al., 2005; Jap et al., 2009):

$$\mathbb{C} \triangleq \ln \left( \frac{\text{PSD}\_{\beta}}{\text{PSD}\_{\alpha} + \text{PSD}\_{\theta}} \right).$$

Where the PSDs are the average power spectral density in α, β and θ bands of the player. In order to win the game, a player should try to pull the target toward herself while depriving other players their chances to grab the target. The game implements a "winnertake-all" strategy: a player is awarded points at a rate proportional to the percentage of total attractive force she exerts on the target, which is calculated by dividing that player's concentration level by the sum of the levels among all the players. However, a player can only start to accumulate points if she contributes at least her fair share to the total sum. A tractor beam will appear between that player and the target when her concentration level passes that threshold. That was when she starts to cumulate her points. **Figure 7** shows a picture of four players engaging in the game across the Pacific Ocean.

The necessary EEG signal processing and the estimation of *concentration level* were performed by the BCILAB/SIFT pipeline (Delorme et al., 2011) running on MATLAB R2013a (Mathworks, 2013) installed in the Fog Servers. **Figure 8** displays the typical processing stages of this brain state estimation pipeline. Its MATLAB code was included in the Appendix for reference. The EEG preprocessing stage aims at cleaning up the raw EEG signals, which was heavily contaminated by artifacts due to eye blinks and head movements. The heavy computation of signal correlation and artifact subspace reconstruction (Mullen et al., 2012) can only be performed on the Fog Servers; these algorithms can quickly drain the batteries in the sensors and the mobile phones. Because players' concentration levels was estimated as the ratios between power spectral density in different EEG frequency bands, multitaper spectral estimation,

**FIGURE 4 | Pilot system architecture of (A) Cloud Computing site at NCHC, Taiwan and (B) Fog Computing sites at NCTU PET Lab, Taiwan and UCSD SCCN, San Diego, California.**

**FIGURE 5 | Time traces of end-to-end synchronous transport of motion and EEG data streams. (A,B)** show the time traces of motion and EEG data transports in two separate sessions. **(C,D)** show the traces of both transports in the same session. The blue lines mark the traces of transmission time while the red lines mark those of reception time. Their slopes give the average transmission and reception intervals of individual messages.

#### **Table 2 | Performance measurements of synchronous BCI data streaming over Internet.**

*aThe average or mean values of transport latency were contaminated by the offset between the system clocks in the mobile phone at NCTU and the desktop computer at UCSD.*

power density calibration1 and averaging were done before the concentration levels were computed. Please note that although we chose to implement the BCI processing pipeline using BCILAB and SIFT, other real-time signal processing software can be used to perform the computation.

To demonstrate the working of our BCI processing pipeline, we showed in **Figure 9** two 1-min scattered plots of a player's centration levels estimated during a 2-min open-eye relaxation period and an equal-length open-eye concentration period. The average concentration level during the relaxation period was μ*R* = −0.19 < 0 as expected while the average level during the concentration period was μ*C* = +0.45. The difference between these values was statistically significant. The estimated values fluctuated notably during both periods. Partially, this was due to the wavering of player's concentration levels, but more likely, the fluctuations were caused by the remaining artifacts of head movements and muscle tension. These artifacts remain as an inevitable component of real-life EEG recording and a challenge to real-world BCI operation. Finally, both plots showed a general downward trend. This was because when the player tried to sustain her concentration, mental fatigue invariably set in after a short while; hence, her EEG power in beta band tended to decrease gradually relative to the power in alpha band. On the other hand, when the player tried to relax, it took some time for her to settle into a relaxed state; hence, we expect her alpha power to increase gradually relative to her beta power. In both cases, gradual decrease in concentration level was expected, especially if the player was untrained to perform the cognitive task.

In all the gaming sessions, the data rates and transport latencies over the Internet have been low since the Fog Servers published short messages merely containing players' identifiers and concentration levels. Also, the game displays among different players were synchronized because they all used Samsung Galaxy phones with compatible computing power. A small but noticeable display lag may appear if a player uses an old Android phone. This display lag can be eliminated using standard game synchronization protocols.

While *EEG Tractor Beam* is a somewhat frivolous demonstration of the capability of the pervasive on-line EEG-BCI system, it does demonstrate some powerful concepts that may have applications far beyond on-line gaming. Foremost, the system has the ability to acquire and process EEG data in real time from large number of users all over the world and feed their brain states back to these individuals as well as any professionals authorized to monitor their cognitive conditions. With distributed Fog and Cloud Servers, our on-line EEG-BCI infrastructure can be scaled indefinitely without adding unsustainable traffic load onto the Internet. Hence, it presents a viable way to realize *interact BCI*. Furthermore, the system has the ability to process, annotate and archive vast amount of real-world BCI data collected during the BCI sessions. Unlike the existing EEG databases, which depend on researchers to donate their data sets, this pervasive EEG-BCI infrastructure collects data sets—with users' approval—as an essential part of its normal operation. This intrinsic data collection provides a natural way to implement *"big data" BCI* as well as *adaptive BCI* in the near future. In the following section, we discuss the potential values and impacts of this pervasive on-line system toward the real-world BCI applications.

#### **DISCUSSIONS**

In this section, we examine the operation scenarios supported by the pervasive on-line EEG-BCI system as well as the costs and benefits of its potential use. This discussion begins with a comparison with the existing BCI systems and on-line physiological data repositories; it is concluded with a highlight of future development.

#### **COMPARISON WITH CURRENT PRACTICE**

Currently, all BCI systems operate in a *standalone* fashion and need to be *personalized* before their use. No matter whether they are used to control patients' wheelchairs, conduct neuromarketing or provide biofeedback, these systems require their users to go through tedious training processes in order to adapt

1The multitaper estimates of EEG power spectral density were multiplied by their sampled frequencies in order to compensate the natural decline of EEG spectral power inversely proportional to its frequency.

**FIGURE 7 | An EEG Tractor Beam game session with four people playing over the Internet: two players at SCCN in San Diego, USA are shown in the foreground while two other players at NCTU in Hsinchu, Taiwan appear in the monitor display.** The inset at the lower right corner shows a captured view of the game display.

them for personal use. Moreover, they often require the training process to be repeated once the use situations are changed. Our on-line EEG-BCI system, however, can download an initial brain state prediction model from the Cloud Server based on the real-world situation in which it operates, and then refine the model progressively using the data gathered from its users (section Adaptive BCI). This *adaptive* capability as well as its *interactive* and *big data processing* capability will distinguish our system from the existing ones.

The biomedical engineering community has been exploiting Cloud Computing and Big Data Mining technologies for years. In the past decade, several on-line physiological data repository including BrainMap (Research Imaging Institute, 2013), PhysioNet (Goldberger et al., 2000), and HeadIT (Swartz Center for Computational Neuroscience, 2013) have been put on line. Among them, PhysioNet earned the best reputation through the offering of a wide-range of data banking and analysis services. However, none of these data repositories are ready to accept real-time streaming data.

Furthermore, as demonstrated in the *EEG Tractor Beam* gaming sessions, our on-line EEG-BCI system also has the ability to support real-time multi-user collaborative/ competitive neurofeedback. This unique ability may lead to many novel applications in cognitive collaboration, e-learning as well as on-line gaming and mind training.

#### **OPERATION SCENARIOS**

As shown in **Figure 10**, the pervasive on-line EEG-BCI system can operate in three different scenarios: Big Data BCI, Interactive (or

Closed-Loop) BCI and Adaptive BCI. Each scenario represents an incremental enhancement of system capability.

#### *Big data BCI*

In this first operation scenario, the pervasive EEG-BCI system is endowed with the capability to collect multi-modal data along with relevant environmental information from real-world BCI applications anytime anywhere. This capability not only enables BCI applications to identify common EEG correlates among different users while they perform the same tasks or exposed to similar stimuli; it also provides a pragmatic way to gather vast amount of BCI data from real-life situations for cross-sectional and longitudinal studies. A linked BCI data repository and a RESTful web service API have been created for maintaining the data collection. Human clients would use the Web Portal (http:// bci.pet.cs.nctu.edu.tw/databank) to access and query the data. Machine or application clients would use the RESTful web service API (http://bci.pet.cs.nctu.edu.tw/api) to perform specific data operations.

Currently, Big Data BCI is the only fully functioning scenario of our pilot system. All our experiments archived their data sets in the linked BCI data repository.

#### *Interactive BCI*

People's brain states and their EEG characteristics can be influenced acutely by the changes in environment conditions. Various visual, auditory, heat and haptic stimuli have long been used to evoke neural responses or modulate users' brain states. Currently, all these stimuli are static in nature as they lack the ability to adapt to users' changing brain states. Hence, the stimuli would become ineffective as habituation dampens users' neural responses or in the worse cases, cause harmful side effects.

Since the on-line EEG-BCI system can perform real-time brain state prediction on the Fog Servers, we can introduce a feedback control loops between the stimuli and the users' EEG responses. This *interactive* operation scenario can improve the accuracy of exogenous brain state prediction and the effectiveness of brain state modulation by applying the most powerful stimuli based on closed-loop feedback control.

#### *Adaptive BCI*

It is well known that people's EEG responses toward the same tasks (or stimuli) often differ significantly from one another and can change drastically over time. Thus, the prediction models employed by our BCI system must adapt to individual user's EEG responses and adjust their parameters continuously to track the changes of their characteristics. Usually, model adaptation and refinement are conducted using a large amount of training data. In order to reduce the amount of training data from individual users, we are exploring the feasibility of adapting the prediction model by leveraging the archived data collected from other users plus a small amount of training data acquired from this new user.

In our system, the adaptive BCI operation is performed through the cooperation between a Fog Server and its associated Cloud Server. The Fog Server will upload the annotated BCI data along with the predicted brain states, the prediction model specification and the confidence level on its prediction onto the Cloud Server. Then, the Cloud Server will issue semantic queries to find similar EEG data fragments among the archived BCI data sets and then apply *transfer learning* techniques on both the acquired and the archived data sets. Through repetitive trials, this *progressive refinement* process will likely produce a prediction model better-adapted to the BCI activity of that user in a specific real-world situation.

#### **PRACTICAL ISSUES**

Users are rightfully concerned about several practical issues such as *cost*, *availability*, *security* and *privacy* that may arise from the daily use of this elaborate infrastructure. Following are the concrete facts we hope may soothe some of these concerns.

First, the technologies we employ have already been used to provide Internet services today. The Cloud Servers have been running Google search and Yahoo web portals all along. Television set-top boxes and game consoles that function as the Fog Servers are popular electronic appliances. Almost without exception, mobile applications are installed in every smartphones these days. From this perspective, pervasive EEG-BCI is a natural outcome of the on-going trend to foster smart living using the state-of-art information and communication technologies. The incremental costs of using pervasive EEG-BCI will be quite affordable. A user only needs to purchase a wearable EEG headset and download a mobile application. The computing engine will be automatically downloaded onto her "fog server" once the user signs a service agreement. It is quite possible that pervasive EEG-BCI would become a fashion very much like the use of fitness gadgets these days.

Second, pervasive EEG-BCI will likely be offered by a supply chain of vendors that can bundle this service with Internet connectivity, content and computing. The huge infrastructure deployment and maintenance costs must be amortized among these service providers. Furthermore, the BCI data repository and the progressive model refinement technologies will take time to develop. Hence, this service must go through a maturing process.

Third, information security and personal privacy should indeed be users' common concerns. However, they must be dealt with as two separate issues. The basic guarantees of user anonymity, secure exchange, save storage and limited access can be provided through the employment of necessary communication and information security measures. These mechanisms are discussed in the following section. However, many users would be terrified by the notion that "the big brother can know not only where I click but also what I *think* when I browse the web!" Protection of personal privacy in that sense must be offered not merely through technical means but by developing and enforcing public policies according to social norms. Surprisingly, the protection of personal cognitive information is not more difficult than the protection of personal behavioral data collected by say Google, and is much easier than preventing information leakage via social networking because unlike individuals, reputable service providers are much more serious and diligent in guarding their clients' personal information.

#### **FUTURE DEVELOPMENT**

The pervasive EEG-BCI pilot system is merely a prototype. We plan to develop it into a field-deployable system within the coming year. Specifically, we will further develop its semantic data model and provide multiple ways to access streaming and archived data via multiple Internet protocols. Moreover, the following capability will be added to the system.

#### *Cloud based progressive model refinement*

Fog Servers will be able to perform adaptive brain state prediction with the aid of *progressive model refinement* carried out by the Cloud Servers. The process begins with *automatic annotation* of EEG data segments with their corresponding brain states according to the outcome of current prediction process. The meta-data annotation will be sent to the Cloud Servers so that cloud-based semantic search can find large number of data segments that match with certain personal, environmental and event specification. These data segments will then be fed into machine learning algorithms to calibrate the prediction model. The calibrated model will be pushed back to the Fog Servers and used to perform the next round of brain state prediction and data annotation. This iterative process will continue to improve the accuracy of prediction and enable the system to track the non-stationary brain dynamics. The Predictive Model Markup Language (PMML v.3.2, 2008; Guazzelli et al., 2009) will be adopted as the interoperable model specification and encoding format.

#### *Information security and user privacy protection*

We are developing a pervasive machine-to-machine communication security infrastructure based on the Internet standards: Host Identity Protocols (HIP) (IETF, 2014) and Host Identity Indirection Infrastructure (Hi3) (Nikander et al., 2004). HIP has become an increasingly popular approach to offer secure communication among the Internet of Things (Kuptsov et al., 2012). In addition, we developed a multi-domain attribute-enriched rolebased access control architecture (Zao et al., 2014). Both of these technologies will be used to offer the essential communication and information security protection.

#### **CONCLUSION**

The pervasive on-line EEG-BCI system we built culminated the development trends of two state-of-art information technologies: *Internet of Things* and *Cloud Computing*. As such, our pilot system can be regarded as a pioneering prototype of a new generation of real-world BCI system. As mentioned in section Operation Scenarios, these on-line systems will not merely connect the existing standalone EEG-BCI devices into a global distributed system; more importantly, they are fully equipped to support futuristic operations including intrinsic real-world data collection, massive semantic-based data mining, progressive EEG model refinement, stimuli-response adaptation. In academic and clinic research, these pervasive on-line systems will cumulate vast amount of EEG-BCI data and thus enable cross-sectional and longitudinal studies of unprecedented scale. Inter-subject EEG correlates of specific tasks and stimuli may be found through these studies. In the commercial world, numerous consumer applications will become feasible as wearable EEG-BCI devices can track people's brain states accurately and robustly in real time.

#### **ACKNOWLEDGMENTS**

This system development project is a team effort. The Pervasive Embedded Technology (PET) Laboratory at the National Chiao Tung University (NCTU) in Taiwan, the SCCN at the University of California, San Diego (UCSD) in the United States of America and the NCHC sponsored by the Taiwan National Research Council have all contributed to the development of this pilot system. In addition, Dr. Ching-Teng Hsiao of the Research Center for Information Technology Innovation (CITI) at the Academia Sinica of Taiwan has served as a technology consult throughout this project. Among the authors: Tchin-Tze Gan, Chun-Kai You, and Chien Yu of PET as well as Yu-Te Wang of SCCN were responsible for the development of the Fog and Cloud computing infrastructure; Sergio José Rodríguez Méndez (PET), Cheng-En Chung (PET), and Ching-Teng Hsiao (CITI) created the Linked Data superstructure and developed the mobile applications to perform the semantic data queries; Tim Mullen, Christian Kothe, and Yu-Te Wang all of SCCN have developed the BCILAB and LSL toolboxes and implemented the EEG signal processing pipelines; San-Liang Chu and his technical team at NCHC set up the cloud servers for this project. Finally, John K. Zao, the Director of PET Lab, was the innovator and the designer of this infrastructure; Tzyy-Ping Jung, the Associate Director of SCCN, first proposed the approach of pervasive adaptive BCI and mobilized this effort; Ce-Kuen Shieh, the Director of NCHC, endorsed and promoted the inter-collegiate deployment of this pilot system.

#### **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/journal/10.3389/fnhum. 2014.00370/abstract

#### **REFERENCES**

DBpedia. (2014). *wiki.dbpedia.org*: *About*. Available online at: http://dbpedia.org/

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 08 December 2013; accepted: 13 May 2014; published online: 03 June 2014.  Front. Hum. Neurosci. 8:370. doi: 10.3389/fnhum. 2014.00370*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2014 Zao, Gan, You, Chung, Wang, Rodríguez Méndez, Mullen, Yu, Kothe, Hsiao, Chu, Shieh and Jung. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Neuroergonomics: a review of applications to physical and cognitive work

#### *Ranjana K. Mehta1\* and Raja Parasuraman2*

1 Department of Environmental and Occupational Health, School of Rural Public Healthy, Texas A&M University, College Station, TX, USA 2 Center of Excellence in Neuroergonomics, Technology, and Cognition, George Mason University, Fairfax, VA, USA

#### *Edited by:*

Klaus Gramann, Berlin Institute of Technology, Germany

#### *Reviewed by:*

Matthew Rizzo, The University of Iowa College of Medicine, USA Andy McKinley, Applied Neuroscience Branch of the 711 Human Performance Wing, USA

#### *\*Correspondence:*

Ranjana K. Mehta, Department of Environmental and Occupational Health, School of Rural Public Healthy, Texas A&M University, Room 106, College Station, TX 77843-1266, USA e-mail<EMAIL_ADDRESS>
Neuroergonomics is an emerging science that is defined as the study of the human brain in relation to performance at work and in everyday settings. This paper provides a critical review of the neuroergonomic approach to evaluating physical and cognitive work, particularly in mobile settings. Neuroergonomics research employing mobile and immobile brain imaging techniques are discussed in the following areas of physical and cognitive work: (1) physical work parameters; (2) physical fatigue; (3) vigilance and mental fatigue; (4) training and neuroadaptive systems; and (5) assessment of concurrent physical and cognitive work. Finally, the integration of brain and body measurements in investigating workload and fatigue, in the context of mobile brain/body imaging ("MoBI"), is discussed.

**Keywords: physical work parameters, physical fatigue, mental fatigue, vigilance, training, neuroadaptive systems**

#### **INTRODUCTION**

Neuroergonomics is defined as the study of the human brain in relation to performance at work and everyday settings (Parasuraman, 2003; Parasuraman and Rizzo, 2007). It integrates theories and principles from ergonomics, neuroscience, and human factors to provide valuable insights on brain function and behavior as encountered in natural settings (Parasuraman, 2011). In this paper, we review neuroimaging techniques applicable to neuroergonomics that has expanded our understanding of the neural correlates of operators' physical and cognitive capabilities and limitations when they interact with work systems. Moreover, while experimental laboratory studies have advanced our knowledge of brain functions during simulated work, it is important to assess operator performance in naturalistic work settings. Understanding brain function in such dynamic and mobile work settings requires the use of ambulatory neuroimaging techniques (Makeig et al., 2009).

There are two main reasons why ambulatory neuroimaging techniques need to be developed for ergonomics research and practice. First, by definition, physical ergonomics requires that participants move their limbs or bodies while carrying out some physical task. Moreover, while cognitive ergonomics studies can be conducted in immobile participants, research on *embodied cognition* has shown that cognitive processing when moving and interacting in the physical world may have unique characteristics that can only be captured with mobile neuroimaging (Clark, 1998; Parasuraman, 2003; Raz et al., 2005). This review discusses the use of neuroergonomics methods to evaluate brain responses in mobile work environments. We discuss the suitability and feasibility of mobile and immobile brain imaging techniques in the context of physical neuroergonomics, cognitive neuroergonomics, and neuroergonomic assessment of concurrent physical and mental work. Finally, we consider the requirements and utility of combined brain and body measurements in investigating workload and fatigue for neuroergonomic investigations.

#### **NEUROERGONOMIC METHODS**

Neuroergonomic studies rely heavily on existing neuroimaging techniques to understand brain structures, mechanisms, and functions during work. Neuroimaging techniques applicable to neuroergonomics fall into two general categories, those that are direct indicators of neuronal activity in response to stimuli, such as electroencephalography (EEG) and event-related potentials (ERPs), and those that provide indirect metabolic indicators of neuronal activity, such as functional magnetic resonance imaging (fMRI), positron emission tomography (PET), and functional near infrared spectroscopy (fNIRS). EEG represents summated post-synaptic electrical activity of neurons firing in response to motor/cognitive stimuli as recorded on the scalp, and thus offers excellent temporal resolution of electromagnetic brain changes, on the order of milliseconds. In comparison, fMRI and PET techniques, that provide information on cerebral blood flow in response to neuronal activity, have low temporal resolution (on the order of about 10 s), but offer excellent spatial resolution (1 cm or better) and unlike EEG, they provide valuable information on location of the neural signal generated.

Since neuroergonomics distinguishes itself from traditional neuroscience in that it evaluates brain functions in response to work, it is important that the neuroergonomic methods provide the flexibility to assess brain function in naturalistic work settings. Some neuroimaging techniques are better designed for and adapted for assessing brain functions in mobile work environments than others. The pros and cons of neuroergonomic methods are discussed in reference to three criteria: (1) temporal resolution, (2) spatial resolution, and (3) degree of immobility. **Figure 1** illustrates how these neuroimaging techniques compare against

"fnhum-07-00889" — 2013/12/19 — 20:28 — page 1 — #1

each other based on the three criteria. In addition, **Table 1** lists these methods and their major characteristics, such as portability, cost, along with spatial and temporal resolution. In this section, we provide a brief review of the various methods that have been used in neuroergonomic evaluations of human work, emphasizing measures of brain function and applicability in mobile experimental/field settings.

Electroencephalography signals are the spatial summation of current density induced by synchronized post-synaptic potentials occurring in large clusters of neurons measured at the scalp (Pizzagalli, 2007). The EEG is recorded as differences in voltage between active electrodes at different positions on the scalp, such as the frontal, parietal, temporal, and occipital lobes of the brain according to the International 10–20 System, and a reference electrode, typically the ear. EEG signals comprises of different frequency bands, each associated with various cognitive and physical states. Spectral analyses of EEG signals can be conducted to assess power in different frequency bands: delta (0.5–3 Hz), theta (4–8 Hz), alpha (8–13 Hz), beta (13–30 Hz), and gamma (40–50 Hz). Another commonly computed EEG-driven spectral metric (i.e., brain) used in conjunction with muscular output (i.e., body) is corticomuscular coherence (CMC). CMC reflects "communications" between the brain and muscle and is determined as the coherence between sensorimotor cortex activation obtained from EEG and muscular activation as measured by electromyography (EMG) during motor activities (Halliday et al., 1995).

Electroencephalography-derived ERPs represent the brain's neural response to specific sensory, motor, and cognitive events. ERPs represent the outcome of signal averaging of EEG epochs time-locked to a particular stimulus or response event. To evaluate mental workload or examine human error (Fedota and Parasuraman, 2010), ERP waveforms are examined for changes in the amplitude and latency of different ERP *components*, typically defined as positive or negative peak activity (such as the P3 and N1 components) or slowly rising activity such as the lateralized readiness potential (Luck, 2005). To assess neural bases of motor activities, motor-related cortical potential (MRCP) ERP components have been studied that are characterized by a slowly rising negative potential, called *Bereitschaftspotential* (BP) or readiness potential, which is followed by a sharp rising negative potential, known as negative slope. As the onset of MRCP occurs prior to the onset of the motor activity, MRCP is considered to indicate premotor activity, which involves specific brain regions that prepare for a desired motor behavior (Kornhuber and Deecke, 1965).

Electroencephalography-driven metrics, both spectral and temporal, in evaluating brain function during naturalistic complex tasks are relatively unobtrusive so that it does not interfere with the operator's work performance. Its compact size and low cost, compared to other neuroimaging techniques such as fMRI and PET, makes it fairly well suited for use in both laboratory and field conditions. While artifacts attributed to movement, eye blinks, and physiological interference accompany EEG data, several algorithms have been developed to allow for the removal of noise in the EEG signal in real time or during post processing of the data (Jung et al., 2000). Recent developments in making "field-friendly" EEG systems include "dry" electrode caps, which do not need extensive participant preparation time, as well as wireless systems that do not require the participant to be tethered to cables. These technical developments have

"fnhum-07-00889" — 2013/12/19 — 20:28 — page 2 — #2

enhanced the relevance and value of EEG for mobile applications (Makeig et al., 2009).

Cerebral hemodynamic techniques such as fMRI and PET provide valuable information on source locations of distinct neural activation patterns associated with simple and complex cognitive, motor, and affective functions. While PET uses injected radioactive tracers to measure the blood flow in response to stimuli, based on their respective magnetic characteristics fMRI focuses on the resulting contrast between oxygenated and deoxygenated blood called the Blood Oxygenation Level Dependent or BOLD signal (Poldrack et al., 2011). Both fMRI and PET have been fundamental in advancing our knowledge on brain functions and mechanisms during simple, and relatively static, cognitive and motor tasks. By leveraging high spatial resolution offered by fMRI measurements, reliable techniques for the fMRI-EEG integration have been made possible that offer greater spatio-temporal resolution of imaging dynamic brain activity as well as significant improvement over the conventional fMRI-weighted EEG source imaging techniques (Liu and He, 2008; Yang et al., 2010). At the same time, fMRI and PET present several limitations in studying brain functions, such as the required supine position that may yield altered hemodynamic changes than seated or standing positions (Raz et al., 2005), limited mobility, and restrictions on synchronized brain-body measurements (Makeig et al., 2009). Moreover, the increasing need to examine brain activation patterns in complex tasks more representative of natural everyday situations have led researchers to adopt alternative neuroimaging techniques that offer better mobility features.

Functional near infrared spectroscopy is a non-invasive optical technique for measuring cerebral hemodynamics similar to PET and fMRI but with lower spatial resolution. By utilizing the tight neurovascular coupling between neuronal activity and regional cerebral blood flow (Villringer and Chance, 1997) fNIRS measures regional cerebral hemodynamic changes (i.e., changes in oxy- and deoxy-hemoglobin levels) (Jobsis, 1977). Since oxygenated and deoxygenated blood can be contrasted by their different optical absorption properties, fNIRS detects the levels of these blood parameters in response to neuronal activity. fNIRS is portable, inexpensive, and has shown to be an effective tool in quantifying cortical activation during static and dynamic motor movements, without causing substantial movement artifact issues (Perrey, 2008). While fNIRS measurements, particularly oxygenated hemoglobin levels, have shown to be strongly correlated to the fMRI BOLD signals, albeit with relatively lower signal to noise ratio (Strangman et al., 2002; Cui et al., 2011), unlike fMRI and PET its effectiveness in mapping neural activations across closely connected regions or within deep cortical areas is limited due to its relatively lower spatial resolution. Multimodal imaging approaches using both fNIRS and EEG systems have demonstrated that fNIRS is capable of enhancing eventrelated desynchronization-based EEG measurements significantly (Leamy et al., 2011; Fazli et al., 2012).

While fNIRS enables measurement of oxygenated and deoxygenated hemoglobin levels in cortical regions, transcranial Doppler sonography (TCDS) uses ultrasound to image cerebral blood flow to the brain hemispheres (Aaslid, 1986). TCDS uses an emitter attached to the head to direct ultrasound toward the middle cerebral artery (MCA) within the brain, and a receiver then records the frequency of the sound wave reflected by red blood cells moving through the artery. The magnitude of the change in frequency (the Doppler shift) varies directly proportional to the velocity of blood flow within the artery (Duschek and Schandry, 2003). In response to increased task-related neuronal activity, MCA blood flow velocity increases to remove by-products of the metabolic exchange, which is captured using TCDS (Aaslid, 1986). TCDS has become increasingly popular in cognitive neuroergonomic studies of vigilance and mental workload (Warm and Parasuraman, 2007). However, because cerebral blood volume and blood flow velocity is influenced by systemic changes such as heart rate and blood pressure during exercise (Ainslie et al., 2007), TCDS is less popular in assessing task-related neuronal activity in physical neuroergonomic studies of fatigue.

In contrast to the excellent temporal resolution offered by EEG techniques (on the order of milliseconds), magnetic resonance imaging (MRI) provides a structural image of the brain and offers excellent spatial visualization of deep internal parts, such as the hippocampus. While MRI provides static images of the brain that is critical in examining structural changes in the brain due to diseases (such as tumor), its application in studying structural changes in the brain over time (i.e., plasticity) has provided important information on learning and training (Huttenlocher, 2002). A relatively newer MRI technique, called diffusion tensor imaging (DTI), uses MRI to target the diffusion of water molecules in the axons that make up white matter in the brain and allows for the computation of *fractional anisotrophy* (FA). FA values can range from 0 to 1, where 0 indicates non-directional (isotropic) and 1 indicates perfectly directional (anisotropic) diffusion. Higher FA values are thought to reflect greater integrity of white matter linking different cortical and subcortical regions of the brain. Several recent studies have assessed the effectiveness of cognitive and motor training on white matter integrity using the DTI technique (Draganski et al., 2004; Takeuchi et al., 2010; Strenziok et al., 2014). In general, the MRI technique does not offer any mobility features, but an MRI static image can be overlaid with more dynamic fMRI images (i.e., blood oxygenation) so that areas of activation can be associated with particular brain regions.

The electromagnetic and hemodynamic neuroimaging techniques discussed thus far are based on sensing brain activity while a human operator is engaged in cognitive or physical work. As such, all such techniques are *correlational*, thus it may be difficult to establish *causal* links between brain activity and performance using these methods. Researchers have therefore turned to non-invasive stimulation techniques that modulate brain activity, such as transcranial magnetic stimulation (TMS) and transcranial direct current stimulation (tDCS), in order to establish such causal associations. These techniques allow for temporary inhibition or activation of specific brain regions thereby allowing researchers to examine the causal role of different brain regions in various cognitive functions (Walsh and Pascual-Leone, 2005). TMS and tDCS can also be used to modulate brain activity so that the performance of a given cognitive or motor task is improved (Coffman et al., 2014). Alternatively, these techniques can also be applied not to enhance performance over baseline, but to

"fnhum-07-00889" — 2013/12/19 — 20:28 — page 3 — #3

reduce or eradicate a normally occurring performance limitation, such as performance decrements that occur in vigilance tasks (Nelson et al., 2014).

Transcranial magnetic stimulation uses a magnetic coil that is positioned over the participant's scalp over a brain region of interest to send electrical current that changes the magnetic field perpendicular to the head. This induces current flow in the underlying cortical issue, sufficient to alter neural firing (Walsh and Pascual-Leone, 2005). The spatial resolution of TMS is relatively high, particularly when the participant's MRI scan is available to guide the TMS coil placement. The temporal resolution is also high, since the TMS pulses can be delivered with millisecond precision. However, due to the equipment setup TMS does not offer a sufficient degree of mobility needed for neuroergonomic assessment in naturalistic work settings. Whereas TMS uses changing magnetic pulses, tDCS uses small DC electric current (1 or 2 mA) with electrodes attached to the scalp. A positive polarity (anode) is typically used to stimulate neuronal function and enhance performance, and a negative polarity (cathode) is used to inhibit neuronal activity. Compared to TMS, tDCS has low spatial and temporal resolution, but has the advantage that it is portable and very inexpensive and thus is more likely to be adopted in applied neuroergonomic studies.

#### **PHYSICAL WORK**

Ergonomics began as the science of work to maximize productivity, particularly in physical work environments, but has since then expanded to become a scientific discipline concerned with the understanding of the interactions among humans and other elements of a system, in order to optimize human well-being and overall system performance. Physical ergonomics focuses on human physical capabilities and limitations, pertaining to anthropometry, physiology, and biomechanics of the human body, as they relate to physical work (Karwowski et al., 2003). Traditional ergonomic evaluations focus solely on peripheral outcomes, such as force or muscle activity, and disregard the contributions of the brain during work. Physical neuroergonomics is an emerging field of study that focuses on the knowledge of human brain activities in relation to the control and design of physical tasks (Karwowski et al.,2003), by taking into consideration an operator's physical, cognitive, and affective capabilities and limitations. Here we consider how neuroergonomic methods have been employed to evaluate different physical work parameters (such as force production and repetition) and physical fatigue (localized muscle fatigue and whole body fatigue).

#### **PHYSICAL WORK PARAMETERS**

The primary goal of ergonomics is to ensure that work demands are always lower than operator capacity, and the conventional assessment of work demands include measuring biomechanical and physiological outcomes, such as joint torque, muscle activity, and heart rate, in laboratory and field settings. There has been recent interest in assessing physical work using neuroergonomic methods in controlled laboratory conditions; however, there is a clear lack of neuroergonomic studies in assessing physical work in actual field/work settings. Like any new field, physical neuroergonomics research first needs to understand the capabilities, limitations, and considerations of existing neuroimaging techniques on simulated work environments that can help build the knowledge base necessary to perform research in naturalistic work environments.

Since physical work can involve both static and dynamic work at different intensities, repetitions, and durations, which in turn can affect autonomic responses, different work parameters can influence the type of measurement technique adopted. For example, dynamic or ambulatory tasks, such as walking or lifting, cannot be assessed using fMRI due to mobility constraints. More appropriate neuroimaging methods to evaluate ambulatory physical work are EEG, ERP, and fNIRS. Of these, EEG appears to be the most common neuroimaging technique since it provides excellent temporal resolution. Effective artifact removal techniques are available that allow for its use in evaluating dynamic tasks. For example, EEG-derived MRCP has provided valuable information on the role of cortical motor commands (represented by the MRCP) on the control of voluntary muscle activation. MRCP from the supplementary motor area and the contralateral sensorimotor cortex has shown to be highly correlated with force production and rate of force production during isometric elbow-flexion, and associated muscle activity (Siemionow et al., 2000). Of note, a recent fNIRS investigation has demonstrated obesity-related alterations in neural patterns of force control (i.e., lower prefrontal cortex activation associated with decreased joint stability) that can shed some light on the increased incidence of injury rates and higher work absenteeism in obese workers (Schulte et al., 2007; Mehta and Shortz, 2013). High repetition is one of the major work-related risk factors that contribute to the development of musculoskeletal disorders (Bernard, 1997). To evaluate the effects of repetition that involves flexion and extension of a joint, traditional ergonomic methods focus on muscular responses such as EMG. In a study investigating thumb flexion and extension movements, EEG-derived MRCP findings from the supplementary motor area and contralateral motor cortex demonstrated that extension and flexion result from separate corticospinal projections to the motor neurons (Yue et al., 2000). Thumb extensions resulted in lower EMG but elicited greater brain responses than flexion movements. This particular finding may be important to our understanding of the etiology of musculoskeletal disorders due to repetitive motion. Real work environments are seldom static, and can require operators to focus not only on the physical work demands but also on the necessary visual/auditory cues associated with the tasks. Such tasks, which are dynamic and require visuomotor control, have shown to increase corticomuscular coupling at higher EEG frequencies (i.e., gamma bands), indicating the adaptive role of cortical oscillations in rapidly integrating visual (or new) information with the somatosensory information (Marsden et al., 2000; Omlor et al., 2007). These findings have important implications for task analysis and design, particularly for work tasks that require visual feedback or fine or precise control of body motions.

#### **PHYSICAL FATIGUE**

Fatigue is defined as the inability to maintain required power after prolonged use of the muscle(s) (Latash et al., 2003), and can be affected by central (i.e., motivation, cortical activity,

"fnhum-07-00889" — 2013/12/19 — 20:28 — page 4 — #4

etc.) and peripheral (i.e., changes in muscle contractile properties) mechanisms. Neuroergonomic methods can help examine the role of central brain mechanisms in fatigue development. Based on the work tasks, fatigue in the workplace can be broadly categorized as localized muscle fatigue, which is the fatigue of specific muscle groups during tasks such as assembly line work or precision work, and whole body fatigue, which is more cardiorespiratory in nature that can occur during manual materials handling tasks. Commonly used ergonomic indicators of localized muscle fatigue include a reduction in force generating capacity (Vøllestad, 1997) and a decrease in EMG power spectrum (Mehta and Agnew, 2012). However these measures do not delineate the contributions of central fatigue from peripheral fatigue. Using EEG-derived MRCP, Johnston et al. (2001) demonstrated a significant increase in the activity of the BP component and the motor potential (MP) component of the MRCP, associated with a decline in force production and reduced EMG activity during a fatiguing grasping task. These increases in the early components of MRCP may reflect development of compensatory cortical strategies to accommodate for the inability to maintain the desired force levels due to peripheral fatigue. Supporting this, Liu et al. (2007) advocated that muscle fatigues well before the brain does; in essence that peripheral fatigue occurs before central fatigue. They demonstrated, by estimating the changes of source locations of high-density EEG signals using a single moving current dipole model, that handgrip muscle fatigue was associated with shifting of brain activation centers from one location to another when neurons in the previous location become fatigued. These studies collectively demonstrate the application of EEG in examining the neural correlates of localized fatigue development of smaller muscles during relatively static, or immobile, tasks.

Of the various neuroimaging techniques, EEG offers the greatest flexibility and mobility features that make it an attractive candidate in assessing whole body fatigue. By simultaneously obtaining information on eye movements and spontaneous EEG signals, Kubitz and Mott (1996) demonstrated increased brain activation (i.e., decreased alpha activity and increased beta activity) during a fatiguing cycling task. While technical advances have been made in minimizing mechanical artifacts from highdensity EEG signals during whole body movements (Gwin et al., 2010), fNIRS has gained rapid attention in evaluating whole body fatigue owing to its methodological advantages over EEG. First, fNIRS provides information on the location of the neural signal generated, whereas with EEG signals, source localization has to be computationally derived. Second, there are no time-sensitive requirements in examining whole body fatigue when compared to fast reaction time tasks; slower hemodynamic responses of fNIRS are thus appropriate when compared to fast EEG responses. As such, fNIRS responses have shown to be less affected by movement artifacts than EEG signals (Perrey, 2008). Several fNIRS studies have reported a significant decrease in relative levels of oxygenated hemoglobin in the prefrontal cortex, accompanied by muscular impairment, at exhaustion during submaximal and maximal fatiguing contractions (González-Alonso et al., 2004; Bhambhani et al., 2007; Nybo and Rasmussen, 2007). In particular, Thomas and Stephane (2008) demonstrated that oxygenated hemoglobin levels in the prefrontal cortex during incremental cycling exercise increased in the early stages, but decreased markedly in the last stage until exhaustion. These findings imply that prefrontal cortex activation is associated with reduction in motor output at the cessation of exercise. However, these fatiguing tasks are accompanied by cardiorespiratory changes in the autonomic system that can affect fNIRS responses (Obrig et al., 1996). Depending on the research questions asked, such systemic influences on cerebral hemodynamic responses may be desired or undesired. Obrig and Villringer (2003) emphasize the importance of analyzing deoxygenated hemoglobin levels as an indicator of "neuronal activation" over the more commonly used oxygenated hemoglobin values. They argue that oxygenated hemoglobin levels are acceptable neuronal activity indicators when cerebral autoregulation is intact, i.e., cerebral blood flow is in homeostasis. Increases in oxygenated hemoglobin during exercise can be attributed not only to neuronal activation but also to exercise-induced increased blood flow to the brain, and as such a decrease in deoxygenated hemoglobin is the most valid parameter. Thus, neuroergonomic investigations of fatigue need to consider these systemic influences, and perhaps collect peripheral measurements such as arterial blood pressure and heart rate to ensure that appropriate inferences are made from fNIRS signals.

#### **COGNITIVE WORK**

The field of human factors and ergonomics had its origins in timeand-motion studies conducted in the early 1900s. With the advent of World War II, increasing attention was paid to evaluation of human psychological processes during work performance, but the dominant approach was behaviorism, or stimulus-response psychology. The advent of the cognitive revolution in the late 1950s lead to the introduction of the cognitive approach in human performance assessment from the 1960s to the present day, but there was still a relative neglect of brain mechanisms. Advances in neuroimaging and related methods that lead to the development of the field of cognitive neuroscience lead to the argument that neural measures should also be considered in human factors and ergonomics (Parasuraman, 2003). Since that time, the neuroergonomic approach has been applied to a number of different issues in cognitive ergonomics.

These historical trends in theoretical frameworks used in ergonomics can be seen clearly in the periodical reviews of the field of engineering psychology in the *Annual Reviews of Psychology*. Fitts (1958) reviewed work conducted mainly within time-andmotion and stimulus-response frameworks; Wickens and Kramer (1985) presented a cognitive or information-processing approach; and the most recent review, by Proctor and Vu (2010), describes the neuroergonomic approach. In this paper, we review a few key issues in cognitive neuroergonomics and on those areas where the most research and development work has been done.

#### **MENTAL WORKLOAD**

The assessment of human mental workload is one of the most widely studied topics in ergonomics (Wickens and McCarley,

"fnhum-07-00889" — 2013/12/19 — 20:28 — page 5 — #5

2008). If operator mental workload is either too high or too low human-system performance may suffer in work environments, thereby potentially compromising safety. Hence, workload must be assessed in the design of new systems or the evaluation of existing ones. Behavioral measures, such as accuracy and speed of response on secondary tasks, or subjective reports (such as the NASA-TLX) have been widely used to assess mental workload. However, measures of brain function offer some unique advantages that can be exploited in mental workload assessment (Kramer and Parasuraman, 2007). Among these is the ability to extract covert physiological measures continuously in complex system operations in which overt behavioral measures may be relatively sparse.

The dominant theory of human mental workload is *resource* theory (Wickens, 1984, 2002). This theory postulates that except for highly overlearned "automatic" tasks, task performance is directly proportional to the application of attentional resources. The theory also proposes that the degree of overlap of multiple pools of resources determines the pattern and amount of interference when two or more tasks are performed simultaneously (such as driving and talking on the cell phone). Dual-task studies have provided abundant support for resource theory (Wickens and McCarley, 2008), but one criticism is that the theory is circular (Navon, 1984), which can be linked to the lack of an independent measure of resources. This criticism can be countered if neural measures of mental resources can be identified.

Measures of cerebral hemodynamics, such as fNIRS and TCDS, have provided validation for the resource construct. In a recent study, Ayaz et al. (2012) tested experienced air traffic controllers (ATC) on a complex ATC task requiring them to keep aircraft in their sector free of conflicts. fNIRS was used to measure activation of the prefrontal cortex. Ayaz et al. (2012) found there was an increase in prefrontal cortex activation as the number of aircraft in their sector increased. These findings suggest that fNIRS can provide a sensitive index of cognitive workload in a skilled group performing a realistic task that was highly representative of their work environment. fNIRS has also been found to index changes in prefrontal cortex activation with skill acquisition in both basic working memory tasks (McKendrick et al., 2014) and more complex piloting tasks (Ayaz et al., 2012). Most recently, portable versions of fNIRS have been developed for use in mobile neuroimaging (Ayaz et al., 2013).

There are many factors, such as cost, ease of implementation, intrusiveness, etc., that must be taken into consideration when selecting neuroergonomic techniques for mental workload assessment. Some of these factors (e.g., cost) may rule out the use of neuroergonomic methods in favor of simpler indexes such as subjective measures. Some workers may also not wish to be "wired up" for physiological recording, so operator acceptance must also be carefully considered. However, with increasing miniaturization and development of dry electrode, wireless, wearable systems, some of these concerns are diminishing.

#### **VIGILANCE AND MENTAL FATIGUE**

The evaluation of operator vigilance and mental fatigue in work environments is a topic closely related to workload assessment. The widespread implementation of automation in many work environments, including air and surface transportation and health care, while often leading to a reduction in operator workload, can also increase workload because of the resulting need for monitoring the automation (Parasuraman, 1987). The typical finding in vigilance studies is that the detection rate of critical targets declines with time on task (Davies and Parasuraman, 1982). Vigilance decrement was originally attributed to a reduction in physiological arousal (Frankmann and Adams, 1962) but more recent neuroergonomic research using TCDS and fNIRS have attributed it to resource depletion (Warm et al., 2008). Warm et al. (2008) reported a series of studies of TCDS and vigilance (for reviews, see Warm and Parasuraman, 2007; Warm et al., 2008). A consistent finding is that the vigilance decrement is paralleled by a decline in blood flow velocity over time, relative to a baseline of activity just prior to beginning the vigilance session. The parallel decline in vigilance performance and in blood flow velocity is found for both visual and auditory tasks (Shaw et al., 2009). These findings have been interpreted using resource theory. A critical control finding in support of resource theory – as opposed to a generalized arousal or fatigue model – is that the blood flow change occurs only when observers actively engage with the vigilance task. When observers are asked to simply watch a display passively without having to detect targets for the same amount of time as in an active vigilance condition – a case of maximal underarousal – blood flow velocity does not decline but remains stable over time.

The deleterious effects of loss of operator vigilance can countered with reduced work hours and more frequent rest breaks, but this may not be practical in all work settings. Another mitigating strategy is to use cueing. Detection performance in vigilance tasks can be improved by providing observers with consistent and reliable cues to the imminent arrival of critical signals, with the extent of the decrement being reduced or eliminated (Wiener and Attwood, 1968). With cueing, observers need to monitor a display only after having been prompted about the arrival of a signal and therefore can husband their information processing resources over time. In contrast, when no cues are provided, observers are never certain of when a critical signal might appear and consequently have to process information on their displays continuously across the watch, thereby consuming more of their resources over time than cued observers. If the vigilance decrement stems from resource depletion due to need to attend continuously to a display, then pre-cues should reduce the decline in cerebral blood flow velocity as measured by TCDS. This was confirmed in a study by Hitchcock et al. (2003). They used no pre-cues or pre-cues that were 100, 80, or 40% reliable in pointing to an upcoming critical event in a simulated air traffic control task. Performance efficiency remained stable when perfectly reliable cues were provides but declined over time in the remaining conditions, so that by the end of the vigil, performance efficiency was clearly best in the 100% group, followed in order by the 80, 40%, and no-cue groups. Blood flow declined in the no cue control condition, but there was a progressive reduction in the extent of the decline with progressively more reliable cues. There was no decline when the cues were perfectly reliable. This pattern of change in blood flow exactly matched that of performance.

"fnhum-07-00889" — 2013/12/19 — 20:28 — page 6 — #6

In addition to cueing, non-invasive brain stimulation could also be used to mitigate vigilance decrement and mental fatigue. Nelson et al. (2014) applied 1 mA anodal tDCS to either the left or right prefrontal cortex while participants performed the same vigilance task used by Hitchcock et al. (2003). tDCS was applied either early or late during the course of the vigilance task. Compared to a control group that showed the normal vigilance decrement, the early stimulation group had a higher detection rate of critical signals. The late stimulation group initially exhibited a vigilance decrement, but this was reversed following application of tDCS. These initial findings are highly encouraging, but need to be followed up with additional research to examine the long-term effectiveness of tDCS as a method to alleviate vigilance problems at work.

#### **TRAINING AND NEUROADAPTIVE SYSTEMS**

While the goal of ergonomic design is to avoid having workers exposed to extremes of workload and to loss of vigilance, this may not always be possible in certain work settings where unexpected events, equipment failures, or other unanticipated factors lead to a transient increase in the task load imposed on the human operator, or long work hours impose demands on operator vigilance. Adaptive automation offers one approach to deal with these issues (Parasuraman, 1987, 2000). In this approach, the allocation of functions to human and machine agents is flexible during system operations, with greater use of automation during high task load conditions or emergencies and less during normal operations, consistent with the approach of dynamic function allocation (Lintern, 2012). The adaptive automation concept has a long history (Parasuraman et al., 1992), but neuroergonomic methods for its implementation have been considered relatively recently (Inagaki, 2003; Parasuraman, 2003; Scerbo, 2007).

Several methods to implement adaptive systems have been examined, including neuroergonomic measures to assess the operator's functional state (Byrne and Parasuraman, 1996; Kramer and Parasuraman, 2007; Wilson and Russell, 2007; Parasuraman and Wilson, 2008; Ting et al., 2010). Many studies have used EEG because of its ease of recording and (relative) unobtrusiveness (compared, say, to secondary tasks or subjective questionnaires). EEG also has the property of being a very high bandwidth measure, offering the possibility of sampling the human operator at up to about 30 Hz (Wilson and Russell, 2003). Workload adaptive systems need to assess operator state in real time, or near real time, so that task allocation or restructuring can be implemented in cases of overload or underload. A number of different statistical and machine learning techniques have been used for this purpose. These include discriminant analysis (Berka et al., 2004), artificial neural networks (Wilson and Russell, 2007; Baldwin and Penaranda, 2012), Bayesian networks (Wang et al., 2011), and fuzzy logic (Ting et al., 2010). These have been implemented in real time and typically provide accuracies of 70–85%.

Implementing neuroergonomic adaptive systems in real settings poses significant challenges. A major issue concerns the detection and removal of artifacts in real time. Furthermore, while initial success has been achieved in using computational techniques to classify workload on the basis of EEG and other neuroergonomic measures, the reliability and stability of these methods within and across individuals needs to be more rigorously tested (Wang et al., 2011; Christensen et al., 2012). Finally, the operational community must be involved in the design of adaptive systems to ensure user acceptance and compliance.

#### **NEUROERGONOMIC ASSESSMENT OF CONCURRENT PHYSICAL AND COGNITIVE WORK**

Both physical and cognitive neuroergonomics have helped advance our understanding on the role of the human brain during physical and cognitive work, respectively. Only a small number of studies have investigated the interaction between physical and cognitive work, which is a big concern since"work"places combined physical and cognitive demands on operators, never either one in isolation. High cognitive demands can influence physical work; and physical activity can in turn influence cognitive processing. In comparison to traditional evaluation techniques in either physical or cognitive ergonomics domain, neuroergonomic methods offer a great advantage in assessing these combined demands. For example, using EEG signals Kamijo et al. (2004) investigated the influence of exercise intensity on cognitive function using the P300 ERP component. They suggested that exercise influenced the amount of attentional resources devoted to a given task and that the changes in P300 amplitude followed an inverted *U*-shaped behavior of differences in exercise intensity. When examining the impact of cognitive demand on physical capacity, a few studies have attributed decreased muscle endurance in presence of a cognitively stressful situation to lower motivation (Marcora et al., 2009), increased neuromotor noise impairing joint steadiness (van Loon et al., 2001; Mehta and Agnew, 2011, 2013; Mehta et al., 2012), or neuronal interference at the prefrontal cortex that is involved in cognitive processing and isometric motor contractions (Dettmers et al., 1996; Rowe et al., 2000; Mehta and Parasuraman, 2013). In particular, using fNIRS to monitor cerebral oxygenation during handgrip exercises, Mehta and Parasuraman (2013) demonstrated that concurrent handgrip exercises in cognitive stressful conditions were associated with lower oxygenated hemoglobin levels in the bilateral prefrontal cortex at exhaustion when compared to the handgrip exercises at the same intensity levels (i.e., no changes observed in peripheral muscular responses of EMG and force exerted) under no stress. Quite similarly, using EEG- and EMG-derived corticomuscular coupling measure, Kristeva-Feige et al. (2002) reported that corticomuscular coupling decreased significantly during a cognitively stressful condition despite no changes observed in traditional measures such as EMG and force production. These studies collectively emphasize the importance of obtaining brain (or central) responses along with the more conventional ergonomic measurements to accurately understand the "total" demands placed on humans during work that requires both physical and cognitive processing. Future investigations on comparing these brain-body responses with the more traditional performance or subjective measures are also needed to understand the underlying neural "cost" of operator functional state. Such studies are also needed so as to develop evaluation tools (surveys, heuristic checklists) that are predictive of the neural and physiological cost associated with optimizing work tasks, which can be

"fnhum-07-00889" — 2013/12/19 — 20:28 — page 7 — #7

used by designers or supervisors to quantify operator workload and fatigue.

#### **MOBILE BRAIN IMAGING CONSIDERATIONS FOR WORKLOAD/FATIGUE ASSESSMENTS**

One of the key distinctions between neuroergonomics and neuroscience is that neuroergonomics is the study of brain and behavior "at work." Thus, it is extremely important that neuroergonomic methods are capable of examining human operators at their naturalistic work settings. In this paper, we discussed the merits and disadvantages of the available neuroimaging techniques applicable to neuroergonomics and a key theme identified was the lack of studies evaluating neural bases of mobile work, particularly in the physical neuroergonomics domain. Recent efforts in developing mobile brain imaging (MoBI) techniques, which consider the physical and environmental impact on human cognitive processing, show great promise. For example, Gramann et al. (2011) reviewed the implications and feasibility of a newly developed MoBI system that was previously employed in examining cognitive processing during human stance and locomotion. In particular, their MoBI investigation included simultaneous brain-body measurements from a 256-channel EEG system and kinematic and kinetic outcomes that are otherwise employed during conventional gait biomechanics using motion capture systems and force plates (Gwin et al., 2011). In their review, Gramann and colleagues identify key requirements for MoBI methods that include: (1) robust mobile sensor technology to measure brain activity, (2) comprehensive "wireless" body measurement system, and (3) powerful computational software to collectively processing and analyze both brain-body responses. While developing an ideal MoBI system may be a challenging goal, understanding current limitations in mobile brain-body imaging and addressing them, albeit painstakingly, is a critical step toward achieving this goal. Future investigations can also include developing similar mobile brain-body imaging systems for hemodynamic neuroimaging techniques, utilizing either fNIRS or TCDS to provide brain imaging measures, and using peripheral measurements such as heart rate and blood pressure to document physiological whole-body responses.

#### **CONCLUSION**

Ergonomics has long since moved from being a science of improving work efficiency to now being focused on enhancing well-being while improving systems performance. To effectively understand how humans interact with work systems, it is not only important to ask how well they perform, but also *why* they perform a certain way. Neuroergonomics have helped fill in the gaps on the neural bases of both physical and cognitive performance that were left unanswered with traditional ergonomic assessments. In this review we discussed the recent developments and adoption of neuroergonomic methods and applications in investigating physical, cognitive, and combined physical and cognitive work. We also reviewed the applicability and feasibility of neuroimaging techniques in evaluating mobile work environments. While some neuroimaging methods are expensive and are immobile, such as the MRI, fMRI, PET, and DTI, portable methods such as EEG, fNIRS, and TCDS, are more likely to be adopted in applied ergonomics research. With the advent of, and recent developments in, MoBI technology, we can be assured that neuroergonomics can continue providing critical information on how/why human interact in ambulatory and naturalistic work settings.

#### **AUTHOR CONTRIBUTIONS**

Both authors contributed equally to this work. Ranjana K. Mehta performed the literature review on neuroergonomics applications to physical work and Raja Parasuraman performed the literature review on cognitive neuroergonomics. Both authors discussed the reviewed implications and commented on the manuscript at all stages.

#### **ACKNOWLEDGMENTS**

Supported in part by Air Force Office of Scientific Research grant FA9550-10-1-0385 to Raja Parasuraman.

#### **REFERENCES**

"fnhum-07-00889" — 2013/12/19 — 20:28 — page 8 — #8

during a maintained motor contraction task. *Clin. Neurophysiol.* 113, 124–131. doi: 10.1016/S1388-2457(01)00722-2

"fnhum-07-00889" — 2013/12/19 — 20:28 — page 9 — #9

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 23 August 2013; accepted: 05 December 2013; published online: 23 December 2013.*

*Citation: Mehta RK and Parasuraman R (2013) Neuroergonomics: a review of applications to physical and cognitive work. Front. Hum. Neurosci. 7:889. doi: 10.3389/fnhum.2013.00889*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2013 Mehta and Parasuraman. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

"fnhum-07-00889" — 2013/12/19 — 20:28 — page 10 — #10

# Continuous monitoring of brain dynamics with functional near infrared spectroscopy as a tool for neuroergonomic research: empirical examples and a technological development

#### *Hasan Ayaz 1 \*, Banu Onaral 1, Kurtulus Izzetoglu1, Patricia A. Shewokis 1,2, Ryan McKendrick3 and Raja Parasuraman3*

*1 School of Biomedical Engineering, Science and Health Systems, Drexel University, Philadelphia, PA, USA*

*2 Nutrition Sciences Department, College of Nursing and Health Professions, Drexel University, Philadelphia, PA, USA*

*3 Center of Excellence in Neuroergonomics, Technology, and Cognition, George Mason University, Fairfax, VA, USA*

#### *Edited by:*

*Klaus Gramann, Berlin Institute of Technology, Germany*

#### *Reviewed by:*

*Craig Speelman, Edith Cowan University, Australia Dietrich Manzey, Technische Universitaet Berlin, Germany*

#### *\*Correspondence:*

*Hasan Ayaz, School of Biomedical Engineering, Science and Health Systems, Drexel University, 3508 Market Street, Monell Suite 101, Philadelphia, PA 19104, USA e-mail<EMAIL_ADDRESS>
Functional near infrared spectroscopy (fNIRS) is a non-invasive, safe, and portable optical neuroimaging method that can be used to assess brain dynamics during skill acquisition and performance of complex work and everyday tasks. In this paper we describe neuroergonomic studies that illustrate the use of fNIRS in the examination of training-related brain dynamics and human performance assessment. We describe results of studies investigating cognitive workload in air traffic controllers, acquisition of dual verbal-spatial working memory skill, and development of expertise in piloting unmanned vehicles. These studies used conventional fNIRS devices in which the participants were tethered to the device while seated at a workstation. Consistent with the aims of mobile brain imaging (MoBI), we also describe a compact and battery-operated wireless fNIRS system that performs with similar accuracy as other established fNIRS devices. Our results indicate that both wired and wireless fNIRS systems allow for the examination of brain function in naturalistic settings, and thus are suitable for reliable human performance monitoring and training assessment.

**Keywords: fNIRS, optical brain monitoring, working memory training, prefrontal cortex, hemodynamic response, wireless NIRS**

#### **INTRODUCTION**

Understanding the neural mechanisms that contribute to cognitive functions such as performing complex cognitive tasks, acquisition, development, and use of cognitive skills, is an important goal for cognitive neuroscience research (e.g., Poldrack et al., 1998) and for applications of neuroscience to work and everyday activities, or neuroergonomics (Parasuraman, 2011). Various magnetic resonance imaging (MRI) methods have provided essential information about the brain systems involved in skill acquisition. These include functional MRI (Karni et al., 1995), resting state functional connectivity (Lewis et al., 2009), and diffusion tensor imaging (Lövdén et al., 2010). Such studies are critical for the development of theories of neuroplasticity, because human brain changes associated with learning can be compared to studies in animals using invasive neurophysiological and pharmacological methods (Sarter and Parikh, 2005). However, MRI has two major limitations; (1) its requirement for participant immobility, and (2) its high operational cost. The former rules out its use for understanding brain dynamics during everyday activities such as walking or running, and while many functional MRI studies have been carried out using virtual reality simulations of such naturalistic activities as spatial navigation (Hartley et al., 2003), flying (Callan et al., 2012), and driving (Calhoun et al., 2002), the concern is that carrying out these activities while prone and immobile may not recruit the same brain networks as those involved when one is mobile and upright (Raz et al., 2005). One consequence of the second limitation of MRI, its high operational cost, is that skill acquisition studies typically image participants pre- and post-training, so that only linear changes in brain structure and function can be assessed. However, skill acquisition is known to be non-linear, e.g., described by power or hyperbolic functions (e.g., Newell and Rosenbloom, 1980; Speelman and Kirsner, 2005). Also, neural changes are likely to occur throughout training, so it is important to examine how such changes are linked to performance at multiple time points during learning, not just before and after training.

One method that is well-suited for such continuous monitoring of brain dynamics is functional near infrared spectroscopy (fNIRS). fNIRS is a non-invasive, safe, and portable optical method to monitor the brain activity within the prefrontal cortex of the human brain. fNIRS has emerged during the last decade as a promising non-invasive neuroimaging tool and has been used to monitor various types of brain activities during motor and cognitive tasks with increasing interest from research communities. fNIRS uses specific wavelengths of light to provide measures of cerebral oxygenated and deoxygenated hemoglobin that are correlated with the fMRI BOLD signal (Cui et al., 2011). While fMRI can monitor the whole brain with high spatial resolution at the sub-millimeter level, fNIRS can only monitor cortical regions with less spatial resolution (usually in the centimeter range). However, unlike fMRI, fNIRS is quiet (no operating sound), provides higher temporal resolution (faster sampling frequency), and participants are not restricted to a confined space or are not required to stay in a supine position motionless. Hence, fNIRS is an ideal candidate for monitoring cortical activity related changes not only in laboratory settings but also in more ecologically valid, everyday working and field conditions.

In this paper we provide results from a number of studies that illustrate the use of this approach to the examination of workload and training-related brain dynamics with human performance assessment. This paper has two major aims. The first is to show that fNIRS provides a sensitive and reliable index of brain activity in skill acquisition, task performance and during the development of expertise in complex tasks. Because occupations involving such tasks often require their human operators to be free to speak, move their eyes or heads, and otherwise be mobile, a second aim of this paper is to describe a mobile fNIRS system that can supplement "tethered" fNIRS in studies of human performance. We first describe the use of fNIRS in monitoring cognitive workload in air traffic controllers. We then describe a study of the acquisition of a dual-tasking skill, in which participants had to perform challenging verbal and spatial working memory tasks simultaneously, as well as a study examining development of expertise in piloting unmanned vehicles. Each of these studies used conventional fNIRS devices in which the participants were tethered to the device while seated at a workstation. Current fNIRS instruments require the participant to be connected to the sensor and the device via cables and/or fiber optic lines which imposes restrictions on the ambulatory nature of the experiment protocol and to the participant. Consistent with the aims of mobile brain imaging (MoBI) (Makeig et al., 2009; Gramann et al., 2011) and neuroergonomics (Parasuraman, 2011), it would be desirable to measure brain dynamics while participants can move freely. To address these limitations we introduce a compact and battery-operated wireless fNIRS system that performs with similar accuracy as other established fNIRS devices.

#### **MONITORING COGNITIVE WORKLOAD IN AIR TRAFFIC CONTROLLERS**

The assessment of cognitive workload using neural measures is a central feature of research and development in neuroergonomics (Parasuraman, 2011). Mental workload is also a critical factor in maintaining safety in air traffic control (ATC), particularly as traffic density increases and new systems and operational procedures are implemented for air traffic management (Loft et al., 2007). Accordingly, there is a need for sensitive, objective methods of measuring cognitive workload in air traffic controllers.

The feasibility of using fNIRS for human performance assessment has been demonstrated in several recent studies. In a prior research project, we incorporated fNIRS to studies conducted at the Federal Aviation Administration (FAA)'s William J. Hughes Technical Center Human Factors Laboratory where certified controllers were monitored with fNIRS while they managed realistic ATC scenarios under typical and emergent conditions (Ayaz et al., 2012b). The primary objective of this work was to use neurophysiological measures to assess cognitive workload and usability of new interfaces developed for ATC systems (see **Figure 1**). Throughout the study, certified professional controllers (CPCs) completed ATC tasks with different interface settings and controlled difficulty levels for verification. The results provide evidence that brain activation as measured by fNIRS provides a valid measure of mental workload in this realistic ATC task (Ayaz et al., 2012b).

For the first part of the study, we used a working memory task (N-back) that has been widely used in the cognitive neuroscience research literature (Owen et al., 2005). The N-back paradigm provides varying task-load conditions to test associations between level of difficulty and cortical activation, and has been shown to activate the dorsolateral (DLPFC) and ventrolateral prefrontal cortex (VLPFC) as assessed with Positron Emission tomography (PET) (Smith et al., 1996) and functional MRI (D'Esposito et al., 1998; Owen et al., 2005). During the task, participants were asked to monitor stimuli (single letters) presented on a screen serially and click a response button when a target stimulus arrives. Four conditions were used to incrementally vary working memory load from zero to three items. In the 0-back condition, subjects respond to a single pre-specified target letter (e.g., "X") with their dominant hand (pressing a button to identify the stimulus). In the 1-back condition, the target is defined as any letter identical to the one immediately preceding it (i.e., one trial back). In the 2- and 3-back conditions, the targets were defined as any letter that was identical to the one presented two or three trials back, respectively. The total test included seven sessions of each of the four n-back conditions (hence, a total of 28 N-back blocks each of 1 min duration, which had 20 letters presented each for 500 ms with a 2500 ms inter-stimulus time) presented in a pseudo-random order.

Results showed that average oxygenation changes due to task engagement (mean for each block with baseline compared to beginning of the block) at optode 2 that is close to AF7 in the International 10–20 System, located within left inferior frontal gyrus in the dorsolateral prefrontal cortex (DLPFC), were associated with task difficulty and increased monotonically with increasing task difficulty (see **Figure 2**, left). Moreover, the significant region within left PFC in this study was implicated in many previous studies of the N-back task using PET (Smith et al., 1996; Reuter-Lorenz et al., 2000), fMRI (Cohen et al., 1997; Owen et al., 2005), and fNIRS (Schreppel et al., 2008).

For the second part of the study, complex cognitive tasks (i.e., ATC) were used. A critical transition defined in the planned future ATC system called NextGen involves augmenting the current auditory-based communications between ATC and the flight deck with text-based messaging, or DataComm systems (Willems et al., 2006, 2010). DataComm systems are expected to allow controllers to manage more air traffic at a lower level of cognitive load, thereby increasing both the capacity of the national airspace system and passenger safety.

Based on the approach and findings of the N-back working memory study, the objective in the second part of the study was to use neurophysiological measures to predict changes in cognitive workload during a complex cognitive task that very closely simulated the activities of air traffic controllers. Two types of communications between the CPCs and pilots, either typical (VoiceComm) or emergent (DataComm) communications were used in ATC simulations in a pseudo-random order (Willems et al., 2006, 2010). For each communication type, task difficulty was varied by the number of aircraft in each sector, containing 6, 12, or 18 aircraft to increase task load. Three simulation pilots supported each sector within voice-based scenarios and entered data at their workstations to maneuver aircraft, all based on controller clearances.

Analysis of data from 24 participants indicated a significant measurement location of optode 8 which is within the medial PFC/frontopolar cortex, and there were two significant main effects, Task Difficulty (number of aircrafts) and Communication (VoiceComm vs. DataComm) (see **Figure 2**, right). The fNIRS results from the main effect of Communication type confirms that VoiceComm condition results in higher oxygenation compared to the DataComm condition with a small to moderate effect size. These results are consistent with the idea that, given the same cognitive workload (within identical scenarios), DataComm required fewer cognitive resources.

#### **BRAIN DYNAMICS DURING EXTENDED WORKING MEMORY TRAINING**

The previously described study by (Ayaz et al., 2012b) showed that frontal cerebral oxygenation as measured by fNIRS increases with working memory load. Specifically, average oxygenation systematically increased in the N-back task as N was varied from 0 to 3 (see **Figure 1**, left). Working memory capacity is predictive of performance on visual attention (Engle, 2002), decision-making (Endsley, 1995), and supervisory control tasks (McKendrick et al., 2013b). Hence it is of interest that recent studies have shown that an individual's working memory capacity is not fixed but can be increased by training (Jaeggi et al., 2008), although whether such training transfers to other general domains of cognition is controversial (Shipstead et al., 2012). MRI studies have also shown that such working memory training is associated with both structural (Takeuchi et al., 2010) and functional (Dahlin et al., 2008) brain changes. However, as mentioned previously, most studies have used MRI in preand post-training designs, so that fine-grained and non-linear changes in brain dynamics have not been studied. Accordingly, we describe a study that used fNIRS to monitor skill acquisition in a dual verbal and spatial working memory task (McKendrick et al., 2013a).

As people perform a task repeatedly, they are likely to experience changes in the degree of mental effort expended, either voluntarily or as required by the task. To distinguish between brain changes as a result of working memory training from increases in mental effort, we compared two training conditions in this study: An adaptive training condition in which working memory load was adjusted based on the trainee's performance, and a yoked condition whose working memory load was adjusted based on the performance of individuals in the adaptive condition. Since task demands are not matched to the capabilities of participants in the yoked group, we predicted that they would expend more mental effort in order to perform the task and show an increase in prefrontal cortex (PFC) total hemoglobin (HbT) as measured with fNIRS. At the same time, because task demands are matched to the capabilities of adaptive-trainees, we therefore expected this group to show little change in hemodynamic response in PFC.

In addition, to improve the efficacy of the working memory training design, we implemented the suggestions for optimal training proposed by Gibson et al. (2012). First, we used a challenging dual verbal-spatial working memory task (see **Figure 3**), in which participants first memorized a string of digits, then a number of spatial locations, and following a delay period, recalled the locations and then the digits. The use of the dual task allowed for taxing the updating and executive control components associated with working memory (Baddeley, 1986). Second, to avoid ceiling effects and challenge participants, the load for verbal and spatial working memory was set to a range beyond what is considered the average capacity limit (spatial: 4 locations, verbal: 7 digits). Participants trained on the working memory task for about 2 h each day for 5 successive days. Daily training was separated into two 1 h sessions with a 15 min break between training sessions. Within a given training sessions participants performed 10 training blocks and each training block consisted of nine trials of the dual working memory task. Finally, we used linear mixed effects modeling of the data to examine both linear and nonlinear changes in performance and brain dynamics with working memory training.

**Figure 4** shows the data, plotted for each individual, for the verbal working memory task. As is clear from **Figure 4**, verbal working memory span increased with training, but in a nonlinear manner. A cubic function provided the best fitting model for training-related changes in performance. As expected, the adaptive group reached higher span levels than the yoked group at the end of training. Differences between training groups were modeled by a significant negative quadratic component for the yoked training condition, representing a slowing of skill development on the third and fourth days of training relative to the adaptive condition (see **Figure 4**).

As predicted, we observed an increase in hemodynamic response for the yoked control condition. This was specifically observed in the right rostral prefrontal cortex during the first 3 days of training. In the same region, in the adaptive condition there was a decrease in hemodynamic response over the same time period. The rostral prefrontal cortex is believed to be involved in the monitoring and processing of sensory stimuli during multitasking (Burgess et al., 2005). The changes in rostral prefrontal cortex suggest that in order to keep pace with the performance of the adaptive group the yoked group had to apply considerably more effort in maintaining and processing dual task representations. This is expected as in the yoked group task demands are not matched with participant capabilities and thus require higher mental effort. Furthermore, toward the end of training the adaptive group had to increase the effort applied to processing dual task representations to improve their performance. The differential quadratic workload effect between adaptive and yoked conditions can be seen in **Figure 5**. During this time the yoked group may have become fatigued due to the high level of effort required on the first 3 days of training. Non-linear increases in left DLPFC and right VLPFC were also observed with increased exposure to working memory training.

These findings point to the sensitivity of fNIRS to track both linear and non-linear changes in cerebral hemodynamics as a result of working memory training. Importantly, non-linear changes over time would not have been observed if a pre-/posttraining design commonly used in MRI studies had been used.

The findings also show that fNIRS provides an efficient and effective way to continuously monitor hemodynamic changes over extended periods of time, as required in training studies. In addition, as described in this paper, portable NIRS systems are being developed that could be used to measure the effects of training in complex real world tasks where the use of fMRI would be challenging or impossible.

#### **MONITORING THE DEVELOPMENT OF EXPERTISE IN PILOTING UNMANNED VEHICLES**

The study by McKendrick et al. (2013a) described above used a basic but challenging cognitive task—dual verbal/spatial working memory. Expertise development in more complex tasks that simulate work and other everyday real settings has also been examined. A majority of the studies examining task practice have found decreases in the extent or intensity of neural activations with ongoing practice, particularly in the attentional control areas (Kelly and Garavan, 2005). This finding is true whether the task is primarily motor [e.g., golf swing (Milton et al., 2004)] or primarily cognitive in nature, as in the Tower of London task (Beauchamp et al., 2003) including more complex tasks such as videogame training (Prakash et al., 2012) or the center-out adaptation task (Gentili et al., 2013). Decreases in activation are thought to represent a contraction of the neural representation of the stimulus (Poldrack, 2000) or a more precise functional circuit (Garavan et al., 2000).

In a recent study, we utilized fNIRS to investigate the relationship of the hemodynamic response in the anterior prefrontal cortex to changes in the level of expertise, and task performance during learning of simulated unmanned aerial vehicle (UAV) piloting tasks (Ayaz et al., 2012a). Novice participants with no prior UAV piloting experience participated in a 9 day training program where they used a flight simulator to execute real world maneuvers. Each day, self-reported measures (with NASA TLX), behavioral measures (task performance), and fNIRS measures (prefrontal cortex activity as mental effort on task) were recorded.

Participants practiced approach and landing scenarios while piloting a virtual UAV. The scenarios were designed to expose novice subjects to realistic and critical tasks for a UAV ground operator directly piloting an aircraft. The first scenario was a turnto-approach task, in which the pilot flies through several waypoints on an approach to land at an airfield. The second scenario was a landing task, in which the pilot performs the actual touchdown. In both scenarios, subjects were told to fly as smoothly as possible, learn the optimal paths, cope with crosswinds, and operate within certain speed and bank angle constraints. The experiment protocol involved a total of nine sessions per subject, one session per day. The first session on day 1 was to allow subjects to become acquainted with the flight simulator; by the end of this session, they needed to demonstrate basic understanding of flight simulator controls. Study data were collected during the following eight practice sessions.

Analysis of data from thirteen participants showed a reduction in the fNIRS measures (see **Figure 6**), which were significantly different across practice levels and matches the same trends reported in behavioral performance and self-reported measures. A valid hypothesis can be derived from the evidence that expertise tends to be associated with overall lower brain activity relative to novices, particularly in prefrontal areas (Milton et al., 2004). Both practice and the development of expertise typically involve decreased activation across attentional and control areas, freeing these neural resources to attend to other incoming stimuli or task demands. As such, measuring activation in these attentional and control areas relative to task performance can provide an index of level of expertise and illustrate how task-specific practice influences the learning of tasks.

Results indicate that level of expertise does appear to influence the hemodynamic response in the dorsolateral/ventrolateral prefrontal cortices confirming previous studies with learning cognitive-motor tasks (Hatakenaka et al., 2007; Leff et al., 2008). Since fNIRS allows development of portable and wearable instruments, it has the potential to be used in future learning environments to personalize the training regimen and/or assess the effort of human operators in critical multitasking settings (Ayaz et al., 2012a,b).

#### **DEVELOPMENT OF A PORTABLE, WIRELESS fNIRS SYSTEM**

The portable optical brain imaging system used in our studies reported here was first described by Chance et al. (Chance, 1998; Chance et al., 1993) further developed at Drexel University (Philadelphia, PA), manufactured and supplied by fNIR Devices LLC (Potomac, MD; www.fnirdevices.com).

The system is composed of three modules: a flexible headpiece (sensor pad), which holds light sources and detectors to enable a fast placement of all 16 optodes; a control box for hardware management; and a computer that runs the data acquisition (see **Figure 7**).

The sensor has a temporal resolution of 500 milliseconds per scan with 2.5 cm source-detector separation allowing for approximately 1.25 cm penetration depth and 16 measurement locations (optodes) on a rectangular grid covering the forehead region (see **Figure 7**) designed to monitor dorsal and inferior frontal cortical areas. The light emitting diodes (LEDs) were activated one light source at a time and the four surrounding photodetectors around the active source were sampled. For data acquisition and visualization, COBI Studio software was used (Ayaz et al., 2011).

#### **EVOLUTION OF DREXEL WIRELESS fNIRS**

The need for development and improvement of fNIRS instrumentation has been growing as fNIRS has been increasingly used in human brain activation studies since it was first described by Jobsis (1977) as an optical method for non-invasively assessing cerebral oxygenation changes. In the 1980s, Delpy et al. designed and tested an fNIRS system for a clinical application with newborn infants (Wyatt et al., 1986). Further efforts improved the methodology and hardware (Delpy et al., 1988; Wray et al., 1988; Cope, 1991; Chance et al., 1993; Elwell et al., 1994) and thus expedited the translation of fNIRS based techniques into a useful

**FIGURE 6 | Changes throughout the practice levels: Self-reported ratings: perceived mental effort as measured by NASA TLX (left), Behavioral performance: average error in banking angle (middle),**

**fNIRS measures: average total hemoglobin concentration changes (right) of all subjects throughout days.** Error bars are standard error of the mean (SEM).

neuroimaging tool (Villringer et al., 1993; Chance et al., 1997, 1998; Hoshi and Tamura, 1997; Villringer and Chance, 1997; Obrig et al., 2000; Strangman et al., 2002). Recent comprehensive reviews on fNIRS technology (Ferrari and Quaresima, 2012) and instrumentation (Scholkmann et al., 2013) confirm that the vast majority of instrumentation development was on continuous wave (CW) type fNIRS which is limited in terms of its information content (i.e., it measures only changes of oxygenated and deoxygenated-Hb) compared to frequency and time-resolved approaches. However, CW fNIRS is also most appropriate for miniaturization and portable system development, because the signal type and acquisition timing requirements are less demanding. Moreover, other than brain imaging, the same approach can also be used for many biomedical approaches (Macnab and Shadgan, 2012) such as muscle assessment and other *in vivo* and clinical applications.

The development of wearable and low cost fNIRS systems began in 1996 for prefrontal cortex brain hemodynamics and muscle measurements (Chance et al., 1997). These systems, were later further developed into the portable systems at Drexel University (Izzetoglu et al., 2004, 2005, 2011; Ayaz et al., 2012b) and used in the studies reported here.

One of the earliest wireless telemetry systems for fNIRS was based on a single channel muscle oximeter developed by Nakase and Shiga (Omron Institute of Life Science, Kyoto, Japan) in collaboration with Chance (Ferrari and Quaresima, 2012). This system was used by Hoshi et al. for assessing cognitive function of children that carried the system in their backpack, allowing them to move untethered (Hoshi and Chen, 2002). More recently, a 4-channel wireless *in vivo* imager has been developed at the University of Zurich, Switzerland (Muehlemann et al., 2008) and utilized for neurorehabilitation fNIRS studies (Holper et al., 2010). Also, an EEG integrated prototype was reported for epilepsy research (Safaie et al., 2013).

Consistent with the MoBI (Makeig et al., 2009; Gramann et al., 2011) and neuroergonomics (Parasuraman, 2011) approaches, development of miniaturized fNIRS systems for ubiquitous monitoring of the brain could benefit studies for understanding brain dynamics in ecologically valid real world environments.

#### **TWO UNIT APPROACH USING SMARTPHONES**

In this initial effort, our aim was to develop a miniaturized and battery operated CW fNIRS system with comparable operation features with the wired/portable system (Yurtsever et al., 2006), such as 16 channel full forehead assessment and 2 Hz sampling. To expedite the development and take advantage of the off-the-shelf embedded systems, a smartphone/pocket pc platform was utilized (Yurtsever et al., 2003). A custom software application and low level driver for direct hardware access was developed to interface a PCMCIA based National Instrument data acquisition card and control the optical imaging hardware unit and sampling of the data. The smartphone system simultaneously transmits the collected signals through a Wi-Fi network to a computer base station running COBI Studio software, thus allowing online monitoring at a remote base station (see **Figure 8**).

#### *Control circuitry*

The circuit board contains a stable current source for LEDs, implemented with a high precision voltage regulator, timing control elements (counter, demultiplexer, multiplexers), amplifiers, filters and is powered by a 7.2 volt Lithium-Ion camcorder battery. A 5 V voltage regulator is used to provide constant voltage to the circuit since battery voltage is decreased throughout operation.

The circuit is designed to use a minimum number of digital and analog channels of the data acquisition card, so that different data acquisition (DAQ) cards can be used with the control box. Only two digital channels and four analog channels are required to operate the system.

Controlling the timing of the LEDs and photodetectors is the key point in the design and were controlled by the data acquisition software in the Pocket PC. The LEDs turn on and off sequentially, one at a time. The LED turn on sequence in one scan cycle is depicted in red, pink and black colors below the timing signals in **Figure 9**. The LED turn on sequence is as follows: Turn on LED1 730 nm, read D1, D2, D3, D4 (Detectors 1–4); turn on LED1 850 nm, read D1, D2, D3, D4; dark, read D1, D2, D3, D4 (read offset); turn on LED2 730 nm, read D3, D4, D5, D6, and so on.

The timing in the circuit is controlled only by two digital signals, DIO0 and DIO1 that feed a negative edge triggered 3 bit counter. A detailed timing diagram is presented in **Figure 9**, a block diagram of the circuit is given in **Figure 10**. One of the two digital control lines, DIO0, is the reset/initialization signal and the other one DIO1 controls the 4:16 demultiplexer and triggers state transitions as shown in **Figure 10**. That demultiplexer controls selection of the light source LEDs by turning on/off the single pole throw switch connecting to the current source. During a scan, one of four LEDs is lit at one wavelength at a time and the surrounding four light detectors are sampled. The operation is repeated for the second wavelength and for the other three LEDs. During a scan, background light level at each detector is measured while all LEDs are off. The outputs of the analog multiplexers which select the detector outputs are amplified, filtered and digitized. The data collected for each scan cycle is transmitted from the pocket PC to the base station computer through wireless connection for data analysis and display.

#### **INTEGRATED SINGLE UNIT APPROACH**

The two unit system developed earlier presented usability challenges, for example, it required participants to carry two

separate hardware pieces (pocket pc and control box) and the experimenter needed to maintain and charge two separate batteries, etc. To address these challenges and further miniaturize the overall system, a unified single part wireless system was developed (Rodriguez et al., 2010; Rodriguez and Pourrezaei, 2011). The single unit wireless fNIRS system was designed to meet a series of requirements including being pocket-sized, lightweight, and compatible with operation in a hospital/clinical or other work settings where other wireless communications are taking place. Design parameters also included working with a rechargeable battery and also with a single charge, minimum of 3 h operation at maximum power consumption settings. The device was also required to interface and allow the use of the fNIRS sensor currently being used with the "wired" versions of the system developed at the optical brain imaging lab of Drexel University. Finally, the system should interface and communicate with the COBI Studio (Ayaz et al., 2011) which is a hardware integrated software platform that is used for all instruments developed at the Optical Brain Imaging Lab of Drexel University. The implemented system is depicted in **Figure 11** below. It provided comparable performance to that of the wired system in terms of signal integrity and signal to noise ratio (SNR) (Rodriguez and Pourrezaei, 2011) and was used in pediatric monitoring for pain assessment of neonatal patients in intensive care units (Izzetoglu et al., 2011).

#### *Implementation*

The instrumentation split into two nodes (i) a "collector node" that connects to a host computer and (ii) a "sensor node" that collects the data (**Figure 12**). Both nodes communicate with each other through wireless. For the wireless communication technology between these two nodes common technologies where assessed including Zigbee, Bluetooth, and Wi-Fi. Zigbee was chosen for this implementation due to its low cost, low-power requirement, and wireless mesh networking capabilities. The low power usage allows longer life with smaller batteries and the mesh

networking provides high reliability and larger network range coverage.

The Texas Instruments CC2430 System-On-Chip (SOC) was selected for this solution; the built in 2.4 GHz IEEE 802.15.4 transceiver, 8051 enhanced core, eight channel 12-bit ADC, and 21 digital IOs were sufficient (See **Figure 12**). A new sensor that contains two monolithic photodiodes with built-in transimpedance amplifiers and a two-wavelength (730 and 850 nm) LED source was developed (see **Figure 13**). The LED driver circuit was designed to be capable of driving each wavelength separately with output currents up to 50 mA. The sensor circuit has a buffer amplifier for each of the two photodiodes, whose gain can be controlled individually by the microcontroller. The input signals from the detectors are multiplexed and then passed through anti-aliasing filter (a low pass analog filter) to remove high frequency noise before being fed to the analog-todigital converter (ADC), which samples at a 1.5 kHz rate per channel.

An optional digital filtering of the data can be performed at the sensor node, prior to packaging and sending the data packets over the wireless network. This system can be used with various types of sensor that have different number of detectors. The "collector node" of the system communicates with a host computer through the Universal Serial Bus (USB) port utilizing a custom communication protocol that allows the COBI Studio software to control the system.

#### *Evaluation of the system*

Testing of the newly designed sensor was performed in terms of noise, accuracy, stability, and effectiveness of obtaining signals using solid and liquid phantoms that mimic the adult human head before measurements in humans.

*Solid phantom tests.* The SNR data calculated for using the modular fNIR sensor at a gain setting of 1 and 10 were compared for the input buffer amplifier. The SNR was higher for gain 10 and increased for higher LED current values from around 60 to 75 dB as the LED current reached 40 mA.

In **Figure 14** the response of both systems are compared when the input gain setting is set to 10 and the LED current is swept from 5 up to 40 mA. It can be seen that the

wireless system performs very close to the wired version: the difference between the measurements is 8% and this difference remains almost constant across the different setting configurations under which the data was collected. The gain linearity of the wireless system was also analyzed. With the LED current held constant at 20mA and the gain changed to values of 1, 10, and 20, the system responded linearly within this range.

*Liquid phantom tests.* To evaluate the dynamic response of the system for changes in oxygenated-hemoglobin (HbO) deoxygenated-hemoglobin (Hb) concentrations, a tissue stimulating phantom was used (Bozkurt et al., 2005; Yurtsever et al., 2006; Rodriguez and Pourrezaei, 2011). Liposyn III solution of 1% was prepared in a cylindrical transparent glass beaker from 30% Liposyn III in 1000 ml phosphate buffered saline at pH 7.4. This solution has reduced the scattering coefficient of 10 cm−1 at 830 nm, which is a good estimate for the human forehead. The mixture was continuously stirred with a magnetic stirring rod to keep the solution homogeneous. To simulate the blood content in tissue, around 50µM, 22 mL of human blood was added to the beaker. The sensor pad was attached to the side of the beaker and baseline was recorded from the wireless unit using COBI Studio. Then, 4 g of baker's yeast was added to the mixture. The yeast respiration led to deoxygenation, so [HbO] decreases and [Hb] increases. Representative data is presented in **Figure 15**. After 13 min, the [Hb] and [HbO] reached a steady state, where oxygenation of hemoglobin and yeast respiration are at equilibrium. Then, we provided oxygen to the solution from an oxygen tank (green line in the graph) to re-oxygenate deoxyhemoglobin. As a result of oxygen bubbling inside the beaker, hemoglobin saturation exceeds the initial saturation and steady state is reached at a higher saturation level.

#### **CONCLUSIONS**

This paper described a range of studies on human performance assessment and skill acquisition monitoring using fNIRS measures of the hemodynamic response of the prefrontal cortex and its relationship to mental workload, expertise, and performance. The results show that the effects of task load and expertise on the hemodynamic response can be reliably and sensitively assessed in a range of tasks, from standardized laboratory tasks to complex cognitive tasks representative of real work settings. With respect to the development of wireless, portable fNIRS, the results, although preliminary, corroborate previous findings and point to the potential of the fNIRS system as a wearable, portable and non-invasive sensor for future neuroergonomics studies. Moreover, miniaturization and wireless system development efforts reported here will benefit future studies that can allow participants to freely navigate in indoor or outdoor environments untethered, consistent with the MoBI approach. Although further work may be needed in specific applications, both wired and wireless fNIRS systems allow for the examination of dynamic aspects of brain function in more natural settings, and are thus suitable for reliable human performance and training assessment.

#### **ACKNOWLEDGMENTS**

The authors would like to acknowledge Drs. Meltem Izzetoglu, Scott Bunce, Kambiz Pourrezaei, and also Davood Tashayyod, Gunay Yurtsever, Mauricio Rodriguez, and Juan Du for their constructive support, contributions, and for taking an active role in of the studies discussed here.This work was supported in part by the U.S. Federal Aviation Administration through BAE Systems Technology Solutions Services Inc. under Primary Contract, DTFA01-00-C-00068, and Subcontract Number, 31–5029862 as well as under U.S. Army Medical Research Acquisition Activity; Cooperative Agreements W81XWH-08-2-0573 and W81XWH-09-2-0104. Additional support was provided by the Air Force Office of Scientific Research grant FA9550-10-1-0385. The views, opinions, and/or findings contained in this article are those of the authors and should not be interpreted as representing the official views or policies, either expressed or implied, of the funding agencies.

#### **REFERENCES**

the adult brain? *Int. J. Psychophysiol.* 35, 125–142. doi: 10.1016/S0167-8760(99) 00048-3

International Airport, NJ: Federal Aviation Administration William J. Hughes Technical Center.

**Conflict of Interest Statement:** fNIR Devices, LLC manufactures the optical brain imaging instrument and licensed IP and know-how from Drexel University. Hasan Ayaz, Kurtulus Izzetoglu, and Banu Onaral were involved in the technology development and thus offered a minor share in the new startup firm fNIR Devices, LLC. The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 03 September 2013; accepted: 28 November 2013; published online: 18 December 2013.*

*Citation: Ayaz H, Onaral B, Izzetoglu K, Shewokis PA, McKendrick R and Parasuraman R (2013) Continuous monitoring of brain dynamics with functional near infrared spectroscopy as a tool for neuroergonomic research: empirical examples and a technological development. Front. Hum. Neurosci. 7:871. doi: 10.3389/fnhum. 2013.00871*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2013 Ayaz, Onaral, Izzetoglu, Shewokis, McKendrick and Parasuraman. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Kinesthetic and vestibular information modulate alpha activity during spatial navigation: a mobile EEG study

#### *Benedikt V. Ehinger 1 \*†, Petra Fischer 1†, Anna L. Gert 1, Lilli Kaufhold1, Felix Weber 1, Gordon Pipa2 and Peter König1,3*

*1 Neurobiopsychology, Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany*

*2 Neuroinformatics, Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany*

*3 Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany*

#### *Edited by:*

*Klaus Gramann, Berlin Institute of Technology, Germany*

#### *Reviewed by:*

*Lutz Jäncke, University of Zurich, Switzerland Tobias Meilinger, Max Planck Institute for Biological Cybernetics, Germany Markus Plank, Brain Products GmbH, Germany*

#### *\*Correspondence:*

*Benedikt V. Ehinger, Neurobiopsychology, Institute of Cognitive Science, University of Osnabrück, Albrechtstra*β*e 28, Osnabrück 49069, Germany e-mail<EMAIL_ADDRESS>
*†These authors have contributed equally to this work.*

In everyday life, spatial navigation involving locomotion provides congruent visual, vestibular, and kinesthetic information that need to be integrated. Yet, previous studies on human brain activity during navigation focus on stationary setups, neglecting vestibular and kinesthetic feedback. The aim of our work is to uncover the influence of those sensory modalities on cortical processing. We developed a fully immersive virtual reality setup combined with high-density mobile electroencephalography (EEG). Participants traversed one leg of a triangle, turned on the spot, continued along the second leg, and finally indicated the location of their starting position. Vestibular and kinesthetic information was provided either in combination, as isolated sources of information, or not at all within a 2 × 2 full factorial intra-subjects design. EEG data were processed by clustering independent components, and time-frequency spectrograms were calculated. In parietal, occipital, and temporal clusters, we detected alpha suppression during the turning movement, which is associated with a heightened demand of visuo-attentional processing and closely resembles results reported in previous stationary studies. This decrease is present in all conditions and therefore seems to generalize to more natural settings. Yet, in incongruent conditions, when different sensory modalities did not match, the decrease is significantly stronger. Additionally, in more anterior areas we found that providing only vestibular but no kinesthetic information results in alpha increase. These observations demonstrate that stationary experiments omit important aspects of sensory feedback. Therefore, it is important to develop more natural experimental settings in order to capture a more complete picture of neural correlates of spatial navigation.

**Keywords: spatial navigation, mobile EEG, alpha band, event related desynchronization, alpha suppression, virtual reality, independent component analysis, time-frequency analysis**

#### **INTRODUCTION**