As would be expected, differences in DOC EEG behavior compared to controls have been noted; specifically a generalized slowing and dominance of slow wave activity with significantly diminished α power (Kulkarni et al., 2007), and peak power in the θ range in MCS (Schiff et al., 2014). However, attitudes as to the prognostic value of EEG are divided. Some pessimistic accounts suggest the heterogeneity of EEG behavior in this population is too confounding for prognostic value (Kulkarni et al., 2007; Kaplan and Bauer, 2011). Alternatively, Lechinger et al. (2013) has found resting EEG patterns (specifically spectral peaks and ratios of frequencies 8 Hz+ to <8 Hz) to correlate well with behavioral diagnostic assessments. Furthermore event related potentials have been used to assess for recovery (Faran et al., 2006; Wijnen et al., 2007), cognitive processing and cortical learning in VS patients (Kotchoubey et al., 2006, 2009; Kotchoubey, 2007), and have revealed evidence of exogenous and endogenous attention in the absence of behavioral signs of awareness (Chennu et al., 2013). A recent audit of DOC registry data from five rehabilitation centers (*n*: 38 patients) recorded EEG reactivity to acoustic, touch, and light pain stimuli in 54% of patients at initial assessment where prior assessments indicated 21% emerging from MCS, 12% MCS and 67% in VS, although the prognostic value of these measures was not reported (Grill et al., 2013).

# **BEHAVIORAL MEASURES WITH DOC**

A number of research studies using behavioral measures to determine responsiveness to auditory stimuli in DOC have drawn promising, if unconfirmed, conclusions. The behavioral characteristics of VS occurring in relation to the presentation of multimodal stimuli have been noted to be heterogeneous, including spontaneous body movement, blinking, and vocalization (Wilson et al., 1996a). Furthermore, contingency between spontaneous body movement and environmental stimuli have been correlated with recovery (Wilson et al., 1996b). Music therapy intervention showed improvements in agitation and interactive behaviors in DOC using blinded behavioral assessment of video behavioral data (Formisano et al., 2001).

From reviewing the literature, it is clear that a range of neurophysiological and behavioral assessment methods are available to investigate the utility of music therapy procedures with DOC assessment, and potential for their use in rehabilitation. The following sections detail a study which addresses the lack of robust research in this area by comparing responses to musical stimuli commonly used in music therapy interventions with other auditory stimuli that provided control and contrasting stimuli.

# **MATERIALS AND METHODS**

A multiple baseline within subjects study was used to compare EEG, HR, HRV, respiration, and behavioral responses contingent to music therapy and other auditory stimuli. Ethical approval was given by an internal research review board and a governmental central ethics agency. 20 healthy controls were recruited firstly (13 female aged 24–52 years, mean 34 years, SD 12.5, and 7 healthy males aged 29–59, mean 41, SD 11). Exclusion criteria comprised individuals with known hearing impairment or a high level of musical proficiency. Patients were then recruited who were medically stable, had no known major hearing impairment according to their medical notes, and who were undergoing assessment for diagnosis of awareness using SMART (Gill-Thwaites, 1997; Gill-Thwaites and Munday, 2004) and MATADOC (Magee et al., 2013) assessments. Twelve patients were diagnosed as VS and nine as MCS using SMART with an 86% agreement to MATADOC outcomes, and patients were seen a mean of 7.3 (SD 2.8) months post injury (see **Table 1** for demographic and diagnostic details). Control subjects were recruited from staff via email at a large neuro-rehabilitation unit, and patients at the unit via contact with next of kin who provided consultee approval.

Five minutes baseline silence (BLS) was followed by the presentation of four contrasting auditory stimuli, with two minutes washout silence separating each stimuli. Music therapy stimuli comprised live performance of preferred song music (LM) and live music featuring simple improvised vocal melody, incorporating the repeated phrase "Hello (*patient's name*). Hello to music" with a basic supporting accompaniment entrained to respiration (EI). Non-music therapy stimuli comprised digital recordings of disliked music (DM) and white noise (WN). DM was included to provide any evidence of nociceptive or discriminatory responses indicative of awareness, and WN as a non-musical auditory control. LM and EI are typically used in music therapy interventions. Information about personal music preferences for the LM and DM stimuli were obtained from relatives for patients, and directly from healthy subjects. LM and EI were performed by the lead author using a Yamaha NP31 digital electric piano. To control for order effects, the sequence of stimuli was randomized, with order series placed in opaque sealed envelopes for blinded order selection by an independent observer for each participant. Data were recorded using a XLTEC 50 channel video EEG and neuro-physiological data acquisition system with a piezoelectric respiratory belt, and analyzed using Mathworks MATLAB, SPSS (Ver20), and BrainVision Analyzer 2 (BVA) software. ECG data were collected for HR and HRV via two chest electrodes together with respiration data within the XLTEC system. Volume was maintained within a 50– 70 dB range for all stimuli using a Tecpal 331 sound level meter.

**Table 1 | Patient demographic details and diagnostic outcomes.**

Healthy subjects were instructed to close their eyes half way into each stimulus presentation to provide both eyes open and closed data.

Behavioral data using video recordings of patient sessions were analyzed by a trained volunteer, who was blinded by removing audio from recordings. Ten second segments were scored for a range of behaviors using a graded system from Wilson et al. (1996a) from "eyes shut and no body movement" to "engaged in activity" (e.g., scratching). Any additional behaviors such as blinking and mouth movement were also documented. Commands from the auditory function scale of the CRS R (Giacino and Kalmar, 2006) were presented after each stimulus to observe for behavioral signs of awareness.

Nineteen channels of EEG data were obtained using a common average montage and 10:20 electrode configuration. Due to the presence of craniotomies amongst 13 patients, free electrode placement was adopted in preference to skull caps. Raw data was sampled at 512 Hz and filtered to a hi/low cut off bandpass filter at 0.5 and 30 Hz. Independent Component Analysis with classic sphering and an infomax algorithm in BVA was used to remove artifacts such as eye blinks, followed by Fast Fourier Transformation (FFT) using a hanning window to produce the frequency spectrum, or amplitude as a function of frequency, for each electrode. Data were segmented to 2 sec units and pooled into 21 different electrode configurations to represent different brain regions (see **Table A1** in Appendix for details).

After exporting to SPSS, one way repeated measures analysis of variance (ANOVA) analysis with Bonferroni corrections was applied to data. For healthy subjects all data was pooled to provide indicators of healthy responses across measures using ANOVA's around means. *Post hoc F* statistics were obtained using simple contrasts in relation to BLS to indicate the strength of association of positive or negative change for individual stimuli in contributing to overall ANOVA significance and *F* statistic levels3 .

Given the clinical and neuro-pathological heterogeneity of those with DOC, particularly with regard to EEG responses (Kulkarni et al., 2007), within-subject statistical analyses were conducted using segmented data that produced individual ANOVAs for the case material.

# **RESULTS**

#### **HEALTHY CONTROL DATA: EEG**

Data presented is here for healthy control data from the "eyes closed" sections of recordings, which produced the most artifact free and significant findings across measures, however missing EEG data from one subject resulted from a corrupted signal. **Table 2** illustrates that where all electrode data pooled to L and R hemispheres, LM provided the greatest effect on EEG amplitude overall in the R hemisphere, and produced similar increases to WN in the left. The peak for WN in δ over the L and R hemispheres is suggestive of a "drowsiness" effect. Examination of each pooled region revealed the most distinct discriminatory responses occurred in frontal and temporal regions across bandwidths,with a bias toward the right (R) hemisphere generally. **Figure 1** illustrates the differences in mean EEG amplitude for frontal and temporal

3*F* statistics are rounded to the first decimal point. ANOVA's were computed using alpha = 0.05.

# regions, most distinct for LM increases compared to other stimuli in the R frontal and temporal regions, with decreases for EI in α and θ in both frontal regions. **Figure 2** highlights significant amplitude differences for θ in the frontal midline region where a contrast between DM and LM was most marked compared to other regions and bandwidths. The ANOVA tests for each pooled region revealed clear significance4 within ANOVA's for changes in δ power in all areas apart from in the left (L) parietal, for θ for all regions except the parietal and R posterior, for α, all regions except the L parietal, and for all individual regions in β frequencies. As the means plots suggest, ANOVA tests revealed right frontal and temporal regions as the most subject to change, most significant for the R frontal region in β [*F*(4, 2740) = 70.7, *p* < 0.001] and α [*F*(4, 2740) = 39.2, *p* < 0.001]. *Post hoc* contrasts with BLS highlighted the dominant contribution of LM to significant ANOVA results in the R frontal region with peak increases in β [*F*(1, 685) = 100, *p* < 0.001] and α [*F*(1, 685) = 50.2, *p* < 0.001].

In summary, for the healthy cohort, EEG responses to LM showed dominance in peak power responses globally and across bandwidths compared to the other stimuli tested, particularly in R frontal and temporal regions.

#### **HEALTHY PHYSIOLOGICAL DATA**

The most significant findings for pooled healthy data were found for respiration rate and variance. **Figure 3** illustrates the significant change found for respiration rate, where LM provided a peak increase5 with a mean respirations per minute rate of 17.8 (SD 3.7)

**Table 2 | Electroencephalogram mean amplitude (**µ**V): stimuli compared.**

SD >100 rounded to 0 decimal point.

BL, baseline silence; LM, liked music; EI, entrained improvisation; DM, disliked music; WN, white noise (standard deviation in parentheses).

4 Significance hitherto defined by *p* ≤ 0.05.

5 "Increases" and "decreases" henceforth in relation to *post hoc* ANOVA contrasts with BLS.

compared to a BLS rate of 12.5 (SD 3.2). A significant peak to peak variance was also observed [*F*(4, 56) = 4.1, *p* = 0.006] where WN provided the largest increase [*F*(1, 14) = 11.5, *p* = 0.005]. *Post hoc* correlation analysis of pooled mean respirations per minute compared to means of beats per minute (BPM) for DM, LM, and EI did not indicate entrainment effects upon tonic respiration rate. Significance for HR and HRV was less clearly defined in relation to the effects of individual stimuli. Similarly, in relation to HRV measures, the "eyes closed" LF measure approached significance [*F*(4, 48) = 2.5, *p* = 0.054], but change was non-specific across stimuli, with similar increases in LM, DM, and WN in relation to BLS.

# **PATIENT DATA**

All patient data were pooled for observation of trends in patient responses. The results were heterogeneous as expected, particularly for physiological measures, however notable exceptions were found within behavioral and EEG data. **Figure 4** highlights that within the VS cohort, pooled eye blink data reached significance, with a peak increase for LM. Similar non-significant trends were observed for LM in eye and mouth movement and "eyes open no body movement" measures in the VS data. Although blink rate change was not significant for the MCS cohort, WN provided a contrasting peak increase [WN mean blinks per min 41.6 (SD 27.1) compared to BLS 34.8 (SD 25.7) and LM 24.3 (SD 25.7)]. Whilst heterogeneous responses and large SD's contributed to a lack of significant change within the VS cohort generally (see **Table 2**), mean FMT increased significantly for LM in half (*n*: 6) of cases where ANOVA's were significant, and peaked significantly in 4 MCS cases (44%).

The data provided in **Table 2** accords with the literature (e.g., Kulkarni et al., 2007; Schiff et al., 2014) with regard to the typical characteristics of MCS, with a spectral peak in the θ range and VS, where power is predominantly observed in very low frequencies. However, one may also observe LM produced L and R hemisphere amplitude peaks across the MCS cohort in α. In reviewing ANOVA's for each pooled region, the most noticeable significant change was observed for pooled MCS frontal α [*F*(4,

**FIGURE 2 | Healthy frontal midline theta activity**.

**FIGURE 4 | Vegetative state blink rate**.

1850.1) = 36.5, *p* < 0.001] with a peak for LM [*post hoc* contrast *F*(1, 809) = 50.6, *p* < 0.001]. **Figure 5** details the frontal α data across cohorts, illustrating expected power differentials between healthy, MCS and VS. An interesting pattern of LM increases for all subjects is also visible, with more distinction from WN increases in MCS than VS subjects. Frontal α peaked for LM in three VS and four MCS subjects, where AVOVAs were significant between *p* = 0.05 and 0.0001. To summarize, whilst pooled physiological patient data were heterogeneous, significant changes were found

in relation to an increased blink rate, maximal for LM, across the VS cohort and frontal α for LM across the MCS cohort, with significant FMT increases for LM for half the VS and four MCS patients.

As noted previously, given the general heterogeneity of the patient data, three case studies follow where noteworthy combinations of behavioral and neuro-physiological data are highlighted.

### **CASE STUDY ONE**

"A" was a 35-year-old female, admitted to the unit with severe TBI following a road traffic accident 10 months previously, who had undergone bi-lateral frontal craniotomies to reduce cranial swelling. Both her SMART and MATADOC assessments gave a diagnosis of VS. Medication adjustments, spiking temperatures, and general poor arousal levels were thought to be impacting on her response levels. The experimental session took place during this assessment period, i.e., when arousal was reduced.

"A" displayed a range of behaviors suggestive of discriminatory responses in relation to LM, EI, and WN. In relation to EEG, significant *post hoc* peaks for LM and EI were observed for temporal, frontal, central, parietal, and occipital regions across bandwidths. **Figure 6** illustrates the θ and α EEG responses for each stimuli in the R temporal region, highlighting the dominance of EI in θ, and LM in α. Furthermore **Figure 7** highlights the marked significant increase in frontal β for LM, which contributed to a significant θ:β ratio [*F*(4, 356) = 47.0, *p* < 0.001] with a marked decrease for LM [*F*(1, 89) = 133.8, *p* < 0.001].

"A" had her eyes open with no body movement throughout the session. Interestingly, she displayed eye movement only for LM and WN with significantly more counts for LM (48 compared to 38, *t*-test *p* = 0.001). Her HR ANOVA showed marked significant change [*F*(4, 36) = 1373, *p* < 0.001] due to the striking increases for mean HR for LM (91 BPM, SD: 2) and WN (86 BPM, SD: 1.2) compared to BLS (68 BMP, SD: 0.69), DM (67 BPM, SD: 1), and EI (65 BPM, SD: 0.9). HRV (RMSSD) also showed significant change [*F*(4, 36) = 101.2, *p* < 0.001] due to decreases for LM [*F*(1, 9) = 141.2, *p* < 0.001] and WN [*F*(1, 9) = 154.4, *p* < 0.001]. LF and HF change was also significant [LF: *F*(4, 36) = 11, *p* = 0.012, HF: *F*(4, 36) = 11.8, *p* < 0.001] accounted for by increases in LF for LM [*F*(1, 9) = 9.8, *p* = 0.012] and WN [*F*(1, 9) = 23.8, *p* = 0.001] and decreases in HF for all stimuli compared to BLS. In summary,"A"displayed significant behavioral responses LM, autonomic responses for LM and WN, and EEG amplitude increases in the R temporal and frontal regions across bandwidths for both music therapy stimuli.

#### **CASE STUDY TWO**

"B" was a 49-year-old male with global hypoxic brain injury following a seizure 10 months previously, with high muscle tone and myoclonus6 . Both SMART and MATADOC assessments gave a VS diagnosis, where he demonstrated mainly reflexive responses, but also some sensitivity to sound and touch, with increased spasm frequency when over stimulated. The experimental session took place during the assessment period.

B's EEG responses were characterized by the dominance of WN in producing amplitude peaks across bandwidths and stimuli. However, notable exceptions were found for FMT, where a marked peak for EI was observed [ANOVA *F*(4, 356) = 82, *p* < 0.001, EI *post hoc* contrast *F*(1, 89) = 138.1, *p* < 0.001], with further peak activity for LM in β in the frontal, parietal, and midline regions. A significant θ:β ratio was found for the pooled frontal region [*F*(4, 356) = 5.8, *p* < 0.001] with a significant *post hoc* decrease

for EI [*F*(1, 89) = 3.7, *p* = 0.57, *F*: 3.7], and increase for WN [*F*(1, 89) = 5.7, *p* = 0.025].

Whilst "B" had his eyes closed throughout the session, various discreet behavioral responses were observed. For example

6A brief, involuntary twitching of a muscle or a group of muscles caused by sudden muscle contractions.

the change in his blink rate (i.e., blinks per 10 s) approached significance [*F*(4, 15.2) = 2.4, *p* = 0.058], due primarily to increases for LM [*F*(1, 17) = 6.6, *p* = 0.02], and head movement changed significantly [*F*(4, 15.2) = 13.1, *p* < 0.001], due to decreases for EI and DM.

Physiological responses for "B" showed significant change in a range of measures, for example his HR decreased for all stimuli compared to BLS [*F*(4, 36) = 4.6, *p* = 0.004], whilst a significant finding for HRV in the RMSSD measure [*F*(4,36) = 4.2,*p* = 0.004] comprised a contrasting increase for LM [*F*(1, 9) = 4.2, *p* = 0.074] with a decrease for DM [*F*(1, 9) = 4.2, *p* = 0.03]. To summarize, neurophysiological and behavioral data showed a range of significant change, where despite generalized EEG power increases for WN, localized increases across bandwidths were observed for LM and EI, with increases in HRV for LM.

#### **CASE STUDY THREE**

"C" was a 24-year-old female with a R temporal hematoma following a traumatic brain injury 6 months previously. Her SMART assessment completed 2 weeks previously gave a diagnosis of VS. However it was clear from the evidence of reactive θ 7 at the start of the experimental session, and some consistent responses to the CRS R commands, that she had emerged from VS to MCS. Interestingly her initial MATADOC assessment undertaken concurrently with the SMART gave a more positive borderline diagnosis.

**Figure 8** details global EEG responses for "C" which were notable for the dominance of music therapy stimuli across θ, α, and β, where ANOVAs for each bandwidth regions indicated significant change. In θ, significant increases were found for LM [*F*(1, 88) = 5.4, *p* = 0.022], and decreases for WN [*F*(1, 88) = 24.4, *p* < 0.001]. In α, discrimination was observed in relation to peak power for LM and EI compared to DM and WN, with a shift to EI as the peak stimulus in β [*F*(1, 88) = 14, *p* < 0.001]. θ:β change was significant [*F*(1.2, 96) = 15.6, *p* < 0.00], comprising an increase for LM [*F*(1, 88) = 3.2, *p* = 0.077], with significant decreases for DM, EI, and particularly WN [*F*(1, 88) = 91.7,*p* < 0.001]. **Figure 9** highlights the topographic changes observed between BLS and LM energy in low α (8–10.5 Hz), illustrating the relatively less damaged left hemisphere provided the location for increased responses, a pattern repeated across bandwidths for LM and EI.

"C"s behavioral responses were interesting in relation to the finding for blink rate noted previously for VS subjects. Whilst blink rate change was significant [*F*(4, 68) = 3.8, *p* = 0.007], this comprised a decrease for LM [*F*(1, 17) = 3.2, *p* = 0.089] and increase for WN [*F*(1, 17) = 3.1, *p* = 0.096]. However "eyes open body movement" counts also changed significantly [*F*(2.1, 36.8) = 3.9, *p* = 0.026], but conversely here there were no counts for WN in comparison to significant increases from baseline level (*n*: 2) for LM [*n*: 32, *F*(1, 17) = 7.2, *p* = 0.015] and EI [*n*: 28, *F*(1, 17) = 8.35, *p* = 0.15].

"C"s HR changed significantly [*F*(4, 36) = 10.5, *p* = 0.001] due to significant increases for LM [*F*(1, 9) = 41.4, *p* = 0.001], EI [*F*(1, 9) = 17.7, *p* = 0.002], and DM [*F*(1, 9) = 5.1, *p* = 0.05], which was accompanied by significant change for LF [*F*(1,

**FIGURE 8 | Case "C": EEG global responses across bandwidths**.

9) = 7.2, *p* = 0.001], with marked increases for LM [*F*(1, 9) = 46, *p* < 0.001]. HF also changed significantly [*F*(4, 36) = 6.7, *p* < 0.001] accounted for by significant decreases with LM and EI [*F*(1, 9) = 11.9, *p* = 0.007 and *F*(1, 9) = 13.1, *p* = 0.006]. The most contrast in relation to individual measures was within the HRV time domain (RMSSD), where significant change [*F*(1.7, 15.2) = 11.9, *p* = 0.001] comprised a marked decrease for LM [*F*(1, 9) = 55.1, *p* < 0.001] in contrast to an increase for WN [*F*(1, 9) = 6.1, *p* = 0.035]. In summary, the combination of global increases in EEG power across bandwidths for music therapy stimuli combined with the changes noted for behavioral and HRV measures suggests music therapy elicited heightened arousal and mental effort compared to other stimuli. Furthermore "C"s

7 i.e., clear θ wave present in the EEG display, showing spiking responses to sudden noise or brief painful stimuli (e.g., pinching).

EEG responses indicated heightened discrimination, or selective attention between music therapy and other stimuli.

# **DISCUSSION**

In parallel with advances in brain imaging techniques such as fMRI, there have been extensive developments in our understanding of how music affects the brain, or "music neuroscience." However our understanding of the neuro-physiological effects of live music stimuli typically used in therapy interventions is limited (O'Kelly and Magee, 2013a). Whilst some scanning techniques are antithetical to naturalistic explorations of the methods described in this paper, we have adopted EEG and physiological monitoring to provide new perspectives on music therapy with both healthy individuals and those with DOC.

The findings of the healthy cohort EEG analysis of the study revealed widespread increases in EEG power across bandwidths in relation to LM, more distinct in the R hemisphere. This hemispheric specialization accords with Tervaniemi and Hugdahl's (2003)review comparing speech with music processing, and Stewart et al.'s (2006) review of disorders in music listening. Interestingly, in relation to the literature on valence, this study's findings are slightly ambiguous. Altenmüller et al. (2002) suggests that greater R frontal activation, as observed here for LM, denotes negative valence in musical listening, particularly in females. This finding may perhaps be accounted for the range of functions preferred music listening serves,such as"self-reflection"(Schäfer et al., 2012). However increased FMT was observed with LM, which is linked to "pleasant" music listening experiences (Sammler et al., 2007). EI, designed to be appropriate for accommodating processing deficits in clinical work, produced a decreases in α compared to silence in the healthy cohort. One may hypothesize that EI lacks the richness and complexity of LM, which may engage more emotional or cognitive responses with healthy subjects. Subjective feedback could have clarified these issues, however this was not gathered as it lay outside the main focus of the study. Valence issues aside, the LM data provides further support to a growing evidence base for preferred music listening and musical activity as beneficial for supporting neuroplasticity through engaging a global system of temporal, frontal, parietal, cerebellar, and limbic brain areas involved in auditory and language processing, emotion, attention, motor control, and memory (Altenmüller et al., 2012; Särkämö et al., 2013). These qualities have been explored with neuro-imaging studies addressing visual neglect, memory, attention, and mood disorders in neurological populations (Särkämö et al., 2008, 2009) and other populations (Koelsch et al., 2010; Fachner et al., 2013), but are as yet untested in this manner in DOC rehabilitation.

The healthy subjects' respiration data points to a useful benchmark for healthy behavior when the rate increases during listening to preferred music, with a lack of entrainment effect to music tempo suggesting to cortically mediated, emotional, or "top down" processes according to Salimpoor et al. (2009). EEG findings for healthy subjects for DM and WN in the R parietal region combined with WN effects on increasing respiration variance are also noteworthy. Boiten et al.'s (1994) finding that increases in respiration variance are found during emotional upset and Heller's (1993) model for the R parietal region modulating autonomic and

behavioral arousal in "emotional states" suggest both responses correspond to processing emotional responses to auditory/musical stimuli which are likely to be unpleasant.

The findings of the patient component of the study provide a range of support for the use of music therapy in assessment of DOC. Furthermore, they highlight the need for less dichotomous thinking in relation to differences between VS and MCS models, as suggested by Bruno et al. (2011). However, the findings on blink rate with VS, although encouraging, need to be interpreted cautiously in relation to the literature. Whilst blink rate is correlated positively with dopaminergic system activity, arousal (Karson et al., 1990), attention (Abe et al., 2011; Irwin, 2011), and creativity (Chermahini and Hommel, 2010), it has also been *negatively* correlated with attention (reading versus resting state) (Bentivoglio et al., 1997) and recognition of saliency in video stimuli (Shultz et al., 2011). Blink rate increased significantly for VS patients presented with LM, however change was not significant for MCS, although WN provided a contrasting peak increase. This, together with the findings of MCS case "C" (decreased blink rate for LM compared a significant increases for WN, but significant increases in body movement for LM and EI), suggests two types of blink behavior: (i) a basic arousal response where blink rate increases, and (ii) a more sophisticated attention response where blink rate decreases where awareness of the stimuli increases and visual attention is recruited. Further questions are raised by the mean VS resting levels of blinking [20 (SD 16.4) per minute] compared to MCS [34.8 (SD 28.5)], which contrasts with the finding of Bonfiglio et al. (2005) that reduced spontaneous blink rate characterizes the early stages of conscious recovery.

The case studies highlight how significant change may be observed in ANS measures in response to music therapy. The contrasting direction of change for these measures highlights the unique responsiveness of each DOC patient. Based on healthy studies of HRV, and dependent on an individual's intact emotional processing, it is plausible that some DOC patients may find LM pleasant and relaxing, whilst others may process the stimuli differently, even to the extent of finding the experience stressful in terms of mental exertion. The divergent, yet significant, changes in θ:β ratios in the case studies further highlight this issue. However, without the benefits of subjective verbal accounts such hypotheses are tenuous. The important issue is perhaps not so much whether a brief exposure to a stimulus produces a reaction akin to a "healthy" stress response, more that significant cortical and ANS activity occurs that might suggest discrimination rather than a reflex response. Following the previously noted findings of Wijnen et al. (2006) and Riganello et al. (2010), patients with these significant ANS responses suggest the need for a comprehensive follow up assessment, as these responses may suggest favorable recovery potential.

Despite the previously noted heterogeneity in DOC EEG data, patterns of discriminatory behavior for FMT and frontal α were observed in VS and MCS subjects. Given the literature on FMT as a marker of visio-spatial navigation and memory activity in the hippocampus (Fell et al., 2003; Ekstrom et al., 2005), and cognitive and emotional functions such as motivation, processing information, and attention found in the anterior cingulate cortex (Devinsky et al., 1995; Wang et al., 2005), this finding is

noteworthy. Certainly for VS subjects, one would not expect to find any evidence of heightened responsiveness to the complex stimuli of LM or EI in preference to DM and WN, where the former may contain unique elements involved in emotional, language, or memory processing. The frontal α increases in pooled MCS data, and widespread increases in α, θ, and β such as those illustrated in case "C" also underpin the utility of music therapy in promoting cortical activity, which may enhance local and long distance connectivity for MCS patients. Furthermore, the level of differentiation between LM and EI compared to WN and DM in MCS highlight more intact cognitive processes such as selective attention in MCS compared to VS, where this differentiation was less evident. Research exploring whether, and how, these responses can be harnessed to promote neuroplasticity in DOC rehabilitation is indicated. Furthermore, the different patterns of discrimination for VS and MCS found highlights the potential of combined music therapy/EEG assessment as providing a noninvasive and widely applicable method to compliment behavioral assessments.

The data collection for the patient cohort in particular posed numerous challenges, such as applying EEG electrodes where patients had craniotomies, to removing numerous EEG artifacts caused by appliances such as feeding machines. Patients did not receive auditory brain stem testing to exclude patients with undiagnosed hearing impairment, which may have provided a confounding element to the findings. Furthermore, all patients were receiving a range of medications, where a common side effect was drowsiness, which may compromise EEG and ANS responses. Conversely one may consider the significant change found in neuro-physiological measures in this study as important from a clinical perspective, where behavioral responses alone, attenuated by drowsiness, might give a different impression. The provision of music therapy methods by one music therapist, and sample size, also suggest caution is needed when interpreting findings, particularly in relation to the potential variance in performance style one might find with other music therapists. It should also be noted that LM and DM comprised heterogeneous tempo, harmonic, rhythmic, and lyrical content. This lack of standardization may have provided a confounding effect, perhaps explaining the divergent HR and HRV results. However, it would be antithetical to provide these musical items in a standardized form, as this would possibly obscure any elicitation of responses based on intact memory function. Further research might involve several therapists trained within standardized protocols, and larger samples to improve the robustness of findings.

# **CONCLUSION**

This study addresses the lack of empirical evidence supporting music therapy in the assessment of those with DOC,with pilot level data on the neurophysiological and behavioral responses of DOC patients to music therapy. By comparing healthy data with findings from the DOC cohorts it is evident that music therapy is capable of eliciting a range of responses indicative of arousal and selective attention. Combined music therapy and neuro-physiological assessment may provide distinctive contribution revealing intact responsiveness to salient stimuli, even in VS patients considered to be "unaware" of themselves and their environment, which merits further investigation for prognostic value. Furthermore, as some VS subjects responded selectively, and beyond chance level to music therapy in neuro-physiological measures, it is clear that blanket assumptions as to the "unresponsive" nature of VS are questioned, especially where these assumptions are predicated on behavioral assessment alone. Given our expanding knowledge of the role of musical activity in promoting neuroplasticity, further research is needed to explore the potential of music therapy to optimize assessment, and promote functional gains within this population.

# **ACKNOWLEDGMENTS**

The authors would firstly like to thank all healthy volunteers and carers who gave consultee approval for their loved ones to be recruited to the study. They would also like to acknowledge the support of Dr. Sophie Duport, Head of Research at the Royal Hospital for Neuro-disability (RHN); the scientific support team at Brain Products GmbH, the RHN multi-disciplinary care team; and funding from Aalborg University, the RHN, and the Music Therapy Charity.

# **REFERENCES**

perception of music. *Ann. N. Y. Acad. Sci.* 1169, 359–362. doi:10.1111/j.1749- 6632.2009.04788.x

Wilson, S. L., Powell, G. E., Brock, D., and Thwaites, H. (1996b). Behavioural difference patients who emerged from vegetative state and those who did not. *Brain Inj.* 10, 509–516. doi:10.1080/026990596124223

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

#### *Received: 30 October 2013; accepted: 05 December 2013; published online: 25 December 2013.*

*Citation: O'Kelly J, James L, Palaniappan R, Taborin J, Fachner J and Magee WL (2013) Neurophysiological and behavioral responses to music therapy in vegetative and minimally conscious states. Front. Hum. Neurosci. 7:884. doi: 10.3389/fnhum.2013.00884*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2013 O'Kelly, James, Palaniappan, Taborin, Fachner and Magee. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# **APPENDIX**

#### **Table A1 | Brain regions and pooled electrodes.**

# Music in disorders of consciousness

#### *Jens D. Rollnik1 \* and Eckart Altenmüller 2*

*2 Institute of Music Physiology and Musician's Medicine (MMM), University of Music, Drama and Media Hannover, Hannover, Germany*

#### *Edited by:*

*Isabelle Peretz, Université de Montréal, Canada*

#### *Reviewed by:*

*Joyce L. Chen, Heart and Stroke Foundation Canadian Partnership for Stroke Recovery, Canada Catherine Y. Wan, Harvard Medical School, USA*

#### *\*Correspondence:*

*Jens D. Rollnik, BDH-Clinic Hessisch Oldendorf, Teaching Hospital of Hannover Medical School (MHH), Institute for Neurorehabilitational Research (InFo), Greitstr. 18-28, 31840 Hessisch Oldendorf, Germany e-mail: prof.rollnik@ bdh-klinik-hessisch-oldendorf.de*

This review presents an overview of the use of music therapy in neurological early rehabilitation of patients with coma and other disorders of consciousness (DOC) such as unresponsive wakefulness syndrome (UWS) or minimally conscious state (MCS). There is evidence that patients suffering from UWS show emotional processing of auditory information, such as listening to speech. Thus, it seems reasonable to believe that music listening—as part of an enriched environment setting—may be of therapeutic value in these patients. There is, however, a considerable lack of evidence. The authors strongly encourage further studies to evaluate the efficacy of music listening in patients with DOC in neurological early rehabilitation. These studies should consider a precise clinical definition and homogeneity of the patient cohort with respect to the quality (coma vs. UWS vs. MCS), duration (rather weeks to months than days) and cause (traumatic vs. non-traumatic) of DOC, a standardized intervention protocol, valid clinical outcome parameters over a longer observation period (weeks to months), monitoring of neurophysiological and vegetative parameters and, if available, neuroimaging to confirm diagnosis and to demonstrate responses and functional changes in the patients' brains.

**Keywords: music therapy, music, coma, unresponsive wakefulness syndrome, minimally conscious state**

# **INTRODUCTION**

Rehabilitation of patients with stroke, hypoxic encephalopathy or severe brain injury is challenging. When considering music as therapy in neurological rehabilitation, one should be aware that there are two distinct groups of patients: First, early rehabilitation patients, frequently comatose (or suffering from other disorders of consciousness). They have a low functional status, high morbidity and are dependent on nursing (Rollnik and Janosch, 2010; Rollnik, 2011, 2014), requiring "passive" therapies (rather listening to music than playing). Second, patients at subsequent stages of rehabilitation, aware, with improving functional status, gaining independence from nursing. These patients require more and more "active" therapies. Along with improvement of consciousness and functional status, their ability to cooperate increases and they may participate in more active therapies (rather playing than listening to music).

The present review focuses on the efficacy of music as a therapeutic tool in early rehabilitation patients with disorders of consciousness (DOC). In Germany, neurological and neurosurgical patients are transferred to specialized early neurological rehabilitation centers, immediately after acute hospital treatment (e.g., brain surgery) (Rollnik and Janosch, 2010; Rollnik, 2011). These centers offer intensive care unit treatment because early rehabilitation patients need to be monitored and are frequently dependent on mechanical ventilation (Rollnik and Janosch, 2010; Rollnik, 2011).

Before reviewing the efficacy of music in early rehabilitation, a precise definition of DOC appears to be useful. First of all, coma is a clinical syndrome characterized by reflex behavior and a disorder of consciousness, no eye opening even to strong painful stimuli may be observed (Bodard et al., 2013). In the unresponsive wakefulness syndrome (UWS)—previously known as vegetative state (VS)—, eyes are open and reflex behavior occurs, but patients are completely unresponsive (e.g., absence of command following) (Bodard et al., 2013). Patients in a minimally conscious state (MCS) can show signs of consciousness, such as command following (even if inconsistent), visual pursuit, localization to noxious stimulation, and appropriate responses to emotional stimuli without being able to functionally communicate (Bodard et al., 2013). It has been suggested to distinguish two groups of MCS patients: Those who show higher-order signs of consciousness as MCS+ (e.g., non-functional communication and command following) from MCS− with only low-level signs of consciousness (e.g., visual pursuit, noxious stimulation localization, appropriate emotional response) (Bodard et al., 2013). These DOC have to be separated from the locked-in syndrome (LIS) which can be found in brain-stem injured patients and is characterized by preserved cognition and eye-coded communication (eye movements) with a lack of any further motor output (Bodard et al., 2013). It has also been suggested to define a functional LIS (fLIS) describing patients with severe brain injury who are behaviorally in an UWS or MCS, but on neuroimaging show better consciousness than expected, with command following and even functional communication) (Bodard et al., 2013). **Table 1** summarizes the clinical features of coma and other DOC.

We know that listening to music influences mood and arousal, which may improve performance on a variety of cognitive tasks (called the "Mozart effect" or "mood and arousal hypothesis")

*1 BDH-Clinic Hessisch Oldendorf, Teaching Hospital of Hannover Medical School (MHH), Institute for Neurorehabilitational Research (InFo), Hessisch Oldendorf, Germany*

*aVisual pursuit, noxious stimulation localization, appropriate emotional response. bCommand following, non-functional communication.*

(Husain et al., 2002). While musical tempo affects arousal, mode (major or minor) may change mood (Husain et al., 2002). There is broad evidence that mood plays a major role in neurological rehabilitation, mood improvement is associated with functional recovery of stroke patients, for instance Bilge et al. (2008). Music listening may be used to facilitate the recovery of cognitive functions and mood after stroke (Särkämö et al., 2008). Listening to self-selected music (at least 1 h daily for 2 months) improved verbal memory, focused attention, depressed, and confused mood (Särkämö et al., 2008). It is reasonable to believe that music listening may be of therapeutic value in neurological rehabilitation of patients without DOC. Improvement of mood and attention seems to be the key component of this "Mozart effect." However, it is unclear whether music listening has any therapeutic effect in DOC. The present review examines if music—through emotional and other processes (e.g., arousal)—might be able to improve consciousness in these patients. To understand its potential it is helpful to focus on some neurobiological aspects of listening in healthy subjects and DOC.

# **NEUROBIOLOGICAL ASPECTS OF LISTENING IN DOC PATIENTS**

Listening to music induces a widespread cortical and subcortical activation of the brain (Altenmüller and Schlaug, 2013). A strongly simplified model of music processing, potential effects of music listening and brain structures involved is presented in **Figure 1**. The model is based on the "mood and arousal hypothesis" which has been described above (Husain et al., 2002).

It has been shown in neuroimaging studies that music listening activates a vast bilateral network of temporal, frontal, parietal, cerebellar and limbic structures related to attention, semantic processing, memory and the motor system (Särkämö et al., 2008; Altenmüller and Schlaug, 2013). Besides speech, music is the most versatile and complex auditory experience integrating input from the auditory, visual, and somatosensory system (Altenmüller and Schlaug, 2013). In addition, the basis and inner surfaces of the frontal lobes, the cingulate gyrus, amygdala, hippocampus and midbrain are involved in the emotional perception of music (Peretz and Zatorre, 2005; Altenmüller and Schlaug, 2013). For a detailed review on neurobiological aspects of music listening, see Peretz and Zatorre (2005), Altenmüller and Schlaug (2013).

So far, music listening seems to be advantageous for alert healthy subjects (Husain et al., 2002). It may stimulate the emotional network and improve attention and cognitive performance. But how about patients with DOC, do they respond to auditory or any other stimulation at all? Recently, it has been shown that patients with UWS do respond to pain cries of other people (Yu et al., 2013).  We know from other neuroimaging studies (functional magnetic resonance imaging fMRI) that UWS patients may have cortical responses to language stimulation (Coleman et al., 2007). It has even been demonstrated that familiar speakers evoked significantly stronger activation in the limbic system (amygdala) than unfamiliar speakers and neutral phrases (Eickhoff et al., 2008). These findings indicate that listening to familiar sounds may not only induce cognitive but also emotional processing in UWS (Eickhoff et al., 2008). Visual stimuli are emotionally processed in UWS patients too (Sharon et al., 2013). Patients displayed more pronounced limbic and cortical activations elicited by presentation of familiar than nonfamiliar faces (Sharon et al., 2013). The fact that limbic and cortical areas have been activated supports the hypothesis that these responses might be a sign of "heightened awareness." The finding of brain responses to emotional stimuli in patients with UWS is of importance because the quality of awareness cannot be evaluated without addressing the question of whether cognitive processes also elicit a subjective emotional experience (Sharon et al., 2013). Emotion and consciousness are considered to be inseparable as each conscious state is endowed with some form of emotion, for a detailed review, see Berkovich-Ohana and Glicksohn (2014). Emotion is regarded as a key component of our experiencing of environment, including our sense of self, serving as an ever-present basic constitute of the quality of human consciousness (Sharon et al., 2013).

# **THERAPEUTIC APPROACHES IN PATIENTS WITH DOC (MULTISENSORY STIMULATION)**

It has been hypothesized that comatose patients might suffer from a condition of "environmental deprivation" (LeWinn and Dimancescu, 1978). This condition could be improved by environmental inputs through all five sensory pathways enhancing the rate and degree of recovery from coma (LeWinn and Dimancescu, 1978). The idea of "enriched environment" inspires therapeutic approaches using sensory stimulation in neurological early rehabilitation (Lippert-Grüner et al., 2007).

A Cochrane systematic review focused on sensory stimulation of brain-injured patients with coma or UWS (Lombardi et al., 2002). The authors identified only three studies which met the well-defined inclusion criteria (coma or UWS patients, brain

injury of traumatic or non-traumatic origin, randomized controlled and non-randomized controlled trials with concurrent controls, comparing sensory stimulation with standard rehabilitation): In one randomized controlled study (RCT), only seven comatose patients (admitted to the ICU within 24 h after traumatic brain injury due to road traffic accident) in the intervention group underwent a multisensory stimulation of all five senses (olfactory, visual, auditory, gustatory, tactile) 20 min per day during their stay on the ICU (medium stay 8.1 days) (Johnson et al., 1993). There was no such stimulation in the control group. Outcome measures were Glasgow Coma Scale (GCS), ventilation, brain stem reflexes, spontaneous eye movements, skin conductance and heart rate assessed 20 min pre and post multisensory stimulation. In a second controlled clinical trial (CCT) with *n* = 30 comatose head injury patients (at least 2 weeks from the trauma), the treatment consisted of 45 min (twice a day) visual, auditory, olfactory, cutaneous, kinesthetic and oral stimulation (six modalities) for a 1–3 months period (Kater, 1989). Outcome was defined as level of cognitive functioning (LCF) measured 2 weeks and 3 months after the trauma. In the third study (CCT), 12 traumatic brain-injured comatose patients (4–12 days after trauma) in the intervention group received 60 min (once or twice a day for up to 4 weeks) multisensory stimulation (visual, auditory, olfactory, tactile, gustatory, kinesthetic, and vestibular) (Mitchell et al., 1990). Outcome measures were GCS and total duration of coma. None of the three studies found any evidence of a therapeutic effect of multisensory stimulation programs in comatose brain injury patients (Lombardi et al., 2002). Despite these negative findings, some limitations of these studies need to be addressed: The duration of coma was quite short in all three studies while early rehabilitation patients frequently suffer from longer lasting DOC, such as UWS or MCS. Further, intervention (intensity and quality of multisensory stimulation) differed substantially between the three studies.

A more recent review focusing on MCS patients after traumatic brain injury included other stimulation techniques such as transcranial magnetic and deep brain stimulation (Lancioni et al., 2010). There is broad evidence that repetitive transcranial magnetic stimulation (rTMS) as well as deep brain stimulation (DBS) may be used for therapeutic purposes and that both types of stimulation interfere with cortical functions (Däuper et al., 2002; Rollnik et al., 2003). One comatose patient was treated with rTMS of the right dorsolateral prefrontal cortex (DLPFC) daily over 6 weeks (thirty sessions with 300 trains) demonstrating slight improvements of awareness (Louise-Bender Pape et al., 2009). The DLPFC is also the focus of rTMS in patients suffering from major depression to improve mood, fatigue and activity (Chen et al., 2013). Given that the thalamus plays a major role in consciousness and has been referred to as the gateway of sensory input, a bilateral DBS of the central thalamus has been tried in a few comatose patients, with only moderate effects (Yamamoto et al., 2005; Schiff et al., 2007). The review also identified more recent case reports focusing on multisensory stimulation in MCS or UWS describing the case of a 24-year old women close to MCS (Canedo et al., 2002).  By the eighth week she started to respond to tactile and auditory stimuli, by the tenth week, she started to communicate (Canedo et al., 2002). In another case, a 20-year old women with UWS was treated with a multisensory stimulation program (visual, auditory, tactile, gustatory and olfactory stimulation) 50 days after brain damage for 63 days (2-h sessions per day) (Bekinschtein et al., 2005). Soon after beginning of the program, the woman made some progress, e.g., following of simple commands (Bekinschtein et al., 2005). These case reports are only anecdotal and cannot replace controlled studies.

Multisensory stimulation is also the basis of the so-called "basal stimulation" which has been established in many German intensive care and early rehabilitation facilities (Menke, 2006). It comprises multisensory stimulation during the nursing process, e.g., somatosensory (initially touching hands, arms, shoulders or chest, body washing), vestibular (moving the head), oral (smell and taste of favorite food), vibratory (vibration of the chest or using an electric shaver), auditory (listening to familiar sounds and music), tactile (putting well known things like a tooth brush or a cup into the patient's hand) and visual stimuli (presenting pictures of relatives). There are, however, no controlled studies available. Basal stimulation is derived from the concept of enriched environment (LeWinn and Dimancescu, 1978). Interventions are not as standardized as in the studies mentioned above with respect to intensity or quality of stimulation (Kater, 1989; Mitchell et al., 1990; Johnson et al., 1993) and are a part of the nursing process.

Several pharmacological interventions have also been studied. The most promising results could be observed with the dopamine releaser amantadine in traumatic brain injury (TBI) patients (Wheaton et al., 2009). It is well known from Parkinson therapy that levodopa improves alertness (Bliwise et al., 2012). In a meta-analysis, 11 pharmacological treatments were investigated by 22 clinical studies, comprising 6472 TBI patients in the treatment groups and 6460 TBI controls. Only one dopamine releaser (amantadine) and 1 bradykinin antagonist (CP-0127 [Bradycor]) produced marked treatment benefits for a single measure of arousal (Glasgow Coma Scale) (Wheaton et al., 2009).

# **MUSIC THERAPY IN NEUROLOGICAL EARLY REHABILITATION PATIENTS WITH DOC**

According to the G-DRG (German Diagnosis Related Groups) system, music therapy may be a part of the therapeutic concept in neurological early rehabilitation1 . Music therapy in neurological rehabilitation has a long tradition in Germany (Muthesius, 2003). Although controlled studies are lacking, about 29% of neurological rehabilitation facilities in Germany have reported to offer music therapy (Jochims et al., 2003). However, most of these therapies refer to the use of live music and singing, for instance, involving the patient as an active part. This form of music therapy makes more sense in aware, conscious patients, not in neurological early rehabilitation subjects suffering from DOC.

However, it has been suggested that music therapy could be used to "communicate" with individuals suffering from DOC and motor disabilities (Magee, 2007). As the auditory modality has been found to be particularly sensitive in identifying responses indicating awareness, a standardized protocol for intervention or measuring patient responses within the music therapy setting has been developed, the so-called "music therapy assessment tool for low awareness states" (MATLAS) (Magee, 2007) and its advanced version "music therapy assessment tool for awareness in disorders of consciousness" (MATADOC) (Magee et al., 2014). MATLAS and MATADOC may be used for MCS or UWS patients and comprise items which rate behavioral responses to sensory stimulation (Magee, 2007; Magee et al., 2014). The 14 items of the MATADOC are: "responses to visual stimuli, responses to auditory stimuli, awareness of musical stimuli, response to verbal commands, arousal, behavioral response to music, musical response, vocalization, non-verbal communication, choicemaking, motor skills, attention to task, intentional behavior, emotional response" (Magee et al., 2014). As an example, the item awareness of musical stimuli is rated from 0 ("no observed response") to 5 ("showed consistent interactive responses within musical exchange") (Magee et al., 2014). The MATADOC has been validated in a small study enrolling only *n* = 21 DOC patients after traumatic, hypoxic-ischemic, hemorrhagic brain damage or viral infection (Magee et al., 2014). In a prospective, non-controlled study with repeated measurements, internal consistency, inter-rater and test-retest reliability and dimensionality were examined (Magee et al., 2014). The five-item scale showed an internal reliability of α = 0*.*76 (Magee et al., 2014). Corrected item-total correlations were all above 0.45, inter-rater intra-class correlations (ICCs) ranged from 0.65 to 1.00 and intra-rater ICCs from 0.77 to 0.90 (Magee et al., 2014). The study showed that diagnostic outcomes had 100% agreement with a validated external reference standard (Magee et al., 2014). However, validity and reliability of the MATADOC should be examined enrolling a larger and homogenous cohort of patients.

Active music therapy ("playing") has been tried in severely brain-injured patients who were already able to cooperate to a certain extent (Formisano et al., 2001). Therapy consisted of musical improvisation between patient and therapist by singing or by playing different musical instruments, according to the vital functions, the neurological condition and motor abilities of the patients (Formisano et al., 2001). *n* = 34 brain-injured patients with a mean coma duration of 52 days and a mean interval from coma onset to the beginning of rehabilitation of 154 days had been enrolled (Formisano et al., 2001). Results showed a significant improvement of the collaboration of the severely brain-injured patients and a reduction of undesired behaviors such as inertia or psychomotor agitation (Formisano et al., 2001).

1http://www.g-drg.de/cms/G-DRG-System\_2014

In one case report, music therapy has been tried in a cerebral hypoxia patient whose diagnosis of UWS was contradicted by purposeful responses within the music therapy assessment, changing the diagnosis to MCS (Magee, 2005). This case illustrates the potential role of music therapy in assisting with diagnosis of patients with DOC (Magee, 2007; Magee et al., 2014). Thus, music therapy might provide a medium which does not rely on language, is non-evasive and elicits emotional responses in these patients (Magee, 2005).

A recently published study applied preferred music exposure in a larger cohort of patients with either UWS or MCS compared to healthy controls (O'Kelly et al., 2013). The neurophysiological and behavioral study was undertaken comparing electroencephalogram (EEG), heart rate variability, respiration and behavioral responses of 20 healthy subjects with 21 individuals with UWS or MCS (O'Kelly et al., 2013). Healthy subjects and patients were presented with live preferred music and improvised music entrained to respiration (procedures typically used in music therapy), recordings of disliked music, white noise, and silence (O'Kelly et al., 2013). ANOVA tests indicated a range of significant responses across healthy subjects corresponding to arousal and attention in response to preferred music including concurrent increases in respiration rate with globally enhanced EEG power spectra responses across frequency bandwidths (O'Kelly et al., 2013). Whilst physiological responses were heterogeneous across patient cohorts, significant *post hoc* EEG amplitude increases for stimuli associated with preferred music were found for frontal midline theta in six UWS and four MCS patients and frontal alpha in three UWS and four MCS subjects (O'Kelly et al., 2013). Furthermore, behavioral data showed a significantly increased blink rate for preferred music within the UWS cohort (O'Kelly et al., 2013). Two UWS patients showed concurrent changes across measures indicative of discriminatory responses to both music therapy procedures (O'Kelly et al., 2013). The results also suggested that music may be used to distinguish MCS from UWS (O'Kelly et al., 2013). However, due to the heterogeneity of the patient group, the study may rather be considered as a case study than a systematic investigation (O'Kelly et al., 2013).

# **DISCUSSION**

Music therapy, in particular music listening, may be used in patients with DOC as part of an enriched environment setting during neurological early rehabilitation (Jochims et al., 2003; Muthesius, 2003; Menke, 2006; Lippert-Grüner et al., 2007; Magee, 2007; O'Kelly et al., 2013). It has been shown that music listening induces a broad activation of several complex neuronal networks and elicits emotional processes in the brain (limbic system) in alert subjects (Peretz and Zatorre, 2005; Altenmüller and Schlaug, 2013). It is evident from neuroimaging studies that even patients suffering from UWS (previously known as vegetative state) show emotional processing of auditory or visual information (Coleman et al., 2007; Eickhoff et al., 2008; Yu et al., 2013). Thus, it seems reasonable to believe that music listening, in particular listening to familiar music [as performed in the "basal stimulation" concept (Menke, 2006)], may be a powerful stimulator in the therapy of patients suffering from DOC. However, there is a considerable lack of evidence. Some controlled studies on multisensory stimulation are available, but they could not prove its efficacy in UWS or coma (Lombardi et al., 2002). In addition, there is only limited evidence that music therapy may be applied when subjects regain consciousness as a means of non-verbal communication or as a diagnostic tool to distinguish between UWS and MCS (Magee, 2005, 2007; Magee et al., 2014). Compared to patients with DOC, there is by far more evidence for the efficacy of "active" music therapy in alert neurological rehabilitation patients, in particular in motor rehabilitation (Altenmüller and Schlaug, 2013).

The therapeutic potential of music therapy in patients with coma, UWS or MCS in neurological early rehabilitation merits further investigation. Currently, there is (like with many other interventions in neurological rehabilitation) a considerable lack of evidence. Proof-of-principle and open-label studies, followed by controlled trials on this topic are strongly encouraged to improve the evidence-base of music as a therapeutic tool in neurological early rehabilitation patients with DOC.

Future studies should consider the following:

# **REFERENCES**

awareness in patients with disorders of consciousness. *Neuropsychol. Rehabil.* 24, 101–124. doi: 10.1080/09602011.2013.844174

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 20 March 2014; accepted: 16 June 2014; published online: 03 July 2014.*

*Citation: Rollnik JD and Altenmüller E (2014) Music in disorders of consciousness. Front. Neurosci. 8:190. doi: 10.3389/fnins.2014.00190*

*This article was submitted to Auditory Cognitive Neuroscience, a section of the journal Frontiers in Neuroscience.*

*Copyright © 2014 Rollnik and Altenmüller. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# **HUMAN NEUROSCIENCE**

# Vowel generation for children with cerebral palsy using myocontrol of a speech synthesizer

# *Chuanxin M. Niu1, Kangwoo Lee2, John F. Houde3 and Terence D. Sanger 2,4,5\**

*1 Department of Rehabilitation, School of Medicine, Ruijin Hospital, Shanghai Jiao Tong University, Shanghai, China*

*2 Department of Biomedical Engineering, University of Southern California, Los Angeles, CA, USA*

*3 Department of Otolaryngology - Head and Neck Surgery, University of California, San Francisco, San Francisco, CA, USA*

*4 Biokinesiology, University of Southern California, Los Angeles, CA, USA*

*5 Neurology, University of Southern California, Los Angeles, CA, USA*

#### *Edited by:*

*Antoni Rodriguez-Fornells, University of Barcelona, Spain*

#### *Reviewed by:*

*Antoni Rodriguez-Fornells, University of Barcelona, Spain Pavel Lindberg, FR3636 Neurosciences Centre National de la Recherche Scientifique, Université Paris Descartes; U894 Inserm, France*

#### *\*Correspondence:*

*Terence D. Sanger, Department of Biomedical Engineering, Viterbi School of Engineering, University of Southern California, 1042 Downey Way, Los Angeles, CA 90089, USA e-mail<EMAIL_ADDRESS>
For children with severe cerebral palsy (CP), social and emotional interactions can be significantly limited due to impaired speech motor function. However, if it is possible to extract continuous voluntary control signals from the electromyograph (EMG) of limb muscles, then EMG may be used to drive the synthesis of intelligible speech with controllable speed, intonation and articulation. We report an important first step: the feasibility of controlling a vowel synthesizer using non-speech muscles.  EMG signals are filtered using a non-linear Bayesian filtering algorithm that provides the high bandwidth and accuracy required for speech tasks. The frequencies of the first two formants determine points in a 2D plane, and vowels are targets on this plane. We focus on testing the overall feasibility of producing intelligible English vowels with myocontrol using two straightforward EMG-formant mappings. More mappings can be tested in the future to optimize the intelligibility. Vowel generation was tested on 10 healthy adults and 4 patients with dyskinetic CP. Five English vowels were generated by subjects in pseudo-random order, after only 10 min of device familiarization. The fraction of vowels correctly identified by 4 naive listeners exceeded 80% for the vowels generated by healthy adults and 57% for vowels generated by patients with CP. Our goal is a continuous "virtual voice" with personalized intonation and articulation that will restore not only the intellectual content but also the social and emotional content of speech for children and adults with severe movement disorders.

**Keywords: electromyography (EMG), speech impairment, speech synthesis, Cerebral Palsy, non-linear Bayesian filtering**

# **INTRODUCTION**

Children with brain injury in the perinatal period, usually referred as Cerebral Palsy (CP), are often left with a combination of weakness, spasticity, dystonia, dyspraxia, and other motor disorders (Cans, 2000). These disorders in CP result from dysgenesis or injury to developing motor pathways in many components of central nervous system, including the cortex, basal ganglia, thalamus, cerebellum, brainstem, central white matter, or spinal cord. Most patients with CP struggle to maintain limb postures or perform voluntary movements due to increased muscle tone or weakness. Concomitant issues in emotional and behavior are also common in CP (Bax et al., 2005). In the most severe cases, the motor disorders in CP can prevent all meaningful voluntary movements of the patient (Sanger et al., 2003), and more than 80% of children with dyskinetic or tetraplegic CP suffer from speech impairments (Odding et al., 2006). While new therapies such as stem cells hold great promise for the treatment of early brain injuries, full restoration of speech for children with CP remains unlikely.

We use the term "language" as the content of human communication, either spoken or written, consisting of the use of words in a structured and conventional way (Oxford English Dictionary); while the term "speech" as the motor function required for vocalizing human language. It is common that patients with CP may have normal language ability (via writing or assistive devices) but are completely incapable of producing speech. Our ultimate goal is to allow children with CP to create intelligible English speech using a portable synthesizer controlled in real-time by body signals such as muscle activity. For children with CP who have preserved language skills but impairment of control of vocal tract musculature, our aim is to create a "virtual voice" to enable them to express language using other muscles for which they may have better voluntary control. The primary engineering challenges for restoring speech are threefold: (1) extracting controllable signals from a diseased neurological system, (2) using these signals to rapidly synthesize sounds resembling human speech, and (3) providing personalized ways of speaking. The third challenge is particularly important for children, since language is used by children for social interaction and emotional communication, much more than for declarative statements (Wing and Gould, 1979; Van Lancker et al., 1989; Patel and Schroeder, 2007). It has been shown that muscle patterns in children with CP are distorted by co-contraction (Young et al., 2011a,b), signal-dependent noise (Sanger et al., 2005; Sanger, 2006), and weakness, which reduce the speed and accuracy of control and lead to a limited effective bandwidth of the voluntary signal (Sanger and Henderson, 2007). Therefore, the challenge is to allow sufficient flexibility in the voice output despite limited bandwidth of the voluntary control signals. We thus seek to maximize the controllability of the produced speech, while minimizing the need for precise control of muscles.

Myocontrol, the control of prosthetic devices using surface electromyographic (EMG) signals, holds great promise for speech production. Previous studies provide the essential support that EMG from limb muscles provides an excellent signal that children with CP can often control (van der Heide et al., 2004). The flexibility and accuracy of muscle activity could potentially approach the quality and flexibility required in speech control. Furthermore, signal processing could potentially transform abnormal muscle patterns of children with CP into much more precisely controllable signals with significantly better performance. In speech science, EMG has been adopted in various studies including the real-time recognition of impaired speech (Jorgensen et al., 2003; Jou et al., 2007); EMGs from neck strap muscles have also been successfully used for driving an artificial larynx in patients who receive laryngectomy (Stepp et al., 2009). In these applications, however, patients have normal neural control and need only bypass abnormal muscles or biomechanics. But for children with CP and absent speech, the neural control itself is impaired so that EMG from the neck muscles is not expected to function for control of speech. Compared to muscles around the neck and vocal tract, limb muscles are defined more clearly and therefore easier to attach surface EMG recording electrodes. Here we will leverage the fact that significant amount of voluntary control can still be reconstructed from limb muscles of children with CP.

Successful application of myocontrol has been limited by the variability in raw EMG signals and the consequent poor quality of estimates. Most existing methods for EMG processing stem from the idea that EMG can be treated as an amplitude-modulated signal with band-limited noise (Hogan and Mann, 1980a,b). Based on this perspective, a procedure of high-pass, rectify and low-pass filtering has been developed and widely adopted for EMG processing (Evans et al., 1984; Merletti, 1999). Using this procedure, however, it is difficult to obtain online control signals that are both responsive and smooth, which is extremely critical for realtime applications such as restoring movement functions. With our recently developed techniques of non-linear Bayesian filtering (Sanger, 2007), we are able to extract high-bandwidth, lowlatency control signals from raw surface EMG. The technique has been applied to studies of biofeedback (Bloom et al., 2010) and motor control (Young et al., 2011b) for children with dyskinetic CP. It provides another essential support for using myocontrol in speech production for CP. See Materials and Methods for details.

In addition to the problem of control, speech restoration for children with CP also requires a technology that can synthesize speech-like voices with a small number of control parameters, and yet still allow for flexible voice output. The technology of speech synthesis has been of interest for more than 200 years (von Kempelen, 1791), and it has evolved into three categories of synthesis approaches: concatenative, articulatory and formant-based. The simplest way to produce synthetic speech is to play back pre-recorded pieces of natural speech following pre-determined concatenations. This *concatenative* approach produces very high quality of voice in text-to-speech applications (Taylor et al., 1998), but pre-recorded voices are usually unavailable from children with CP. Even though in some cases the patients' voice can be pre-recorded, such systems usually require accurate selection of speech elements; thus the control task may be harder than necessary.

As a continuous alternative to concatenative synthesis, *formant-based* synthesis uses relatively few control parameters and allows for full control of intonation and inflection (Klatt, 1980). In the case of vowels, for instance, it has been suggested that frequencies of the lowest two formants (F1 and F2) are sufficient for vowel intelligibility, while formant bandwidths and other parameters are less important (Stevens, 2000; Ladefoged and Johnstone, 2010). Therefore, in the case of vowel synthesis, it becomes possible to reduce the number of parameters for control in CP using a low dimensionality of control inputs. In a recent study (Larson et al., 2013), the investigators succeeded in using two channels of surface EMG from orbicularis oris muscles to control F1 and F2 for vowel synthesis. The EMGs were categorized into pre-defined syllables and therefore would not provide continuous auditory feedback. In this study, because we convert EMGs moment-by-moment to continuously varying formant frequencies, our system will produce continuous voice even though the target phoneme has not yet fully developed. As a result, even though real-time myocontrol allows significant flexibility that could potentially lead to personalized intonation and articulation, the very first step should be testing the intelligibility of vowels using myocontrol.

Ultimately, the goal of myocontrolled speech synthesis is to generate both vowels and consonants in continuous speech. In reaching this goal, generation of consonants presents several important challenges. First, many consonants (e.g., plosives and fricatives) are rapid, dynamically changing acoustic events. Such consonants thus require high bandwidth in the control signal to produce the fast temporal transitions. Second, many consonants involve controlling more acoustic features than just F1 and F2, e.g., frication noise for fricatives and F3 for /r/ (Espy Wilson, 1992; Heinz and Stevens, 2005). Our technique of myocontrolled vowel synthesis can be directly used for producing liquid consonants (or semi-vowels) due to the acoustic similarity between vowels and liquid consonants (Ladefoged and Johnstone, 2010). For other consonants that have dynamics and richer acoustic features (plosives, fricatives, etc.), we expect that certain dimensionality reduction methods will be required to simplify control of the formant trajectory and sound production process such that the consonant production can be mapped to activation of a limited number of muscles. The focus of the current study, however, is to answer the initial feasibility question of whether vowel synthesis can be controlled in real-time using non-speech muscles.

This paper first introduces the design and methodology of real-time vowel generation using EMG from non-speech muscles. Next, we describe testing the quality of myocontrolled speech synthesis in 10 healthy adults and 4 patients with dyskinetic CP. The two populations are not age matched since our purpose is validation of myocontrolled vowel-synthesis in a wide range of subjects. We hypothesize that for healthy adults the vowels synthesized via myocontrol will be recognized by naïve listeners with statistical significance; in patients with CP, the intelligibility is expected to be lower than healthy adults, but the listeners should still be able to identify the vowels with statistical significance. The fraction of vowels correctly identified by 4 naive listeners is calculated to test the feasibility of intelligible speech restoration using myocontrol. Preliminary results have been reported as an abstract (Niu et al., 2013). Our main innovations are (1) using non-linear Bayesian filtering to extract high-bandwidth, low variability control signals and (2) mapping the vowel generation to moving a cursor on a 2D plane, which is intuitive even for children with CP.

# **MATERIALS AND METHODS**

All research participants signed written informed consent to participate as well as U.S. Health Information Portability and Accountability Act (HIPAA) authorization for use of medical and research records, according to approval of University of Southern California Human Subjects Review Board. Both healthy subjects and patients with CP were recruited. Healthy subjects without known neurological disease were recruited as control. Patients with CP were required to have normal cognition such that they could understand the experimental instructions, and at least one side of their upper extremities should show motor deficits. Ten healthy adult subjects (7 male, 3 female, age 21–29 years), four subjects with dyskinetic CP (2 male, 2 female, age 12–20 years) and four naïve listeners (2 male, 2 female, age 25–30 years) were recruited. The clinical diagnosis and motor deficit analyses of 4 patients with CP are summarized in **Table 1**.

### **SYNTHESIZING VOWEL IN REAL-TIME BY CONTROLLING FORMANTS**

The overall design of our myocontrolled formant-based synthesis platform is shown in **Figure 1**. Raw muscle EMGs are monitored using surface EMG electrodes (Biometrics SX230) attached over the belly of the chosen muscle. EMG signals are sampled at 1 kHz and processed online using a non-linear Bayesian filter. The screen provides visual feedback of the muscle activity, and also shows the level of activity required for a given vowel. Both auditory and visual feedback are continuously provided during experimental tests.

In human speech, vowel production can be described as a broadband sound source generated at the glottis (i.e., the glottal source) that is filtered by the vocal tract (e.g., the pharynx, tongue, palate, teeth, lips) (Titze, 2000). The glottal source is periodic with a fundamental frequency F0 that is lower for male voices and higher for female voices. The vocal tract has several resonances that speakers change by moving the articulators, especially the tongue. These resonances filter the glottal source, so that the output speech spectrum has harmonics of the pitch modulated by broad peaks of the vocal tract resonances called formants. The frequencies of the lowest two formants (F1 and F2) determine which vowel is being spoken, while

**FIGURE 1 | Design of the system (top) and a snapshot of the working set-up (bottom).** The EMG electrodes are placed on the flexor pollicis brevis muscles of both hands. Pinching the thumb and index finger will activate the EMG to move both the screen cursor and formant frequency.

#### **Table 1 | Diagnosis of four patients with dyskinetic CP.**

higher formants usually reflect non-phonetic speaker characteristics (e.g., vocal tract length) (Stevens, 2000; Ladefoged and Johnstone, 2010). Thus, in formant-based speech synthesis, we can artificially synthesize different vowels by tuning the lowest two resonances of a filter driven by a broadband periodic (e.g., impulse trains) acoustic source (Parsons, 1987; Stevens, 2000).

In this project, we adapted an open-source formant-based synthesizer (Wavesurfer, KTH Royal Institute of Technology, Sweden) such that F1 and F2 can be directly controlled over an ethernet link in real-time. The reason of selecting a welldeveloped synthesizer is to keep our goal modest, such that if the feasibility test fails it is likely due to our myocontrol design but not the synthesizer. The source code of the Wavesurfer synthesizer is located at http://www*.*speech*.*kth*.*se/wavesurfer/formant/. The adapted version is available upon request from the authors of this paper.

#### **NON-LINEAR BAYESIAN FILTERING OF EMG**

Under the assumption that the rectified EMG signal results from random depolarization events of multiple muscle fibers, the average amplitude of rectified EMG in a small time window will be proportional to the number of depolarization events during that time. One representative non-linear model of such depolarization events is a non-homogeneous Poisson process with n events per second, controlled by the muscle drive *x*:

$$P\left(EMG \mid \text{x}\right) \propto \frac{\text{x}^n e^{-\text{x}}}{n!} \tag{1}$$

where the driving signal x is unknown and thus must be estimated by the filtering algorithm.

Since the drive signal x is determined by voluntary behavior, we model this behavior as a jump-diffusion process that includes the possibility of gradual changes in muscle drive with occasional sudden jumps at the time of force onset or offset:

$$d\mathfrak{x} = \mathfrak{a}\ (dW) + (U - \mathfrak{x})\,dN\_{\beta} \tag{2}$$

where the stochastic differential equation is to be interpreted in the Ito sense, *dW* is the differential of a standard Brownian motion, *dNβ* is the differential of a counting process with rate *β* events per second, and x is a random variable uniformly distributed on [0,1]. Equation 2 models the gradual drift of x determined by the drift rate *α*, and rare jumps that occur at transitions in the counting process *dNβ*, at which times the value of *x* will change to a new random value drawn from the distribution of *x*.

Using Equation 1, we can derive a posterior estimate for *x* based on measurement of EMG and a prior estimate of the density of *x* using Bayes' rule. Between measurements of EMG, *x* will change according to the stochastic differential equation Equation 2, and the distribution of *x* will propagate forward in time according to a corresponding partial differential equation similar to the Fokker-Planck equation. After each measurement, the maximum a posteriori estimate of *x* is calculated using Bayes' rule, and this provides the output estimate from the algorithm at that time.

An example of non-linear Bayesian filtering applied to rectified EMG is shown in **Figure 2**. As can be seen, the output

from non-linear Bayesian filter has captured the rapid changes in force, but still with sufficient smoothness and dynamic range. For details about this EMG model and filtering algorithm refer to Sanger (2007). There exist a variety of EMG filtering algorithms that may all allow myocontrolled speech synthesis. We adopt nonlinear Bayesian filtering due to its benefits of fast time-domain response and low variability. Detailed comparison between EMG filtering algorithms for speech synthesis may be tested if the basic feasibility is proven.

#### **FORMANT PLACEMENT IS A 2D REACHING MOVEMENT**

The five common English vowels and the frequencies of their lowest two formants (F1 and F2) are shown in **Table 2** according to Hillenbrand et al. (1995) On a plane defined by F1 and F2, each of the five vowels represents a point, as can be seen in **Figure 3A**. The mapping between vowel and locations on a 2D plane is visualized and shown to the subject for learning the use of myocontrolled vowel production. It is important to realize that perfect accuracy is not required; vowels will be recognizable in a region near the targets, and the tolerance for error will be greater when vowels are incorporated as part of a word or phrase in which meaning can be inferred.

To familiarize with the task without the need to learn myocontrol simultaneously, we first asked all subjects to repetitively move between /i/ and /A/ using the finger to swipe across the surface of a touch-screen. To demonstrate the feasibility of placing F1 and F2 using myocontrol, we asked the same subject to move the cursor on the formant plane using flexor pollicis brevis EMGs from both hands. The EMG signals were transformed to F1 and F2 using the Cartesian transform (explained below). See Results for details.

#### **CONTROL STRATEGIES FOR PLACING FORMANTS**

Since F1 and F2 frequencies represent points on a 2D surface, filtered EMGs must be mapped to positions within the same surface. Due to the non-negativity of filtered EMG, we chose two straightforward position mappings either in Cartesian coordinates within the first quadrant, or polar coordinates within the entire 2D surface. Cartesian transform is given by:

$$F\_1 = \left(F\_1^{hi} - F\_1^{lo}\right)EMG\_1 + F\_1^{lo} \tag{3}$$

$$F\_2 = \left(F\_2^{li} - F\_2^{lo}\right) EMG\_2 + F\_2^{lo} \tag{4}$$

**Table 2 | Average formant frequencies (Hz) for U.S. English vowels produced by 45 males (from Hillenbrand et al., 1995), with the corresponding EMG magnitude under Cartesian and polar transform.**

*\*High muscle contraction level required.*

where *Fhi* 1 , *Flo* 1 , *Fhi* 2 and *Flo* 2 are the boundary frequencies chosen such that all vowels listed in **Table 2** are reachable with normalized *EMG*1*, EMG*2 ∈ [0*,* 1]. When using the Cartesian transform, a dot cursor representing the current coordinate of (*F*1*, F*2) was shown on the screen (**Figure 3B**).

Another way of transforming non-negative normalized EMG into formant space is to use polar coordinates. The transform is given by:

$$F\_1 = \frac{F\_1^{hi} + F\_1^{lo}}{2} + K\_{F1} EMG\_1 \sin \theta\_E \tag{5}$$

$$F\_2 = \frac{F\_2^{hi} + F\_2^{lo}}{2} + K\_{F2} EMG\_1 \cos \theta\_E \tag{6}$$

$$
\theta\_E = 2\pi E \text{MG}\_2 + \frac{\pi}{2} \tag{7}
$$

where *KF*1 and *KF*2 are scaling factors for adjusting the size of the ellipse defined by the transform. When using the polar transform, the visual feedback was provided by a vector line, of which the magnitude was primarily driven by *EMG*1 while the angle driven by *EMG*2 (**Figure 3C**). In the current setup we choose that when *EMG*1 = 1*.*0 and *EMG*2 = 0*.*0 the vector is pointing to the negative direction of *F*1.

The normalized EMGs required for each vowel are listed in **Table 2**, calculated according to Equations (3–7). When using Cartesian transform, the cursor starts from the top-right corner of the screen; when using polar transform, the cursor starts from the center of the screen. It is worth noting that the origin of the coordinate system directly affects the magnitude of *EMG*1 and *EMG*2 required for each vowel. Therefore, reaching certain targets may require an EMG level higher than 60% of maximum voluntary contraction (MVC, see asterisk terms in **Table 2**), which has been known to cause fatigue (Bigland-Ritchie et al., 1981). Simple increasing the gain of non-linear filter will reduce the maximum EMG level, but this will reduce the accuracy when producing low

level EMGs. Therefore, one goal in the future is to minimize the EMG amplitude requirement while maintaining the accuracy at low EMG levels.

## **EXPERIMENTAL PROCEDURE OF RANDOMIZED VOWEL GENERATION**

Subjects were seated in front of a laptop computer, with both hands resting on the knees. Activities from bilateral upper-limb muscles were monitored using two surface EMG electrodes, one for each hand. For healthy subjects with normal hand and arm function, signals from brachioradialis (BR, an elbow flexor) and flexor pollicis brevis (FPB, a thumb flexor) muscles were separately tested for myocontrol; for subjects with dyskinetic CP who had trouble using hand muscles, only BR was tested. In FPB conditions, subjects were instructed to pinch their thumb and index finger to activate the muscle; in BR conditions, the muscles were activated by the subjects lifting their forearms against the desk. Before the experiment, raw EMG amplitude was normalized to MVC (maximum voluntary contraction) determined for each electrode using the highest activation in any 250 ms period during five 5-s attempts, with visual feedback and encouragement.

All subjects received two test sessions (Cartesian transform and polar transform) in random sequence, illustrated in **Figure 4**. During each session, the subject was first given a 10-min practice to familiarize with vowel generation. During practice, the locations in formant space of all 5 vowels were displayed on the screen, and continuous audio feedback was provided. Subsequently, subjects were required to produce a series of 25 movements to targets in formant space corresponding to random vowels containing 5 occurrences of each common English vowel (/A/, /e/, /i/, / c /, /u/). In each trial, the target in formant space was shown, and the subject was allowed 5 s to produce the target vowel. A 2-s break was given between each trial.

The 5-s sound clips recorded from each trial were played to four naive listeners (native U.S. English speakers) to identify the attempted vowel. At the beginning of each clip, the initial sound represent the two formants when *EMG*1 = 0*, EMG*2 = 0; the initial sound then transitioned to the steady-state sound produced by each subject. The listener was informed that during the 5 s the subject was trying to speak one vowel out of five possibilities. The listener was forced to make a choice by inferring from the steady-state sound and ignoring the transition. The fraction of vowels correctly identified by the listener, given by *Ncorrect/Ntotal*, was calculated to show the overall quality of myocontrolled vowel generation.

## **STATISTICAL ANALYSIS**

The responses from each of the 4 naive listeners are analyzed using Cohen's κ-test (Viera and Garrett, 2005) to determine whether the synthesized vowels are intelligible. The purpose of Cohen's κ-test is to identify whether there is a difference in responses between the naive listener and an imaginary perfect listener who has exact knowledge of which vowel was spoken. In our case the listener was required to pick an answer from 5 candidate vowels, therefore an imaginary perfect listener would achieve 100% fraction of correctness, while a random guesser would be expected to achieve, on average, 20% correctness. In order to test the agreement among 4 listeners, we also analyzed their responses using Fleiss's κ-test (Fleiss, 1971).

# **RESULTS**

We first asked the subjects to repetitively move between /i/ and /A/ using the finger to swipe across the surface of a touch-screen. The trajectories resulting from the finger movements of a healthy subject are displayed in **Figure 5A**. In agreement with previous studies (Atkeson and Hollerbach, 1985; Uno et al., 1989), the trajectories of finger swiping movements were clustered around a straight line, even though the subject was free to choose any possible trajectory. Once the trajectory shown in **Figure 5A** was connected to our speech synthesizer it immediately produced the vowel sequence of "ee-ah-ee-ah." It is hence suggested that when using formant-based synthesis, producing a target vowel can be

equivalently interpreted as reaching to a target position within a 2D plane.

We then asked the same subject to move the cursor on the formant plane using flexor pollicis brevis EMGs from both hands using the Cartesian transform. The myocontrolled trajectory is shown in **Figure 5B**. As can be seen, although the variability of the trajectory is significantly higher, the two targets (/i/ and /A/) were still reached. This demonstrates the feasibility of using EMGs to move a cursor on the 2D formant plane. Notice the abrupt jump in the bottom-left part of the trajectory shown in **Figure 5B**, this is because non-linear Bayesian filtering allows for rapid jumps even though the main purpose is still acquisition of smooth control signals. We argue that the task remains intuitive to the subject, which will facilitate learning.

We first compared the spectrogram of a vowel sequence "eeah-ee-ah" produced by both myocontrolled synthesizer and natural human speech of a healthy subject (**Figure 6**). It can be seen that the spacing between the lowest two formants are qualitatively similar. This is not surprising since the synthesizer is designed to closely match human voice, but the test is still useful for the validation of whether subjects can indeed drive the synthesizer using limb muscles.

The responses from the naive listener for all 10 healthy subjects are summarized in **Table 3**. For each condition, the fraction of correctness is averaged across all vowels produced by all 10 subjects. As can be seen, in all cases the fractions of success are higher than 60%, suggesting that it is feasible to produce intelligible English vowels using myocontrol. Cohen' κ-tests show that in all cases the κ-values are higher than 0.50, suggesting at least "Moderate" ("Almost Perfect" when using Cartesian transform) agreement between the naïve listeners and an imaginary perfect listener.

Two-Way repeated measures ANOVA shows that in healthy subjects (**Figure 7A**, healthy), Cohen's κ is significantly higher when using Cartesian transform compared with polar transform [main effect of transform, *F*(1*,* 9) = 169*.*7, *p <* 0*.*00001];

also our design favors the finger flexor (FPB) over the elbow flexor (BR) [main effect of muscle, *F*(1*,* 9) = 9*.*831, *p <* 0*.*05]. The interaction between transform and muscle is also significant [*F*(1*,* 9) = 20*.*51, *p <* 0*.*01]. These results suggest that both the EMG-formant mapping and muscle selection are important for future improvements of myocontrolled speech synthesis.

For the patients with dyskinetic CP, the responses of the listeners are also shown in **Table 3**. Since the subjects with dystonia had difficulty controlling their FPBs, only brachioradialis muscles (BR) were tested. Although the fraction correct for subjects with dyskinetic CP was lower than for healthy subjects, the naive listeners were still able to identify almost half of the vowels

**Table 3 | The responses of 4 listeners to vowels from 10 healthy subjects and 4 patients with dyskinetic CP.**

*\*The quality of raters agreement is assessed according to Landis and Koch (1977).*

**FIGURE 7 | (A)** Cohen's κ calculated from the responses of 4 listeners for the 10 healthy subjects. Each data point represents the responses from 1 listener across 25 vowels. For healthy subjects, each box covers 4 (listener) × 10 (subject) = 40 data points. **(B)** Cohen's κ calculated from the responses of 4 listeners for 4 patients with CP, each box covers 4 (listener) × 4 (patient) = 16 data points.

generated by subjects with dyskinetic CP. Cohen' κ-test also shows either "moderate" or "fair" agreement between the naive listeners and an imaginary perfect listener, suggesting that the vowels were unlikely to have been identified from pure speculation. Similar to healthy subjects, One-Way repeated measures ANOVA shows that for these 4 patients (**Figure 7A**, CP), our design favors Cartesian transform compared to polar transform [main effect of transform, *F*(1*,* 3) = 10*.*9, *p <* 0*.*05].

The performances of individual patients measured in κ are shown in **Figure 7B**. Although patient #4 performed better than other patients, the κ-values of all patients across all listeners using Cartesian transform were higher than 0.3 [one-tailed *t*-test, *t*(15) = 3*.*5, *p <* 0*.*01], suggesting a Fair quality of fit according to Landis and Koch (1977). Similarly, the κ-values for patients using polar transform were higher than 0.2 with Fair quality of fit [one-tailed *t*-test, *t*(15) = 2*.*2, *p <* 0*.*05]. It is unlikely that the statistical outcomes were due to the outperforming outliers.

Since both visual and auditory feedback were provided during the task, we compared the performance measured by both visual error and auditory intelligibility. We measured the visual error (*VE*) as the distance between the instantaneous cursor position and the final target in the 2D formant space:

$$VE(t) = \sqrt{(F\_1(t) - F\_{10})^2 + (F\_2(t) - F\_{20})^2} \tag{8}$$

The average *VE* during the last second of the 5-s task is shown in **Figure 8A**. Two-Way repeated measures ANOVA shows that the visual error was significantly higher in patients than in healthy subjects [main effect of population, *F*(1*,* 3) = 10*.*9, *p <* 0*.*05]. Suppose a linear measurement of the performance (p) that is inversely proportional to the visual error (*VE*):

$$p = \frac{1}{VE} \tag{9}$$

The relative increase in *VE* is related to the relative decrease in *p*:

$$
\Delta VE = \frac{VE\_2 - VE\_1}{VE\_1} \tag{10}
$$

$$
\Delta p = \frac{p\_2 - p\_1}{p\_1} = \frac{\Delta V E}{1 + \Delta V E} \tag{11}
$$

Take Cartesian transform as an example, the mean visual error increased by 114.9% from healthy subjects (179.42 ± 94.26, mean ± sd) to patients with CP (385.65 ± 293.93, mean ± sd), therefore the measurement of performance should decrease by 53.3%. Nevertheless, the performance (measured in κ, see **Figure 8B**) in myocontrolled vowel generation only decreased by 44.7% from healthy subjects (0.85 ± 0.1, mean ± sd) to patients with CP (0.47 ± 0.2, mean ± sd). Similar patterns were found in polar transform. This suggests that our paradigm could tolerate a certain amount of variability in the subject's input. It is also suggested that the subject did not try to minimize the visual error during the task, but instead they tried to rely on the auditory feedback and use the auditory intelligibility of the vowel as the control criteria.

**FIGURE 8 | (A)** Visual error across population and transform. Only data from the brachioradialis muscle are shown, since the patients with CP were only tested with this muscle. **(B)** Cohen's κ across population and transform. Only data from brachioradialis muscle are shown. ∗*p <* 0*.*05, ∗∗*p <* 0*.*001.

**Table 4 | Inter-rater agreement across 4 listeners.**

*\*The quality of raters agreement is assessed according to Landis and Koch (1977).*

Inter-rater agreements measured in Fleiss' κ are shown in **Table 4**. For healthy subjects the agreement level represented by κ-value is higher than 0.80 when using Cartesian transform, suggesting "Almost Perfect" agreement among listeners according to Landis and Koch (1977); when using polar transform, the κ-value decreases but still shows "Moderate" to "Substantial" agreement among listeners. For patients with CP, the κ-values are lower compared to healthy subjects, but the results still suggest "Moderate" agreements among listeners.

## **DISCUSSION**

We have shown that it is feasible to produce English vowels using myoelectric signals from non-speech muscles. When using Cartesian transformation between EMG and formant frequency, the fraction of correctly identified vowels is greater than 80% in healthy adults and 50% in two subjects with dystonia. The κtest suggests that a naive listener is able to identify the vowel and the intelligibility is unlikely due to pure guessing. We have succeeded in extracting high-bandwidth, low variability control signals from non-speech muscles by using the non-linear Bayesian filtering algorithm. Our other main innovation is mapping the vowel generation to a virtual reaching movement on a 2D plane, which is intuitive even for children with movement disorders (see explanations below). With only 10 min of practice the subjects were able to produce intelligible English vowels.

We point out that the goal of this study is not to prove that myocontrolled speech synthesis can improve the intelligibility of speech in CP, but to test in this population whether it is feasible to produce the simplest human speech using flexible myocontrol provided by non-speech muscles. Our results first suggest that it is feasible to use upper-limb muscles to produce intelligible English vowels in real-time. Secondly, the success rate in CP (57% in Cartesian transform) after less than 20 min of familiarization is comparable with the reported speech intelligibility (∼60%) in adult patients with CP (Platt et al., 1980), allowing us to further optimize the intelligibility by testing various EMG-formant mappings and muscle selections.

#### **LINKAGE TO VOCAL MUSIC AS INNOVATIVE THERAPIES**

We plan on expanding our EMG-auditory paradigm to "virtual singing" for children with CP. Our EMG-formant mapping is one of its first kind to enable such goal. We suggest the use of assisted vocal music as a therapeutic approach to restore the social, emotional and cultural aspect of patients with CP, so to enhance their qualities of life (Flanagan, 1978). In this study we investigated the efficacy of EMG-formant mapping for controlling the intonation and tempo of speech. This major feature also naturally enables the "virtual singing." To the audience of music training, we hope this seminal study can be of help and inspiration.

# **ADVANTAGES AND CONSTRAINTS DUE TO SPEECH-REACHING MAPPING**

One exciting discovery is that when using formant-based synthesis, the continuous production of vowels can be mapped to a continuous 2D reaching task. Similar ideas were presented in computer-aided education for pronunciation (Hiller et al., 1993), and commercialized software such as Vowel Viz (SmartPalate International, LLC, available for iOS devices). In these applications, the position of the cursor / finger was compared to predefined targets, and the best-fit vowel was chosen as the current sound. This category-selection approach did not allow modifications of sub-category features such as formant undershoot features that healthy speakers actively control according to their desired speaking style (Hardcastle and Hewlett, 1999). Also the synthesized speech would be insensitive to fine details of limb reaching movement, such as the smoothness of trajectory, or the variability around the intended vowel, if using category selection. In particular, our results show that the synthesized speech transitioned from /i/ and /A/ naturally and smoothly compared to human speech (**Figure 6**); such transition would not be controllable by the user if using category selection. Overall, our design highlights the value of continuous speech synthesis with controllable sub-vowel features, which may only be reproduced from a continuous limb movement such as reaching.

Due to the prevalence of reaching movements during dayto-day life, we argue that by associating vowel production with reaching it will make the task significantly easier to learn. Furthermore, myocontrolled speech synthesis creates a new paradigm that allows us to test human proprioception in the context of speech. For example, the role of proprioception in jaw muscles have been studied in previous studies (Ostry et al., 1997; Larson et al., 2000; Shiller et al., 2002), we can now ask more questions such as whether the proprioceptive feedback from the limb engages more, similar or less modulation compared to jaw. In future work we will use muscles directly involved in reaching movements, presumably from the same arm but from different joints. Such a multi-muscle control paradigm will require decoding the latent control signals from multi-channel EMGs, which has been an intriguing but challenging goal in myocontrol studies.

Our immediate next step is to test whether the quality of vowel generation can be improved through practice. In the current study, most patients with CP (3 out of 4) requested more practice after 10 min. They were given 5–10 extra min to familiarize with the task. The amount of practice required for fluent vowel generation is not the focus of the current study but remains an important topic for future studies. Furthermore, vowel generation may only improve within a narrow range of muscle contraction, because high muscle contraction may induce fatigue whereas low muscle contraction generally has significant variability. This, however, provides an important criterion for optimizing the mapping between EMG and formant frequency.

# **COMPARISON WITH OTHER TECHNOLOGIES**

Unlike in spinal cord injury or brainstem stroke, in CP there remains a connection between the brain and the spinal cord, which makes EMG a useful read-out of movement-related activities in motor cortex. When filtered with a non-linear Bayesian algorithm, we argue that electromyographic control (myocontrol) is more advantageous for motor restoration in this patient population compared with alternative paradigms such as brain computer interface (BCI), eye gaze control, buttons, or touchscreens. In particular, BCI is either invasive or low in bandwidth when using scalp electrodes. Eye gaze control is non-invasive but many children with CP have poor oculomotor control; also gaze interfaces restrict where the child can look. Buttons are low bandwidth and have limited ability for flexible tasks like speaking. Touchscreens require accurate reaching and multi-muscle control of the arm, which are often not possible for children with severe CP. Using formant-based synthesis we are able to achieve our standards for vowel production using only 2 independent EMG channels. We expect to be able to find 4–6 independent EMG channels in children with CP and explore the sufficiency to produce consonant-vowel syllables using up to 6 independent channels.

# **FACTORS THAT MAY AFFECT THE PERFORMANCE**

Our results showed that in both healthy subjects and patients with CP, Cartesian transform suits the vowel generation task better than polar transform. The subjects generally reported that Cartesian transform was more "intuitive" to control. This might be because that only one muscle from each limb was used in the current set-up. We predict that if we can access all the muscles involved in hand supination/pronation, the polar transform will be significantly easier to control. In addition, the thumb flexor (FPB) is better than brachioradialis for the task in healthy subjects, this might be due to that the precision of hand muscles is usually high among skeletal muscles.

In patients, the worst performer was a 12-year-old boy (patient #1 in **Table 1**) who had severe dysarthria, but it was noticed that he was easily distracted by the surroundings. Therefore, it is unclear whether the poor performance of patient #1 was due to inherent motor deficit or attention. The best performer was the 12-year-old girl who showed no less impairment in her diagnosis (patient #4 in **Table 1**). It has been well documented that significant muscle co-contractions are present in CP (Sanger et al., 2003), therefore the decreased performance of CP in this study could be due to the various cocontraction levels across patients, especially when the required muscle activation is high (sometimes *>* 60% MVC, **Table 2**). More experiments are needed to test what factors affect the performance in CP. It can be seen from **Table 1** that the 4 patients showed a variety of motor deficits. It suggests that the feasibility of EMG-driven vowel synthesis is unlikely to be an anecdotal success specialized to only certain types of motor deficits.

#### **CONSONANT AND SYLLABLE SYNTHESIS**

Toward the goal of synthesizing intelligible English language, the most challenging step is perhaps creating consonants and eventually consonant-vowel syllables. Unlike vowels, the consonants are not only dominated by the steady-state frequency of F1 and F2, but the formants also must be produced with precise timing. Even more challenging is that, for some consonants, the entire consonant production occurs in only tens of milliseconds, meaning that millisecond-level control of formants may be required. In addition to the timing, several consonants will have to be synthesized with more formants than just F1 and F2 (Klatt, 1980) (e.g., F3 in /r/) and other types of acoustic output (e.g., frication in fricatives). Our design does not restrict the number of formants or types of acoustic outputs, but since adding more acoustic outputs to control will require additional EMG channels, this will increase the difficulty and the need for training for users. For comparison, even in normal children, the acquisition of intelligible speech production takes several years (Sander, 1972). In the current setup, the sound production is non-stop as there is no direct control of the volume. This makes it difficult to simulate stop consonants that require momentary break in the voice. Therefore, in consonant synthesis it will be necessary to map the volume (at least the on/off of voice) into movement space. Finally, we will need to test the overall speed of vowel and consonant production in order to determine the number of muscles needed, and whether synthesized speech can approach the speed of normal human communication. If we are successful, these techniques will produce a new technology for assistive communication that will allow children to communicate not only the declarative content of language, but also the individual social and emotional content that is so important for interacting with peers, teachers, and parents.

#### **ACKNOWLEDGMENTS**

The authors thank Adam Feinman for help with the EMG setup. Funding was provided by the Don and Linda Carter Foundation, the Crowley Carter Foundation, and the USC Department of Biomedical Engineering.

# **REFERENCES**

Stevens, K. N. (2000). *Acoustic Phonetics*. Cambridge, MA: MIT Press.

Taylor, P., Black, A. W., and Caley, R. (1998). "The architecture of the Festival speech synthesis system," in *Proceedings of Third ESCA/COCOSDA Workshop on Speech Synthesis* (Jenolan Caves House, Blue Mountains: International Speech Communication Association), 147–151. Available online at: http://hdl*.*handle*.* net/1842/1032

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 21 June 2014; accepted: 31 December 2014; published online: 22 January 2015.*

*Citation: Niu CM, Lee K, Houde JF and Sanger TD (2015) Vowel generation for children with cerebral palsy using myocontrol of a speech synthesizer. Front. Hum. Neurosci. 8:1077. doi: 10.3389/fnhum.2014.01077*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2015 Niu, Lee, Houde and Sanger. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Rapidly learned identification of epileptic seizures from sonified EEG

# **Psyche Loui \*, Matan Koplin-Green, Mark Frick and Michael Massone**

Program in Neuroscience and Behavior, Music, Imaging, and Neural Dynamics Laboratory, Department of Psychology, Wesleyan University, Middletown, CT, USA

#### **Edited by:**

Eckart Altenmüller, University of Music and Drama Hannover, Germany

#### **Reviewed by:**

Lutz Jäncke, University of Zurich, Switzerland Mikko E. Sams, Helsinki University of Technology, Finland

#### **\*Correspondence:**

Psyche Loui, Program in Neuroscience and Behavior, Music, Imaging, and Neural Dynamics Laboratory, Department of Psychology, Wesleyan University, Judd Hall, 207 High Street, Middletown, CT 06459, USA e-mail<EMAIL_ADDRESS>
Sonification refers to a process by which data are converted into sound, providing an auditory alternative to visual display. Currently, the prevalent method for diagnosing seizures in epilepsy is by visually reading a patient's electroencephalogram (EEG). However, sonification of the EEG data provides certain advantages due to the nature of human auditory perception. We hypothesized that human listeners will be able to identify seizures from EEGs using the auditory modality alone, and that accuracy of seizure identification will increase after a short training session. Here, we describe an algorithm that we have used to sonify EEGs of both seizure and non-seizure activity, followed by a training study in which subjects listened to short clips of sonified EEGs and determined whether each clip was of seizure or normal activity, both before and after a short training session. Results show that before training subjects performed at chance level in differentiating seizures from non-seizures, but there was a significant improvement of accuracy after the training session. After training, subjects successfully distinguished seizures from non-seizures using the auditory modality alone. Further analyses using signal detection theory demonstrated improvement in sensitivity and reduction in response bias as a result of training. This study demonstrates the potential of sonified EEGs to be used for the detection of seizures. Future studies will attempt to increase accuracy using novel training and sonification modifications, with the goals of managing, predicting, and ultimately controlling seizures using sonification as a possible biofeedback-based intervention for epilepsy.

**Keywords: epilepsy, music, seizure, signal detection theory, learning, psychophysics, signal processing, sound design**

# **INTRODUCTION**

Since, Hans Berger recorded the first human brainwaves in 1924, electroencephalography (EEG) has established itself as one of the most useful non-invasive methods for clinical and scientific investigations of the brain. EEG offers high temporal resolution in investigating how electrical activity of the brain relates to cognition, sleep, emotion, and various neuropathologies such as dementia and epilepsy. Data from an electroencephalogram are typically represented visually, with time and voltage fluctuations on the *x*- and *y*-axes, respectively. In this study, we seek first to represent EEGs from normal and pathological brain rhythms in the auditory modality. Having defined a simple sonification algorithm for EEGs, we show that naïve human listeners can learn to distinguish epileptic seizures from normal brain rhythms using audition alone.

Sonification, in the case of EEG, refers to a process of datadriven sound composition that aims to make certain characteristics of the EEG waveform perceptible (Kramer, 1994). Techniques are being developed for both on-line and offline applications, in the scientific and artistic disciplines (Väljamäe et al., 2013). Sonification is being tested for a wide range of uses including monitoring of biological signals (Glen, 2010), diagnostic work (particularly in cases of epileptic seizure) (Khamis et al., 2012), auditory feedback of motion (Cheng et al., 2013), neurofeedback (McCreadie et al., 2012), and musical composition (Arslan et al., 2005). Here, we present a simple EEG-to-sound mapping algorithm and investigate its potential in monitoring EEGs by determining whether a non-expert population can use these sonifications to detect a seizure, based purely on basic musical comprehension skills. These results will inform the design of seizure monitoring algorithms that rely on abnormal electrical activity in the brain.

#### **ADVANTAGES OF EEG SONIFICATION**

One might ask what the utility of sonified EEG might be, when compared to the existing standard of visual EEG assessments. Sonification may have unique advantages for monitoring physiological rhythms due to the nature of human auditory perception. Compared to visual perception of EEG data, auditory perception – specifically music perception – may be more suitable for biofeedback therapeutic approaches for three reasons. Firstly, musical sounds and seizure EEGs both have strong frequency patterns; this correspondence offers a natural mapping system in translating EEGs to music. For example, pitch control, volume, and duration of a tone can be determined by any combination of parameters from the EEG data (Kramer, 1994).

Secondly, our ears are constantly open, unlike our eyes, and thus the ear acts as a more natural constant monitor that does not require foveation to function. In conjunction with this ability, human beings are surprisingly adept at focusing on important aural information even in noisy environment (e.g., the cocktail party effect,Arons, 1992). Studies have demonstrated that subjects can perform faster and more accurately at complex monitoring of physiological data when the data were presented sonically rather than by visual display (Fitch and Kramer, 1994; Barrass and Kramer, 1999; Watson and Sanderson, 2004). The authors suggest that this advantage of the auditory system can be explained by the fact that auditory recognition of objects occurs simultaneously in multiple parallel streams, in contrast to the visual system, which processes multiple objects serially. Sonification of EEG would prioritize temporal cues and thus allow persons to detect changes in parallel streams of activity as they occur.

Thirdly, people may find listening to music (especially as generated by their own neural activity) more motivating than visually monitoring their EEGs. The enjoyment of listening to esthetically pleasing sonifications might be an important factor in developing therapeutic uses and in improving the relationship patients have with EEG technology. A pleasant esthetic experience for potential end users of sonification technology is important for clinical utility and should thus be central to our goal. Taken together, while sonification does not add new information *per se*, the advantages to sonifying EEG lie in its user interface and increased usability: e.g., sonifying EEG may direct the user's attention to features of the EEG that are not as readily available to the eye, and thus future users, who may be epilepsy patients themselves or their caregivers rather than trained experts in reading EEG, might be able to detect seizures with minimal training. Sonification may also increase the options available for future biofeedback interventions.

#### **EARLY USES OF EEG SONIFICATION IN SOUND DESIGN**

Perhaps due to these natural characteristics of the auditory system as a data monitor, electroencephalography has also been revolutionary in the field of experimental music and sound design. Alvin Lucier's piece "Music for Solo Performer" (1965) is the first welldocumented instance of using an EEG for sonification purposes. His composition used two electrodes, attached to Lucier's temples, to transmit electrical activity (most notably alpha waves) to microphones placed inside various percussive instruments. The amplified frequencies recorded from the electrodes then caused the instruments to resonate at those same frequencies. This transformation from the electrical waveform of the brain to the acoustic waves produced by a drum's membrane occurred in real-time and represents one of the earliest successful sonification techniques. By modifying his own state of alertness, Lucier was able to modulate the level of energy in the alpha band, thus changing the levels of sound output. Thus, Lucier used his own music composition as an early biofeedback system.

Since Lucier's "Music for a Solo Performer," many composers have broadened the scope and output of similar explorations,looking into making more controlled and tonal sonifications of brain waves as recorded by EEG. Pulling from work done by Dr. R. Furth and E. A. Bevers in the 1940s, Bakerich and Scully filed a patent in 1971 for the "electroencephalophone," which they originally described as a device that can "enable the user to listen to his own brain-wave generation" (1971). In the same decade, Pauline Oliveros and David Rosenboom became seminal figures in experimental music composition by using the electroencephalophone, and other EEG-based forms of synthesis, in their compositions for

sonification purposes, yielding works such as Rosenboom's"Brainwave Music" (1976) and "On Being Invisible" (1977). Our hope is to draw on this history of ingenuity in experimental music to craft an elegant new system for the conversion of the brain's electrical potentials into sound, for the purpose of creating positive clinical and esthetic outcomes.

## **APPROACHES IN EEG-TO-SOUND PARAMETER MAPPING**

The processes of sonification depend crucially not only on the type of data input but also on the data-to-sound mapping process. If the pertinent data can easily be rendered as a simple variable/time graph such as the typical EEG time-voltage readings, then the signal should be easily translatable into sound. If, however, the data lend itself to simple audification or sonification that adheres to the general acoustic wave formula by displaying some form of periodicity in its sequence, then one must consider how much of the recorded data are significant and how much can be considered noise. In these cases, an algorithm that includes filtering of irrelevant noise and/or specific periodicities is necessary for optimal sonification (Hermann and Hunt, 2011).

The ability of the human auditory system to distinguish multiple voices and instruments from background noise make it well adapted to processing sonified EEG. Methods to sonify EEG data remain relatively unique as some have devised means but no method has shown extreme utility compared to any other. Over the past 10 years, new techniques such as parametric orchestral sonification have arisen that allow for the use of multiple channels of data to be sonified from EEG (Hinterberger and Baier, 2005). These approaches to processing multiparametric data allow for experimentation with parameter mapping, where the researcher can match different parameters of the EEG waveform with auditory parameters, such as pitch, duration, and volume. This level of control surpasses basic audification and allows for composition of novel musical scores from EEG data. Hinterberger and Baier (2005) have demonstrated that this technology can be used in real time, using a sample of 0–40 Hz divided into six frequency bands (namely alpha, beta, gamma, theta, and delta) assigned to individual voices in a musical instrument digital interface (MIDI) device. Subjects were able to control these voices and produce music in real time through a brain– computer interface. Musical compositions, created using similar techniques, have the potential to support clinical applications. In the case of applications toward epilepsy, these parameter mapping techniques may form the basis of monitoring systems that inform caregivers of partial seizures that might otherwise go undetected.

In fact, research is already beginning to show that sonifications are successful at representing important EEG data for diagnostic purposes. Khamis et al. (2012) showed that with limited training, non-experts were able to recognize temporal lobe seizures using audified EEG at a rate comparable to expert technicians using visual displays of the EEG waveform. Khamis's sonification process required compression of EEG signals over time, and the frequencies were limited to 1–10 Hz range prior to time-compression. Using this algorithm, the authors successfully demonstrated the relative ease of detecting seizures from audified EEG by non-experts with minimal training. However, due to the time-compression inherent in their sonification algorithm, this technique poses challenges for real-time sonification.

In contrast to the time-compressed sonifications of Khamis et al., experiments by Baier et al. (2007) have demonstrated the potential of sonified EEG for diagnosis of epileptic seizure in *real-time* using a process of sonification that does not require compression. Their technique of event-based sonification works on the principle of suppressing background noise and highlighting both normal and pathological rhythms. Identifying epileptic rhythms was accomplished by exploiting amplitude of the waveform and inter-maxima intervals to trigger specific sonic elements such as volume, tone duration, and the number of harmonics. This technique relies on stereotyped EEG rhythms and may not to be used from patient to patient without adjustment. Despite this drawback, this work demonstrates that on-line use of EEG sonification is possible and can exploit the diagnostic advantages demonstrated by Khamis et al. (2012).

While certain EEG parameters are used for sonification by almost all researchers in the field, some groups have developed complex parameter analysis that happens before the data are fed into the sonification algorithm. One universal parameter is the time-frequency dimension, and different projects have found different ways to manipulate this parameter. For example, some research has utilized a sliding-window technique, where data are analyzed in increments of several milliseconds or seconds (Arslan et al., 2005). Although this technique helps remove unwanted artifacts, it introduces latency into the system, thus reducing effectiveness for real-time sonification. Another dimension used in sonification is signal amplitude. As the amplitude of EEG signals corresponds to the firing rates of neurons, this parameter is vital in sonifying ictal brain activity, which manifests itself in increased firing rate (Blumenfeld, 2003). Because of the high level of background noise during normal brain activity, measures must be taken to attenuate this noise if a system attempts to diagnose an epileptic seizure. Noise may include intense spiking caused by jaw clenching or head moving, along with other artifacts. For example, some groups have found success by linking extreme maxima and maxima values to separate noise from target activity (Väljamäe et al., 2013). Aside from the time-frequency dimension and amplitude dimension, various groups have used high-level processing of EEG data pre-sonification. These processes include, but are not limited to, quadratic distance in the feature space (McCreadie et al., 2012), Gaussian kernel based on a normal distribution (Hermann et al., 2008), and calculation of timedomain parameters (Hjorth parameters, Miranda and Brouse, 2005).

In summary, current EEG sonification applications can be placed on continuum from *functional* to *esthetic* (Väljamäe et al., 2013). While our work certainly lies on the former side, we would like to make our sonifications esthetically pleasing as well, for two main reasons. First, if an EEG sonification system is to be implemented in a public setting (e.g., a hospital or nursing home), sonifications that are dissonant and cacophonous could be undesirable. Second, creating sonifications that sound pleasing may help non-experts hear fluctuations that correspond to seizures; the more strange and unfamiliar our sonifications are, the harder

it will be for someone to hear important developments in the sonified score. Thus, while the clinical outcome of seizure detection is undoubtedly the central goal of the present research, the creation of esthetically pleasing sonifications will serve the clinical goal, as sonifications will be far easier to use as a clinical device over extended periods of time if they sound pleasing, i.e., if they adhere to perceptual and cognitive principles that underpin our appreciation of music [see Lerdahl (1992)]. Thus, our aim is to develop, and test, an algorithm for real-time EEG sonification that provides an esthetically pleasing perceptual experience, while being functionally diagnostic of seizures by a listener with minimal training.

#### **GOALS OF THE PRESENT STUDY**