with fixed SNRs (Zekveld and Kramer, 2014), grammatical complexity (Schluroff, 1982) or perceptual effort with hearing loss (Kramer et al., 1997). Certainly the results of the current study support an increase in pupil dilation for the most challenging SNRs with the 16-channel vocoded sentences. However, the relationship between pupil dilation and SNR for the 6-channel vocoded sentences in the current study was not simple, where the mean pupil dilation across subjects plateaued for moderately negative SNRs and showed an increase with increasing speech intelligibility. It is possible that the changes in the pupil size across the 15 SNRs for the 6-channel vocoded sentences could reflect the non-linear behavior of the pupil size that has been observed when task difficulty exceeds capacity (Peavler, 1974; Granholm et al., 1996; Zekveld and Kramer, 2014). For example, it has been demonstrated that pupil dilation systematically increases with task difficulty (such as with a digit recall task), until it reaches or exceeds the limits of available cognitive resources, whereby it either asymptotes (Peavler, 1974), declines (Granholm et al., 1996), or shows both a decline followed by an asymptote for the most challenging intelligibility conditions (Zekveld and Kramer, 2014). An alternative explanation is that the noise levels per se could have influenced pupil dilation at the more negative SNRs (noise levels reached a maximum of 72 dB), where mean pupil dilation for both 16- and 6-channel vocoded sentences was similar. While Zekveld and Kramer (2014) attempted to reduce the likelihood of noise affecting pupil dilation by controlling the overall signal level while changing the SNR, in the current study, a fixed signal level was used with modulated levels of noise. Pupil dilation has been shown to be modulated by acute stress (Valentino and Van Bockstaele, 2008; Laeng et al., 2012) and animal studies have demonstrated that long-term effects of non-traumatic noise is associated with increased cortisol levels, hypertension and reduced cardiovascular function (see Gourévitch et al., 2014 for a review). A recent study looking at physiological measures of stress during listening in noise found that adults with hearing loss, who are constantly exposed to degraded speech, had higher autonomic system reactivity compared to adults with normal hearing, at similar performance levels (Mackersie et al., 2015). Therefore, while the noise levels in the current study were short-term, this may have caused a phasic stress reaction which could have influenced pupil dilation. This hypothesis, however, is not supported by studies suggesting that the pupil dilates with negative affect (Partala and Surakka, 2003).

The change in mean alpha power, relative to baseline, showed an enhancement of alpha activity in both 16-channel and 6 channel vocoding conditions, consistent with the inhibition hypothesis, where activity that is not related to the goal-directed task is actively inhibited (Klimesch et al., 2007). Therefore, it has been suggested that alpha enhancement which occurs during a speech-in-noise task results from the enhancement of auditory attention through the active suppression of noise (Strauß et al., 2014). However, most studies assessing alpha power change with vocoded speech material (Obleser and Weisz, 2012; Becker et al., 2013; Strauß et al., 2014) or during the processing of semantic information (Klimesch et al., 1997) have shown a reduction of alpha power, which is consistent with active cognitive processing of speech information. Specifically, the results of the current study appear contradictory to those reported by Obleser and Weisz (2012) using noise vocoded (2-, 4-, 8-, and 16-channels) mono- bi- and tri-syllabic words. They showed less alpha power suppression, of posterior-central alpha power with decreasing intelligibility measured between 800 and 900 ms post word onset. However, the task across the two studies was not the same. In the current study, participants were required to repeat the vocoded sentences, whereas in the Obleser and Weisz (2012) study, participants were asked to rank the comprehension of vocoded words without attending to the linguistic or acoustic aspects of the speech materials. While previous studies have shown a very high correlation between SRSs and rating scores, it is unclear whether the pattern of event-related oscillatory cortical activity measured during these different tasks is the same. Further, the types of analyses conducted across studies are not the same. For example, while Becker et al. (2013) demonstrated that mean alpha power during the region of interest (ROI) between 480 and 620 ms is reduced as speech intelligibility is increased (using monosyllabic French words), this was an absolute measure of alpha power rather than a change relative to the baseline. Variability of whether alpha power was increased or decreased was observed within studies. For example, Becker et al. (2013) showed the mean trajectory of change in alpha power during noise-vocoded monosyllabic words and demonstrated that alpha power is enhanced in the less intelligible conditions (similar to our results) but is suppressed in the most intelligible conditions (similar to the results shown by Obleser and Weisz, 2012). Further, using an auditory lexical decision task, Strauß et al. (2014) demonstrated mean increases of alpha power occurred for clear pseudo-words but a reduction was observed for ambiguous and real-words, which parametrically changed as the clarity of the words increased. Finally, using 18 younger and 20 older healthy adults, Wöstmann et al. (2015) demonstrated that decreases in mean alpha power which occurred as speech intelligibility increased (using four syllable digits masked by a single speaker) appeared to be driven by the older adults rather than an effect across the entire population. Given the differences in the types of speech stimuli used across the different studies, the task required, as well as the ROI used to assess alpha power changes (i.e., during or after the speech tokens), and the different populations assessed (older versus younger adults), further investigation of alpha power is needed to better understand the changes observed and how this might be used as an objective measure of attentional effort and/or cognitive load for the individual.

Within the current study, while a significant interaction was found between 6- and 16-channel vocoding for both alpha power and pupil size change, the trend patterns differed. The magnitude of the difference between both vocoding levels was greater in the most challenging SNRs for alpha power, but in the least challenging SNRs for the pupil size. This could suggest that these physiological responses are driven by different neurophysiological or attentional networks (Corbetta and Shulman, 2002; Corbetta et al., 2008; Petersen and Posner, 2012). There is a vast literature on attentional effort which suggests that discrete neuroanatomical areas encode specific cognitive operations ("processors") that are involved in attention, which are modified by "controllers" depending on the type of

attentional tasks required (see Power and Petersen, 2013). While the majority of the literature in this field focuses on the visual modality, there is evidence to suggest that similar processes should be evident when listening to degraded speech, such as listening in noise (Spagna et al., 2015). The main determinants of attentional allocation would then be; the identification of the appropriate processing strategy needed to undertake the speech perception task, the maintenance of attention during the task, and the processing of errors to increase (or, at least, reduce declines in) performance. Further, these processes may work synergistically under less cognitively demanding conditions but diverge under more challenging conditions, or conditions which have different types of attentional requirements (Vossel et al., 2014). It is also possible that different processors and controllers are used by different individuals to undertake these cognitively demanding task, which may have led to a lack of correlation between alpha power change and pupil dilation change within individuals. Corbetta and Shulman (2002) proposed the existence of two anatomically distinct attention networks; the dorsal fronto-parietal network, which is involved in the top–down voluntary or goal-directed allocation of attention (which includes preparatory attention and orienting within memory), and the ventral fronto-parietal network, which is involved in the involuntary shifts in attention. It is proposed that under normal circumstances, the ventral network is suppressed but is activated by unexpected, novel, salient, or behaviorally relevant events. Where this occurs, it is assumed that a "circuit-breaking" signal is sent to the dorsal attention network, resulting in reorienting, or shifting in attention toward this new event (Corbetta et al., 2008). It has been proposed that the locus coeruleus-norepinephrine (LC-NE) system modulates the functional integration of the entire cortical attentional system (Corbetta et al., 2008; Sara, 2009), whereby NE released by the LC triggers the ventral network to interrupt the dorsal attention network (Bouret and Sara, 2005) and reset attention. This ensures a coordinated rapid and adaptive neurophysiological response to spontaneous or conditioned behavioral imperatives (Sara and Bouret, 2012).

Pupil dilation is under the control of the LC-NE system, therefore it may be reasonable to assume that indirect attention tasks may be associated with the changes in pupil dilation observed in the current study. It has been proposed that pupil dilation is modulated by both staying on task and choosing between alternatives (exploration; Aston-Jones and Cohen, 2005). Therefore, a complex task, such as the perception and comprehension of a moderately intelligible (vocoded) speech signal, may result in changes in pupil dilation that reflect the interaction between different processing strategies. Alpha power changes have been associated with top–down inhibition of task irrelevant brain regions, and it has been suggested that alpha power is under the control of the dorsal attention network (Zumer et al., 2014). Further, increases in alpha power may inhibit the ventral attention network, preventing reorienting to irrelevant stimuli during goal-directed cognitive behavior (Benedek et al., 2014). While other models of attention exist (Seeley et al., 2007; Petersen and Posner, 2012), it is clear that a simple association between a physiological measure of attentional effort and task difficulty (e.g., changes in speech intelligibility) fails to consider the multiple autonomic cognitive operations as well as the voluntary control of attention that reflects effortful cognitive control (see Sarter et al., 2006). It is recognized that there is a dynamic interplay between the bottom–up sensory information and the top–down cognitively controlled factors (which may be either under automatic or voluntary control), such as knowledge, expectations and goals, that can be modulated by motivational factors, such as payment for participations (Tomporowski and Tinsley, 1996) and genetic influencers (Fan et al., 2003). Therefore, it is reasonable to assume that considerable variability in attentional allocation could exist between individuals undertaking a highly complex task.

An alternative explanation is that the within-subject variability of sustaining on-task attention toward sentences with unpredictable levels of intelligibility, was greater under the more challenging noise vocoding conditions (6-channel) where the effort-reward balance was not as high compared with the 16-channel vocoded materials. Sustaining attention on a complex task is challenging (Warm et al., 2008) and requires suppression of internal tendencies of mind-wandering, a default network activation that typically occurs during low task demands (Christoff et al., 2009; Gruberger et al., 2011), with concomitant activation of the goal-directed dorsal fronto-parietal attentional network (Corbetta and Shulman, 2002). Fluctuations in sustained attention can occur with stress, distraction with competing stimuli, fatigue, or lack of motivation toward the task, and are commonly associated with a decline in performance (Hancock, 1989; Esterman et al., 2012). As stated by Esterman et al. (2012) "as the neural systems supporting task performance appear to shift with one's attentional state, failure to account for attentional fluctuations may obscure meaningful information about underlying mechanisms". Certainly, some people have a preponderance to mind-wandering (Mason et al., 2007). This may be a confounder to the results of the current study comparing physiological responses to a range of SNRs, despite the ecological validity that this may have to their ability to follow conversations within multi-talker environments. That is, the variability in the physiological measures may, in fact, provide important information about the individual's processing of degraded speech that is not captured within more common behavioral measures of speech perception. For example, a recent study by Kuchinsky et al. (2016), suggests that individual differences in the pupillary response of older adults with hearing loss during a monosyllabic word recognition task was related to task vigilance (less variability in response time) and to the extent of primary auditory cortical activity. Therefore, pupil dilation may index the magnitude of the engagement between bottom–up sensory and top–down cortical processing which is increased with greater degradation of the speech signal (influenced by poorer SNR, reduced spectral information, or hearing loss).

Significant differences in the baseline data were also observed between the 6- and 16-channel vocoding for pupil size, but not for alpha power. These two levels of vocoding were assessed during different sessions for all subjects, therefore this could be due either to a session effect, or to a difference in the level of cognitive effort that was maintained throughout the session. Given that the results are consistent with an increase in cognitive load during

the 6-channel vocoded session, it is likely that the difference in the tonic pupillary response across the two physiological measures sessions (16- versus 6-channel vocoded-sentence tasks) resulted from differences in vigilance or the awareness of errors in performance during the more cognitively challenging task (Critchley, 2005; Ullsperger et al., 2010).

Limitations of the study include the relatively small number of participants included in the final data analysis (particularly for pupillary measures), and that only 16 sentences were presented for each SNR level (scored as 50 words across the set of 16 sentences) in each condition, reducing statistical power. Further, the test set-up restricted people from responding normally to an effortful task (i.e., a number of participants tended to close their eyes during the stimuli presentation but were instructed to keep their eyes opened). Explicitly investing effort in trying to keep their eyes opened despite the natural tendency to want to close them may have in itself created changes in pupil size and alpha oscillations. This may also have added an additional stressful component to the task.

### CONCLUSION

The results of this study suggest that the relationship between task difficulty and both pupil dilation and alpha power change was similar for the 16-channel vocoded sentences (high intelligibility), which might suggest that the attentional networks are operating with high concordance, or in a consistent and predictable manner across the SNRs. However, further degradations in the speech intelligibility, using the 6 channel vocoded materials, could have produced a discordant relationship between the attention networks, or different processors (such as linguistic strategies) may have been used to comprehend the speech signal. Importantly, however, given the considerable interest in assessing listening effort within clinical settings (see McGarrigle et al., 2014), it is important to ensure that we have a solid understanding of what these physiological measures are assessing, and how to interpret the

### REFERENCES

responses for the individual. Certainly, the results of this study do not currently support the clinical use of these physiological techniques as sufficiently sensitive to provide complementary information about listening effort to existing measures of speech perception performance. To be clinically viable in a hearing rehabilitation setting, such objective indices of effort should be more sensitive to changes in auditory input than existing measures of speech perception performance or subjective ratings of effort. The behavior of these indices should also be predictable across a range of performances and speech degradation to be applicable to the range of hearing loss and devices available, including hearing aids, and cochlear implants.

### AUTHOR CONTRIBUTIONS

Original idea: PL, CM, IB. Protocol development: PL, IB, CM, RI. Data collection: CYL, LG, PL, IB. Writing of manuscript: CM, IB, LG, KM, CYL. Data processing and analyses: RI, PL, PG, IB, CM, KM.

### FUNDING

This study was supported by the HEARing CRC, established and supported by the Cooperative Research Centres Programme – Business Australia.

### ACKNOWLEDGMENTS

The authors thank Jörg Buchholz for support with protocol development, Mike Jones for support with statistical analyses, and Gabrielle Martinez for help with data collection. Contributions- Original idea: PL, CM, IB. Protocol development: PL, IB, CM, RI. Data collection: CYL, LG, PL, IB. Writing of manuscript: CM, IB, LG, KM, CYL. Data processing and analyses: RI, PL, PG, IB, CM, KM.

influence of age and hearing impairment. Brain Res. Bull. 91, 21–30. doi: 10.1016/j.brainresbull.2012.11.005

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 McMahon, Boisvert, de Lissa, Granger, Ibrahim, Lo, Miles and Graham. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Impact of Background Noise and Sentence Complexity on Processing Demands during Sentence Comprehension

Dorothea Wendt1,2 \*, Torsten Dau1 and Jens Hjortkjær1,3

1 Hearing Systems, Hearing Systems Group, Department of Electrical Engineering, Technical University of Denmark, Kongens Lyngby, Denmark, 2 Eriksholm Research Centre, Snekkersten, Denmark, 3 Danish Research Centre for Magnetic Resonance, Centre for Functional and Diagnostic Imaging and Research, Copenhagen University Hospital Hvidovre, Hvidovre, Denmark

Speech comprehension in adverse listening conditions can be effortful even when speech is fully intelligible. Acoustical distortions typically make speech comprehension more effortful, but effort also depends on linguistic aspects of the speech signal, such as its syntactic complexity. In the present study, pupil dilations, and subjective effort ratings were recorded in 20 normal-hearing participants while performing a sentence comprehension task. The sentences were either syntactically simple (subjectfirst sentence structure) or complex (object-first sentence structure) and were presented in two levels of background noise both corresponding to high intelligibility. A digit span and a reading span test were used to assess individual differences in the participants' working memory capacity (WMC). The results showed that the subjectively rated effort was mostly affected by the noise level and less by syntactic complexity. Conversely, pupil dilations increased with syntactic complexity but only showed a small effect of the noise level. Participants with higher WMC showed increased pupil responses in the higher-level noise condition but rated sentence comprehension as being less effortful compared to participants with lower WMC. Overall, the results demonstrate that pupil dilations and subjectively rated effort represent different aspects of effort. Furthermore, the results indicate that effort can vary in situations with high speech intelligibility.

Keywords: effort, processing demands, pupillometry, syntactic complexity, background noise, working memory capacity, reading span, digit span

# INTRODUCTION

Speech communication provides a major basis for human interaction. Speech intelligibility has traditionally been measured in terms of the speech reception threshold (SRT) which reflects the signal-to-noise ratio (SNR) at which 50% of the words or sentences have been correctly recognized. However, these measures are typically obtained at low SNRs which do not correspond to everydaylistening situations that typically take place at SNRs of +5 to +15 dB (Smeds et al., 2015). In such more realistic communication situations, despite the fact that speech intelligibility is high, people may experience considerable difficulties when listening to speech. There has recently been growing interest in identifying the factors that cause these difficulties and attempts have been made

### Edited by:

Jerker Rönnberg, Linköping University, Sweden

### Reviewed by:

Jonathan E. Peelle, Washington University in St. Louis, USA Frederick Jerome Gallun, National Center for Rehabilitative Auditory Research, USA

> \*Correspondence: Dorothea Wendt<EMAIL_ADDRESS>
### Specialty section:

This article was submitted to Auditory Cognitive Neuroscience, a section of the journal Frontiers in Psychology

Received: 26 November 2015 Accepted: 24 February 2016 Published: 10 March 2016

### Citation:

Wendt D, Dau T and Hjortkjær J (2016) Impact of Background Noise and Sentence Complexity on Processing Demands during Sentence Comprehension. Front. Psychol. 7:345. doi: 10.3389/fpsyg.2016.00345

to characterize the processing demand or processing load (Johnsrude and Rodd, 2015) involved in speech comprehension (Gosselin and Gagné, 2010, 2011; McGarrigle et al., 2014).

Processing demands can be imposed by two factors: stimulusrelated factors that are associated with properties of the stimulus (e.g., noise degradation or linguistic complexity), and listener-related factors that reflect the perceptual and cognitive abilities of the listener [e.g., hearing impairment or working memory capacity (WMC)]. Regarding stimulus-related factors, the degradation of the speech signal due to the presence of background noise has been demonstrated to have an impact on the processing demand (e.g., Rabbitt, 1968; Pichora-Fuller and Singh, 2006). Varying the SNR can thus be used to induce higher or lower processing demand during speech comprehension, such that a higher amount of noise imposes a processing demand. Linguistic aspects, such as syntactically complex sentence structures, have been shown to decrease speech comprehension (Just and Carpenter, 1992), decrease sentence intelligibility (Uslar et al., 2013) and increase the sentence processing duration (Wendt et al., 2014, 2015). Hearing impairment, as a listenerrelated factor, typically degrades the representation of the speech signal in the auditory system which, in turn, can affect speech recognition (e.g., Plomp and Mimpen, 1979; Wingfield et al., 2006) and the sentence processing duration (Wendt et al., 2015). Moreover, cognitive abilities, such as a person's WMC, have been related to speech recognition performance (e.g., Lunner, 2003; Akeroyd, 2008). It has been suggested that individual cognitive recourses can be utilized to partly compensate for changes in the processing demand imposed by stimulus-related factors, even though the relationship between cognitive abilities and processing demand remains controversial (Ahern and Beatty, 1979; Verney et al., 2004; van der Meer et al., 2010).

The amount of cognitive resources utilized by a listener in a speech comprehension task can be defined in terms of effort (see also Johnsrude and Rodd, 2015). In other words, effort is a measure indicating the amount of resources deployed when processing speech, which depends on the interplay of the processing demand imposed by the stimulus-related factors (e.g., background noise, sentence complexity) and the listener-related cognitive abilities (such as WMC). A person's effort involved in speech comprehension has been measured using various methods and techniques (see McGarrigle et al., 2014 for a review). Subjective measures, such as perceived effort experienced during speech comprehension, have been tested using rating scales or questionnaires. Rudner et al. (2012) tested the effect of both noise level (in terms of SNR) and noise type (stationary vs. fluctuating) on subjective ratings of the perceived effort experienced in a sentence recognition task. It was found that the subjectively rated effort was affected by both the type of the background noise and the SNR. Although a fluctuating noise masker typically provides a release from masking (e.g., Festen and Plomp, 1990; Wagener et al., 2006), implying increased recognition rates compared to the condition with a stationary noise, listeners rated speech recognition in this noise condition to be more effortful. Rudner et al. (2012) also reported that rated effort increased with decreasing SNR consistent with other studies (Humes et al., 1999; Hällgren et al., 2005; Zekveld et al., 2010). Physiological correlates of processing effort include pupillary responses measured during speech tasks (see Kahneman and Beatty, 1966; Kahneman, 1973; Poock, 1973; Beatty, 1982; Granholm et al., 1996). More recently, there has been an increasing interest in measuring pupil dilations during speech perception in acoustically challenging situations (Kramer et al., 1997; Zekveld et al., 2010, 2011; Koelewijn et al., 2012; Kuchinsky et al., 2013). Zekveld et al. (2010, 2011) reported increased pupil dilations as an index of effort depending on speech intelligibility and type of background noise. Some studies have recorded subjective ratings of effort and pupil dilations in the same listeners (Zekveld et al., 2010, 2011; Koelewijn et al., 2012), but the relationship between the two measures has not yet been clarified. While Koelewijn et al. (2012) showed that the subjective ratings were positively correlated with pupil dilations during a speech recognition task, Zekveld et al. (2010) reported significant correlations between the rated effort and intelligibility but did not find any correlation between the subjectively rated effort and pupil dilations.

Working memory capacity has also been related to both subjective ratings and pupil dilations. Zekveld et al. (2011) reported a positive correlation between digit span test scores (as an index of WMC) and pupil dilations. Moreover, van der Meer et al. (2010) showed that listeners with higher fluid intelligence scores showed larger pupil dilations while performing a difficult task compared to individuals with lower scores. This led to the "resource hypothesis" (van der Meer et al., 2010) stating that individuals with better cognitive abilities, including higher WMC, allocate more resources, leading to a higher processing effort as reflected by larger pupil dilations. However, individuals with greater WMC have also been shown to rate listening as being less effortful (e.g., Rudner et al., 2012). This led to the "efficiency hypothesis" stating that individuals with higher cognitive resources report lower perceived effort due to more efficient processing (Ahern and Beatty, 1979; van der Meer et al., 2010). In line with this, the ease of language understanding (ELU) model suggests that it is less effortful for individuals with a high WMC to process a distorted speech signal (Rönnberg, 2003; Rönnberg et al., 2013). This seems to be in conflict with the resource hypothesis arguing that individuals with higher WMC engage more cognitive resources leading to higher effort. However, whereas the resource hypothesis is based on studies employing pupil response as a physiological correlate of effort, the ELU model refers to studies using subjective ratings as the indicator of effort. Thus, it may be that the two metrics represent different components of processing demand.

The present study attempted to distinguish between the outcomes obtained with rated effort vs. pupil dilation. Here, subjective ratings of effort, termed "perceived effort" (McGarrigle et al., 2014), were considered as an indicator of how effortful the process of speech comprehension is experienced by the participants. In contrast, pupil responses were considered as an indicator of "processing effort". Perceived effort and processing effort were measured in an audio-visual picturematching paradigm. In this paradigm, the participant's task was to match a spoken sentence with a picture presented before the sentence. This paradigm was designed to capture several levels of speech processing involved in the comprehension of speech

in background noise. This includes both lower-level perceptual processing, such as the separation of the speech signal from the background noise (Johnsrude and Rodd, 2015) and higher-level cognitive processes including linguistic and syntactic operations, such as a thematic assignment of the characters' role in the spoken sentence (see e.g., Wingfield et al., 2005). In the applied picturematching paradigm, a mental assignment of the characters' roles (i.e., who is doing something to whom) is required to accomplish the comprehension task. By employing the paradigm, it was investigated how different levels of the SNR (at high speech intelligibility levels) and the variation of the syntactic complexity of the sentence structure affect perceived effort and processing effort. Furthermore, it was examined how individual participants' cognitive test scores were related with perceived effort and processing effort.

# MATERIALS AND METHODS

# Participants

Eleven female and nine male participants with normal hearing participated in the experiment, with an average age of 23 years (ranging from 19 to 36 years). The participants had pure tone hearing thresholds of 15 dB hearing level (HL) or better at the standard audiometric frequencies in the range from 125 to 8000 Hz. All participants performed better than 20/50 on the Snellen chart indicating normal or corrected to normal vision (according to Hetherington, 1954). All experiments were approved by the Science Ethics Committee for the Capital Region of Denmark.

# Stimuli

### Speech Material

Thirty-nine items from the German Oldenburg Linguistically and Audiologically Controlled Sentence corpus (OLACS, see Uslar et al., 2013) were translated into Danish language and recorded. Each sentence describes two characters and an action being performed by one of the characters.  Each speech item was recorded with two different sentence structures in order to vary the complexity of sentences without changing word elements. Each sentence was either realized with a subject-verb-object structure (SVO I and II in **Table 1**) as well as with a syntactically complex objectverb-subject structure (OVS I and II in **Table 1**). While the SVO structure is canonical in Danish syntax and considered to be easy to process, written and spoken OVS sentences in Danish are more difficult to process (see Boeg Thomsen and Kristensen, 2014; Kristensen et al., 2014).

In both (SVO and OVS) sentence structures, the participants need to identify the semantic roles of the involved characters. The role assignment of the character that carries out the action (the agent) and the character that is affected by the action (the patient) is possible only after the auxiliary verb vil. Until the auxiliary verb, both sentence structures are ambiguous with respect to the grammatical roles of the involved characters and, thus, no thematic role assignment can be made. The auxiliary verb vil is either followed by the transitive verb filme ("film" see word 5 in **Table 1**), indicating a subject noun phrase at the beginning of the sentence, or by the article den ("the" see word 5 for the OVS I and II), informing the listener about the object role of the first noun. Since word 5 within each sentence provided the information required performing the comprehension task, the onset of word 5 is defined as the point of target disambiguation (PTD) for all sentence structures (see **Table 1**). Care was taken in selecting actions, agents, and objects that were non-stereotypical for any of the characters (for example, baking is a typical action of a baker). This constraint was employed to make sure that the participants did not make premature role assignments based on any anticipation of an agent's characteristic action.

### Visual Material

Pictures from the OLACS picture set were used, which were created for eye-tracking purposes (see Wendt et al., 2014, 2015). Each sentence was presented with either a target or a competitor picture. The picture illustrating the situation as described in the spoken sentence was defined as target picture (left panel of **Figure 1**). The competitor picture showed the same characters

TABLE 1 | Examples of the two sentence structures that were presented in the audio-visual picture-matching task.

All sentences contained eight words and have either a subject-verb-object (SVO) or an object-verb-subject sentence (OVS) structure. The onset of word five is defined as point of target disambiguation (PTD).

and action but interchanged roles of the agent and patient (right panel of **Figure 1**). Both the competitor and the target picture were of the same size, and within each picture, the agent was always shown on the left side in order to facilitate fast comprehension of the depicted scene. There were always two sentences that potentially matched a given sentence (i.e., a SVO and an OSV sentence for each picture). For instance, the left picture shown in **Figure 1** was used as target picture for sentence SVO I and OVS II in **Table 1**. All pictures were presented to the participants before they performed the audio-visual picture matching paradigm to familiarize them with the visual stimuli. All pictures are publicly available1 .

# Audio-Visual Picture-Matching Paradigm

The trial procedure for the audio-visual picture matching paradigm is shown in **Figure 2**. After an initial silent baseline showing a fixation cross (for 1 s), the participants were shown a picture (either target or competitor) for a period of 2 s. This was followed by a 3-s long background noise baseline after which a sentence was presented in the same background noise. After the sentence offset, the background noise continued for additional 3 s. A fixation cross was presented during the sound stimulus presentation. After the final noise offset, the participants were prompted to decide whether the sentence matched the picture or not via a button press (left or right mouse button). After the comprehension task, the participants were instructed to rate how

1http://www.aulin.uni-oldenburg.de/49349.html

FIGURE 2 | Trial structure of the audio-visual picture-matching paradigm. Participants saw a picture on screen for 2000 ms, followed by a visual fixation cross and a simultaneous acoustical presentation of a sentence in background noise. Background noise was presented 3000 ms before and ended 3000 ms after sentence offset. After the acoustic presentation, participants' task was to decide whether the picture matched with the sentence or not. Pupil dilations were measured from the picture onset until the participants' response in the comprehension task. The comprehension task was followed by a subjective rating of the experienced difficulty.

difficult it was to understand the sentence using a continuous visual analog scale (McAuliffe et al., 2012). They were asked to indicate their rating by positioning a mouse on a continuous slider marked "easy" and "difficult" at the extremes.

First, the participants performed one training block, which contained 10 trials. After training, each participant listened to 159 sentences, divided into two blocks. Both SVO and OVS sentences were presented in a lower-level noise condition (+12 dB SNR) or in a higher-level noise condition (−6 dB SNR). The noise masker was a stationary speech-shaped noise with the long-term frequency spectrum of the speech. Filler trials were included were the picture either did not match the character or the action of the spoken sentences.

### Cognitive Tests

At the end of the test session, the participants performed two cognitive tests: a digit-span test and a reading span task. The digit span test was conducted in a forward and a backward version. The forward version is thought to primarily asses working memory size (i.e., number of items that can be stored) whereas the backward version reflects the capacity for online manipulation of the content of working memory (e.g., Kemper et al., 1989; Cheung and Kemper, 1992). In the forward version, a chain of digits was presented aurally and the participants were then asked to repeat back the sequence. In the backward version, the participants were asked to repeat back the sequence in reversed order. To calculate the scores for the digit span test, one point was awarded for each correctly repeated sequence (according to the traditional scoring; see Tewes, 1991). The scores were presented in percentages correct, i.e., how many sequences out of the 14 sequences were repeated correctly. In addition, while the participants performed the digital span tests, pupil dilations were recorded to obtain a physiological correlate of effort.

In the reading span task, the participants were presented with sequences of sentences on the screen and instructed to determine, after each sentence, whether the sentence made sense or not (Daneman and Carpenter, 1980). After each sentence, a letter was presented on the screen and the participant was asked to remember the letter. After a set of sentences (length of the set varied between 3 and 11 sentences), the participant was prompted to recall the letters presented between sentences. The number of letters that were correctly recalled were scored regardless of the order in which they were reported. The reading span score was defined as the aggregated number of letters correctly recalled across all sentences in the test. Letters were used as targets rather than sentence words in order to make the task less reliant on reading abilities.

### Apparatus

The experiment was performed in a sound-proof booth. Participants were seated 60 cm from the computer screen and a chin rest was used to stabilize their head. Visual stimulus was presented on a 2200 computer screen with a resolution of 1680 × 1050 pixels. The stimuli were delivered through two loudspeakers (ADAM, A5X), located next to the screen. An eyetracker system (EyeLink 1000 desktop system, SR Research Ltd.) was used to record participants' pupil dilation with a sampling rate of 1000 Hz throughout the experiment. The eye-tracker was calibrated at the beginning of the experiment using a nine-point fixation stimulus. During each trial, pupil size and pupil x- and y-traces were recorded for detecting horizontal and vertical eyemovements, respectively. The eye tracker sampled only from the left eye.

# Pupil Data Analysis

The recorded data were analyzed for 20 participants in a similar way as reported in previous studies (Piquado et al., 2010; Zekveld et al., 2010, 2011) 2 . First, eye-blinks were removed from the recorded data by classifying samples for which the pupil value was below 3 standard deviations of the mean pupil dilation. After removing the eye-blinks, linear interpolation was applied starting 350 ms before and ending 700 ms after a detected eyeblink. Trials for which more than 20% of the data required interpolation were removed from the further data analysis. For one participant more than 50% of the trials required interpolation and, therefore, this participant was excluded from the further data analysis (Siegle et al., 2003). The data of the de-blinked trails were smoothed by a four-point moving average filter. In order to control for individual differences in pupil range, each trial data point was subtracted by the minimum pupil value of the entire trial time series (from trial onset of the picture presentation until the comprehension task) for each individual participant. Afterward, the pupil data were divided by the range of the pupil size within the entire trial. Finally, the pupil data were normalized by subtracting a baseline value which was defined as the averaged pupil value across 1 s before sentence presentation (when listening to noise alone, see **Figure 3**). The pupil responses were averaged across all participants for each condition. Averaged pupil data were analyzed within three different time epochs (see **Figure 3**). Epoch 1 describes the time from the start of the sentence until the point of disambiguation. Epoch 2 is defined as the time after the point of disambiguation until the sentence offset. Epoch 3 defines the 3 seconds following the sentence offset when the participants are asked to retain sentences in memory until the comprehension question.

# RESULTS

## Speech Comprehension in the Audio-Visual Picture-Matching Task Comprehension Accuracy

**Figure 4** shows the mean response accuracy across participants in the audio-visual picture-matching paradigm. The highest accuracy was found for the SVO sentences (93.1% in the lower-level noise condition and 87.8% in the higher-level noise condition). For the OVS structure, the response accuracy was between 57.2% (in the higher-level noise condition) and 58.1% (for the lower-level noise condition). The comprehension accuracy was analyzed using two separate repeated-measures analyses of variance (ANOVA) with complexity (simple,

2All data exclusions, all manipulations, and all measures were reported in this study.

complex) and noise level (high, low) as within-subject factors. The ANOVA revealed a main effect of complexity [F(1,18) = 15.8, p = 0.001, ω = 0.53] showing that the processing of OVS sentences resulted in more comprehension errors compared to the processing of SVO sentences. No effect of the noise level on the accuracy scores was found [F(1,18) = 1.8, p = 0.2] indicating that speech intelligibility was high in both noise conditions.

### Subjective Ratings

Averaged subjective ratings across all participants were calculated for each condition. The subjective ratings were analyzed using two separate repeated-measures ANOVA with complexity and noise level as within-subject factors. The ANOVA revealed a main effect of noise level [F(1,18) = 56.3, p < 0.001, ω = 0.779] indicating that the higher-level noise condition was rated as being more difficult compared to the lower-level noise condition. In addition, a small but significant effect of complexity on rating was also found [F(1,18) = 4.6, p = 0.048, ω = 0.223].

### Time-Averaged Pupil Dilation

fpsyg-07-00345 March 8, 2016 Time: 17:51 # 7

Averaged pupil dilations across all participants were calculated for each epoch (see **Figure 5**). The dilations were analyzed using separate repeated-measures ANOVA treating complexity and noise level as within-subjects factors. Separate ANOVAs were performed for each epoch. In epoch 1, there was a significant effect of noise level on the time-averaged pupil dilation [F(1,18) = 12.1, p = 0.03, ω = 0.41], but no effect of complexity was found [F(1,18) = 0.93, p = 0.35]. In epochs 2 and 3, significant effects of complexity [F(1,18) = 10.8, p = 0.004, ω = 0.39; epoch 3: F(1,18) = 12.8, p < 0.001, ω = 0.52] were revealed. Furthermore, an interaction of complexity and noise level was found in epoch 3 [F(1,18) = 9.0, p = 0.008, ω = 0.35].

# Cognitive Data

Pearson correlation coefficients between the subjective ratings and the performance in the cognitive tests [digit span forward (DF) score, digit span backward (DB) score, and reading span (RS) score in **Table 2**] were computed. A statistically significant correlation between the subjectively rated effort and the DB score was found for the SVO sentences presented at the higher noise level (p < 0.05, see **Table 2** and **Figure 6**).

In addition, Pearson correlation coefficients between the mean pupil response in epoch 3 and the performance in the cognitive tests were computed. Significant correlations were only found between the DB score and the pupil dilations in epoch 3 (p < 0.05, see **Table 2**), indicating that participants with higher DB scores had larger pupil dilations in the speech task.

Finally, correlations between the pupil dilations in the digit span test and the pupil dilations in the speech task (during epoch 3) were calculated. Pearson correlation coefficients revealed statistical significance (see **Figure 6**; p < 0.05), i.e., participants with enlarged pupil dilations in the speech task also showed higher pupil dilations in the span test.

# DISCUSSION

# Effects of Stimulus-Related Factors on Effort

A small but significant increase in pupil dilation due to the increased noise level was found in epoch 1, i.e., while the participants were listening to the first part of the sentence. The changes in pupil dilation due to the noise level were similar for both sentences structures, i.e., independent of the syntactic complexity. Moreover, a clear effect of the noise level on the perceived effort was found, i.e., the listeners reported speech processing as being more effortful when the sentences

FIGURE 5 | Mean pupil dilation observed for all four conditions. Time-averaged pupil dilation was calculated for three different epochs. Epoch 1 is the time when the first part of the sentence was presented. Epoch 2 includes the time after the sentence was disambiguated until the comprehension question. The third epoch is defined as the time from sentence onset until participants' response. The error bars show the standard deviations.

TABLE 2 | Correlation coefficient between the span tests and both the mean pupil dilation in the audio-visual picture matching paradigm and the subjective ratings for all four condition.

DB, digit span backward; DF, digit span forward; RS, reading span; Scores, scores in the span tests; Pupil, averaged pupil response in the digit span backward tests. Bolded values indicate significant correlation coefficients.

were presented at lower SNRs. These results are in line with studies that reported changes in pupil dilation and subjective ratings to be dependent on the SNR (Rudner et al., 2012). Zekveld et al. (2010, 2014) observed that pupil dilations and subjective ratings of effort increased with decreasing SNR. However, the current findings also clearly indicate an effect of noise level on effort in listening situations even when speech intelligibility is high. As reflected by the performance in the comprehension task, the participants were able to perform the task equally well at low and high noise levels. To the authors' knowledge, this is the first study demonstrating that effort increases with decreasing SNR even when speech intelligibility is still high.

Higher processing effort due to the increased syntactic complexity was expected for the OVS sentences compared to the syntactically less complex SVO sentences. Syntactic complexity came into play in epoch 2 when the participants listened to the second part of the sentence. In epochs 2 and 3, the participants were required to process and interpret the sentence by mentally assigning the grammatical roles of agent and patient and matching the spoken sentence content with the scene depicted in the picture. A pupil enlargement was measured for

the OVS sentences during epoch 2 and during the retention interval in epoch 3. These findings are consistent with other studies that showed increased effort while processing syntactically complex sentences (Piquado et al., 2010; Wendt et al., 2014). For example, Piquado et al. (2010) reported significantly larger pupil dilations during the retention of complex sentences. In the current study, the sentences were presented in noise in order to test the combined effects of sentence complexity and background noise level. The pupil data demonstrated distinct effects of noise and sentence complexity during epochs 1 and 2. Whereas a main effect of noise was observed in epoch 1, increased pupil dilations induced by the complexity of the sentence were measured in epoch 2. These results suggest that an increased processing effort due to an increased noise level occurs only if the sentence complexity is irrelevant for the task. As soon as the listeners start to process and retain syntactically complex information, the effect of the noise becomes negligible. Interestingly, an interactive effect of noise and complexity on the pupil dilation was found in epoch 3. This interaction was characterized by a steep decrease of the pupil response in the acoustically challenging listening situation (see epoch 3 in **Figure 3**). In other words, although a high pupil size induced by the noise level was detected in epoch 1, the pupil size decreased faster back to the baseline value in the retention period in epoch 3. This observation may suggest that listeners were able to recover faster from the high processing demand in the acoustic more challenging listening situation. However, this fast recovery occurred only for the simple sentence structures. When processing more linguistically complex sentences, this interactive effect was not found.

Whereas the pupil response indicated a clear impact of syntactic complexity, the effect of complexity on the subjective ratings was rather small. This suggests that subjective ratings and pupil dilations reflect different aspects of effort involved in speech comprehension. The pupil responses, interpreted as a physiological correlate of processing effort, were mainly sensitive to the syntactic complexity during sentence comprehension but were not indicative of the subjectively perceived effort. The perceived effort, in contrast, was more influenced by the degradation of the speech signal resulting from the increased background noise level. This is consistent with previous studies also suggesting that potentially different aspects of the effort may be measured when testing different methods and measures of effort (e.g., McGarrigle et al., 2014).

# The Influence of Listener-Related Factors on Effort

In the present study, different span tests were used to measure cognitive abilities of the participants. A moderate correlation between the digit span scores and the pupil dilations was found (**Figure 6**). Higher scores in the backward digit span test were found to correlate with higher pupil dilations in the speech comprehension task in the higher-level noise condition.

This could indicate that individuals with higher WMC allocate and engage more cognitive resources compared to individuals with smaller WMC. Previous studies have also reported higher pupil enlargement during speech processing for individuals with higher scores in cognitive tests (e.g., Zekveld et al., 2011). The results thus are consistent with the notion that individuals with higher cognitive capacities mobilize more working memory resources in acoustical challenging conditions as stated by the resource hypothesis (van der Meer et al., 2010). It is noticeable that significant correlations between WMC and pupil dilations were found specifically in epoch 3 comprising the retention period. This could suggest that pupil dilations specifically indicate the mobilization of working memory resources while storing speech information (and preparing for the upcoming comprehension task). Interestingly, significant correlations appeared only when processing sentences in the acoustically more challenging condition, suggesting that pupil response may further relate to the ability of listeners to rely on some form of working memory processing for compensating increased demands due to challenging acoustics. However, further research is needed to specifically explore these mechanisms.

Interestingly, the subjective ratings were found to be negatively correlated with WMC such that participants with a higher WMC tended to report lower perceived effort when processing SVO sentences in the higher-level noise condition. This suggests that individuals with greater WMC are able to use their resources to cope with the acoustically degraded speech signals and therefore report less effort, as argued by the efficiency hypothesis and the ELU model (Rönnberg et al., 2013). The presented data indicate that the relationship between individual WMC and effort depends on the employed measure. While listeners with a larger memory capacity may engage more resources, as indicated by increased pupil responses, this is not perceived as being effortful. Predictions made by the resource hypothesis with regard to processing effort (and its pupil response correlate) may be interpreted in terms of engagement of enhanced WMC, but not in terms of perceived effort. Predictions about effort made by the ELU model may be interpreted in terms of the subjective experience of effort.

Significant correlations between effort (both perceived effort and processing effort) and the digit span scores were only measured for the SVO sentences in the higher-level noise condition. This indicates that the WMC was only relevant when the induced demands increased due to the acoustic degradation of the speech signal. For the OVS sentences, no correlations between the digit span scores and the rated effort were found both in the higher-level noise and the lower-level noise condition. This suggests that the effort reached a plateau in situations when the cognitive resources could not compensate for the increased processing demands any longer (Johnsrude and Rodd, 2015). Thus, it may be that the available cognitive resources are exhausted when processing OVS sentences, which would further explain why no correlation between the digit span scores and effort were found in neither the higher-level noise nor the lowerlevel noise condition. No correlations were found between the reading span and the pupil response in the speech task. Note, however, that the procedure for the reading span test differed to the procedure applied in more recent studies (e.g., Lunner, 2003; Rönnberg et al., 2014; Petersen et al., 2016). A revised procedure

of the reading span test was developed to include having to remember either the first or the last word of each sentence in the list (see e.g., Lyxell et al., 1996). Since the participants do not know beforehand whether it will be the first or the last word, this revised procedure is suggested to increase the task difficulty and, therefore, the reading span score is supposed to reflect a more sensitive measure of the WMC. Thus, the missing correlation between the reading span score and the speech task might be explained by the procedure applied in the current study.

In this study, WMC was considered as a listener-related factor that potentially influences effort. However, there may be other listener-related factors that have not been considered in the current study. Interestingly, positive correlations between the pupil dilations in the digit span test and the pupil dilations in the speech task were found (**Figure 6**). Listeners that allocated more resources in the speech task also tended to mobilize more resources in the digit span test. This may indicate that some listeners generally engaged more resources than others when performing a task. Other potential listenerspecific factors affecting effort have been discussed in the literature. For instance, the level of motivation of individual participants could further influence the intensity of effort mobilization (Brehm and Self, 1989; Gendolla and Richter, 2010). With increasing success importance or with increasing motivation intensity, the amount of effort involved in a task can increase. It is possible that those participants who showed increased pupil dilations in both tasks were more motivated than those who exhibited smaller pupil dilation in both tasks. However, since motivation and success importance were not tested in the present study, the potential contribution of motivation to the results from this study remains to be clarified.

### Implications for Future Research

The audio-visual picture-matching paradigm presented in this study is well suited for studying speech processing in realistic communication situations. Monitoring increased effort during speech processing when intelligibility is high is crucial since it indicates challenges that constantly appear in everyday life. Moreover, in order to perform the task, listeners need to conduct a syntactic analysis of the sentence. This is in contrast to many speech intelligibility studies where the participants are typically asked to repeat back the recognized words of a sentence (Hagerman, 1984; Plomp, 1986). However, repetition does not necessarily involve any processing of the sentence structure or meaning that may constitute an important component of the challenges experienced in every day speech comprehension.

Extensive engagement of cognitive resources in everyday speech processing may eventually lead to fatigue or tiredness. Previous research suggested that hearing-impaired listeners are particular challenged in adverse conditions both with regard to speech perception performance and in terms of their effort required to achieve successful speech perception (Plomp, 1986; Rönnberg et al., 2013; Wendt et al., 2015). Consequences of increased effort can be, for example, a higher level of mental distress and fatigue leading to stress (Gatehouse and Gordon, 1990; Kramer et al., 2006; Edwards, 2007; Hornsby, 2013). Since traditional speech recognition tests are not sensitive to detect changes in effort in more realistic communication situations, there seems to be a need for new methods and measures to examine effort for hearing-impaired people. The findings of the present study suggest that pupil responses and subjective ratings are independent measures addressing different aspects of effort. Thus, when testing one measure of effort, the other measure is not necessarily reflected. This should be taken into account by researchers and clinicians when applying either one or the other method in their studies (McGarrigle et al., 2014).

# SUMMARY AND CONCLUSION

Three main observations were made in the present study. First, effects of increased demands due to background noise level and syntactic complexity were reflected in both the subjective ratings and pupil dilations. Second, the interaction between background noise level and syntactic complexity was rather small. Instead, separable effects of noise level and complex syntax on the subjective ratings and the pupil dilations were found: Increased syntactical complexity resulted in enlarged and prolonged pupil dilations, whereas a higher background noise level resulted in the task being rated as more effortful. Third, individual differences in cognitive abilities of the participants correlated differently with perceived effort and processing effort. Participants with higher scores in the backward span test (indicating higher WMC) showed increased pupil dilations but also reported the speech task to be less effortful than participants with lower scores. Overall, these findings demonstrate that pupil dilations and subjectively rated effort can vary in situations when intelligibility is at a high level and represent different aspects of effort. The methods and measures employed to investigate effort therefore need to be chosen carefully depending on the specific research question and hypothesis.

# AUTHOR CONTRIBUTIONS

The author DW developed the conception and design of the study, supervised the data acquisition, analyzed, and interpreted the data, and wrote the paper. DW gives the final approval of the version to be published, and agrees to be accountable for all aspects of the work. The author TD was involved in developing the design of the study, enabled the data acquisition, and substantial contributed to the interpretation of the data. TD was involved in writing and critically revising this version of this manuscript for important intellectual content. TD gives the final approval of the version to be published, and agrees to be accountable for all aspects of the work. The author JH developed the conception and design of the study, supervised the data acquisition, analyzed, and interpreted the data. JH was involved in writing and critically revising

this version of this manuscript for important intellectual content. JH gives the final approval of the version to be published, and agrees to be accountable for all aspects of the work.

# FUNDING

This research was supported by the Oticon Centre of Excellence for Hearing and Speech Sciences (CHeSS).

# REFERENCES

# ACKNOWLEDGMENTS

We thank Line Burholt Kristensen for her help with developing and recording the sentence material, the AULIN project for providing the picture material, and Wiebke Lamping for performing the measurements. In addition, thanks to Hartwig Siebner for his valuable input to the study design. JH was supported by the Oticon Foundation.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Wendt, Dau and Hjortkjær. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Autonomic Nervous System Responses During Perception of Masked Speech may Reflect Constructs other than Subjective Listening Effort

Alexander L. Francis1 \*, Megan K. MacPherson2 , Bharath Chandrasekaran3 and Ann M. Alvar1

1 Department of Speech, Language and Hearing Sciences, Purdue University, West Lafayette, IN, USA, 2 School of Communication Science and Disorders, Florida State University, Tallahassee, FL, USA, 3 Department of Communication Sciences and Disorders, University of Texas at Austin, Austin, TX, USA

### Edited by:

Rachel Jane Ellis, Linköping University, Sweden

### Reviewed by:

Nai Ding, Zhejiang University, China Carol Mackersie, San Diego State University, USA

> \*Correspondence: Alexander L. Francis<EMAIL_ADDRESS>
### Specialty section:

This article was submitted to Auditory Cognitive Neuroscience, a section of the journal Frontiers in Psychology

Received: 25 September 2015 Accepted: 10 February 2016 Published: 01 March 2016

### Citation:

Francis AL, MacPherson MK, Chandrasekaran B and Alvar AM (2016) Autonomic Nervous System Responses During Perception of Masked Speech may Reflect Constructs other than Subjective Listening Effort. Front. Psychol. 7:263. doi: 10.3389/fpsyg.2016.00263 Typically, understanding speech seems effortless and automatic. However, a variety of factors may, independently or interactively, make listening more effortful. Physiological measures may help to distinguish between the application of different cognitive mechanisms whose operation is perceived as effortful. In the present study, physiological and behavioral measures associated with task demand were collected along with behavioral measures of performance while participants listened to and repeated sentences. The goal was to measure psychophysiological reactivity associated with three degraded listening conditions, each of which differed in terms of the source of the difficulty (distortion, energetic masking, and informational masking), and therefore were expected to engage different cognitive mechanisms. These conditions were chosen to be matched for overall performance (keywords correct), and were compared to listening to unmasked speech produced by a natural voice. The three degraded conditions were: (1) Unmasked speech produced by a computer speech synthesizer, (2) Speech produced by a natural voice and masked byspeech-shaped noise and (3) Speech produced by a natural voice and masked by two-talker babble. Masked conditions were both presented at a −8 dB signal to noise ratio (SNR), a level shown in previous research to result in comparable levels of performance for these stimuli and maskers. Performance was measured in terms of proportion of key words identified correctly, and task demand or effort was quantified subjectively by self-report. Measures of psychophysiological reactivity included electrodermal (skin conductance) response frequency and amplitude, blood pulse amplitude and pulse rate. Results suggest that the two masked conditions evoked stronger psychophysiological reactivity than did the two unmasked conditions even when behavioral measures of listening performance and listeners' subjective perception of task demand were comparable across the three degraded conditions.

Keywords: listening effort, psychophysiology, informational masking

# INTRODUCTION

fpsyg-07-00263 March 1, 2016 Time: 17:6 # 2

In the normal case, understanding speech may seem to be effortless and automatic. However, even small changes in hearing acuity, signal quality or listening context can substantially reduce recognition performance and subsequent understanding or recall of the message (Van Engen et al., 2012) and presumably therefore increase perceived listening effort. Chronic effortful listening may, in turn, lead to long-term stress and fatigue as well as potentially serious health issues including hypertension and increased risk of stroke (Hogan et al., 2009). In the audiology clinic, listening effort is increasingly being seen as a significant factor for hearing aid users, both as it relates to intelligibility and as a potentially independent quality associated with willingness to adopt and continue using hearing aids (Picou, 2013). Listening effort is often associated with the allocation of limited supplies of cognitive "resources" such as working memory capacity or selective attention (Hicks and Tharpe, 2002), such that increased listening effort is associated with poorer performance on simultaneous or immediately subsequent cognitively demanding tasks (McCoy et al., 2005; Sarampalis et al., 2009). However, there is still a great deal of disagreement regarding the source of listening effort, or how to best characterize and quantify it (McGarrigle et al., 2014). The present article addresses these questions by quantifying psychophysiological responses to stimulus manipulations that are associated with different possible sources of increased listening effort.

According to one prominent proposal, the effortfulness hypothesis, the increase in perceived effort (and the decrease in downstream task performance) that is associated with listening in adverse conditions is linked to the acoustic phonetic degradation of the signal. Listeners confronted with a phonetically ambiguous or misleading acoustic signal must engage cognitively demanding mechanisms of repair or compensation in order to successfully decipher the intended message. Operating these mechanisms is assumed to require the commitment of cognitive resources that are in limited supply, and the consumption of these resources is typically associated with the concept of "effort." Thus, for present purposes, effortful processes may be thought of as those cognitive processes that involve the active commitment of cognitive resources such as working memory. Because such resources are in limited supply, listeners will have fewer resources remaining for subsequent processing of the linguistic information encoded in that signal (Rabbitt, 1968, 1991; Pichora-Fuller et al., 1995; McCoy et al., 2005; Wingfield et al., 2005; Pichora-Fuller and Singh, 2006; Surprenant, 2007; Lunner et al., 2009; Tun et al., 2009). However, not all sources of signal degradation have the same effect on the signal, and it is possible that different repair or compensation mechanisms may be engaged (or the same mechanisms may be engaged to differing degrees) to achieve the same level of performance under different circumstances. That is, different types of signal degradation may incur different demands on cognitive resources, or demands on different resources, and thus may differentially affect perceived effort even when performance is comparable. The goal of the present study was to investigate this possibility by quantifying physiological responses associated with task demand while listening to three similarly intelligible but differently degraded speech signals. If different types of degradation that result in the same performance are nevertheless associated with different patterns of psychophysiological reactivity, this would suggest that listeners are engaging different compensatory cognitive mechanisms to cope with the different sources of degradation.

Three types of degradation were chosen to represent three different ways in which a signal might be degraded. The first two involve masking, and represent examples of energetic and informational masking, respectively, while the third, computer speech synthesis, represents a complex form of signal degradation accomplished without masking.

Energetic masking is the simplest type of masking, in which one signal (the masker) physically obscures some part of the meaningful (target) signal. The source of difficulty in this case is simply the physical interaction between the two competing signals in the auditory periphery (Brungart et al., 2006). Adding speech-shaped noise to the target signal is a prototypical example of energetic masking, as the decrease in performance with respect to unmasked speech is arguably due entirely to the overlap of the excitation patterns of the target and masker signals on the basilar membrane. From a listener's perspective, the difficulty in understanding speech in noise arises mainly from the loss of information contained within those parts of the target signal that are obscured by the noise. Although listeners are likely to recognize that there are two separate sound sources in the combined signal, namely the target speech and the masking noise, they generally have little difficulty distinguishing between the two, meaning that demands on selective attention should play a minimal role in this condition (Shinn-Cunningham and Best, 2008). Similarly, the noise signal has no informational content, and therefore, in itself, is assumed to add no appreciable load to listeners' working memory (though cf. Sörqvist and Rönnberg, 2014, who suggest that attention, and hence working memory, is still involved even in simple noise-masking conditions). In principle, the effortfulness hypothesis would thus account for any increase in listening effort related to added noise as primarily due to the need to cope with the less informative (degraded) target signal itself.

Informational masking, in contrast, is often used as a catchall term covering all cases of interference that cannot be explained purely in terms of energetic masking (Cooke et al., 2008). In the present case we will consider a specific type of informational masking, namely the use of one or more tobe-ignored speech signals (speech maskers) to interfere with listeners' understanding of a target speech signal, a condition under which performance has been shown to dissociate from performance under energetic masking (Brungart, 2001; Brungart et al., 2006; Van Engen et al., 2012). In this case, in addition to the energetic masking that occurs when the masking signal(s) interfere acoustically with the target signal, there is also some interference occurring at a more linguistic or cognitive level of processing (Mattys et al., 2009). For example, speech masked by two-talker babble not only presents listeners with the challenge of dealing with a partially obscured target speech signal, it also imposes greater demands on selective attention as listeners must choose to which of the three voices to attend (Freyman et al.,

2004; Brungart et al., 2006; Ihlefeld and Shinn-Cunningham, 2008; Shinn-Cunningham, 2008). In addition, demands on working memory likely increase, as listeners probably retain some of the content of the masking signal in working memory and this must subsequently be selectively inhibited at the lexical level (Tun et al., 2002; Van Engen and Bradlow, 2007; Cooke et al., 2008; Mattys et al., 2009; Dekerle et al., 2014). Neuropsychological and genetic studies further suggest that populations that are predisposed to show poorer selective attention, as indexed either by increased degree of depressive symptoms (Chandrasekaran et al., 2015) or genetic markers associated with poorer executive function (Xie et al., 2015) experience greater interference in conditions that emphasize informational masking as compared to those involving primarily energetic masking.

Finally, synthetic speech represents a different sort of signal degradation, one that has been less well-studied in the effort literature but that has been shown to introduce cognitive demands on speech perception (Pisoni et al., 1985; Francis and Nusbaum, 2009). Unmasked synthetic speech, like foreign accented, dysarthric, and noise-vocoded speech consists of a single signal, thus eliminating issues of selective attention at the signal level. However, synthetic speech is distorted in ways that not only represent a lack of information, but potentially introduce misleading information (Francis et al., 2007), a property shared with accented and dysarthric speech, but not necessarily vocoded speech. Thus, listening to synthetic speech, like listening in competing speech, may require the application of additional cognitive resources for the retention and eventual inhibition of a larger number of competing lexical items in working memory (Francis and Nusbaum, 2009); but, unlike competing speech conditions, in the case of synthetic speech there is no benefit to applying selective attentional processes to filter out competing signals before their content interferes.

Thus, these three types of degradation allow for the possibility of distinguishing between listening effort due to the increased cognitive demands associated with informational masking (noise- vs. speech-masked), and of listening to a single challenging signal as compared to selectively attending to multiple signals (speech-masked vs. synthetic speech).

In order to quantify listening effort, three general methods of assessment have been identified in the literature: subjective (self-report) measures of task demand using instruments such as the NASA Task Load Index (TLX, Hart and Staveland, 1988) and the Speech, Spatial and Qualities of Hearing Scale (SSQ, Gatehouse and Noble, 2004); measures of behavioral interference between dual tasks (Sarampalis et al., 2009; Fraser et al., 2010); and physiological assessments of central nervous system function using fMRI (Wild et al., 2012) and EEG/ERP methods (Bernarding et al., 2012) and of autonomic nervous system arousal based on measurements of a variety of systems, such as those that reflect pupillary (Zekveld et al., 2011), electrodermal, and cardiovascular function (Mackersie and Cones, 2011; Mackersie et al., 2015; Seeman and Sims, 2015).

The autonomic nervous system is a division of the nervous system controlling functions vital to survival including respiration, digestion, body temperature, blood pressure, vasoconstriction, heart rate and sweating (Hamill et al., 2012). It is divided into three major branches: the sympathetic, parasympathetic, and enteric nervous systems. The enteric nervous system primarily governs digestion and will not be further discussed here. The sympathetic nervous system (SNS) is typically associated with fight-or-flight responses such as the cool, damp palms associated with confronting a physical or emotional threat, while the parasympathetic nervous system (PNS) is typically associated with rest, relaxation, and recovery from stressors. The sympathetic and parasympathetic branches interact to preserve a homeodynamic balance within the body, maintaining a stable internal state and adjusting bodily functions to respond to internal and external stimuli (Kim and Kim, 2012).

Thus, physiological measures of autonomic nervous system reactivity were selected for the present study because such measures, especially those reflecting SNS arousal, are associated both with increased cognitive demand and with emotional stress, and may therefore constitute an important link between the momentary demands of listening to speech under adverse conditions and long-term health issues associated with hearing impairment. For example, chronic stress associated with living in a noisy environment has been linked to both higher levels of SNS arousal and increased risk of adverse health outcomes (Babisch, 2011). Similarly, measures of peripheral vasoconstriction due to SNS arousal are associated with subjective measures of annoyance by noise (Conrad, 1973) which, in turn, may be among the better predictors of compliance in hearing aid users (Nabelek et al., 2006; though cf. Olsen and Brännström, 2014). Moreover, anxiety also affects speech perception, potentially increasing demand on cognitive processing (Mattys et al., 2013). Thus, developing a better understanding of autonomic nervous system responses to different sources of listening effort will also provide insight into the possibility that chronically heightened listening effort may contribute to broader issues of health and wellbeing. In this study, four measures of autonomic nervous system reactivity were assessed: skin conductance response (SCR) rate and amplitude, fingertip pulse amplitude (PA), and pulse rate (PR).

### Skin Conductance Response

The SCR refers to a phasic increase in the conductivity of the surface of the skin, especially on the palms of the hands or the feet, reflecting increased eccrine sweat gland activity. The eccrine sweat glands are innervated solely by the SNS. Skin conductance is collected by running a slight (0.5 V) current between two electrodes across the surface of the skin. As eccrine sweat gland activity increases, the concentration of negative ions on the skin surface increases, increasing conductivity between the two electrodes (Boucsein, 2012). Although SCRs are not elicited in all trials (Andreassi, 2007), their frequency and amplitude have long been associated with a wide range of psychological responses. The simplest of these is the orienting response (OR), an involuntary response to any sufficiently large change in the sensory environment, reflecting stimulus novelty and degree of surprise, but also affected by stimulus significance. In this context, the SCR is also potentiated by the arousing quality of the stimulus content (irrespective of positive or negative affective valance), such that more significant or more emotionally arousing stimuli induce a stronger SCR (Bradley, 2009). For example, Mackersie and Cones (2011) showed that increasing task demands on selective attention by increasing the complexity of a dichotic digits repetition task increased the amplitude of the SCR, suggesting that as the listening task became more attentionally demanding, listeners' SNS arousal increased.

## Pulse Amplitude

fpsyg-07-00263 March 1, 2016 Time: 17:6 # 4

Fingertip pulse amplitude (PA) is a measure of the volume of blood in the capillary bed of the fingertip at the peak of the heartbeat. Like the SCR, it is governed purely by the sympathetic branch of the autonomic nervous system, with increasing arousal leading to peripheral vasoconstriction and therefore decreased amplitude of the blood pulse volume signal (Iani et al., 2004; Andreassi, 2007) (henceforth PA). Phasic PA has been shown to decrease in response to increasing demands of cognitive tasks such as the Stroop task (Tulen et al., 1989) and mental arithmetic (Goldstein and Edelberg, 1997), and such decrease has been linked specifically to the increased investment of mental effort in a task, such that PA decreases parametrically with increase in working memory load (Iani et al., 2004).

## Pulse Rate

Changes in heart rate have been used extensively to study arousal related to sensory and cognitive processing. The period (and thus frequency or rate) of the heart beat is governed by both sympathetic and PNSs, with acceleration primarily under the influence of the sympathetic branch (Andreassi, 2007). Phasic cardiac acceleration and deceleration (momentary increase and decrease of heart rate) are each associated with different aspects of mental demand. Deceleration within the first few heart beats following presentation of a stimulus is typically characterized as part of an automatic OR, and is often interpreted as reflecting the holding of resources in reserve to prepare for stimulus encoding and processing (Lacey and Lacey, 1980; Lang, 1994) or even as an indication of a defensive response to threatening or unpleasant information in the stimulus (Bradley, 2009). Thus, listeners anticipating the need to process more complex or perceptually demanding stimuli, or who are experiencing the stimulus as threatening or aversive, might be expected to show a greater degree of cardiac deceleration during the initial OR. That is, to the extent that cardiac deceleration constitutes a component of an automatic OR, it is not, in itself, a reflection of the operation of an effortful (i.e., controlled, resource-demanding) process but it may nevertheless be expected to occur more strongly in conditions in which the stimulus is perceived to be aversive and/or is expected to be demanding to process further. On the other hand, heart rate has also been observed to increase as a mental task becomes more difficult, for example when doing increasingly complex mental arithmetic (Jennings, 1975), and this acceleration generally persists throughout the duration of the task. Thus, different aspects of cardiac response may reflect different ways in which a given task may be perceived as effortful: deceleration may be associated with tasks that are perceived as effortful because they involves processing stimuli that are

# Summary

The purpose of the present study was to quantify psychophysiological responses that might reflect differences in the degree or type of effortful cognitive mechanisms listeners employ to perceive speech under three conditions of increased difficulty compared to listening to unmasked, undistorted speech. The conditions differed in terms of the source of the difficulty (energetic masking, informational masking, and distortion) but were chosen to be matched for overall performance (keywords correct). We hypothesized that, although speech recognition performance should not differ significantly across conditions, listeners would exhibit greater psychophysiological reactivity in conditions involving informational masking and distortion, because these conditions, more so than simple energetic masking, increase demands on cognitive mechanisms of working memory and attention. In fact, results suggested instead that greater psychophysiological reactivity across degradation conditions was mainly associated with conditions involving masking (whether informational or energetic) as compared to either unmasked condition.

### MATERIALS AND METHODS

# Subjects

Fourteen native speakers of American English gave informed consent and participated in this study under a protocol approved by the Purdue University Human Research Protection Program. They ranged in age from 20 to 32 years (mean = 26.0). There were 11 women and 3 men and all were right-handed. All were recruited from the Purdue University community and either had at least a Bachelor's degree level of education (13) or were currently in college. No participant reported fluency in any language other than English. All were nonsmokers in good health by self-report, and none were currently taking any medications known to influence cardiovascular or electrodermal responses. All reported having minimal or no caffeine consumption on the day of testing (though cf. Barry et al., 2008). Participants were screened for anxiety and depression which may affect or be associated with autonomic nervous system function (Dieleman et al., 2010), using scales that would be suitable for both younger and elderly individuals because this study was intended as part of a larger project including geriatric participants. All participants scored within normal limits on the Geriatric Depression Scale (GDS, Yesavage et al., 1983) and the Geriatric Anxiety Inventory (GAI, Pachana et al., 2007). All exhibited auditory thresholds within age-normal limits, passing a pure tone screening test of 20 dB SPL at 250 and 500 Hz, and 25 dB SPL at 1000, 2000, 4000, and 8000 Hz. All reported normal or corrected-to-normal vision. All participants scored

TABLE 1 | Scores range from 1 to 20 where 1 = "very low" and 20 = "very high" for ratings of mental demand, effort, and frustration, and 1 = "perfect" and 20 = "failure" for performance.

GDS, Geriatric Depression Scale (Yesavage et al., 1983); GAI, Geriatric Anxiety Index (Pachana et al., 2007); PTA, Pure Tone Average [average of pure tones at 0.25, 0.50, 1, and 2 kHz, in the left (L) and right (R) ear]; CLQT Attention normal limits (NL) between 180-215; CLQT Memory NL between 155 and 185; CLQT Executive Function NL between 24 and 40; CLQT Language NL between 29 and 37; CLQT Visuospatial Skills NL between 82 and 105. Values are presented in the form of Mean (SD).

within normal limits on all subscales of the Cognitive Linguistic Quick Test (CLQT, Helm-Estabrooks, 2001). Basic demographic information and test results are shown in **Table 1**.

# Apparatus and Materials

### Testing Environment

During the speech perception task, participants were tested in a quiet room, seated comfortably approximately 1.5 m directly in front of a speaker (Hafler M5 Reference). All stimuli were played via speaker at a comfortable listening level (approximately 76 dBA measured at the location of the seated participant's head, averaged over four test sentences). Although this overall level is higher than is typical in speech audiometry, it corresponds to a signal (speech) level of 67 dBA combined with a masking noise level 8 dB louder (necessary to achieve comparable performance across the two masked conditions). Stimulus presentation was controlled by a program written in E-Prime 2.0 (Psychology Software Tools, Inc. [E-Prime 2.0], 2012). Responses were made verbally, and were scored on-line by the experimenter.

### Stimuli

Stimuli were selected from a database of sentences originally developed by Van Engen et al. (2014). The subset used here consisted of 80 semantically meaningful sentences based on the Basic English Lexicon sentences (Calandruccio and Smiljanic, 2012) spoken in a conversational style by a young, female native speaker of American English. Sentences always contained four key words. For example (keywords underlined) The hungry girl ate a sandwich. Masking stimuli were derived from a set of 30 different sentences (not in the target set) produced by eight different female native speakers of American English (not the target talker). Two talker-babble was created by concatenating sentences from two of these talkers, removing silences, and adding them together using the mix paste function in Audacity 1.2.51 . The speech shaped noise was generated by filtering white noise to match the long-term average spectrum of all of the masking sentences.  Stimuli were mixed to present the target at a challenging SNR of −8 dB, and then all stimuli were normalized to the same RMS intensity level using Praat 5.3. At this SNR, prior studies with these stimuli in our labs have shown that comparable performance is typically elicited across the two listening conditions tested here.

Synthetic speech was generated using ESpeak 1.462 . Espeak is a publicly available formant-style text-to-speech synthesizer that runs under Windows and Linux. Stimuli were generated by presenting a text file with one sentence per line to the synthesizer, producing a single sound file containing all sentences spaced at regular intervals. The default voice (male) and speaking rate were used because preliminary, informal testing suggested that these were sufficiently difficult to be comparable to the masked speech stimuli in terms of overall intelligibility, even though the synthetic sentences were noticeably shorter than the natural ones. The resulting wave file was segmented into separate files using Praat 5.3, and these were subsequently RMS amplitude normalized to the same level as the masked and unmasked stimuli generated with natural speech. Thus, there were two masked conditions: speech-shaped noise and two-talker babble, and two unmasked conditions: synthetic speech and natural speech.

### Design

Participants completed two sessions, with inter-session intervals averaging 6.4 days (SD = 5.5; ranging from later in the same day for one participant, to 16 days later for another). In the first session, participants completed the process of informed consent, provided background demographic information, and completed screening tests for hearing thresholds, anxiety, depression, and cognitive function. In the second session participants were played two sentences (not otherwise used in the experiment) in each of the four conditions, and then completed the speech perception task, which consisted of four conditions. Each condition presented one type of stimulus: unmasked natural speech, unmasked synthetic speech, natural speech masked by speech-shaped noise, or natural speech masked by two-talker babble. Conditions were presented in random order across participants.

**Figure 1** shows a schematic diagram of the experiment design. In each condition, there were three sequences of stimuli, which we will refer to here as "runs." The first run in each condition was originally intended to permit the collection of a variety of preliminary physiological data as well as to familiarize the participant with the experimental paradigm. It consisted of 2 min of silence followed by a 0.25 s tone (400 Hz), 0.75 s of silence, 6 s of presentation of the masker (in the two masked trials) or silence in the unmasked trials. The idea was to enable the collection of data

1www.audacity.sourceforge.net

2http://espeak.sourceforge.net/index.html

under true baseline conditions (in silence) as well as in a noiseonly condition (see Parsons, 2007). This was followed by 60 s of silence and then two trials using sentences not otherwise used in the rest of the experiment. Preliminary analyses conducted after the first three participants had completed the study suggested that there was little benefit to analyzing physiological responses during the various portions of this run because some participants did not remain sufficiently still during the silent periods, so although it was included for all subsequent participants in order to maintain a consistent experimental protocol across participants, it was not further analyzed.

The second and third runs presented the experimental stimuli, and had identical formats. Each experimental run began with 30 s of silence, followed by eight experimental trials (sentences). Each trial began with a 0.25 s beep (400 Hz), followed by 0.5 s of silence, and then the start of the masking sound which began 0.75 s before the speech stimulus, resulting in a total duration of 1.5 s between the onset of the warning beep and the onset of the speech signal to be repeated. In the two unmasked conditions the period between the warning beep and the start of the speech stimulus was also 1.5 s, but the period following the beep was silent up to the beginning of the target sentence. The target sentence ended 0.25 s before the end of the noise, between 2.768 and 3.503 s after the sentence began (or 1.208–1.904 s for the synthetic speech). Twelve seconds after the initial warning beep, a second, identical beep was played to indicate to the listener that they should repeat the sentence they heard, or as much of it as they could remember. Eight seconds later the next trial began. Thus, each trial, from initial warning beep to beginning of the next trial, lasted 20.5 s while each run consisted of the presentation of eight sentences and lasted 3 min, 14 s. In total, each condition (three runs, including two containing eight sentences each) lasted 10 min, 16 s, and the entire session required a minimum of 41 min, 4 s (although exact durations varied somewhat because of different times spent between runs and between conditions). All participants finished the second session in under an hour.

# Behavioral Measures

During the speech perception task, the experimenter scored the number of key words repeated correctly on each sentence. Each sentence contained four keywords, so there were a total of 64 possible correct responses in each condition (four words per sentence, eight sentences per run, two runs per condition). The experimenter also administered an abbreviated version of the NASA Task Load Index (TLX; Hart and Staveland, 1988) after each block. Following Mackersie et al. (2015), the present study included only four of the six subscales from the orginal TLX (Mental Demand, Performance, Effort, and Frustration) and slightly revised the questions to make them more appropriae for the current listening task context. The other two dimensions, Physical Demand and Temporal Demand, were excluded in the interest of time, and because this listening task did not impose any physical or response time demand on participants. This task was administered orally, asking participants to rate each measure on a scale of 0–20, in order to permit the participant to remain still during performance of the task.

# Physiological Recordings

Immediately prior to the speech perception task period, participants washed their hands carefully with soap and water, and let them dry thoroughly. During the task, autonomic nervous system responses were collected using a Biopac MP150 Data

Acquisition System, including a Biopac GSR100C amplifier (electrodermal response) and PPG100C (pulse plethysmograph) amplifiers. Acquisition and analysis was conducted using AcqKnowledge 4.3 software (Biopac Systems, Inc.) running on a Dell Latitude E6430 running Windows 7.

### Electrodermal Response Measures

Self-adhesive Ag/AgCl electrodes for measuring skin conductance were affixed to the palmar surface of the medial phalanges of the first (index) and second (middle) finger on the participant's right hand. Following recommended procedures, the electrodes were left in place for at least 5 min before data collection began (Potter and Bolls, 2012). The tonic conductance, in microSiemens (µS), between the two electrodes was recorded with an initial gain of 5 µ/V and at a sampling rate of 2.5 kHz. The signal was subsequently resampled to 19.5 Hz to facilitate digital processing (see Arnold et al., 2014 for comparable methods). The resulting tonic skin conductance level (SCL) curve was then smoothed using the built-in AcqKnowledge algorithm with baseline removal (baseline estimation window width of 1 s), and phasic SCRs were automatically identified from this signal as peaks greater than 0.01 µS occurring within a window beginning 1 s after the warning tone (to avoid including responses to the tone itself) and ending 10 s later (about 1 s before the tone indicating that participants should begin speaking). Two SCR-related measures were examined:

### Blood Pulse Measures

A pulse plethysmograph transducer (TSD200) was affixed securely but comfortably using a Velcro band to the palmar surface of the distal phalange of the participant's right ring (third) finger. This transducer emits an infrared signal and calculates the amount that has been reflected by the blood volume in the capillary bed it faces (Berntson et al., 2007). Reflectance, and thus signal level, increases with increased capillary blood volume. This signal, in volts, was initially digitized at a sampling rate of 2.5 kHz, was subsequently down-sampled to 312.5 Hz to facilitate digital analysis, and was then digitally band pass filtered (Hanning) between 0.5 and 3 Hz to remove potential artifacts. The resulting signal is periodic, with a frequency (PR) corresponding to heart rate. However, because this is a measure derived from capillary volume rather than directly from the heart signal, we will refer to it as PR rather than heart rate.

Following a combination of methods used by Potter et al. (2008) and Wise et al. (2009), PR and volume were calculated in 1 s increments over the 10 s beginning at the first warning beep for each trial, and referenced to the baseline (pre-stimulus) respective PR or volume calculated over the 2 s immediately preceding the beep for each trial.3 This resulted in scores centered around 1, with values greater than 1.0 indicating a heart rate acceleration or increase in PA and scores less than 1.0 indicating deceleration or decrease in PA. Two blood pulse measures were examined:

# RESULTS

### Keyword Recognition (Intelligibility)

In order to meet the criteria for application of analysis of variance, proportion correct responses were transformed into rationalized arcsine units (RAU; Studebaker, 1985), shown in **Table 2**. This is simply a linear transformation of the results of a traditional arcsine transformation, with the goal of putting the transformed values into a range that is comparable to that of the original percentages over most of the range of values (i.e., between the "stretched" tails of the distribution). Results of a generalized linear model analysis of variance (ANOVA) with condition treated as a repeated measure showed a significant effect of condition, F(3,39) = 44.47, p < 0.001, η 2 p = 0.77, with the unmasked, natural condition (115.5 RAU) being significantly better understood (p < 0.001 in all cases by Tukey HSD post hoc analysis) than the other three (speech-shaped noise = 93.3, twotalker babble = 91.9, and synthetic speech = 98.2, all values in RAU). There was also a significant difference between two-talker babble and synthetic speech (p = 0.04). However, there was no significant difference between speech-shaped noise or two-talker babble conditions, suggesting that, as intended, the two masked speech conditions were comparable in terms of intelligibility and both were significantly less intelligible than the unmasked speech.

### Subjective Task Demand (Self-Report)

Scores on the four Task Load Index questions (Mackersie et al., 2015) were relatively similar across three difficult conditions, as shown in **Table 3**.

Because the different sub-scales of the NASA TLX address distinct theoretical constructs related to task load, separate analyses of variance were conducted to determine whether listeners' subjective ratings of mental demand, performance, effort or frustration differed across the four conditions. Results showed significant main effects of condition for all four scales: Mental Demand, F(3,39) = 28.13, p < 0.001, η 2 p = 0.68; Performance, F(3,39) = 10.13, p < 0.001, η 2 p = 0.44;

3A 10 s window was used because preliminary observations suggested that some listeners were (physiologically) anticipating the signal to begin speaking that occurred 12 s after the beginning of the trial. By ending the analysis window 2 s before the signal to begin speaking, it was possible to avoid including response properties that might pertain mainly to such anticipation.

### TABLE 2 | Behavioral and physiological measures obtained for each condition.

Standard deviations shown in parentheses for all measures.

### TABLE 3 | Mean scores on the NASA TLX subscales in each condition.

Effort, F(3,39) = 22.38, p < 0.001, η 2 p = 0.63; Frustration, F(3,39) = 5.42, p = 0.003, η 2 p = 0.29. The only posthoc (Tukey HSD) pairwise comparisons between conditions that were statistically significant were those that included the unmasked, natural speech condition (p < 0.001) for all scales except Frustration, for which the comparison between unmasked natural speech and synthetic speech was significant only at the p = 0.047 level, while the comparisons of unmasked natural speech with the speech-shaped noise masking and two-talker babble masking conditions were both significant (p = 0.03 and p = 0.003, respectively). Although extremely tentative at this point, these results suggest that future research exploring task load for listening to speech masked by other speech might benefit from focusing specifically on listeners' sense of frustration in addition to broader subjective measures of overall task load. Overall, these results suggest that, at least as far as can be determined by self-report, listeners found the degraded speech conditions to be comparatively more demanding than the unmasked natural speech, but not differently demanding compared to one another. However, it must be noted that all scores were relatively low (below 10 on a 20-point scale) suggesting that the overall task was not perceived as particularly demanding.

### Physiological Measures

Results from the four physiological measures, SCR frequency, SCR amplitude, PR, and PA, calculated for all four conditions are shown in **Table 2**. There were no significant (p < 0.05) (uncorrected) Pearson product-moment correlations between any of the measures within each of the four conditions, nor were there any significant correlations across conditions within any of the four measures. These scores were submitted to linear mixed model (SAS 9.3 PROC MIXED, SAS Institute Inc, 2011) ANOVA with repeated measures.4

### Skin Conductance Response

A comparison of SCR frequency across the four conditions showed no significant effect of condition, F(3,39) = 2.03, p = 0.13, η 2 p = 0.14. However, the ANOVA of SCR amplitude showed a significant effect of condition, F(3,36.2) = 3.02, p = 0.04, η 2 p = 0.21. Note that three cells were omitted from this design because there were no SCR peaks for those subjects in those conditions. Post hoc (Tukey HSD) analyses showed that this effect was carried entirely by a significant difference between the two-talker babble and the speech-shaped noise conditions (padj = 0.031). This suggests that listeners showed a stronger electrodermal response when presented with speech in a background of two-talker babble as compared to a background of speech-shaped noise.

### Pulse Rate

A graph of mean PR (calculated over 10 consecutive 1 s windows beginning at the warning beep prior to the stimulus and referenced as a proportion of the average PR calculated over the 2 s immediately preceding the beep) is shown in **Figure 1**.

Results of a linear mixed model analysis of variance (ANOVA) with two within-subjects measures (condition and time period) showed no significant effect of condition, F(3,39) = 1.31,

4Effect sizes were calculated independently from F and p statistics by first estimating type III sums of squares within the PROC MIXED procedure, and then applying the methods described by Bakeman (2005).

p = 0.28, η 2 p = 0.09, and no significant interaction, F(30,520) = 0.79, p = 0.79, η 2 p = 0.04. However, the effect of Time Period was significant, F(10,520) = 4.65, p < 0.001, η 2 p = 0.08. Post hoc (Tukey HSD) analyses show that point T + 7 (7 s after the start of the trial, roughly 6 s after the start of the sentence, and between 2 and 3 s after the end of the sentence/stimulus) exhibited a significantly lower PR when compared to all other time points except T + 6. T + 3 and T + 6 were also significantly different, but no other pairwise comparisons were significant at the p < 0.05 level. Thus, there appears to be a slight (but non-significant) increase in PR about 2–3 s after the beginning of the trial, approximately when we might expect the beginning of a response to the onset of the stimulus, followed by a significant decline in relative heart rate approximately when we might expect to see a response subsequent to the end of the stimulus. Note that a change of about 4%, as seen here, reflects a change of approximately 3 beats (or cycles) per minute, given an observed grand average PR of 74.7 beats per minute across all participants and conditions. Although this amount of change may seem small, it is relatively large compared to changes in PR seen in response to auditory stimuli in previous studies, e.g., Potter et al. (2008) (mean change < 1 BPM).

Even though the lack of a significant interaction effect does not strictly license examination of post hoc test results involving pairwise differences within the interaction (i.e., time point × condition), such planned comparisons may be informative in guiding the design of future research. Indeed, comparison of the lowest PRs for the speech-shaped noise, synthetic speech, and two-talker babble conditions vs. the Unmasked natural speech PR at the same time point (i.e., T + 6 for speech-shaped noise vs. T + 6 for Natural Speech, and T + 7 for synthetic speech and two-talker babble vs. T + 7 for Natural Speech) show large differences. Testing these differences using uncorrected post hoc comparisons5 and comparing the resulting p value to a threshold corrected for sequential multiple comparisons (Holm, 1979) shows that the difference for speech-shaped noise, synthetic speech and twotalker babble are all significant (puncorrected = 0.011, 0.013, and 0.024, respectively), suggesting that degradation of speech induces significantly greater decrease in heart rate than does unmasked speech, and evidence of this increased reactivity is found approximately 6–7 s following the beginning of the stimulus.

### Pulse Amplitude

A graph of mean PA over 10 consecutive 1 s windows and referenced as a proportion of the average PA over the 2 s immediately preceding the beep in a manner comparable to that of PR in **Figure 1**, is shown in **Figure 2**.

Results of a linear mixed models ANOVA with two repeated measures (condition and time period) showed a significant main effect of condition, F(3,39) = 3.52, p = 0.02, η 2 p = 0.21, and of time period, F(10,520) = 59.06, p < 0.001, η 2 p = 0.53, but no interaction, F(30,520) = 0.95, p = 0.54, η 2 p = 0.05. Post hoc (Tukey HSD) analyses show no significant pairwise differences between conditions (p > 0.08 in all cases). However, post hoc (Tukey HSD) analyses of pairwise differences in time point were found to be significant (p < 0.05 in all cases reported here) as follows: T vs. T + 5 and beyond; T + 1 vs. T + 4 and beyond; T + 2 vs. T + 4 and beyond; T + 3 vs. T + 5 and beyond; T + 4 vs. T + 5 and beyond; T + 5 vs. T + 6 and beyond; T + 6 and T + 7 and beyond.

Although the interaction between time-point and condition was not significant, and none of the pairwise comparisons between conditions overall or at the same time point were significant in a corrected (Tukey HSD) analysis, as with the PR date discussed above, unlicensed examination of subsidiary effects may provide guidance for subsequent research. In this spirit, examination of the graph combined with pairwise comparisons between conditions suggest that the significant effect of condition is possibly being carried by a difference between masked and unmasked conditions. According to these analyses, there does not appear to be any meaningful difference between the two masked conditions: speech-shaped noise vs. two-talker babble, puncorrected = 0.748; Unmasked natural speech vs. synthetic speech, puncorrected = 0.968, but there are visible differences between the two unmasked conditions that are significant by uncorrected post hoc analyses (although these are not significant when compared to a Bonferroni–Holm-corrected threshold): speech-shaped noise vs. unmasked natural Speech, puncorrected = 0.042; speech-shaped noise vs. synthetic speech, puncorrected = 0.038; two-talker babble vs. unmasked natural speech, puncorrected = 0.020; two-talker babble vs. synthetic speech, puncorrected = 0.018).6 Further, it appears that the preponderance of any such effects occurs in the last 5 or 6 time periods, a time at which the masked stimuli (speech-shaped noise and two-talker babble) exhibit considerably lower PA values than do the unmasked stimuli (Natural Speech and synthetic speech). Specifically, the greatest difference appears to be occurring around time point T + 8 or T + 9, with the divergence beginning around time T + 5 or T + 6. It may be noted that the peak PA response (at T + 9) is occurring about 2 s later than the peak PR deceleration (T + 7), though they begin at about the same time. This may be due to differences in the speed of response of the two measures or to the cognitive phenomena to which they are related, or both (see Discussion). Although these results must be considered preliminary due to the increased probability of Type 1 error through the reliance on uncorrected post hoc statistical analyses, overall it can be said that there appears to be a difference in the magnitude of the PA response to masked as compared to unmasked speech, and this difference begins to become apparent approximately 5–6 s after stimulus onset, and peaks 2–3 s after that.

5Uncorrected comparisons were used because standard post hoc corrections take all pairwise comparisons into account, drastically increasing corrected p-values to compensate for comparisons that are irrelevant to the present analysis. Instead, we have chosen to report raw p-values along with the critical p-value as determined by Holm–Bonferroni sequential correction as implemented for Excel by Justin Gaetano (Gaetano, 2013) for the number of pairwise comparisons that are actually relevant to the present analyses.

6 Similar results are obtained when examining pairwise comparisons at specific time points, e.g., T + 8 and T + 9.

### Correlations

In order to explore possible relationships between subjective measures of task demand and individual physiological responses, Pearson product-moment correlations were carried out for each of the four conditions between all four subscales of the TLX collected here (Mental Demand, Performance, Effort and Frustration) and six physiological measures: SCR Frequency, SCR Amplitude, Mean Pulse Rate, Mean Pulse Amplitude, and Pulse Rate and Amplitude at the respective minima shown in **Figures 2** and **3** (for Pulse Amplitude this was time T + 9 for all four conditions, while for Pulse Rate this was time T + 7 for all conditions except speech-shaped noise masking, for which it was T + 6). Due to the large number of comparisons, none of these tests were significant at a level corrected for multiple comparisons (p < 0.002). However, a general trend was observed suggesting that the measure of Mean Pulse Volume might be more likely to correlate with TLX subscales, in that it correlated with ratings of Performance (unmasked natural speech, r = 0.66, puncorrected = 0.01; synthetic speech, r = 0.70, p = 0.005), Effort (two-talker babble Masker, r = 0.53, puncorrected = 0.05; synthetic speech, r = 0.56, puncorrected = 0.04), and Frustration (unmasked natural speech, r = 0.69, puncorrected = 0.007). The only other physiological measures correlating with a TLX subscale measure with a significance at or below p = 0.05 were Mean Heart Rate (with Performance in the unmasked natural speech condition, r = 0.71, p = 0.005) and Pulse Amplitude at time T + 9 (with Performance in the Speech-shaped noise masking condition).

### DISCUSSION

Behavioral measures of performance (proportion of key words repeated correctly) and subjective task demand showed that all degraded conditions were significantly less intelligible and imposed greater task demands than the unmasked natural speech condition. Additional findings also suggest that the synthetic speech condition may have been marginally less difficult, as reflected in performance, than the two-talker babble condition, and it may also have been somewhat less frustrating in comparison to unmasked natural speech than were the two masked conditions. These findings suggest that finer-grained assessments of subjective task load and behavioral performance might be informative in future research with stimuli like those used here.

In the present study, participants showed a significant increase in SCR to sentences presented in two-talker babble as compared to those presented in speech-shaped noise. Mackersie and Cones (2011) interpreted their finding that SCRs were elevated in more difficult dichotic digit task conditions as confirming that the SCR may index task demand, but Mackersie et al. (2015), who found no effect of changing SNR (and therefore presumably task demand), moderated these findings by suggesting that SCR may only be sensitive to task demand when performance is very good and/or effort is low. The present results, however, suggest a slightly different interpretation, namely that the SCR may be most indicative of the operation of selective attention. In the present experiment, performance and ratings of task

demand were comparable across the two masked conditions, yet the SCR response was significantly stronger when the masker contained intelligible speech. This also highlights a significant difference between the conditions used by Mackersie and Cones (2011) and Mackersie et al. (2015): In the former, the task involved listening to streams of spoken digits presented simultaneously to each ear (i.e., speech in the presence of intelligible masking speech). In the latter, the masker consisted of a mixture of speech signals from 5 different talkers, two of which were time-reversed, making the mixture potentially much less intelligible, and perhaps closer in intelligibility to the current speech-shaped noise condition. Further research is necessary to investigate the possibility that an increase in skin conductance may correspond to the engagement of attentional mechanisms involved in separating acoustically similar streams of speech.

In contrast, physiological measures of blood PR and PA suggested the possibility that there might be some differences between one or more of the degraded conditions and the umasked natural condition. With respect to PR, the appearance of a significant deceleration approximately 5–6 s after the start of the stimulus is consistent with the expectation that the stimuli in question require some degree of mental processing. Such deceleration is consistent with the appearance of an OR indicating the holding in reserve of cognitive resources in anticipation of having to encode a perceptually demanding stimulus. The lack of any apparent increase in PR during the span of the analysis window suggests that processing these stimuli, once they are encoded, does not require significant additional mental elaboration. Notably, in-depth (but speculative) examination of the main effect of condition suggested a difference between the synthetic and unmasked natural conditions, suggesting that the OR to the synthetic stimuli was stronger (perhaps indicating an anticipation that the stimuli would be perceptually more complex) than for the unmasked natural speech. Further inspection of the data suggested that the same might be true for the other two degraded conditions as well. Even more speculatively, it is possible that there is a slight deceleration within the first 1–2 heart beats after trial onset (time T + 1) followed by a small acceleration (T + 2, T + 3) prior to the large deceleration discussed here. Such a triphasic response (deceleration, acceleration, deceleration) would be consistent with results observed from studies with shorter and less meaningful auditory stimuli (Keefe and Johnson, 1970; Graham and Slaby, 1973; cited in Andreassi, 2007 p. 354). In short, it seems likely that all three types of degraded speech required greater commitment of cognitive resources in the service of initial encoding of the signal (as indicated by a stronger OR for these stimuli), but that synthetic speech may have incurred the greatest demand. Further research is

necessary to better specify the structure of the heart rate response associated with auditory stimuli of sentence length, and to better quantify factors that affect different components of this response.

Finally, there is a clear decrease in PA peaking approximately 4–5 s following the end of the stimulus. Decreased PA has long been associated with an increased demand on working memory capacity (Iani et al., 2004), so this pattern is consistent with the hypothesis that listeners were engaging working memory systems in processing the speech stimuli presented here. Other studies, however, have shown that decreased PA is a physiological response associated with the presence of noise even when task performance is unaffected (Kryter and Poza, 1980; Millar and Steels, 1990). This is then interpreted in terms of the "adaptive costs" model of physiological response to performance under stress, such that decreased PA (and other SNS responses) are considered to reflect "active coping," that is, the application of increased effort to maintain performance in the presence of an environmental stressor (see discussion by Parsons, 2007). Indeed, research by Mattys et al. (2013) suggests that exogenously induced anxiety or stress can influence the application of capacity-demanding processes to speech perception. This interpretation would be consistent with the tentative determination that there may be a difference in the decrease in PA associated with conditions containing added noise (two-talker babble and speech-shaped noise) as compared to that associated with conditions without noise (unmasked natural speech and synthetic speech). If the reliability of this distinction is borne out by future research, its appearance here may be interpreted as reflecting either a greater commitment of working memory resources to the listening task in the two masked speech conditions as compared to the unmasked conditions, or (also) a more complex response incorporating both an autonomic stress response associated with performing a task in noise as well as the greater cognitive effort required to maintain performance when listening to degraded speech. Further research is necessary to determine whether there is in fact a reliable distinction between the PA response to speech in noise as compared to similarly difficult unmasked speech, and, if so, to further untangle direct and indirect effects of noise on the application of working memory to speech perception in both masked and unmasked conditions.

While the determination that there is an overall increased commitment of working memory capacity to speech perception in degraded conditions would be completely consistent with the predictions of the effortfulness hypothesis, the apparent discrepancy between the conclusions drawn from the different pulse measures (rate vs. amplitude) must still be considered. That is, why does the synthetic speech condition, which was significantly more difficult to understand than the unmasked natural speech condition according to both self-reported effort ratings and performance measures, seem to incur greater demand on mental processing as indexed by PA, but not according to the measure of PR? One clue to an answer to this question lies in the observation that the peak of the PA marker seems to be occurring somewhat earlier during the window of analysis than did the PR response. This temporal difference likely reflects some combination of: (1) a relative delay in the responsivity of the two systems (cardiac deceleration vs. peripheral vasoconstriction), (2) differential contribution of sympathetic arousal affecting both end organs as compared with the combined effects of parasympathetic and sympathetic systems on PR, and (3) each measure reflecting a response to different stimulus processing demands.

While it is entirely likely that the two systems respond on different timescales, the fact that they show discrepant patterns of reactivity for different sorts of stimuli is also quite consistent with the idea that the two measures reflect responses to different aspects of speech processing. In this regard, it is important to note first that previous research comparing physiological responses associated with the perception of degraded (but unmasked) speech to those associated with masked speech has already suggested that these tasks may differ in terms of the degree to which cognitive processes are applied. In particular, Zekveld et al. (2014) found that noise-vocoded speech (degradation without masking) evoked a smaller pupillary response (a measure of ANS reactivity reflecting both sympathetic and parasympathetic contributions) than did noise- and speechmasked natural speech, even when performance was matched. Moreover, regional brain activity, as measured with the BOLD response, in regions associated with speech perception and selective attention (bilateral superior- and medial-temporal gyri, and dorsal anterior cingulate cortex) changed parametrically with pupil dilation, suggesting that different types of degradation result in different degrees of demand on attentional and speech processing systems specifically related to segregating target speech from competing signals. Thus, the differences between responses to masked vs. unmasked stimuli observed here in the PA measures may reflect differences in the engagement of selective attentional mechanisms associated with segregating target from masking signals. On the other hand, the response pattern observed in the PR measures may reflect overall differences in the difficulty of encoding degraded signals as such, or perhaps even differences in the degree to which masked signals are perceived as stressful, arousing or emotionally evocative (Bradley and Lang, 2000). The fact that the one pattern (PA, related to segregation) appears later in the pulse record than the other (PR, related to orienting and preparation for stimulus encoding) even though one might arguably expect segregation to incur demand earlier in processing than encoding, may be a result of differences in the speed of response of the two systems. Further research is necessary to determine whether stimulus differences that lead to differences in PR vs. PA measures are in fact associated with differential demands on segregation vs. encoding, and, if so, whether they have similar or differing effects on downstream performance (i.e., recall or understanding of the target speech, or processing of subsequent speech), as might be predicted by the effortfulness hypothesis.

In summary, the present results suggest that listening to speech in the presence of a masking sound or sounds introduces additional, or different, processing demands beyond those associated with the simple difficulty of understanding degraded speech. From the present results it cannot be determined

whether these additional demands derive from the application of additional, or different, cognitive mechanisms such as those involved in selective attention (as suggested by Zekveld et al., 2014 in explaining related findings) or whether they instead reflect aspects of an affective or emotional stress-like response to the presence of a noxious stimulus (the masker). Given that anxiety may also introduce changes in the cognitive processes applied to speech perception (Mattys et al., 2013), further research is necessary to distinguish between psychophysiological and behavioral consequences of both stress and cognitive demand on speech processing in adverse conditions.

### AUTHOR CONTRIBUTIONS

AF and MM designed the study with input from BC and AA. BC provided the stimuli. AF and AA collected the data and conducted all analyses with input from MM and BC. AF wrote the paper, and MM, BC, and AA provided comments on drafts.

### REFERENCES

# FUNDING

This research was partially supported by funding from the Department of Speech, Language and Hearing Sciences and the Office of the Provost, Purdue University, under the auspices of a Provost's Fellowship for Study in a Second Discipline granted to AF.

### ACKNOWLEDGMENTS

We are grateful to our participants for their time and interest, to Anne Smith, Bridget Walsh and Janna Berlin for assistance with equipment and study design, to Rob Potter and Annie Lang for advice on data analysis, to Bruce Craig and Rongrong Zhang for assistance with statistical analyses, and to Audrey Bengert, Allison Gearhart, Jessica Lorenz, Alyssa Nymeyer, Nikeytha Ramsey, Jim Rodeheffer, Ashlee Witty and McKenzie Wyss for assistance with running the experiment and conducting data analysis.

(Cambridge, MA: Academic Press), 17–26. doi: 10.1016/B978-0-12-386525- 0.00004-4

style, and masker. J. Speech Lang. Hear Res. 57, 1908–1918. doi: 10.1044/JSLHR-H-13-0076

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Francis, MacPherson, Chandrasekaran and Alvar. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Effects of Aging and Adult-Onset Hearing Loss on Cortical Auditory Regions

Velia Cardin1, 2 \*

1 Department of Experimental Psychology, Deafness, Cognition and Language Research Centre, University College London, London, UK, 2 Department of Behavioural Sciences and Learning, Linnaeus Centre HEAD, Swedish Institute for Disability Research, Linköping University, Linköping, Sweden