Modeling and MEG evidence of early consonance processing in auditory cortex
[['Pitch is the perceptual correlate of the periodicity in a sound’s waveform , and thus a fundamental attribute of auditory sensation .', 'It forms the basis of both music and speech perception .', 'However , understanding pitch processing as elicited by concurrent sounds in human auditory cortex is still a major challenge in auditory neuroscience [1–5] .', 'A combination of two sounds that simultaneously elicits two different pitches is called a dyad , and the pitch interactions within the dyad give rise to a sensation that can be characterized by its consonance or dissonance .', 'Loudness , timbre , and the fundamental periodicities of the two sounds can have subtle effects on whether a dyad is perceived as consonant or dissonant .', 'However , the dominant factor in determining the degree of a dyad’s consonance is the relationship between the fundamental periods of the sounds that make up the dyad: simple periodicity ratios result in more consonant sensations .', 'In contrast , the sensation becomes more and more dissonant as the complexity of the periodicity ratio increases [6 , 7] .', 'It has been previously proposed that dissonance correlates with the beating or roughness sensation that is elicited by the interfering regularities of the dyad components [6 , 7] .', 'However , listeners who showed impaired pitch perception but were sensitive to beating and roughness were unable to differentiate between consonant and dissonant dyads [1 , 8] .', 'This suggests that pitch- rather than roughness-related auditory processing is responsible for consonance perception .', 'Neurophysiological evidence for a close link between consonance and pitch has recently been provided by Bidelman and colleagues [2] .', 'Their study showed , using electroencephalography ( EEG ) , that the amplitude of the cortical pitch onset response ( POR ) is strongly modulated by a dyad’s perceived consonance .', 'The POR is a pitch-selective component of the transient auditory evoked potential/field ( AEP/AEF ) that occurs within the time range of the well-known N100 deflection , around 100 ms after pitch onset [9] .', 'The morphology of the POR is strongly correlated with the perceived pitch in single tones: its latency scales linearly with the period of the sound and its amplitude increases with the strength of the pitch percept [9 , 10] .', 'The neural sources of the POR are located in the anterolateral section of Heschl’s gyrus ( alHG ) in auditory cortex [2 , 9] , in agreement with the anatomical location of pitch-selective neurons in non-human primates ( e . g . , [11–13] ) , and with pitch-selective regions that were reported for human listeners [14–18] .', 'Further experiments in human subjects demonstrated that the dyad-evoked frequency-following response in the brainstem is predictive for the perceived consonance of a dyad ( for a review , see [19] ) .', 'However , functional magnetic resonance imaging ( fMRI ) studies showing selective activation to consonance/dissonance contrasts in the superior temporal gyrus [20] and in frontal cortex [21] led the auditory community to link neural representations of consonance and dissonance with higher cognitive processes [22] .', 'In this study , we used a combined experimental and theoretical approach to assess whether consonance and pitch share similar processing mechanisms in human auditory cortex .', 'Towards this goal , we first developed an ensemble model of cortical pitch responses , specifically designed to understand the mesoscopic representation of pitch in alHG .', 'The model can account , mechanistically , for the POR latency effects that have been reliably reported in numerous experimental settings [9] but remained poorly understood .', 'Second , we recorded the AEF elicited by consonant and dissonant dyads using magnetoencephalography ( MEG ) .', 'Our experimental results revealed a strong correlation between the POR latency and the degree of consonance , extending previous EEG findings [2] .', 'Finally , we aimed to replicate the results from the MEG experiment using our model .', 'If the hypothesis that consonance and pitch are processed by similar mechanisms in cortex is correct , we would expect the model to explain the dependence of POR latency on the degree of consonance without the inclusion of higher processing stages within the auditory hierarchy [20 , 21] .', 'In line with this hypothesis , the model provides a quantitative explanation for the relationship between the POR dynamics and consonance , suggesting that consonance and dissonance perception might be linked to pitch processing regions in auditory cortex , prior to higher-order processing .'], ['Next , we recorded neuromagnetic fields evoked by six different dyads from 37 normal hearing subjects .', 'Data were preprocessed using standard MEG procedures and equivalent current dipoles were fitted for the POR , independently in each subject and hemisphere , and pooled over conditions ( see Methods ) .', 'Dipole locations in Talairach space are plotted in Fig 4B .', 'Dyads consisted of two IRN sounds .', 'The lower note pitch was 160 Hz; the pitch of the upper note was adjusted accordingly to form either a consonant dyad ( unison , P1; perfect fifth , P5; major third , M3 ) or a dissonant dyad ( tritone , TT; minor seventh , m7; minor second , m2 ) .', 'To dissociate the energy onset response in planum temporale from the POR in alHG , the dyads were preceded by an energy-balanced noise segment , cross-faded with the dyad to avoid discontinuous waveforms ( like for the single IRN sounds analyzed in the previous section; see Methods ) .', 'Fig 4A presents the MEG grand-mean source waveforms , for both hemispheres , in response to the six stimulus conditions .', 'The noise onset from silence ( depicted in grey below the source waveforms ) was followed by a transient P1m-N1m-P2m AEF complex .', 'Since the first stimulus segment did not vary between conditions , we did not expect to find any significant differences in the corresponding neuromagnetic activity at this point .', 'In contrast , the transition to the second stimulus segment ( IRN dyads , black signal below the source waveforms ) elicited prominent POR waves and the morphology of the POR varied considerably between conditions .', 'Fig 4C shows close-up views of the POR .', 'Consonant dyads ( pooled conditions [P1+P5+M3] ) elicited a much earlier ( p < . 0001 ) and larger ( p < . 0001 ) POR than dissonant dyads ( pooled conditions [m7+TT+m2] ) .', 'Fig 4D depicts 99% bootstrap confidence intervals for the POR amplitudes and latencies pooled over hemispheres in response to the experimental conditions; the activity pattern observed here also points to a close relationship between the degree of a dyad’s consonance and the morphology of the respective POR .', 'When pooling across conditions , we found a difference between the left and the right hemisphere in the POR amplitude ( p = . 01 ) , but not in the POR latency ( p = . 36 ) ; also , the difference between the neuromagnetic responses to consonant or dissonant dyads did not significantly vary between hemispheres ( latency: p = . 58 , amplitude: p = . 48 ) .', 'The POR latency difference in response to consonant and dissonant dyads in alHG suggests that consonance and dissonance are computed at relatively early stages of the cortical auditory hierarchy .', 'We used our model of cortical pitch processing , designed to reproduce the neuromagnetic responses elicited by iterated rippled noises , to test this interpretation .', 'If the differential responses to consonance and dissonance in alHG were intrinsic to pitch processing , we would expect our model to be able to reproduce this behavior .', 'First , we verified that the model was able to provide a joint representation of the two pitches comprised in the dyads; the results are shown in Fig 5A–5C .', 'As in Fig 3 , the plots show the average activation of the different ensembles ( x-axis ) for the 13 dyads in the chromatic scale ( y-axis ) .', 'The vertical line in Fig 5C indicates the first note common to all dyads; the diagonal the neural representations of the second notes .', 'It should be emphasized that even phenomenological ( biologically inspired but with low realism ) models of pitch perception are generally unable to decode sounds with concurrent pitches ( e . g . , [45 , 46]; see [47] for a review ) .', 'Fig 5J shows the latency predictions of the model and the experimental data for the respective dyads ( see also Fig 5D–5I ) .', 'Note that this is a genuine out-of-sample test , since model parameters previously fitted ( see Methods ) were held fixed to account for this new data .', 'Although the model predicted a slightly shorter POR latency for the semitone ( m2 ) dyad than observed ( see Discussion ) , latency predictions match the experimental trend; moreover , the differential response to consonant ( P1 , M3 , P5 ) and dissonant ( m2 , TT , m7 ) dyads found in the MEG data was accurately replicated by the model ( latency of P1 and P5 < latency of dissonant dyads: p < 10−7 , W > 5050; latency of M3 < latency of m2: p = . 00002 , W = 4414; latency of M3 < latency of TT: p = . 38 , W = 3688; latency of M3 < latency of m7: p = . 096 , W = 3878; one-tailed Wilcoxon rank-sum tests performed over the results of N = 60 runs of the model ) .', 'The full temporal dynamics of the dipole moment predicted by the model is shown in Fig 5D–5I .', 'Last , we extended the POR latency predictions to all 13 dyads comprising the entire chromatic scale ( see Fig 5K ) , and tested if the differential responses to consonance and dissonance were generalizable to additional dyads .', 'Following Helmholtz [6] , we considered an extended set of consonant dyads , including the octave ( P8 ) and the perfect fourth ( P4 ) ; and an extended set of dissonant dyads , including the major seventh ( M7 ) and the major second ( M2 ) .', 'Once again , consonant dyads produced shorter latencies than dissonant dyads ( latencies of P1 , P4 , P5 and P8 < latencies of the extended set of dissonant dyads: p < . 0003 , W > 4445; latency of M3 < latency of M2: p = . 00002 , W = 4420; latency of M3 < latency of M7: p = . 75 , W = 3501; one-tailed Wilcoxon rank-sum tests , N = 60 ) .', 'These results , fully in line with our previous findings , suggest that the differential response of our model to consonance and dissonance is a consequence of the harmonic relationships between the periodicities of the two dyad components .', 'These analyses are extended to further families of dyads in S9 Fig , yielding similar results .', 'The model can also be used to explain how interactions between the components of the dyads influence processing time: consonant dyads consist of tones that share a larger number of lower harmonics than the ones in dissonant dyads .', 'For instance , in the just intonation , the perfect fifth of a given fundamental shares one in every two harmonics with that fundamental , whilst only one in 16 harmonics are shared by a minor second and its fundamental .', 'The proposed mechanism is based on the idea that cortical pitch processing is triggered by the joint activation of , at least , three periodicity detectors characterizing a specific harmonic series .', 'Consonant dyads elicit a dramatically larger signal-to-noise ratio in the periodicity detectors tuned to their common harmonics , resulting in a collaborative effort towards pitch extraction that effectively speeds up processing dynamics ( see video S2 Video for an animation depicting the full process ) .'], ['A new systematic interpretation of the POR latency can be deduced from the dynamics of the decoder network: the POR might reflect the amount of time that is necessary for the network to robustly stabilize in a state representing a unequivocal pitch ( see Fig 2 ) .', 'Although an association between POR latency and processing time has been previously hypothesized in experiments ( such as in [9] ) and in a phenomenological model [45] , a detailed understanding of this mechanism was still lacking .', 'In our model , the latency of the POR coincides with the instant in which the net inhibition at the decoder network overtakes the excitatory activity from the periodicity detectors .', 'From a dynamic systems perspective , this is equivalent to the instant in which the trajectory in the phase-space is unequivocally directed towards the attractor state dominated by the neural ensemble that is characterized by the perceived pitch ( see the phase space portrait in S1 Video ) .', 'The model only performs a robust perceptual decision concerning stimulus pitch after the cortical system identifies three peaks from the harmonic series of the stimulus period in the representation of the periodicity detectors .', 'This accounts for the relation between the POR latency and the stimulus period [9] .', 'In addition , this also explains why pitch identification is only robust when the stimulus duration exceeds four times the pitch periodicity [9] .', 'Although previous studies had postulated that cortical pitch processing mechanisms must integrate along several period cycles in order to make a perceptual decision [9] , a specific mechanism for such an integration has not been proposed to the date .', 'Moreover , since phase-locked activity is not robustly present above 50–200 Hz in the cortex [15] , integration along several repetition cycles is only possible in subcortical areas .', 'The decoder network in our model takes advantage of the input harmonic representations provided by an autocorrelation model that does not require phase-locking to transmit information concerning several repetition cycles [28] , and thus provides a parsimonious solution to this problem .', 'Combined , our results suggest that cortical processing of dissonant dyads is slower than the processing of consonant dyads; i . e . , it requires a longer processing time .', 'The model constitutes a physical rationale for this phenomenon: cortical extraction of consonance is based on the accumulation of activity in the columns with preferred periods characterizing the lower harmonics of the target sound; thus , concurrent pitch frequencies sharing common lower harmonics contribute to the build-up of each other’s representation , thereby speeding up the stabilization of the network .', 'Since consonant dyads are characterized by simpler frequency ratios , their components share a larger number of lower harmonics than the components of dissonant dyads , and hence this stabilization is promoted .', 'Early phenomenological models based on Helmholtz’s roughness theory described dissonance as the beating sensation produced by tones with fundamental periodicities that were not harmonically related [6 , 7] .', 'More recent explanations of consonance , based on pitch processing , have linked the regularity of the autocorrelation harmonic patterns elicited by dyads to their evoked consonance and dissonance percepts [1 , 4 , 19] .', 'Thus , previous phenomenological consonance models have consistently described the degree of consonance as the perceptual correlate of the degree of overlap between the dyad components’ lower harmonics .', 'Our model introduces a potential explanation for the biophysical rationale underlying this description .', 'Although our modeling results generally show a good fit with the data from the MEG experiment , the model prediction falls around 5 ms short when explaining the POR latency evoked by the minor second dyad .', 'This underestimation might result from the limited number of harmonics considered during the integration step in the decoder network: dissonant dyads , whose components do not share any common harmonic within the first three peaks of their harmonic series , present comparable processing times .', 'More accurate results would occur if an adaptive mechanism adjusted the number of harmonics required to trigger the decoding process according to the degree of peak overlap in the input .', 'This adaptive mechanism would be necessary to explain how humans can differentiate dyads that differ in a quarter of a semitone .', 'Our study did not assess whether the general ( yet not universal [3 , 5] ) association between consonance and pleasantness might be a consequence of the differential responses of the decoder network to consonant and dissonant dyads .', 'Future work should investigate whether this link could be better explained by processes at higher levels of the auditory hierarchy that might be more sensitive to cultural and background modulations .', 'Our neuromagnetic findings concerning the POR morphology in response to consonant and dissonant dyads resemble and extend recent EEG data reported by Bidelman and Grall [2] , and by Proverbio et al . [48] .', 'Specifically , Bidelman and Grall [2] applied EEG in a smaller sample ( N = 9 ) of musically trained listeners and revealed a close relation between their subject’s consonance/dissonance ratings and the morphology of the POR that was elicited by the respective dyads in alHG .', 'In their study , the POR latency difference between consonant and dissonant dyads ( cf . their Fig 4b ) appears to have a non-significant ( p = . 22 ) effect size , smaller than the results that were obtained in our study by means of MEG .', 'One reason for this might be that Bidelman and Grall [2] applied shorter IRN stimuli with a higher number of iterations , resulting in an increased saliency of the pitch percept; moreover , they employed a dichotic stimulation paradigm in which each ear was presented with only one dyad component , whereas in our experiment sounds were delivered diotically to the listeners .', 'Since our model does not predict an effect of the number of iterations of the IRNs on latency ( predictions for dyads of 32 iterations IRNs are shown in S9D Fig ) , we speculate that the diotic presentation of the dyads is responsible for the stronger effect shown in our data .', 'This hypothesis cannot be explored by our current model because it does not consider binaural integration .', 'The modeling of this process could be informed by the divergence between Bidelman and Grall’s and the present results in future work .', 'In line with results from previous experiments ( e . g . , [9 , 11 , 30 , 32 , 49] ) , our findings are consistent with the notion that lateral HG acts as a cortical pitch center .', 'Many of these earlier studies employed IRN stimuli; however , a number of fMRI experiments ( e . g . , [50 , 51] ) have argued that the activity observed in HG might be confounded by slow fluctuations in the IRN spectrum .', 'Indeed , pitch-sensitive cell ensembles seem to overlap , within HG , with other neural populations that are more sensitive to other ( e . g . , spectral ) sound features [52 , 53] .', 'However , this does not speak against the existence of a pitch-specialized subregion in HG since overlapping neuron ensembles are difficult to disentangle by means of fMRI [54] .', 'Thus , based on the relatively homogeneous pattern of results in the current study and in experiments using different stimuli and neuroimaging methods [2 , 18 , 31 , 35] , the existence of a pitch center in HG might be viewed as highly probable .', 'Numerous phenomenological models have been designed to predict pitch for a wide range of complex sounds ( e . g . , [24 , 27 , 28 , 45 , 46 , 55] , see [47] for a review ) .', 'These models can account for a variety of perceptual phenomena [24] .', 'The weaker pitch of frequency-transposed harmonic complex sounds [56] , for example , was explained using nonlinear filters to simulate the compression taking place in the basilar membrane [24] .', 'Although the present work constitutes a first effort towards a mechanistic model of early pitch and consonance processing , future efforts should focus on broadening the model scope to pitch phenomena not addressed in the current model implementation .', 'The correlation between pitch and cortical AEFs has been qualitatively studied in the Auditory Image Model’s buffer [46] and its derivative [57] , and quantitatively in the derivative of the model output in [45] and [10] .', 'However , these models did not provide a mechanistic explanation of the processes underlying the generation of the POR or its latency dependence with pitch .', 'Other models , designed to explain the biophysical mechanisms of pitch perception , focused primarily on subcortical processing .', 'Two of these models describe how neurons , mainly in subcortical nuclei , might process periodicities from the auditory nerve activity: Meddis and O’Mard’s model [23] proposes a biophysical implementation of the summary autocorrelation function [27 , 28] , based on the joint action of chopper neurons in the cochlear nucleus and coincidence detectors in the inferior colliculus .', 'More recently , Huang and Rinzel [58] described a neural implementation of an array of coincidence detectors able to detect periodicities by comparing neural activity across different cochlear channels .', 'Despite their mechanistic differences , both models present an output comparable to that of the autocorrelation function [58] .', 'The model presented here is downstream with respect to Meddis’ and Huang’s models because it focuses on explaining how pitch decisions based on the later subcortical representation are made in alHG .', 'In our model , pitch processing is mediated by a connectivity pattern among interacting columns specialized in characteristic periods .', 'Similar connectivity patterns were found in mice AC , stemming from L4 and targeting L6 neurons [36] , and in the cat AC in earlier studies [37] .', 'Neurons that respond selectively to harmonically related input have been recently identified in the core region of the marmoset’s auditory cortex [13] .', 'Inhibitory and facilitatory interactions between neurons encoding harmonically related frequencies are often reported in the mammal auditory cortex ( see [59] for a review ) .', 'Specifically , intracranial recordings in marmoset AC revealed that activation elicited by a given tone resulted in the facilitation of neurons encoding higher harmonics , and in the suppression of neurons encoding lower harmonics [60] , in line with the decoder network mechanisms of our model .', 'Harmonic co-activation has also been shown in human AC [61] .', 'In a more speculative vein , we suggest that this connectivity pattern might result from spike-timing-dependent plasticity ( STDP ) operating over cortical neurons receiving the upstream outputs of periodicity detectors .', 'To illustrate this , let us consider the processing of a sound of fundamental period T . After tone onset , the first periodicity detector responding to the sound provides input to the upstream excitatory ensemble encoding T , which subsequently activates its inhibitory counterpart .', 'Assuming an initial all-to-all connectivity , this inhibitory drive propagates in the network and provides concurrent input to neurons receiving excitatory drive from periodicity detectors 2T , 3T , and so forth .', 'Input synchrony would result in a stronger connectivity change though STDP in these harmonically related ensembles , whilst the uncorrelated asynchronous inputs to the remaining ensembles would result in a net decreased connectivity weight .', 'A similar STDP mechanism for spectral pitch integration was proposed earlier in [62] .', 'The decoding strategy or our model is based on the well-known winner-takes-it-all architecture [29 , 63 , 64]: excitatory populations in the decoding network compete with each other , while the inhibitory ensemble arbitrating this competition is the one in the column that is sensitive to the fundamental period ( Fig 1 ) .', 'In this way , multiple fundamentals can be simultaneously decoded ( Fig 5 ) .', 'Moreover , akin to recent models of sensory integration [29] , once a fundamental period is represented in the decoder network , the activity of the winner column is reinforced by the sustainer network ( rather than the pitch being repeatedly decoded ) .', 'This ensures stability until a significant change in the subcortical input triggers a new decoding process ( see S2 Fig ) .', 'This sustaining strategy is also related to predictive coding-related strategies [45 , 65 , 66] , where top-down efferent convey expectations about the input , whereas bottom-up afferents convey prediction errors [65] .', 'Additional top-down expectations could coexist at higher cognitive levels based on , for example , prior knowledge , experience , or focused attention .', 'Such biases could modulate the sustainer network by increasing the baseline activity of the inhibitory ensembles that characterize the target pitch values , thereby facilitating pitch processing in the decoder network .', 'To summarize , in this study we proposed a model specifically designed to understand the neural mechanisms of cortical pitch processing at a mesoscopic scale .', 'We introduce a possible mechanistic link between the latency of the POR component in the N100 deflection and the processing time required for the system to achieve convergence , explaining the classical result that tones with a lower pitch elicit PORs with longer latencies .', 'More intriguingly , our modeling and experimental results indicate that processing time varies with the degree of consonance in dyads , suggesting that the sensation of consonance and dissonance might start early in auditory cortex , prior to higher-order processing .']]
['Introduction', 'Results', 'Discussion']
['Pitch is a fundamental attribute of auditory perception .', 'The interaction of concurrent pitches gives rise to a sensation that can be characterized by its degree of consonance or dissonance .', 'In this work , we propose that human auditory cortex ( AC ) processes pitch and consonance through a common neural network mechanism operating at early cortical levels .', 'First , we developed a new model of neural ensembles incorporating realistic neuronal and synaptic parameters to assess pitch processing mechanisms at early stages of AC .', 'Next , we designed a magnetoencephalography ( MEG ) experiment to measure the neuromagnetic activity evoked by dyads with varying degrees of consonance or dissonance .', 'MEG results show that dissonant dyads evoke a pitch onset response ( POR ) with a latency up to 36 ms longer than consonant dyads .', 'Additionally , we used the model to predict the processing time of concurrent pitches; here , consonant pitch combinations were decoded faster than dissonant combinations , in line with the experimental observations .', 'Specifically , we found a striking match between the predicted and the observed latency of the POR as elicited by the dyads .', 'These novel results suggest that consonance processing starts early in human auditory cortex and may share the network mechanisms that are responsible for ( single ) pitch processing .']
['In this work , we argue that human auditory cortex processes pitch and consonance by means of a common neural network mechanism operating at early cortical stages .', 'We introduce a neural population model of cortical pitch processing that contains biophysically realistic synaptic and neural parameters .', 'The model quantitatively explains , for the first time , the neuromagnetic responses observed in human auditory cortex during pitch perception .', 'The model is subsequently used to elucidate the cortical processing of musical dyads , in which concurrent pitches lead to the perception of consonance or dissonance .', 'Interestingly , the model predicts that sounds perceived as dissonant need more time for cortical processing than consonant sounds .', 'This prediction is experimentally validated by recording cortical neuromagnetic fields in response to consonant and dissonant dyads .', 'Taken together , our results suggest a novel mechanistic explanation for early cortical processing of musical harmony , in the sense that the differential response to consonance and dissonance starts early , simultaneously to ( single ) pitch processing , in auditory cortex .']
['auditory cortex', 'linguistics', 'medicine and health sciences', 'neurochemistry', 'neural networks', 'brain', 'social sciences', 'neuroscience', 'hearing', 'brain mapping', 'neurotransmitters', 'neuroimaging', 'research and analysis methods', 'sensory physiology', 'consonants', 'computer and information sciences', 'imaging techniques', 'animal cells', 'phonetics', 'gamma-aminobutyric acid', 'auditory system', 'biochemistry', 'psychology', 'cellular neuroscience', 'anatomy', 'cell biology', 'physiology', 'neurons', 'biology and life sciences', 'sensory systems', 'sensory perception', 'magnetoencephalography', 'cellular types', 'pitch perception']