Let us now discuss the most effective acoustic features out of the above mentioned large set for single- and cross-domain emotion recognition. To this end, besides correlation coefficients (*r*) of features with the arousal or valence ratings,we introduce the crossdomain correlation coefficient (CDCC) as criterion. As we strive to identify features which carry similar meaning with respect to emotion in different domains, and at the same time provide high correlation with emotion in the domains by themselves, the purpose of the CDCC measure is to weigh high correlation in single domains against correlation deviations across different domains. Let us first consider a definition for two domains *i* and *j*, namely

$$\text{CDCC}^2\_{f,i,j} = \frac{\left| r^{(i)}\_f + r^{(j)}\_f \right| - \left| r^{(i)}\_f - r^{(j)}\_f \right|}{2} \tag{2}$$

where *r* (*i*) *f* is the correlation of feature *f* with the domain *i*, and "domain" refers to the arousal or valence annotation of a certain data set. We only consider the CDCC across the data sets (speech, music, and sound), not CDCC across arousal and valence.

**Table 2 | ComParE acoustic feature set: 64 provided low-level descriptors (LLD).**

#### **Table 3 | ComParE acoustic feature set: functionals applied to LLD contours (Table 2).**

1Arithmetic mean of LLD/positive 1 LLD. 2Not applied to voice related LLD except F0. 3Only applied to F0.

It is obvious that the CDCC measure is symmetric in the sense that CDCC2 *f* ,*i*,*j* = CDCC2 *f* ,*j*,*i* , and that it ranges from −1 to 1. If a feature f exhibits either strong positive or strong negative correlation with both domains, the CDCC will be near one, where as it will be near −1 if a feature is strongly positively correlated with one domain yet strongly negatively correlated with the other. A CDCC near zero indicates that the feature is not significantly correlated with both domains (although it might still be correlated with either one). Thus, we can expect a regressor to show similar performance on both domains if it uses features with high CDCC.

Next, we generalize the CDCC2 to *J* domains by summing up the CDCCs for domain pairs and normalizing to the range from −1 to +1,

$$\text{CDCC}\_f^J = \frac{\sum\_{i=1}^{J} \sum\_{j=i+1}^{J} \left( \left| r\_f^{(i)} + r\_f^{(j)} \right| - \left| r\_f^{(i)} - r\_f^{(i)} \right| \right)}{J(J-1)}. \tag{3}$$

Intuitively, a regression function determined on features with high CDCC *J f* is expected to generalize well to all *J* domains.

In **Tables 4** and **5**, we now exemplify the CDCC3 across the three domains on selected features, along with presenting their correlation on the individual domains. Note that for the purpose of feature selection, we treat the union of VAM and GEMEP as a single domain ("speech"). Further, in our analysis we restrict ourselves to those features that exhibit high (absolute) correlation in a single domain (termed *sound, speech,* or *music features* in the table), and those with high CDCC3 (termed *cross-domain features*). Thereby we do not present an exhaustive list of the top features but rather a selection aiming at broad coverage of feature types. To test the significance of the correlations, we use t-tests with the null hypothesis that feature and rating are sampled from independent normal distributions. Two-sided tests are used since we are interested in discovering both negative and positive correlations. Significance levels are adjusted by Bonferroni correction, which is conservative, yet straightforward and does not require independence of the individual error probabilities.

Looking at the top sound arousal features (**Table 4**), we find loudness to be most relevant – in particular, the (root quadratic) mean, the linear regression offset (corresponding to a "floor value") and the 99-percentile. This is similar to the ranking for speech. Interestingly, loudness is stronger correlated than RMS energy, indicating the importance of perceptual auditory frequency weighting as performed in our loudness calculation. For music, these three loudness features are not as relevant, though still significantly correlated.

The overall best speech arousal feature is the root quadratic mean of spectral flux – indicating large differences of consecutive short-time spectra – which is interesting since it is independent of loudness and energy, which have slightly lower correlation (cf. above). The "second derivative" of the short-time spectra (arithmetic mean of 1 spectral flux) behaves in a similar fashion as spectral flux itself. However, the correlation of these features with arousal in sound and music is lower. Further, we find changes in the higher order MFCCs, such as the root quadratic mean of delta MFCC 14 to be relevant for speech and music arousal, relating to quick changes in phonetic content and timbre. Finally, mean F0, a "typical" speech feature characteristic for high arousal, is found to be relevant as expected, but does not generalize to the other domains.

The best music arousal features are related to mean peak distances – for example, in the loudness contour and the spectral entropy contour resembling occurrence of percussive instruments, indicating positive correlation between tempo and arousal. In contrast, the peak distance standard deviation is negatively correlated with arousal – thus, it seems that "periodic" pieces of music are more aroused, which can be explained by examples such as dance music. However, it seems that all these three features have a mostly musical meaning, since they only show weak correlations in sound and speech. Yet, a notable feature uniting speech and music is the (root quadratic) mean of the first MFCC, which is related to spectral skewness: arguably, a bias toward lower frequencies (high skewness) is indicative of absence of broadband (mostly percussive) instruments, and "calm" voices, and thus low arousal.

Summarizing cross-domain features for arousal, we find that the"greatest common divisor"of speech, sound, and music is loudness (and – relatedly – energy), but the behavior of functional types is interesting: the quadratic regression offset is much more relevant in the case of music than the mean loudness, which is mostly characteristic in speech and sound. In the NTWICM database of popular music, in fact we often find parabola shaped loudness contours, such that this offset indicates the intensity of the musical climax. A suitable cross-domain feature not directly related to loudness or energy is the spectral flux quadratic regression offset (the ordinate of the "high point" of spectral change).

Judging from the results in **Table 5**, we see that loudness is also indicative of *valence* in sound, music, and speech, but the correlations have different signs: on the one hand, loud sounds as identified by high root quadratic mean of loudness are apparently perceived as unpleasant, as are loud voices. For music, on the other hand, loudness can be indicative of high valence ("happy" music).

Among relevant speech valence features, we find mean energy (change) in the speech frequency range (1–4 kHz) and F0 (quartiles 1 and 2) – F0, however,is a"speech only"feature which exhibits low correlation in the other domains (similarly to the observations for arousal above).

Music valence features overlap with music arousal features, due to the correlation in the ratings. Among the music valence features, the median first MFCC (related to spectral skewness – cf. above) is particularly noticeable as it has "inverse" correlation on speech and music – "percussive"music with a flat spectrum is connotated with positive emotion (high valence) while "noisy" voices are characteristic of negative emotion (low valence).

Cross-domain features for valence are generally rarely significant on the individual domains and hard to interpret – here, in contrast to arousal, it seems difficult to obtain descriptors that generalize across multiple domains.

We now move from discussion of single features to a broader perspective on automatic feature selection for cross-domain emotion recognition. To this end, we consider automatically selected subsets of the ComParE feature set by the CDCC criteria. In particular, for each of arousal and valence, we choose the 200 features that show the highest CDCC2 for the (sound, music), (sound, speech), and (music, speech) pairs of domains. Furthermore, for each of arousal and valence, we select a set of 200 features by highest CDCC3 across all three of the sound, music, and speech domains.

**Table 4 | Cross-domain feature relevance for arousal: top features ranked by absolute correlation (r) for single domain, and CDCC across all three domains (CDCC3 ).**

Significance denoted by \*\*p < 0.00, \*p < 0.01, °p < 0.05, −p ≥ 0.05; Bonferroni corrected p-values from two-sided paired sample t-tests.

#### **Table 5 | Cross-domain feature relevance for valence: top features ranked by absolute correlation (r) for single domain, and CDCC across all three domains (CDCC3 ).**

Significance denoted by \*\*p < 0.001, \*p < 0.01, op < 0.05), −p ≥ 0.05; Bonferroni corrected p-values from two-sided paired sample t-tests.

In **Figure 2**,we summarize the obtained feature sets by the share of cepstral, prosodic, spectral, and voice quality LLDs, as well as by the share of modulation,moment, peak, percentile, regression, and temporal functionals (see **Tables 2** and **3** for a list of descriptors in each of these groups). We compare the cross-domain feature sets to the full ComParE feature set as well as the "single domain" feature sets that are obtained in analogy to the cross-domain feature sets by applying the CDCC2 to a 50% split of each corpus. A feature group is considered particularly relevant for a recognition task if its share among the selected features is larger than its share of the full feature set.

We observe notable differences in the importance of different LLD groups; it is of particular interest for the present study to highlight the results for the considered cross-domain emotion recognition tasks: cepstral features seem to be particularly relevant for cross-domain speech and music emotion recognition. In contrast, cross-domain emotion recognition from speech and sound, and from sound and music, are dominated by"prosodic"and spectral cues such as loudness, sub-band energies, and spectral flux. Regarding relevant functional types, the summarization reveals less evident differences between the tasks; still, percentile type functionals seem to be particularly promising for all of the tasks considered.

#### **3.2. AUTOMATIC CLASSIFICATION EXPERIMENTS**

Finally,we demonstrate the predictive power of the obtained crossdomain feature sets in automatic regression. In automatic regression, the parameters of a regression function on N-dimensional feature vectors are optimized to model the assignment of L "learning" vectors (e.g., feature vectors of emotional utterances) to the gold standard (e.g., the arousal observer rating). Then, the regression function is evaluated on a disjoint set of test vectors and the correlation of the function's predictions and the test set gold standard is computed as a measure of how well the regression function generalizes to "unseen" test data. In the present study, it is of particular interest to consider cross-domain evaluation, i.e., training on data from one domain (e.g., enacted speech) and evaluating on another domain (e.g., sound). In this context, we also treat spontaneous and enacted speech as different domains, as such analysis is receiving increasing attention at the moment (Bone et al., 2012) also due to practical reasons: for instance, it is of interest to determine if training on "prototypical" data from a controlled experiment (such as the GEMEP database) can improve automatic emotion recognizers applied "in the wild," e.g., to media analysis (such as given by theVAM database). For reference,we also consider within-domain regression in a twofold cross-validation manner.

For each learning set, we determine a multivariate linear regression function by means of support vector regression (SVR) (Smola and Schölkopf, 2004), which defines a real valued mapping

$$f(\mathbf{x}) = \mathbf{w}^T \mathbf{x} + b \tag{4}$$

of *N*-dimensional feature vectors **x** to a regression value *f*(**x**). **w** is the normal vector of the *N*-dimensional hyperplane describing the regression function, and *b* is a scalar offset. Specifically for SVR, the primary optimization goal is *flatness* of the regression function, which is defined as low norm of the weight vector **w**.

This is related to the notion of *sparsity* and crucial to avoid over-fitting of the model parameters in the present case of high dimensional feature spaces. The trade-off between flatness of the weight vector and deviation of the regression values from the gold standard on the learning set is modeled as a free parameter *C* in the optimization (cf. Smola and Schölkopf, 2004 for details). In our experiments, *C* is set to 10−3 for within-domain regression and 10−5 for cross-domain regression. The optimization problem is solved by the frequently used Sequential Minimal Optimization algorithm (Platt, 1999). To foster reproducibility of our research, we use the open-source machine learning toolkit Weka (Hall et al., 2009). Unsupervised mean and variance normalization of each feature per database is applied since SVR is sensitive to feature scaling.

In **Table 6**, the correlation coefficients (*r*) of automatic withindomain and cross-domain regression with the arousal observer ratings are displayed. First, we consider regression using the full 6373-dimensional ComParE feature set. In within-domain regression, results ranging from *r* = 0.54 (sound) up to *r* = 0.85 (enacted speech) are obtained,which are comparable to previously obtained results on sound, music, and spontaneous speech (Grimm et al., 2007b; Schuller et al., 2011b, 2012). Especially the result for music is notable, since we do not use any "hand-crafted" music features such as chords or tempo. In cross-domain regression, significant correlations are obtained except for the case of training on music and evaluating on sound. However, the mean r across all training and testing conditions (0.50) is rather low.

Considering automatic feature selection by CDCC2 for each combination of two domains, results in **Table 6B** indicate a drastic gain in performance especially for cross-domain regression. However, also the results in within-domain regression are improved. All correlations are significant at the 0.1% level. Particularly, using CDCC based feature selection robust regression (achieving *r* > 0.76) is possible across enacted and spontaneous speech. Further, it is notable that the average result across the four testing databases does not vary much depending on the training database used, indicating good generalization capability of the selected features. The overall mean r in this scenario is 0.65.

Finally, if we select the top features by CDCC3 on all databases (treating speech as a single domain for the purpose of feature selection), it is notable that we still obtain reasonable results (mean r of 0.58) despite the fact that the top features by CDCC3 exhibit comparably low correlation with the target labels on the single domains (cf. **Table 4**).

Summarizing the results for valence regression (**Table 7**), we observe that using the full feature set, we cannot obtain reasonable results in cross-domain regression. Among the cross-domain results, the only significant positive correlations are obtained in evaluation on spontaneous speech, however, these are lower than the correlation of the single best speech features. Interestingly, we observe significant negative correlations when evaluating on music and training on another domain, which is consistent with the fact that some of the music valence features are "inversely" correlated with the target label in the other domains (cf., e.g., the discussion of median MFCC 1 above). In the within-domain setting, it can be observed that regression on valence in music is possible with high robustness (*r* = 0.80). This is all the more noticeable since this **Table 6 | Results of within-domain and pair-wise cross-domain support vector regression on arousal observer ratings for sound (emotional sound database), music (NTWICM database), and spontaneous and enacted speech (VAM/GEMEP databases).**

Significance denoted by \*\*p < 0.001, \*p < 0.01, −p ≥ 0.05; Bonferroni corrected p-values from two-sided paired sample t-tests. Full ComParE feature set (cf. **Tables 2** and **3**); 200 top features selected by CDCC2 for specific within-domain or cross-domain regression tasks; Generic features: 200 features selected by CDCC3 across sound, music, and speech domains (cf.**Table 4**).

correlation is higher than the one obtained in arousal regression, while for the other domains, valence seems to be harder to recognize than arousal. This can partly be attributed to the fact that in the analyzed music data, the valence rating is correlated to the arousal rating.

Concerning feature selection by CDCC2 (**Table 7B**), we observe a boost in the obtained correlations (mean = 0.44, compared to 0.12 without feature selection). For instance, when training on enacted speech and evaluating on music, we obtain a significant *r* of 0.60. This result is interesting in so far as the best selected feature for this particular cross-domain setting, namely the flatness of the loudness contour, only exhibits a correlation of 0.28, respectively 0.27, with the valence rating on the NTWICM (music) and GEMEP (enacted speech) databases. Thus, the 200 CDCC2 -selected features for this regression task seem to be of complementary nature. Furthermore, by applying feature selection in the within-domain setting, best results are obtained for sound (*r* = 0.51), music (*r* = 0.82), and enacted speech (*r* = 0.50) valence recognition. However, regarding the issue of enacted vs. spontaneous speech,we find that regressors trained on one type do not generalize well to the other, which is in contrast to the finding for arousal.

Weninger et al. Acoustics of emotion in audio

**Table 7 | Results of within-domain and pair-wise cross-domain support vector regression on valence observer ratings for sound (emotional sound database), music (NTWICM database), and spontaneous and enacted speech (VAM/GEMEP databases).**

Significance denoted by \*\*p < 0.001, \*p < 0.01, °p < 0.05, −p ≥ 0.05; Bonferroni corrected p-values from two-sided paired sample t-tests. Full ComParE feature set (cf. **Tables 2** and **3**); 200 top features selected by CDCC2 for specific withindomain or cross-domain regression tasks; Generic features: 200 features selected by CDCC3 across sound, music, and speech domains (cf.**Table 5**).

Finally,when applying the"generic valencefeature set"obtained from the CDCC3 ranking across sound, music, and speech, we obtain an average correlation of 0.32. Results are considerably below the CDCC2 results particularly for sound and enacted speech. This – again – points at the difficulty of finding features that generalize to valence recognition across domains. However, it is notable that robust results (*r* = 0.75) are obtained in withindomain music recognition using the generic feature set, of which the "best" feature (rise time of spectral centroid) only has an (absolute) correlation of 0.16 with the music valence rating.

#### **4. DISCUSSION**

We have presented a set of acoustic descriptors for emotion recognition from audio in three major domains: speech (enacted and spontaneous), music, and general sound events. Using these features, we have obtained notable performances in within-domain regression – particularly, these surpass the so far best published results on the NTWICM database (Schuller et al., 2011b) despite the fact that the latter study used hand-crafted music features rather than the generic approach pursued in the present paper.

We have found that it is rather hard to obtain features that are equally well correlated across the three domains. For arousal, such features comprise mostly loudness-related ones. In contrast, we have not been able to obtain features that are significantly correlated with the valence rating in all domains. A further notable result for valence is that some features have an "inverse" meaning in different domains (i.e., significant correlations with different signum), while this does not occur for arousal. It will be subject of further research whether this is simply due to the correlation of intended arousal and valence in popular music or to more fundamental differences.

This phenomenon has motivated the introduction of a "crossdomain correlation coefficient" which summarizes the differences in correlation across multiple domains. Using this coefficient, we were able to provide an automatic method of selecting generalizing features for cross-domain arousal and valence recognition. In the result, cross-domain arousal and valence regression has been proven feasible, achieving significant correlations with the observer annotations.

The degree of cross-domain consistency in encoding the two main dimensions of affect – valence and arousal – demonstrated in this article is quite astounding. Music has often been referred to as the "language of emotion" and a comprehensive review of empirical studies on the expression of emotion in speech and music (Juslin and Laukka, 2003) has confirmed the hypothesis that the acoustic parameters marking certain emotions are quite similar in music and speech (cf. also Ilie and Thompson, 2006). Scherer (1991) has suggested that speech and emotion may have evolved on the basis of primitive affect bursts serving similar communicative functions across many mammalian species. Ethological work shows that expression and impression are closely linked, suggesting that, in the process of conventionalization and ritualization, expressive signals may have been shaped by the constraints of transmission characteristics, limitations of sensory organs, or other factors. The resulting flexibility of the communication code is likely to have fostered the evolution of more abstract, symbolic language, and music systems, in close conjunction with the evolution of the brain to serve the needs of social bonding and efficient group communication.

As vocalization, which remained a major modality for analog emotion expression, became the production system for the highly formalized, segmental systems of language and singing, both of these functions needed to be served at the same time. Thus, in speech, changes in fundamental frequency (F0),formant structure, or characteristics of the glottal source spectrum can, depending on the language and the context, serve to communicate phonological contrasts, syntactic choices, pragmatic meaning, or emotional expression. Similarly, in music, melody, harmonic structure, or timing may reflect the composer's intentions, depending on specific traditions of music, and may simultaneously induce strong emotional moods. This fusion of two signal systems, which are quite different infunction and in structure,into a single underlying production mechanism, vocalization, has proven to be singularly efficient for the purpose of communication, and the relatively high degree of convergence as demonstrated by the correlations found in our study suggests that it might be possible to identify elements of a common code for emotion signaling. Recently, Scherer (2013)

has reviewed theoretical proposals and empirical evidence in the literature that help to establish the plausibility of this claim, in particular, the evolutionary continuity of affect vocalizations, showing that anatomical structuresfor complex vocalizations existed before the evidence for the presence of representational systems such as language.

As to the cross-domain consistency with different kinds of environmental sounds, it seems quite plausible to assume that once speech and music were decoupled from actually occurring affect bursts and took on representational functions, different kinds of nature sounds were used in speech and music both for reference to external events and expressive functions. It seems reasonable to assume that the type of representational coding was informed by

#### **REFERENCES**

*Proceeding of the ASRU* (Cancún: IEEE), 381–385.

the prior, psychobiological affect code, particularly with respect to the fundamental affect dimensions of valence and arousal.

Empirical studies like the one reported here, using machine learning approaches, may complement other approaches to examine the evolutionary history of affect expression in speech and music by empirically examining, using large corpora of different kinds of sound events, the extent to which auditory domains exhibit cross-domain consistency and which common patterns are particularly frequent.

#### **ACKNOWLEDGMENTS**

This study has received funding from the European Commission (grant no. 289021, ASC-Inclusion).

of emotion,"in *Handbook of Affective Sciences*, eds R. J. Davidson, K. R. Scherer, and H. H. Goldsmith (Oxford, NY: Oxford University Press), 433–456.

*Annual Conference of the International Speech Communication Association* (Lyon: ISCA).

*Book, Version 3.4.1*. Cambridge: Cambridge University Engineering Department.

Zwicker, E., and Fastl, H. (1999). *Psychoacoustics – Facts and Models*. Heidelberg: Springer.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 28 March 2013; paper pending published: 15 April 2013; accepted: 06 May 2013; published online: 27 May 2013.*

*Citation: Weninger F, Eyben F, Schuller BW, Mortillaro M and Scherer KR (2013) On the acoustics of emotion in audio: what speech, music, and sound have in common. Front. Psychol. 4:292. doi: 10.3389/fpsyg.2013.00292*

*This article was submitted to Frontiers in Emotion Science, a specialty of Frontiers in Psychology.*

*Copyright © 2013 Weninger, Eyben, Schuller, Mortillaro and Scherer. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in other forums, provided the original authors and source are credited and subject to any copyright notices concerning any third-party graphics etc.*

## The "Musical Emotional Bursts": a validated set of musical affect bursts to investigate auditory affective processing

#### *Sébastien Paquette1 \*, Isabelle Peretz 1 \* and Pascal Belin1,2,3*

*1 Department of Psychology, International Laboratory for Brain Music and Sound Research, Center for Research on Brain, Language and Music, University of Montreal, Montreal, QC, Canada*

*2 Voice Neurocognition Laboratory, Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, UK*

*3 Institut des Neurosciences de La Timone, Aix-Marseille Université, Marseille, France*

#### *Edited by:*

*Petri Laukka, Stockholm University, Sweden*

#### *Reviewed by:*

*Tuomas Eerola, University of Jyväskylä, Finland Cesar F. Lima, University College London, UK; University of Porto, Portugal*

#### *\*Correspondence:*

*Sébastien Paquette and Isabelle Peretz, International Laboratory for Pavillon 1430 Boulevard Mont-Royal, Montréal, QC H2V 4P3, Canada e-mail: sebastien.paquette.1@ umontreal.ca<EMAIL_ADDRESS>Brain Music and Sound Research,*

The Musical Emotional Bursts (MEB) consist of 80 brief musical executions expressing basic emotional states (happiness, sadness and fear) and neutrality. These musical bursts were designed to be the musical analog of the Montreal Affective Voices (MAV)—a set of brief non-verbal affective vocalizations portraying different basic emotions. The MEB consist of short (mean duration: 1.6 s) improvisations on a given emotion or of imitations of a given MAV stimulus, played on a violin (10 stimuli × 4 [3 emotions + neutral]), or a clarinet (10 stimuli × 4 [3 emotions + neutral]). The MEB arguably represent a primitive form of music emotional expression, just like the MAV represent a primitive form of vocal, non-linguistic emotional expression. To create the MEB, stimuli were recorded from 10 violinists and 10 clarinetists, and then evaluated by 60 participants. Participants evaluated 240 stimuli [30 stimuli × 4 (3 emotions + neutral) × 2 instruments] by performing either a forced-choice emotion categorization task, a valence rating task or an arousal rating task (20 subjects per task); 40 MAVs were also used in the same session with similar task instructions. Recognition accuracy of emotional categories expressed by the MEB (*n*:80) was lower than for the MAVs but still very high with an average percent correct recognition score of 80.4%. Highest recognition accuracies were obtained for happy clarinet (92.0%) and fearful or sad violin (88.0% each) MEB stimuli. The MEB can be used to compare the cerebral processing of emotional expressions in music and vocal communication, or used for testing affective perception in patients with communication problems.

#### **Keywords: music, emotion, auditory stimuli, voices**

#### **INTRODUCTION**

With increasing knowledge in the field and new methods to explore the human brain, emotions are no longer too obscure or subjective to be studied scientifically. In neuroscience, many research projects are now entirely dedicated to the study of emotion. Thus, it appears timely to construct a standardized and validated set of stimuli and to make these freely and easily available (www.brams.umontreal.ca/plab\_download; http://vnl.psy.gla.ac. uk/resources.php) in order to facilitate the comparability of future studies.

A great amount of work has been achieved in the field of visually perceived emotions, utilizing validated stimuli like the International Affective Picture System and the Ekman faces (Ekman and Friesen, 1978; Lang et al., 1988; Dailey et al., 2001; Ekman et al., 2002), which were designed to portray basic emotions (anger, disgust, fear, happiness, sadness, and surprise as well as a neutral expression). These validated sets of stimuli have provided highly useful tools for the study of brain structures (e.g., amygdala: Adolphs et al., 1994) involved in emotional processing and its developmental trajectory (Charlesworth and Kreutzer, 1973). With the same objectives, an increasing number of studies are being conducted in the domain of aurally perceived emotions, thus calling for validated stimuli sets.

A large part of the research on auditory affective processing has been conducted on speech prosody utilizing words or sentences spoken with various emotional expressions (Monrad-Krohn, 1963; Banse and Scherer, 1996; Buchanan et al., 2000; Kotz et al., 2003; Mitchell et al., 2003; Schirmer et al., 2005; Pell, 2006). Another way to express an emotion vocally is via non-verbal affect bursts (Scherer, 1994; also sometimes called non-verbal interjections: Schröder, 2003). Non-verbal affect bursts are vocal expressions (e.g., screams, laughter) that usually accompany intense emotional feelings. Affect bursts are minimally conventionalized, thus a relatively universal means of spontaneous human communication (see Sauter et al., 2010; Koeda et al., 2013, for cross-cultural studies). They are believed to reflect more of a biological push than a sociological pull (Scherer, 1986); they are closer to the primitive affect expressions of babies and animals than to emotional speech.

Recently, a validated set of auditory affect bursts designed as an auditory counterpart of Ekman faces was recorded and validated by Belin et al. (2008). The so-called Montreal Affective Voices (MAV) consist of a set of short vocal interjections on the vowel /a/ expressing anger, disgust, fear, pain, sadness, surprise, happiness, sensual pleasure, and neutrality. The MAV represent short primitive expressions of these emotions with minimal semantic information, providing useful stimuli for the study of the psychological mechanisms underlying auditory affective processing with minimal interaction with linguistic processes (e.g., Bestelmeyer et al., 2010).

However, vocal affect bursts are not the only means of transmitting auditory emotions. Music is often described as the "language of emotions," and recent research on basic musical emotions has shown that emotion recognition in music is consistent across listeners (Vieillard et al., 2008). The terms "basic emotions" correspond to a limited number of innate and universal emotion categories (happiness, sadness, anger, fear, and disgust) from which all other emotions can be derived (Ekman, 1982). Moreover, many studies have demonstrated that emotions in music fit Ekman's definition of basic emotions, they are recognized quickly [only a quarter of a second of music; one chord or a few notes (Peretz et al., 1998; Bigand et al., 2005)], early in development (Terwogt and van Grinsven, 1991; Flom et al., 2008), and across different cultures (Balkwill et al., 2004). The latter is even true for cultures without previous exposure to western music (Fritz et al., 2009).

Perception of specific musical emotions (e.g., fear and sadness) can also be lost after damage to the amygdala (Gosselin et al., 2005, 2007), suggesting that damage to the limbic system affects perception of basic musical emotion just as reported for other domains (e.g., vocal expression: Dellacherie et al., 2011; facial expression: Adolphs et al., 1994).

An important question that ensues is why music moves us? Recent studies have shown that certain brain areas [e.g., the striatum (Salimpoor et al., 2011), the amygdala (Gosselin et al., 2007)] are associated with musical emotional processing. These same areas have also been associated with basic biological functions (sex, pain). How can we conceptualize the relationship between music and these neurobiological substrates? One possibility is that music co-opts or invades emotional circuits that have evolved primarily for the processing of biologically important vocalizations [e.g., laughs, screams; Peretz (2010)]. There is currently little experimental data supporting or invalidating the existence of a common musical and vocal channel.

For example, Lima and Castro (2011), demonstrated that musical expertise enhances the recognition of emotions in speech prosody, suggesting that expertise in one domain could translate to the other. Conversely, Thompson et al. (2012), reported that amusics (individual with a pitch perception deficit; Peretz et al., 2002) were also impaired in perceiving emotional prosody in speech.

More specifically, Ilie and Thompson (2006) compared domains by evaluating the effect of manipulating acoustic cues common to both the voice and music [intensity, rate (tempo), and pitch height] on emotional judgments. They found that loud excerpts were judged as more pleasant, energetic and tense compared to soft excerpts, and that fast music and speech were judged as having greater energy than slow music and speech. However, it was also found that tempo and pitch had opposite effects on other emotional scales. Their results support the view that the processing of musical and vocal emotion could utilize common circuitry, but that some of this circuitry might be domain specific.

The existence of domain-specific processes for decoding emotion is consistent with neuropsychological dissociations found between music and language (Peretz and Coltheart, 2003; Omar et al., 2011; Lima et al., 2013). These dissociations could be explained by the fact that musical emotion needs to be actively decoded by the brain based on associations learned via exposure to a musical culture (Peretz et al., 1998; Juslin and Västfjäll, 2008) and past experience with music (Eschrich et al., 2008); since not all musical emotional acoustic parameters are present in emotional vocalizations (e.g., harmony: Juslin and Laukka, 2003), it is possible that these additional cues require additional processing.

 Although such stimuli have been quite useful to help exploring aurally perceived emotions in their respective channel, many characteristics set current musical and vocal stimuli apart making them hard to compare in a controlled study. This is especially true for factors such as musical structure (limited by mode or tempo), length, level of complexity as well as the context in which they have been created. The use of pre-existing music can introduce uncontrolled variability of many acoustic parameters, with various demands on attention and memory. Such acoustic and cognitive differences are likely to recruit different neural networks (Peretz and Zatorre, 2005). This is why it is important to create and validate musical stimuli that would be as similar as possible to the MAV to allow for a more proper comparison of aurally (musical and vocal) perceived emotions.

The purpose of the present study is to make available for future research a validated set of brief musical clips expressing basic emotions, designed as a musical counterpart of the MAV. We chose to only include happiness, sadness, and fear because these emotions are among the easiest to recognize from music (Gabrielsson and Juslin, 2003; Juslin and Laukka, 2003; see Zentner et al., 2008, for a more nuanced range of musically induced emotions).

Brief "musical emotional bursts" (MEB) depicting neutral and emotional (happy, sad, and fear) expressions have been recorded from different musicians. The violin and the clarinet were chosen as instruments, not only because they are representative of two different classes of instruments (strings and woodwind) but also because they share important similarities with the voice: "The quasi-vocal quality implied by a seamless progression between notes is a characteristic that can be cultivated in both the clarinet and the violin" (Cottrell and Mantzourani, 2006:33). These recordings were then pre-selected and validated based on listeners' emotion categorization accuracy, as well as on valence and arousal ratings.

#### **MATERIALS AND METHODS RECORDING**

### *Participants*

Twenty professional musicians (10 violinists, 10 clarinettists) participated in the recording sessions, after providing written informed consent. They received a compensation of 20\$ per h.

#### *Procedure*

The musicians were first instructed to perform 10 short improvisations with different levels of expressiveness. They were not told in advance what the recording session was about; on the day of the recording they were told one after the other the emotion they were supposed to improvise on, [fear (as if they were scared), happiness, sadness, and neutrality]. They were told their improvisation had to last around a second (they could practice with the metronome), when ready they realized 10 renditions of the emotion. Neutral stimuli were presented just like any other category of stimuli, but characterized as "without emotion." After improvising, the same musicians were asked to imitate one after another four MAV stimuli depicting fear, happiness, sadness, and neutrality; they could listen to the stimuli as often as they wished. If the emotional category of the musical burst was not clearly recognized by the experimenter (SP) or if the improvisations were too long they were discarded.

The musical bursts were recorded in a sound-treated studio using a TLM 103 large diaphragm microphone Neumann (Georg Neumann, Berlin, Germany) at a distance of approximately 30 cm. Recordings were pre-amplified using a Millennia Media HV-3D preamplifier and digitized at a 44-kHz sampling rate at 24-bit resolution, using Apogee AD16X. Subsequently they were edited into short segments and normalized at peak value (90% of maximum amplitude), using Adobe Audition 3.0 (Adobe Systems, Inc. San Jose, CA).

We ended up with more stimuli than expected, because each musician gave us more excerpts than we asked for. In total, 1505 improvisations [a minimum of 10 × 4 emotions (happy, sad, fear, and neutral) per musician] and 319 imitations of the MAV [a minimum of 4 × 4 emotions (happy, sad, fear, and neutral) per musician] were recorded.

#### *Stimulus pre-selection*

Improvisations lasting longer than 4 s were excluded. Improvisations or imitations containing an artifact (breathing, vocal sounds, breaking bow hair sounds) were also excluded. In the end, the clearest and most representative stimuli (120 Violin-MEB and 120 Clarinet-MEB) were selected for the validation phase, regardless of their type (improvisation or imitation).

#### **VALIDATION**

#### *Participants*

Sixty participants (19 males) aged from 19 to 59 years (*M*: 28.8; *SD*: 9.2), with normal hearing participated in an on-line validation test. Each participant gave informed consent and filled out a socio-demographic information questionnaire prior to the judgment phase. Fifteen participants had 6 years or more of musical education and 45 had 5 years or less of training. They were compensated 3£ for their participation.

#### *Procedure*

Participants were instructed to evaluate each of the 240 MEB and 40 MAV (The MAV were included for comparison with the vocal stimuli). There were 30 violin-MEB, 30 clarinet-MEB and 10 MAV per emotion, and all were presented in a random order. Twenty of the 60 participants performed a four alternative forced-choice identification task "*Please choose the emotion you think this stimulus represents" among* fear, happiness, sadness, and neutrality labels, 20 participants gave arousal ratings "*Please rate on the scale below the perceived arousal of the emotion expressed (from 1 not at all aroused to 9 extremely aroused)*" and another group of 20 participants gave valence ratings "*Please rate on the scale below the perceived valence of the emotion expressed (from 1 extremely negative to 9 extremely positive).*"

#### **RESULTS**

The stimuli (40 violin-MEB and 40 clarinet-MEB) that were best identified (by being categorized in the intended emotion) by the largest amount of participants were selected (10 MEB; 7 improvisations, 3 imitations- per emotion). In the presence of identical ratings, the briefest stimuli were selected. Due to the small number of stimuli in each category, improvisations and imitations were not analysed separately (separate Tables can be found in the Supplementary Material).

Acoustical analyses were also performed to allow users to individually select their stimuli (Supplementary material).

#### **EMOTIONAL CATEGORIZATION**

Overall accuracy in the four-alternative emotions categorization task is 85.5% (*SD*: 15.8) for the violin-MEB, 75.4% (23.9) for the clarinet-MEB, and 94.8% (12.1) for the voice-MAV. The average percentage of correct recognition of each intended emotion for the selected stimuli are presented in **Table 1**. As can be seen, timbre had a greater effect on certain emotional intentions than on others. For example, fear was more difficult to recognize when expressed on a clarinet than on any other timbre.

The ANOVA conducted on the recognition scores (see values in bold in **Table 1**) with Timbre (violin, clarinet, and voice) and Emotion (happiness, sadness, fear, and neutrality) as within-subject factors yielded a main effect of timbre [*F*(2, 38) = 79.51, *p* < 0.001, η2 = 0.81] and of emotion [*F*(3, 57) = 6.81, *p* < 0.005, η2 = 0.26]; however, they are modulated by a significant interaction between Timbre and Emotion, [*F*(3.4, 64.4) = 16.41, *p* < 0.001, η2 = 0.46, corrected Greenhouse-Geisser].

Recognition scores were compared using Tukey's honestly significant difference. Scores averaged across emotions for each timbre were all significantly different (all *p* < 0.005) from one another: voices yielded the highest recognition scores and clarinet the lowest. Comparing emotions, fear was overall significantly (*p* < 0.01) less accurately recognized than all other emotions.

Using binomial tests to determine if the emotions conveyed by each of the 80 stimuli were recognized above chance level (25%), we found that 87.5% (70/80) of the MEB were recognized above chance (*p* < 0.05; bonferroni corrected). Thus, most MEB are effective in expressing an emotion on a musical instrument. Eight of the 10 stimuli that failed to be recognized belonged to the clarinet-fear category; the other two stimuli were from the violin-joy category.

#### **EMOTIONAL RATINGS**

The arousal and valence ratings averaged across participants for each stimulus are presented in **Figure 1**. The individual ratings are provided in the Supplementary Material.

The same ANOVA with Timbre and Emotion as betweensubjects factors as the one performed on the recognition scores

*Each row represents the percentage of choices for each emotion in each timbre. Percentage of correct recognition is presented in bold, (SE).*

was computed on the arousal ratings. A main effect of timbre [*F*(2, 38) = 10.05, *p* < 0.001, η2 = 0.35] and of emotion [*F*(3, 57) = 33.94, *p* < 0.001, η2 = 0.64] were observed; however as previously an interaction between Timbre and Emotion was obtained, [*F*(6, 114) = 5.85, *p* < 0.001, η2= 0.24].

In general, the clarinet stimuli were judged to be less arousing than the violin and the vocal ones (all *p* < 0.05; by Tuckey's tests), whereas the latter two were judged to be equally arousing (*p* = 0.67). Neutral expressions were overall significantly less arousing (*p* < 0.001) than all other emotions, and happy stimuli were found to be more arousing (*p* < 0.001) than the sad ones.

It is important to note that the stimuli played on a clarinet were rated differently than the violin and vocal stimuli. Happy clarinet stimuli were rated as more arousing than all the other emotions played on clarinet (all *p* < 0.05); [fear was also significantly (*p* < 0.005) more arousing then the neutral stimuli]. In contrast however, the only significant difference for violin and vocal emotional bursts was that neutral stimuli were significantly less arousing (all *p* < 0.01) than all other stimuli.

Regarding valence ratings, we found qualitatively a similar pattern for both the violin and vocal stimuli (Happy > Neutral > Fear > Sad). The clarinet stimuli showed however a slightly different pattern, where fear was rated as being more positive than neural stimuli (Happy > Fear > Neutral > Sad). Again, both a main effect of timbre [*F*(2, 38) = 6.13, *p* < 0.05, η2 = 0.24] and of emotion [*F*(3, 57)= 116.65, *p* < 0.001, η2 = 0.86] were observed, while the interaction between Emotion and Timbre was again found to be significant [*F*(6, 114) = 31.64, *p* < 0.001, η2 = 0.63]. Overall, violin MEB were judged to be less positive than the vocal ones (*p* < 0.005), but globally emotions were significantly different from one another in terms of their valence ratings (*p* < 0.005).

This interaction can be explained by the fact that some differences were observed within timbre. Among the vocal stimuli, the happy ones were judged to be more positive than the neutral ones which were rated as more positive than fear, which in turn was also rated more positively than sadness (all *p* < 0.01). When played on a musical instrument, the happy stimuli were also judged as most pleasant (all *p* < 0.001), whereas only the sad stimuli were rated as significantly more negative than the neutral ones when played on violin (*p* < 0.05), and also as more negative than the stimuli expressing fear played on the clarinet (*p* < 0.005).

#### **DISCUSSION**

Here we validate the MEB—a set of short music clips designed to express basic emotions (happy, sad, fear, and neutral). Despite their short duration (1.6 s on average), the MEB stimuli were correctly categorized by emotion with high accuracy (average recognition score of 80.4%). The highest accuracy was obtained on the violin for stimuli expressing fear and sadness (88%) and on the clarinet for those conveying happiness (92%). Although, the MAV stimuli were best recognized, the newly created MEB were still accurately portraying the desired emotions.

Only three emotions were tested here to allow for direct comparison between basic vocal (MAV) and musical (MEB) emotions. Our limited selection of emotions does limit voicemusic comparison, but it is a first step in making that comparison. We acknowledge that there are multiple declinations of positive and negative emotions in the musical and vocal literature, our aim was to use the most easily recognized common to both domains. From a dimensional approach, basic emotions can be distinguished on the dimensions of valence and arousal; variations of these (and other) emotions also differ in valence and arousal and can easily be represented along basic emotions.

The arousal and valence ratings obtained here fit well with this dimensional representation of emotions, with happy stimuli as conveying positive and arousing emotions, fear stimuli as conveying negative and arousing emotions (with the exception of a few clips played on clarinet), sad stimuli as conveying moderately arousing and negative emotions, and the neutral stimuli as conveying an emotional valence that is neither positive or negative with little arousal.

Although the valence scale had a highest rating possible of 9, it is important to note that the maximal average arousal elicited by our stimuli is 6.8 (7.1 for voice), Perhaps the short duration of our stimuli limited their arousing capabilities and could potentially explain the partial overlap in arousal observed in **Figure 1** between our two negative emotions (fear, sadness). Also, the fact that the valence scale ranged from "extremely negative" to "extremely positive" (Belin et al., 2008; Aubé et al., 2013), and not from "unpleasant" to "pleasant" could explain why the sad stimuli are differently positioned on the scale than in previous studies (e.g., Vieillard et al., 2008). Nevertheless, our results are still quite similar to those of Vieillard et al. (2008), which were obtained with longer and more conventional musical stimuli (inspired from film music), suggesting that the MEB may tap into similar emotional processes as those evoked by more elaborate film music clips. Yet, the MEB consist of brief expressions and are less likely to involve high-level cognitive mechanisms such as dividedattention and sophisticated knowledge of musical structure than more conventional musical stimuli. The MEB are not limited by tonality or defined by a specific rhythm; they were created as short musical bursts, by professional musicians on their instrument.

Our stimuli can be viewed as a primitive form of musical emotion, situated somewhere in between long musical excerpts from recordings (e.g., Peretz et al., 1998) or short musical segments extracted from these (Dalla Bella et al., 2003; Filipic et al., 2010) and synthesized frequency-modulated tones designed to mimic key acoustic features of human vocal expressions (Kantrowitz et al., 2013). Our novel stimuli were created to be exactly where they are in this spectrum by representing the most basic form of musical emotion that can be closely related to vocal expressions. Although exact replicas of the MAV could have been used instead, by digitally transposing the MAV to another timbre, we chose to produce new recordings in order to keep the stimuli as natural (realistic) as possible.

The timbre, or instrument on which music is played, is known to have an important impact on emotion recognition (Behrens and Green, 1993; Gabrielsson and Juslin, 1996; Balkwill and Thompson, 1999; Hailstone et al., 2009). For example, Hailstone et al. (2009) have found that melodies sound less happy when played on the violin than on other instruments, as we found here. This effect was particularly clear in the imitations of vocal expressions (see Supplementary Material). A range of timbres were used in prior studies (including violin and voice) and each instrument seemed to present its own possibilities and limitations when it came to expressing specific emotions. For instance in our study, we observed that fear was not well recognized when expressed on the clarinet.

Other limitations will also need to be addressed. For example, a forced-choice emotion recognition task was used here, and such tasks can have an impact on statistical analyses, such as increased co-linearity (if one response is chosen, the others are not), which generates artificially high recognition rates (Cornwell and Dunlap, 1994; Frank and Stennett, 2001). This method was selected to facilitate the web-based validation procedure of a large number of stimuli (280), and we believe the technique has served its purpose, as significant differences were observed between the timbres and within each timbre as revealed within the confusion matrix.

In addition, musicians were explicitly asked to imitate vocalizations (3/10 MEB per emotion). Such imitations produced on an instrument with voice-like characteristics may limit the chance to obtain domain-specific responses. In contrast, by using such a setup, finding evidence for domain-specificity would be compelling, even more so if parameters like pitch, emotion recognition scores and valance/arousal ratings are controlled for and used as regressors (Supplementary material) to compensate for the observed differences.

Here we propose a validated set of auditory stimuli designed as a musical counterpart of the MAV to allow a better comparison between auditory (musical and vocal) stimuli designed to convey emotions. We hope that the MEB will contribute to the understanding of emotions across domains and modalities.

#### **ACKNOWLEDGMENTS**

This work was supported by an Auditory Cognitive Neurosciences, Erasmus Mundus Mobility Scholarship by the European Union to Sébastien Paquette, by grants from Natural Sciences and Engineering Research Council of Canada, the Canadian Institutes of Health Research and from the Canada Research Chairs program to Isabelle Peretz and BBSRC grants BB/E003958/1 and BB/I022287/1, and ESRC/MRC large grant RES-060-25-0010 to Pascal Belin. We thank Patrice Voss for editing a previous version of the manuscript.

#### **SUPPLEMENTARY MATERIAL**

The Supplementary Material for this article can be found online at: http://www.frontiersin.org/Emotion\_Science 10.3389/fpsyg.2013.00509/abstract

### **REFERENCES**

*Research in Review*, ed P. Ekman (New York, NY: Academic Press), 91–168.

P. N., Butler, P. D., et al. (2013). Reduction in tonal discriminations predicts receptive emotion processing deficits in schizophrenia and schizoaffective disorder. *Schizophr. Bull.* 39, 86–93. doi: 10.1093/schbul/sbr060

anticipation and experience of peak emotion to music. *Nat. Neurosci.* 14, 257–262. doi: 10.1038/nn. 2726

*Commun.* 40, 99–116. doi: 10.1016/ S0167-6393(02)00078-X

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 04 April 2013; accepted: 18 July 2013; published online: 13 August 2013. Citation: Paquette S, Peretz I and Belin P (2013) The "Musical Emotional Bursts": a validated set of musical affect bursts to investigate auditory affective processing. Front. Psychol. 4:509. doi: 10.3389/fpsyg. 2013.00509*

*This article was submitted to Frontiers in Emotion Science, a specialty of Frontiers in Psychology.*

*Copyright © 2013 Paquette, Peretz and Belin. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## A vocal basis for the affective character of musical mode in melody

#### *Daniel L. Bowling\**

*Department of Cognitive Biology, University of Vienna, Vienna, Austria*

#### *Edited by:*

*Anjali Bhatara, Université Paris Descartes, France*

#### *Reviewed by:*

*Sébastien Paquette, University of Montréal, Canada Meagan Curtis, Purchase College, USA*

#### *\*Correspondence:*

*Daniel L. Bowling, Department of Cognitive Biology, University of Vienna, Alserstrasse 14, 1090 Vienna, Austria e-mail<EMAIL_ADDRESS>
Why does major music sound happy and minor music sound sad? The idea that different musical modes are best suited to the expression of different emotions has been prescribed by composers, music theorists, and natural philosophers for millennia. However, the reason we associate musical modes with emotions remains a matter of debate. On one side there is considerable evidence that mode-emotion associations arise through exposure to the conventions of a particular musical culture, suggesting a basis in lifetime learning. On the other, cross-cultural comparisons suggest that the particular associations we make are supported by musical similarities to the prosodic characteristics of the voice in different affective states, indicating a basis in the biology of emotional expression. Here, I review developmental and cross-cultural studies on the affective character of musical modes, concluding that while learning clearly plays a role, the emotional associations we make are (1) not arbitrary, and (2) best understood by also taking into account the physical characteristics and biological purposes of vocalization.

**Keywords: music, emotion, voice, mode, interval-size**

#### **INTRODUCTION**

One of the oldest and most prevalent ideas in the history of music is that there is a special connection between music and emotion (Kivy, 2002, 14). Today we know that the emotional coloring of a piece of music results from a complex interplay between acoustical properties, expectation, context, and a listener's personal experience. Efforts to understand the role of acoustical properties have been particularly successful. A great deal is now known about the affective contributions of tempo, intensity, pitch, spectral energy distribution, tone attack, and microstructural irregularity—and a coherent biological framework for understanding why they affect us has been offered in terms of similarity to corresponding properties in vocal affect expression (reviewed in Juslin and Laukka, 2003). The focus here is on the affective contribution of musical *mode*, which has proven a greater challenge to understand. In musicology, the term "mode" is used to refer to a variety of different concepts, only loosely related by their usage in the study of scales and melodies (Randel, 1986, 499). This ambiguity of meaning has generated some confusion, particularly when comparing different musical traditions. Thus, as a first step, I define mode as a set of musical tones and tone-relationships that are used to create a melody. By avoiding additional related concepts such as rules for ornamentation or pre-specified melodic motifs, this simple definition can be applied to music from a wide variety of traditions.

The notion that the mode of a melody influences the perception of its emotional coloring has been a part of music theory since antiquity. In *The Republic*, Plato treats it as common sense that certain modes are best suited for the representation of particular feelings or states of character (Plato, 375 BCE/1955, 93–95). Similarly, in Indian musical traditions, descriptions of which modes are appropriate for various emotions are found in ancient Sanskrit texts, such as the *Naty* ¯ s´*astra* ¯ (∼200 CE; Capwell, 1986, 779; Devy, 2002, 3–4). Further examples come from the Middle East and East Asia, where the affective connotations of different modes are documented in Persian and Japanese musical traditions (Nettl, 1986, 531; Hoshino, 1996).

Despite these rich historical and cross-cultural opportunities, the affective character of musical modes has almost exclusively been studied in the context of post-renaissance Western music, where the most salient examples of mode-emotion associations are based on the major and minor modes. When other factors such as tempo and intensity are carefully controlled, music composed using the major mode is typically heard as relatively positive and excited (e.g., joyful), whereas music composed using the minor mode is heard as relatively negative and subdued (e.g., sad; Hevner, 1935; Crowder, 1984, 1985; Kastner and Crowder, 1990; Gerardi and Gerken, 1995; Gregory et al., 1996; Peretz et al., 1998; Dalla Bella et al., 2001; Gagnon and Peretz, 2003). These connotations appear to have persisted for over 400 years the 16th century Italian music theorist and composer Gioseffo Zarlino similarly described the effect of the major mode as "gay and lively," and the minor mode as "sad and languid" (Zarlino, 1558/1968, 21–22).

The historical, cross-cultural, and repeatedly verified relationships between musical modes and emotions make it clear that these associations must be addressed by any theory of musical emotions. In spite of this, most modern theories downplay mode, relegating it to a catch-all of cultural convention, and in so doing pay little attention to the possibility of biological roots. In what follows, I advance a biological approach to mode-emotion associations by: (1) demonstrating the logical error of using evidence that mode-emotion associations are learned, as evidence against biological underpinnings; (2) presenting the evidence that mode-emotion associations exhibit acoustical similarities across cultures; and (3) extending the biological framework of vocal imitation proposed by Spencer (1857, 1890) to account for why we associate modes and emotions.

#### **THE EVIDENCE FOR LEARNING**

Most modern accounts of why music expresses emotion suggest that emotional associations with mode are learned through exposure to the conventions of a particular musical culture (e.g., Lundin, 1967, 166–169; Juslin and Laukka, 2003; see also Huron, 2008; Trainor and Corrigall, 2010). This position is supported by evidence that mode-emotion associations develop with age, and are more reliably made by individuals with musical training. One experimental paradigm involves presenting children of different ages with major and minor music and asking them to "match the feeling" by selecting among schematic drawings of various facial expressions (Dolgin and Adelson, 1990). Using this approach, a number of studies have found that culturally appropriate modeemotion associations do not develop until 6–8 years of age, before which they are not reliably made (Gerardi and Gerken, 1995; Gregory et al., 1996; Dalla Bella et al., 2001; but see Kastner and Crowder, 1990 for emergence at age 3). The perception of major and minor chords has also been studied in infancy (6 months) using a preferential looking paradigm (Crowder et al., 1991). In accord with the results for young children, no reliable preferences were observed for major over minor (or vice versa). With respect to musical training, a number of studies have shown that individuals who have had explicit instruction in Western music make culturally appropriate mode-emotion associations more reliably than those who have not (e.g., Heinlein, 1928; Hevner, 1935). Importantly however, these studies also show that musical training is not necessary for appropriate associations to be made (see also Dalla Bella et al., 2001). These findings, together with the fact that mode-emotion associations are well established in adulthood (Hevner, 1935; Crowder, 1984, 1985; Gerardi and Gerken, 1995; Gregory et al., 1996; Peretz et al., 1998; Dalla Bella et al., 2001; Gagnon and Peretz, 2003), suggest a pattern of learning over the course of development.

Evidence that mode-emotion associations strengthen with age and musical training is often taken to imply that such associations are arbitrary, arising solely through learning the conventions of a given musical culture (Heinlein, 1928; Lundin, 1967, 166–169; Gregory et al., 1996). Over the past 70 years, however, ethologists have repeatedly demonstrated that evidence for learning does not necessarily imply an absence of biological preparation. On the contrary, learned associations are often supported by inherited predispositions. For example, honeybees must learn to associate specific flower types with food, but they instinctively attend flowerlike objects (Gould and Marler, 1987). Similarly, rats must learn whether specific foods are safe for consumption, but they are predisposed to attend olfactory as opposed visual or auditory cues in doing so (Garcia and Koelling, 1966). Another example comes from songbirds that despite having to learn local song dialects from adult conspecifics, are predisposed to recognize and preferentially learn species-typical song patterns (Marler, 1991). Among primates, young Vervet monkeys must learn to emit "eagle" alarm calls to objects that pose an aerial

threat (such as eagles or hawks), but are predisposed to emit these calls to a wide-range of objects moving overhead, including for example, falling leaves (Seyfarth et al., 1980). Human children must also learn about potential sources of danger, a process that is in part supported by perceptual biases to associate certain classes of animals (e.g., spiders or snakes) with fear responses (Lobue and DeLoache, 2008). Finally, and perhaps of even greater significance, we are somehow predisposed to learn the myriad associations and rules that constitute language (Tomasello, 1999; Chomsky, 2006). In sum, these examples underscore the importance of biology in explaining perception and behavior and caution against the assumption that evidence for learning in mode-emotion associations precludes a specieswide predisposition to make them.

#### **MODE-EMOTION ASSOCIATIONS ACROSS CULTURES**

Interpretation of developmental studies is complicated by the confounding roles of perceptual and cognitive development. Thus, in the developmental studies alluded to above, it remains possible that mode-emotion associations are somehow made by young children but were not observed due to perceptual or cognitive limitations, such as difficulties with infering harmonic structure (see Gerardi and Gerken, 1995) or because the task of pointing to schematic faces representing emotional reactions to music requires cognitive skills poorly developed at the ages examined (see Dalla Bella et al., 2001). Cross-cultural studies offer a clear advantage here because adults from different cultures presumably have comparable perceptual and cognitive abilities. Accordingly, a number of studies have examined how adults respond to emotion in music from unfamiliar cultures. Few of these studies however, have been specifically designed to assess the influence of mode, which is thus typically confounded with other variables such as tempo and intensity. Nevertheless, this body of work shows that adult listeners perceive the emotions intended by composers/performers in culturally unfamiliar music with considerable accuracy (particularly for joy and sadness), and that their emotional judgments vary with mode in a way that is either in agreement with unfamiliar traditions (Balkwill and Thompson, 1999; Balkwill et al., 2004), or at least matches the judgments of listeners native to the culture in question (Fritz et al., 2009; Zacharopoulou and Kyriakidou, 2009). Only one cross-cultural study has isolated the influence of mode on perceived emotion. Hoshino (1996) played simple major and minor melodies to Japanese adults who grew up before WWII and were reportedly unfamiliar with Western musical conventions. Although Hoshino's methods were somewhat unorthodox—subjects were asked to associate melodies with colors, which they later described with emotional labels—she found some evidence for cross-cultural similarity in modal perception: major melodies were described as bright and warm, whereas minor melodies were described as dark and melancholic.

A different approach to studying mode-emotion associations is to compare the structure of music composed using modes from different cultures that are associated with similar emotions. Features that are held in common are candidates for mediating emotional associations. Accordingly, Bowling et al. (2012) compiled melodies composed in modes associated with either negative-subdued or positive-excited emotions in classical South Indian (*Carnatic*) and classical Western music and compared their structure in terms of the sizes of the intervals that occur between adjacent melody notes (see also Huron, 2008). In both traditions, it is apparent that the use of modes associated with negative-subdued emotion results in melodies with a significantly greater proportion of smaller intervals (<200 cents1 ), whereas the use of modes associated with positive-excited emotion results in melodies with a significantly greater proportion of larger intervals (≥200 cents). These results are not specific to classical music, as the same pattern has also been observed in major and minor Finnish Folk melodies (Bowling et al., 2010). These comparisons cast further doubt on the idea that the associations between mode and emotion are inherently arbitrary.

#### **A VOCAL BASIS FOR THE AFFECTIVE CHARACTER OF MUSICAL MODES**

What are cross-cultural similarities in mode-emotion associations based on? In his essay "On the Origin and Function of Music," Herbert Spencer (1857) proposed that music expresses emotion by imitation and exaggeration of acoustical properties of emotional expression in the voice. His logic entails two steps. First, he argued that the physiological components of emotion affect the mechanisms of vocal production and thus the acoustical properties of the voice, resulting in routine associations between those properties and the subjective components of emotional experience. And second, he argued that by employing the same acoustical properties, music gains access to the same emotional associations. Although Spencer's theory co-exists with several others that attempt to explain how music conveys emotion [reviewed in Davies (2001); see also Crowder (1984) and Juslin and Västfjäll (2008)], increasing evidence of acoustical similarities between music and voice, paired with careful consideration of the various relationships that might explain them (Juslin and Laukka, 2003), have provided strong support for Spencer's vocal imitation hypothesis.

While Spencer (1857, 399) did not specifically consider musical mode, he did comment on interval size, noting that "calm speech is comparatively monotonous" whereas "emotion makes use of fifths, octaves, and even wider intervals." Distinguishing between different types of emotion, more recent studies have found that frequency contours in sad speech are relatively flat and stable, whereas frequency contours in joyful speech are more dynamic and variable [reviewed in Scherer (1986, 2003)]. With respect to interval-size, Bowling et al. (2012) calculated the frequency differences between adjacent voiced intensity maxima in recordings of sad and joyful speech. In parallel with the pattern of interval-sizes found in modal melodies (see above), sad speech comprised a greater proportion of smaller intervals, while joyful speech comprised a greater proportion of larger intervals. Furthermore, the interval-size at which the reversal in prevalence occurred between the two emotional conditions was roughly the same in speech and music, between 100 and 200 cents.

Evidence of similarities in interval-size between musical and vocal expression have also been found by Curtis and Bharucha (2010), who report that descending minor thirds (-300 cents) frequently occur between the first and second syllables of twosyllable expressions conveying sadness (e.g., "come on"). This result is of particular interest because minor thirds play a central role in distinguishing music composed in the minor modes (Bowling et al., 2010). Accordingly, Curtis and Bharucha argue for the possibility of an "interval code" by which the occurrence of specific intervals signal specific emotions in music and the voice. Two issues complicate this interpretation. First, only descending minor thirds ending on the tonic (the "tonal center" of a mode) are emphasized in minor vs. major music. Defined more generally (i.e., not limited to those intervals ending on the tonic), descending minor thirds are only marginally more prevalent in minor vs. major melodies: by 0.1% in classical Western music and 0.4% in Finnish folk music (Bowling et al., 2010, 2012). Furthermore, the opposite pattern occurs in Carnatic melodies, where descending minor thirds are actually more prevalent in melodies composed in modes associated with joy (accounting for 8% of melodic intervals) than they are in melodies composed in modes associated with sadness (accounting for 4.2%; Bowling et al., 2012). It remains possible that tonic minor thirds tap into some form of interval code, but further research is needed to explore whether or not the concept of a tonic has any bearing in speech. Second, there is conflicting evidence as to whether or not specific musical intervals are emphasized in speech prosody. In contrast to Curtis and Bharucha, Bowling et al. (2012) found no evidence of emphasis at descending minor thirds in two-syllable expressions of sadness. Methodological differences in speaker selection, method of interval calculation, and histogram-bin size are likely sources of this difference. However, aside from these conflicting findings over the importance of relative vs. specific intervalsizes in mode-emotion associations, the available evidence clearly indicates that interval-size, like many other acoustical properties, varies with emotion in music and the voice in a parallel fashion.

#### **THEORETICAL ACCOUNTS**

Five theoretical frameworks can account for the relationship between interval-size in modal music and vocal expression. In what follows, I argue for the exclusion of four of them in favor of the fifth, i.e., Spencer's theory that music expresses emotion by imitating the voice. Juslin and Laukka took a similar approach in their 2003 review of acoustical similarities between musical and vocal expression, although they did not explicitly consider interval-size.

#### **EXPLANATION 1 - INTERVAL-SIZES IN MUSICAL AND VOCAL EXPRESSION ARE ENTIRELY UNRELATED, OBSERVED SIMILARITIES ARE COINCIDENTAL**

The principal argument against this possibility is that the same relationship between interval-size and emotion is observed in

1Cents are units of frequency interval size. One cent is 1/100 of an equally tempered semitone. An octave thus comprises 1200 cents. The formula for calculating the size of an interval between two frequencies (F1 and F2) in cents C, is C = 1200 × log2(F1/F2).

music and speech from different musical traditions and cultures. Whereas we can be fairly certain that basic acoustical properties of vocal expression are conserved across cultures (Elfenbein and Ambady, 2002), assessments of interval-size in mode-emotion associations from additional musical traditions could potentially strengthen the argument against spurious correlation (Persian music offers perhaps the best opportunity here). If such assessments are undertaken in the future, the prediction from the Carnatic and Western music comparisons (see above) is that musical traditions that systematically associate smaller intervals with positive-excited emotion, and/or larger intervals with negative-subdued emotion, are either non-existent or exceptionally rare. Additionally, the creation of new tonal traditions with these "backward" emotional associations should be considerably more difficult than the creation of new tonal traditions with "normal" associations.

A second argument against a coincidental interpretation of interval-size similarities in musical and vocal expression is that interval-size is part of a larger pattern of overall acoustic similarity between vocal and musical expression that includes a host of other properties. The possibility that all of these similarities are coincidental is remote (Juslin and Laukka, 2003), and there is no apparent reason to make an exception of interval-size.

#### **EXPLANATION 2—INTERVAL-SIZES IN MUSICAL AND VOCAL EXPRESSION ARE RELATED, BUT ONLY THROUGH THE COMMON INFLUENCE OF A THIRD CAUSAL FACTOR**

The arguments against this explanation are best made using examples. Thus, one potential third factor that could influence music and the voice is the expression of emotion through body postures and movements (Kivy, 2002, 37–48). According to this hypothesis, acoustical properties such as tempo or intensity, express emotion in both music and the voice because they imitate analogous properties of bodily expression, such as the speed and force of movements. However, it is unclear how this hypothesis would apply to the sizes of frequency intervals, as there is no clear analogue in body postures or movements. This problem reflects a general limitation of *third causal factor* accounts of interval-size similarities: frequency intervals are essentially unique to music and the voice. It is true that a metaphorical similarity between frequency intervals in music and the size/extent of body movements could be drawn, but it is unclear why this would be more plausible than a direct similarity with frequency intervals in the voice.

Another type of third causal factor interpretation considers influence apart from explicit imitation. For example, it is possible that a physiological factor such as arousal determines the relationship between emotion and interval-sizes in music and the voice. With respect to the voice one could postulate that changes in arousal determine interval-size by regulating the energy available to the muscles that control breathing and laryngeal posture. With respect to music however, this link between arousal and interval-size does not necessarily apply. The reason is that the forms of musical instruments are not physiologically constrained. Accordingly, the relationship between muscular effort and interval-size may be decoupled (as it is to some extent with a piano or guitar, and to a fuller extent with computer software used for composition) or even reversed (imagine an instrument on which the more forcefully one plays, the slower, softer, and smaller frequency intervals become). The lack of necessary connection between arousal and interval-size in instrumental music poses a problem for an arousal-based third causal factor account because it makes clear that arousal does not always determine interval-sizes in music. This example thus provides another general argument against third causal factor accounts of interval-size similarities in musical and vocal expression. For any cause that does not explicitly comprise frequency intervals, a necessary link to interval-size must be demonstrated in both domains.

#### **EXPLANATIONS 3–5**

Each of the remaining explanations posits some form of causal relationship between interval-size in music and the voice. Before considering them, it is necessary to draw a distinction between vocal and non-vocal music since any argument claiming that emotional expression in the voice derives from vocal music (or vice versa) is inherently circular. Keeping this in mind, there are three possible causal relationships that could exist between interval-size in musical and vocal expression: (3) *the voice imitates music*; (4) *both music and the voice imitate each other, with neither being primary*; and (5) *Spencer*'*s theory that music imitates the voice*.

The critical evidence for deciding between these alternatives concerns the evolutionary primacy of affective vocal expression. Despite the well-known debate between Spencer and Darwin over whether music derives from speech or speech derives from music, both men agreed on the primacy of some form of vocalization in auditory affective communication (Darwin, 1879/2004, 638; 1889/1998, 88–99; Spencer, 1857, 1890), and the evidence has only grown stronger since. With respect to musical instruments the earliest uncontested archaeological evidence—a pair of flutes made from the wing bones of a swan—dates to approximately 37000 years ago (reviewed in Fitch, 2006). In contrast, many of the neural and physiological mechanisms responsible for vocal affect expression (e.g., descending motor control from the brainstem, and various aspects of laryngeal anatomy) are shared by a large variety of mammals, suggesting relatively ancient phylogenetic roots (Jürgens, 1992; Ploog, 1992).

Even if one pushes the origins of instrumental music back to our common ancestor with *H. neaderthalensis* (as suggested by certain interpretations of slightly older flute-like objects found with other Neaderthal artifacts; Kunej and Turk, 2000), or even further back to our last common ancestor with chimpanzees and gorillas, (which both display some forms of drumming behavior; Fitch, 2006), the primacy of vocal affect expression remains incontrovertible. Explanations that the voice imitates music, or that both imitate each other with neither having primacy (explanations 3 and 4 above), are both incompatible with these facts. Only Spencer's theory, that interval-size in music expresses emotion by imitation and exaggeration of interval-size in the voice (explanation 5 above), is compatible with current archeological and phylogenic data relevant to the ages of vocal affect expression and instrumental music.

#### **A NOTE ABOUT HARMONY**

The present discussion has focused on changes in frequency over time (i.e. melody), but the affective character of musical modes is also realized in the simultaneous presentations of multiple frequencies, i.e., harmonies or chords (Heinlein, 1928; Crowder, 1984, 1985). Nevertheless, the relationship between melodic interval-size and emotion discussed here may also have relevance for the affective character of mode in harmony. Krumhansl (1990) provides extensive evidence that the notes in a melody line are perceived in the harmonic context of previously occurring notes, even though there is no physical simultaneity. Accordingly, when this context is made explicit in the form of chords, simultaneous intervals may in some sense be interpreted in relation to their melodic counterparts, i.e., the response of the nervous system may be partially redundant between these situations, activating associations originally made in melodic context in harmonic context and vice versa. For alternative theories on the affective impact of mode in harmony, based on sensory dissonance, familiarity with harmonic spectra, or sound symbolism, see Helmholtz (1877/1895) (211–219), Crowder (1984), and Cook (2007), respectively.

#### **CONCLUSION**

Explanations of mode-emotion associations that rely solely on lifetime learning of arbitrary cultural conventions are incompatible with evidence suggesting that the particular

#### **REFERENCES**

harmonies. *Music Percept.* 24, 315–319.

associations we make are similar across cultures and constrained by the interval-size properties of modes as expressed in melody. The parallel patterns observed between intervalsize and emotion in modal music and the voice suggest a relationship between these domains. Upon consideration of the possible explanations that could account for interval-size similarities between musical and vocal expression, Spencer's theory that music imitates the voice is the only explanation that is entirely consistent with the available evidence. Accordingly, it seems reasonable to conclude that the affective character of modes as realized in melody is—like many other aspects of music—best understood by also taking into account the physical characteristics and biological purposes of vocalization.

#### **FINANCIAL DISCLOSURE**

This work was supported by a grant from the National Science Foundation [BCS-0924181] awarded to Dale Purves, and a European Research Council advanced grant [No. 230604 "SOMACCA"] awarded to Tecumseh Fitch.

#### **ACKNOWLEDGMENTS**

I thank Douglas Bowling, Dale Purves, Sudara Williams, Marisa Hoeschele, and Bruno Gringas for their comments on the manuscript. Thanks are also due to Tecumseh Fitch and the University of Vienna for financial support during writing.

*Basis for the Theory of Music*. Translated by A. J. Ellis. London: Longmans, Green, and Co.

Oxford University Press. doi: 10.1017/S0140525X08005293

in *Music Perception (Springer Handbook of Auditory Research)*, eds M. R. Jones, A. N. Popper, and R. R. Fay (New York, NY: Springer), 89–127.

**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 08 April 2013; accepted: 03 July 2013; published online: 31 July 2013. Citation: Bowling DL (2013) A vocal basis for the affective character of musical mode in melody. Front. Psychol. 4:464. doi: 10.3389/fpsyg.2013.00464*

*This article was submitted to Frontiers in Emotion Science, a specialty of Frontiers in Psychology.*

*Copyright © 2013 Bowling. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Animal signals and emotion in music: coordinating affect across groups

#### *Gregory A. Bryant\**

*Department of Communication, Center for Behavior, Evolution, and Culture, University of California at Los Angeles, Los Angeles, CA, USA*

#### *Edited by:*

*Daniel J. Levitin, McGill University, Canada*

#### *Reviewed by:*

*Rajagopal Raghunathan, University of Texas at Austin, USA Charles T. Snowdon, University of Wisconsin–Madison, USA*

#### *\*Correspondence:*

*Gregory A. Bryant, Department of Communication, Center for Behavior, Evolution, and Culture, University of California at Los Angeles, 2303 Rolfe Hall, Los Angeles, CA 90095, USA e-mail<EMAIL_ADDRESS>
Researchers studying the emotional impact of music have not traditionally been concerned with the principled relationship between form and function in evolved animal signals. The acoustic structure of musical forms is related in important ways to emotion perception, and thus research on non-human animal vocalizations is relevant for understanding emotion in music. Musical behavior occurs in cultural contexts that include many other coordinated activities which mark group identity, and can allow people to communicate within and between social alliances. The emotional impact of music might be best understood as a proximate mechanism serving an ultimately social function. Recent work reveals intimate connections between properties of certain animal signals and evocative aspects of human music, including (1) examinations of the role of nonlinearities (e.g., broadband noise) in non-human animal vocalizations, and the analogous production and perception of these features in human music, and (2) an analysis of group musical performances and possible relationships to non-human animal chorusing and emotional contagion effects. Communicative features in music are likely due primarily to evolutionary byproducts of phylogenetically older, but still intact communication systems. But in some cases, such as the coordinated rhythmic sounds produced by groups of musicians, our appreciation and emotional engagement might be driven by an adaptive social signaling system. Future empirical work should examine human musical behavior through the comparative lens of behavioral ecology and an adaptationist cognitive science. By this view, particular coordinated sound combinations generated by musicians exploit evolved perceptual response biases – many shared across species – and proliferate through cultural evolutionary processes.

**Keywords: emotion in music, arousal, nonlinearities, music distortion, coalition signaling**

"fpsyg-04-00990" — 2013/12/31 — 11:24 — page 1 — #1

#### **INTRODUCTION**

Musical sounds can evoke powerful emotions in people, both as listeners and performers. A central problem for researchers examining music and emotion is to draw clear causal relationships between affective acoustic features in music and the associated responses in listeners. Behavioral ecologists have long studied emotional communication in non-human animals, and one guiding principle in this research is that the physical forms of evolved signals are shaped by their respective communicative functions (Morton, 1977; Owren and Rendall, 2001). Signals evolve as part of signaling system – that is, the production of a signal is necessarily tied to a systematic response by target listeners. This basic fact of animal signaling leads us to an inescapable conclusion regarding music and emotion: the physical structure of musical forms must be related in important ways to people's perceptions and behavioral responses to music. The complex question thus arises: does music, in any way, constitute a signal that is shaped by selection to affect listeners' behavior and potentially convey adaptive information to conspecifics (i.e., members of the same species)? Alternatively, perhaps music is a by-product of a variety of cognitive and behavioral phenomena. In any case, comparative analyses examining acoustic signals in non-human animals can shed light on musical behaviors in people.

Here I will describe research that explores the perception of arousal in music from a comparative perspective, and frame this work theoretically as the exploration of one important proximate mechanism (i.e., an immediate causal process) among many underlying our special attention and attraction to affective properties in musical sound. Music is a cultural product that often exploits pre-existing perceptual sensitivities originally evolved for a variety of auditory functions including navigating sonic environments as well as communication. Cultural evolution has led to increasingly complex, cumulative musical developments through a sensory exploitation process. I suggest that humans have evolved an adaptive means to signal relevant information about coalitions and collective affect within and between social groups. This is accomplished through the incorporation of elaborate tonal and atonal sound, combined with the development of coordinated performance afforded by rhythmic entrainment abilities.

A key issue for understanding the nature of music is to explain why it is emotionally evocative. Darwin (1872)famously described many affective signals in humans and non-human animals, and biologists have since come to understand animal emotional expressions not as cost-free reflections of internal states, but rather as strategic signals that have evolved to alter the behavior of target organisms in systematic ways (Maynard Smith and Harper, 2003). Receivers have evolved response biases that allow them to react adaptively to these signals resulting in co-evolutionary processes shaping animal communication systems (Krebs and Dawkins, 1984). Many scholars have noted the clear connections between human music and emotional vocalizations (Juslin and Laukka, 2003), as well as the connections between human and animal vocalizations (Owren et al., 2011). Snowdon and Teie (2013) recently outlined a theory of the emotional origins of music from a comparative perspective. But researchers examining emotion in music do not typically draw explicit connections to animal vocal behavior.

#### **FORM AND FUNCTION IN ANIMAL SIGNALS**

Recently there has been an increased focus on the form–function relationship between acoustic structure in animal signals and their communicative purposes. The principle of form and function has been indispensable in the study of, for example, functional morphology, but is also crucial for understanding animal signaling. Morton (1977) in his classic paper described the convergent evolution of specific structural features in animal signals based on the behavioral communicative context, and the motivations of senders. Low, broadband (i.e., wide frequency range) sounds are often honestly tied to body size and hostile intent, and can induce fear in receivers. Conversely, high pitched tonal sounds are related to appeasement, and are often produced to reduce fear in listeners. These motivational–structural (MS) rules apply widely across many species and have provided an evolutionary basis for studying the acoustic structure of animal signals (see Briefer, 2012 for a recent review). MS rules illustrate nicely how sound is often much more important than semantics in animals signals. Owren and Rendall (2001) described researchers'frequent reliance on linguistic concepts in understanding primate vocalizations. Animal signals have often been studied as potentially containing "meaning" with referential specificity. An alternative approach is to examine patterns of responses to closely measured non-referential acoustic features of signals. Many signals can affect perceivers in beneficial ways that that do not require the activation of mental representations, analogs to "words," or the encoding of complex concepts. Owren and Rendall (2001) encouraged researchers to rule out simple routes of communication before invoking necessarily more complex cognitive abilities that would be required of the signaling organism. That is not to say that complex meanings are never instantiated in non-human animal signals, but that we should not begin with that assumption.

So how do specific acoustic parameters in vocal signals underlie the communicative purposes for which they are deployed? Consider the interactive affordances of the acoustic-startle reflex. Many animal calls consist of loud bursts of acoustic energy with rapid onsets, loudness variation, and nonlinear spectral characteristics that often give the signals a harsh or noisy sound quality. These features serve to get the attention of a target audience, and can effectively interrupt motor activity. The direct effect of this kind of sound on the mammalian nervous system is a function that has been phylogenetically conserved across many taxa. Humans

"fpsyg-04-00990" — 2013/12/31 — 11:24 — page 2 — #2

rely on this reflexive principle in vocal behaviors such as infantdirected (ID) speech, crying, pain shrieks, and screams of terror. For instance, in the case of ID speech, prohibitive utterances across cultures contain similar acoustic features – including fast rise times in amplitude, lowered pitch (compared to other ID utterances), and small repertoires (e.g., No! No! No!; Fernald, 1992). These directed vocalizations are often produced in contexts where caretakers want to quickly interrupt a behavior, and must do so without the benefit of grammatical language.

In studies examining the recognition of speaker intent across disparate cultures, subjects are quite able to identify prohibitive intentions of mothers speaking to infants, and other adults as well (Bryant and Barrett, 2007; Bryant et al., 2012). This ability is not a function of understanding the words, but instead due to the acoustic properties of the vocalizations (Cosmides, 1983; Bryant and Barrett, 2008). In the case of ID prohibitives, proximate arousal in senders contributes to the generation of particular kinds of sound features, including rapid amplitude increases and lowered pitch for the authoritative stance. People, including infants, respond in predictable ways to high arousal sounds, such as stopping their motor activity and re-orienting their attention to the sound source. Research with animal trainers also reveals the systematic relationships between specific communicative forms and desired outcomes in animals such as sheep, horses, and dogs (McConnell, 1991). Vocal commands to initiate motor activity in a variety of species typically contain multiple short and repeated broadband calls, while signals intended to inhibit behavior tend to be longer and more tonal. McConnell (1991) also draws an explicit connection to music and cites several older studies from the 1930s showing the above characteristics in music correlating with physiological changes in human listeners. Short repeating rising notes are associated with increased physiological responses such as pulse rate and blood pressure, while longer, slower musical pieces have the opposite effects.

Research has shown that non-human animals respond predictably to musical stimuli, if the music is based on affective calls of their species. Snowdon and Teie (2010) created synthesized musical excerpts that were based on acoustic features of cotton-top tamarin affiliation and threat signals, and they played these compositions, as well as music made for humans, to adult tamarins. Musical stimuli based on threat calls resulted in increased movement, and huddling behavior shortly after exposure. Conversely, the tamarins reacted to affiliation-based music with calming behavior and reduced movement. There was little response to human music, except some reduced movement in response to human threat-based music, suggesting that species-specific characteristics were crucial in eliciting predictable reactions. Because the stimuli did not contain actual tamarin vocalizations, the responses were likely due to structural features of their vocal repertoire, and not merely the result of conditioning. The acoustic structure in the music clearly triggered tamarin perceptual systems designed for perceiving conspecific vocalizations, but importantly, this work demonstrates how acoustic forms can be readily transposed into stimuli we would consider musical, and that it can be affective for non-human listeners. There is some evidence that human music can have effects on non-human animals. Akiyama and Sutoo (2011) found that exposure to recordings of Mozart reduced blood pressure in spontaneously hypertensive rats, and the effect was driven by relatively high frequencies (4 k–16 kHz), an optimal range for rat hearing sensitivity. The authors proposed that the blood pressure reduction was a result of accelerated calcium-dependent dopamine synthesis. These data again show the importance of species-specific response biases in examinations of the effects of musical stimuli on humans and non-humans alike.

Universal form and function relationships are due to the fact that emotional communication systems in animals are evolutionarily conserved (Owren et al., 2011; Zimmermann et al., 2013), and recent work examining the perception of non-human animal affective vocalizations by humans shows that even when people cannot accurately recognize the affect in an animal vocal expression, brain structures react differentially as a function of the emotional valence in the vocalizations. Belin et al. (2008) found that judges could not reliably judge rhesus monkey or cat vocalizations on a positive–negative scale, but still had varying activation in right ventrolateral orbitofrontal cortex (OFC) in response to the recorded vocalizations. There was also greater overall activationfor negative affect in the vocal samples, whether produced by human or non-human animals. Other research shows that experience also matters when humans can accurately judge affect in non-human vocal signals. Trained pig ethologists were more accurate than naïve students at classifying the behavioral context of domestic pig vocalizations, and caretakers also systematically judged intensity features as being lower overall (Tallet et al., 2010). Chartrand et al. (2007) found that bird experts had unique brain responses (using EEG) to birdsong than naïve listeners, but the difference extended to environmental sounds and voices as well suggesting that expertise in one domain of auditory processing can affect how people hear sounds in other ways.

#### **SOUND OF AROUSAL**

Excitement in mammals is often characterized by physiological activation that prepares the animal for immediate action. An emotional state characterized by heightened arousal occurs in context-specific ways, but often motivates vocal communication shaped by selection to affect others' behavior in an urgent manner. Animals produce pain shrieks, alarm calls, and urgent contact calls, each demanding particular responses perceptually and behaviorally. Specifically in vocalizations, the physiology of high arousal results in increased activation of upper body musculature (including vocal motor systems and respiration) that can cause increased subglottal air pressure and heightened muscle tension. Consequently, vocal folds can vibrate at their natural limit, generating sound waves that reach their maximum amplitude given particular laryngeal and supralaryngeal structural constraints. This saturating nonlinearity (e.g., deterministic chaos) correlates perceptually with a harsh, noisy sound – a sound that effectively penetrates noisy environments, and is hard for listeners to habituate to. **Figure 1** shows a single coyote (*Canis latrans*) contact call that contains subtle deterministic chaos, subharmonics, and a downward pitch shift.

Nonlinearities can be adaptive features of conspicuous signals that require a quick response or certain attention (Fitch et al., 2002). As is the case with many acoustic features of emotional vocalizations, the sound of arousal in scared or excited animals has been conserved across numerous mammal species (Mende et al., 1990; Blumstein et al., 2008; Blumstein and Récapet, 2009; Zimmermann et al., 2013). Researchers examining how noisy features manifest in particular communicative contexts have found that results are not always predictable (e.g., Slaughter et al., 2013) but responses to noisy vocalizations are typically consistent with the idea that these sounds invoke fear in listeners and prepare them for a quick response. Accurate recognition of high arousal in a vocalizer can provide valuable cues concerning threats in the immediate environment, predicting events such as an imminent attack by a conspecific, or an external danger like the approach of a predator. Signaling behavior can evolve from these cues when senders and receivers mutually benefit from the communicative interaction (Maynard Smith and Harper, 2003), and behavioral features often become ritualized in a co-evolutionary process of production enhancement and perceptual sensitivity (Krebs and Dawkins, 1984).

The sound of arousal example provides a very clear logicfor why specific sound features (i.e., forms) are associated with systematic emotional reactions and likely subsequent behavioral responses (i.e., functions). Audio engineers and musicians have exploited the sound of arousal in music, and as a result, instrumentation and performances across a variety of music genres seem well-suited to invoke arousal in listeners including inducing fear, excitement, anger, and exhilaration. For the same reasons people watch horror films, ride roller coasters, or surprise each other for amusement, particular sounds in music are interesting and sometimes exciting.

#### **IS MUSIC SPECIAL?**

"fpsyg-04-00990" — 2013/12/31 — 11:24 — page 3 — #3

A complete explanation of the sound features of music is most likely going to be developed from an adaptationist cognitive science informed by a cultural evolutionary framework. The perception and appeal of music is currently best characterized as the co-occurring activation of a collection of by-product perceptual and judgment processes (McDermott, 2009). Pinker (1997) famously described music as "auditory cheesecake" – the theory to beat when proposing adaptive functions for music. It is clear that many systems designed to solve adaptive auditory problems faced recurrently by mammalian species are triggered by phenomena most people would call music. That is, the melodic and rhythmic properties of "musical" sounds satisfy input conditions in a variety of auditory processing mechanisms. Auditory scene analysis research has examined in great detail many fundamental sound perceptual processes and how they relate to navigating the sonic environment (Bregman, 1990). We can segregate sound streams, locate sound sources, and categorize sounds efficiently – abilities that clearly contribute to our perception of music.

Musical forms affect the full range of human emotions. I will focus on the sound of arousal, which often induces fear, as one good example of how a specific vocal phenomenon can manifest itself in music and be perpetuated culturally. This is not intended to explain other emotional phenomena in music, although I would certainly expect similar principles to apply widely across the emotion spectrum. Theories such as these, however, do not fully explain the appeal of Mozart or Bach, for example. Formal accounts of musical structure have laid out in rich detail the hierarchical patterning in tonal organization (e.g., Lerdahl and

"fpsyg-04-00990" — 2013/12/31 — 11:24 — page 4 — #4

Jackendoff, 1983), so a complete account of the nature of music must incorporate connections with other aspects of our cognition beyond emotional vocalizations. Snowdon and Teie (2013) proposed four categories of elements to explain the various factors contributing to music. The first two categories involve the development of auditory perception and sensitivity to vocal emotion information. But in the other two categories they point to elements such as melody, harmony, counterpoint, and syntax that are fundamental to the complexity and beauty in music (see also Patel, 2008).

#### **SPEECH AND MUSIC**

Speech is often cited as an important domain contributing to music perception. Speech communication in people has likely resulted in many refinements of phylogenetically older vocal production and perception abilities shared with many non-human animals (Owren et al., 2011). Models of efficient coding of sound also suggest that any specialized auditory processes for speech could be achieved by integrating auditory filtering strategies shared by all mammalian species (Lewicki, 2002). Human hearing sensitivity, however, appears particularly well-attuned to the frequency range of normal speech (Moore, 2008) just as all vocalizing species' auditory abilities are adapted to conspecific vocalization characteristics. Based on modeling work examining potential filtering strategies of peripheral auditory systems, Lewicki (2002) proposed that the representational coding of speech could be effectively instantiated using schemes specialized for broadband environmental sounds combined with schemes for encoding narrowband (i.e., tonal) animal vocalizations. That is, evolutionarily conserved auditory processes might have constrained speech production mechanisms such that speech sounds fell into frequency and temporal ranges exploiting prelinguistic perceptual sensitivities.

Speech perception is quite robust in normal speakers even in cases where high degradation or interruption is occurring (e.g., Miller and Licklider, 1950), and the temporal rate at which speech can be reliably understood far exceeds the production capability of the most efficient speakers (Goldman-Eisler, 1968). These facts hint at perceptual specialization. But a good deal of our speech processing ability is likely due to auditory abilities widely shared across mammals (Moore, 2008). Cognitive neuroscience research has shown repeatedly that music and speech share brain resources indicating that speech perception systems accept music as input (for recent reviews seeArbib,2013), though evidence exists for separate processing as well (Zatorre et al., 2002; Peretz and Coltheart, 2003; Schmithorst, 2005). The relationship between speech and music is certainly more than a coincidence. Amplitude peaks in the normalized speech spectrum correspond well to musical intervals of the chromatic scale, and consonance rankings (Schwartz et al., 2003). Many parallels also exist between music and speech development (McMullen and Saffran, 2004).

The physical properties of the sounds are not the only dimensions that link speech and music. The structure of various sound sequences also seems to activate the same underlying cognitive machinery. Research examining rule learning of auditory stimuli demonstrates the close connection between perceiving speech and music. Marcus et al. (2007) found that infants could learn simple rules (e.g., ABA) in consonant–vowel (CV) sequences, and the learning can apply to non-speech stimuli such as musical tones or non-human animal sounds. However, extracting rules from sequences of non-speech stimuli was facilitated by first learning the rules with speech, suggesting that the proper domain (see below) of rule learning in sound sequences is speech, but musical tones and other sounds satisfy the input conditions of the rule learning system once the system is calibrated by spoken syllables. Studies exploring the acquisition of conditional relations between non-adjacent entities in speech or melodic sequences show similar patterns (Creel et al., 2004; Newport and Aslin, 2004).

A good deal of music perception is likely due to the activity of speech processing mechanisms, but perception is only half of the system. We should be concerned with how production and perception systems evolved together. There are clear adaptations in place underlying breathing processes in speech production and laryngeal and articulator control (MacLarnon and Hewitt, 1999). Moreover, we have fine cortical control over pitch, loudness, and spectral dynamics (Levelt, 1989). These production systems, as a rule of animal signaling, must have complementary adaptive response patterns in listeners. Many perceptual biases were in place before articulated speech evolved, such as the categorical perception of continuous sounds (Kuhl, 1987). But other response biases might be new, such as sensitivity to the coordinated isochronic (i.e., steady, pulse-based repetition) rhythms produced by multiple conspecifics. Sperber (1994) made a distinction between the proper domain of a mechanism and its actual domain. Proper domain refers to those specific features that allow a system to solve an adaptive problem. Depending on the nature of the dynamics (i.e., costs and benefits) of the adaptation, systems will vary in how flexible the input conditions are to respond to a stimulus. The actual domain of a system is the range of physical variation in stimuli that will result in a triggering of that mechanism, something that is often a function of context and the evolutionary history of the cognitive trait. In these terms, the actual domain of speech processers presumably includes most music.

Domain specificity in auditory processing can illuminate the nature of people's preferences for certain sounds, including why certain musical phenomena are so interesting to listeners. But how these preferences manifest themselves as social phenomena remains to be explained. One possibility is that cultural evolutionary processes act on those sound characteristics that people are motivated to produce and hear. For example, rhythmic sound that triggers spatial localization mechanisms could be preferred by listeners, and consequently be subject to positive cultural selection resulting in the feature spreading through musical communities. Other examples include singing patterns that exaggerate the sound of affective voices, or frequency and amplitude modulations that activate systems designed to detect speech sounds. The question becomes, of course, is any sound pattern unique to music?

#### **CULTURAL TRANSMISSION OF MUSICAL FEATURES**

Researchers are starting to explore how listeners' specific sound preferences can lead to the evolution of higher order structure that can constitute eventual musical forms. MacCallum et al. (2012) created a music engine that generates brief clips of sounds that

"fpsyg-04-00990" — 2013/12/31 — 11:24 — page 5 — #5

were judged by listeners – clips that started out quite non-musical. Passages that were preferred in forced-choice trials "reproduced," that is, were recombined with other preferred passages. This evolutionary process resulted in several higher order structures manifesting as unquestionably musical attributes. For instance, an isochronic beat emerged. Understanding perceptual sensitivities (i.e., solutions to auditory processing adaptive problems) that are relevant in music listening contexts will help explain preference patterns, and evolutionary cultural processes can provide a framework for understanding the proliferation of these sensitivities (Merker, 2006; Claidière et al., 2012). The sound of fear represents one dimension of auditory processing relevant for music which is in place because of conserved signaling incorporating arousal. As a consequence, people are interested in sounds associated with high arousal, and cultural transmission processes perpetuate them.

Consider the form and function of punk rock in western culture. The relevant cultural phenomena for a complete description of any genre of music are highly complex, and not well understood. But we can clearly recognize some basic relationships between the sonic nature of certain genres of music and their behavioral associations in its listeners. Like much music across culture, there is a strong connection between music production and movement in listeners, epitomized by dancing, resulting in a cross-cultural convergence on isochronic beats in music traditions. The tight relationship between musical rhythm perception and associated body movement is apparent in babies as young as seven months (Phillips-Silver and Trainor, 2005). Punk rock is no exception. Early punk is characterized by a return to fundamentals in rock music (Lentini, 2003). It began as a reaction to a variety of cultural factors, and the perceived excesses of ornate progressive music in general. The initial creative ethos was that anybody can do it, and it was more of an expression of attitude than the making of cultural artifacts. In short, it was intense (and sometimes aggressive) in many ways, and whatever one's interpretation of the cultural underpinnings, the energy is apparent. The music is characterized by fast steady rhythms, overall high amplitude, and noisy sound features in all instruments – attributes that facilitate forceful dancing. But the distortion noise is especially distinct and key for the genre. Of course, many genres of rock use noise – the punk example is just preferred here for many cultural and explanatory reasons, but the same principle applies to many variations of blues and rock music.

Noisy features in rock took a life of their own in the No Wave, post punk, and experimental movements of the 1980s and beyond (e.g., O'Meara, 2013). In rock music, what originally likely arose as a by-product of amplification (i.e., attempting to be loud along with an intense style of playing) soon became conventionalized in ways that are analogous to ritualization in the evolution of animal signals (Krebs and Dawkins, 1984). Particular manifestations of noisy features (forms) were directly related to compositional and performance goals of musicians (functions). Products were developed that harnessed particular kinds of distortion in devices (e.g., effects pedals) that modified the signal path between an instrument and the amplifier. This allowed artists to achieve the desired distortion sounds without having to push amplifiers beyond their natural limit. The use of noise quickly became a focus of a whole family of musical styles, most being avant garde and experimental. Continuing the trend of rejecting aspects of dominant cultural practices, artists could signal their innovation and uniqueness by using this new feature of music in ways that set them apart. The sound affordances of broadband noise provide a powerful means for artists to generate cultural attractors fueled by discontent with mass market music. Moreover, the creative use of distortion and other effects can result in spectrally rich and textured sounds. Cultural evolutionary forces will tap into any feature that allows socially motivated agents to differentially sort based on esthetic phenomena (Sperber, 1996; McElreath et al., 2003). Simple sound quality dimensions like intensity might be excellent predictors of how people are drawn to some genres and not others (Rentfrow et al., 2011). Listeners also often find moderate incongruities (as opposed to great disparities) between established forms and newer variations the most interesting (Mandler, 1982). For example, modern noise rock with extreme distortion that is quite popular today would likely have been considered much more unlistenable in 1960 because it is such a dramatic departure from the accepted sounds for music at the time. But today it is only slightly noisier than its recent predecessors. What gets liked depends on what is liked.

#### **DISTORTION, AROUSAL, AND MUSIC**

Distortion effects in contemporary music mimic in important ways the nonlinear characteristics we see in highly aroused animal signals, including human voices. Electronic amplification, including the development of electro-magnetic pick ups in guitars, was arguably the most important technological innovation that led to the cultural evolution of rock music, and the situation afforded an incredible palette of sound-making that is ongoing well over half a century later (Poss, 1998). Just in the same ways that an animal's vocal system can be "overblown," so can the physical hardware of amplification systems. Early garage rock music, the precursor to punk rock, was likely the first genre to systematically use this overblown amplification effect on purpose. Specific manipulations of electronic signal pathways were developed that allowed musicians to emulate in music what is an honest feature of a vocalization: high arousal. A basic distortion pedal works as follows. The first process is typically an amplitude gain accompanied by a low-pass filter, pushing the signal toward a saturation point where nonlinear alterations will occur. This saturating nonlinearity is filtered again, resulting in output that becomes a multi-band-passed nonlinearity. **Figure 2** shows the effect of a wave shaping function on a 4 s recording of an acoustic guitar and **Figure 3** shows a 78 ms close-up segment of several cycles of the complex waveform in both unaltered and distorted treatments. Yeh et al. (2008) have used ordinary differential equations (ODEs) to digitally model this analog function suggesting that analog distortion used by musicians closely approximates noisy features in vocalization systems that are also well described by the same mathematics. **Figure 1** shows the spectrogram of a coyote vocalization with subtle nonlinear phenomena that appear quite similar to broadband noises generated by ODEs.

Recently, we produced musical stimuli to examine the role of noise in emotional perceptions of music, and used digital

"fpsyg-04-00990" — 2013/12/31 — 11:24 — page 6 — #6

models created for musicians as our noisy source (Blumstein et al., 2012). Twelve 10 s compositions were created that were then manipulated into three different versions: one with added musical distortion noise, one with a rapid frequency shift in the music, and one unaltered control. The manipulations were added at the halfway point in the pieces. These stimuli were played to listeners and they were asked to rate them for arousal and valance. We expected that distortion effects approximating deterministic chaos would cause higher ratings of arousal, and negative valence judgments – the two dimensional description of vocalized fear (Laukka et al., 2005). This is precisely what we found. Subjects also judged rapid pitch shifts up as arousing, but not pitch shifts down. Downward pitch shifts were judged as more negatively valenced which is what we should expect given the acoustic correlates of sadness in voices (Scherer, 1986). Surprisingly, previous work had not explored the role of distortion in affective judgments of music, but an animal model of auditory sensitivity afforded a clear prediction which was confirmed.

We were interested in how these effects occurred in the context of film. Previous work had found that horror soundtracks contained nonlinearities at a much higher rate than other film genres (Blumstein et al., 2010). Film soundtrack composers were exploiting people's sensitivity to noisy features in their efforts to scare or otherwise excite their viewers. Of course, for the most part the direct connection is not consciously made between the ecology of fear screams in animals and the induction of fear in a human audience. But composers and music listeners have an intuitive sense of what sounds are associated with what emotions, and this intuition is rooted in our implicit understanding of form and function in nature – a principle that is strongly reinforced by cultural processes bringing these sounds to us repeatedly generation after generation.

But would sound features alone be sufficient to invoke fear even in the context of an emotionally benign film sequence? We created simple 10-s videos of people engaged in emotionally neutral actions, such as reading a paper, or drinking a cup of coffee. The videos were edited so that the key "action" happened at the exact midpoint, the same time that our nonlinear features in the music clips occurred. Subjects viewed these videos paired with the same music as described above, and we found something interesting. Judgments of arousal were no longer affected by the nonlinear features in the music clips when viewed in the context of a benign action, but the negative valence remained. Clearly, decision processes used in judgments of affect in multimodal stimuli will integrate these perceptual dimensions. One obvious possibility for our result is that the visual information essentially trumped the auditory information when assessing urgency, but the emotional quality of a situation was still shaped by what people heard. Future research should explore how consistent fearful information is processed, and we should expect that auditory nonlinearities will enhance a fear effect as evidenced by the successful pairing of scary sounds and sights in movies. Currently, we are examining psychophysiological responses to nonlinearities, with the expectation that even when judges do not explicitly report greater arousal while hearing nonlinear musical features in certain contexts, there will be measurable autonomic reactions, similar to how brain (OFC) responses to non-human animal

"fpsyg-04-00990" — 2013/12/31 — 11:24 — page 7 — #7

voices do not correspond to people's judgments (Belin et al., 2008).

As mentioned earlier, nonlinear characteristics in music represent one dimension in sound processing that plays a role in music perception and enjoyment. Our sensitivity to such features is rooted in a highly conserved mammalian vocal signaling system. I argue that much of what makes music enjoyable can be explained similarly. But one aspect of music that is not well explained as a by-product is the conspicuous feature that it is often performed by groups – coordinated action of multiple individuals sharing a common cultural history, generating synchronized sounds in a context of ritualized group activity.

#### **MUSIC AS COALITION SIGNALING**

Humans are animals – animals with culture, language, and a particular set of cognitive adaptations designed to interface with a complex social network of sophisticated conspecifics. Pinker (2010) called this the "cognitive niche" taking after ideas earlier proposed by Tooby and DeVore (1987). Information networks and social ecologies have co-evolved with information processors, and thus, a form–fit relationship exists between the cognitive processes in the human mind and the culturally evolved environments for social information. Humans cooperate extensively – in an extreme way when viewed zoologically – and we have many reliably developing cognitive mechanisms designed to solve problems associated with elaborate social knowledge (Barrett et al., 2010). Because many of the adaptive problems associated with extreme sociality involve communicating intentions to cooperate as well as recognizing cues of potential defection in conspecifics, we should expect a variety of abilities that facilitate effective signaling between cooperative agents.

Many species, ranging from primates, to birds, to canines, engage in coordinated signaling. By chorusing together, groups can generate a signal that honestly communicates their numbers, and many other properties of their health and stature. Chorusing sometimes involves the ability to rhythmically coordinate signal production. When two signaling systems synchronize their periodic output (i.e., enter a phase relationship), it can be described as entrainment – an ability that is phylogenetically old, and evolutionarily widespread (Schachner et al., 2009; Phillips-Silver et al., 2010). Fitch (2012) described the paradox of rhythm, which is the puzzle of why periodic phenomena are so ubiquitous in nature, but overt rhythmic ability in animals is so exceedingly rare. The answer, Fitch argued, lies in how we conceptualize rhythm in the first place. When we consider the component abilities that contribute to our capacity for rhythmic entrainment, the complexity in the neurocomputational underpinnings makes the capacity much less paradoxical, and instead understandably rare.