Automatic Sound Event Detection and Classification of Great Ape Calls
  Using Neural Networks
We present a novel approach to automatically detect and classify great ape
calls from continuous raw audio recordings collected during field research. Our
method leverages deep pretrained and sequential neural networks, including
wav2vec 2.0 and LSTM, and is validated on three data sets from three different
great ape lineages (orangutans, chimpanzees, and bonobos). The recordings were
collected by different researchers and include different annotation schemes,
which our pipeline preprocesses and trains in a uniform fashion. Our results
for call detection and classification attain high accuracy. Our method is aimed
to be generalizable to other animal species, and more generally, sound event
detection tasks. To foster future research, we make our pipeline and methods
publicly available.
Zifan Jiang, Adrian Soldati, Isaac Schamberg, Adriano R. Lameira, Steven Moran
# Automatic Sound Event Detection and Classification of Great Ape Calls Using Neural Networks

###### Abstract

We present a novel approach to automatically detect and classify great ape calls from continuous raw audio recordings collected during field research. Our method leverages deep pretrained and sequential neural networks, including wav2vec 2.0 and LSTM, and is validated on three data sets from three different great ape lineages (orangutans, chimpanzees, and bonobos). The recordings were collected by different researchers and include different annotation schemes, which our pipeline preprocesses and trains in a uniform fashion. Our results for call detection and classification attain high accuracy. Our method is aimed to be generalizable to other animal species, and more generally, sound event detection tasks. To foster future research, we make our pipeline and methods publicly available.[1]

S000 000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 00000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 00000 0000 0000 0000 0000 0000 0000 000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 00000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 00000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 000 00000 0000 0000 0000 0000 0000 0000 0000 0000 000 0000 0000 0000 0000 00000 0000 0000 00000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 000 0000 0000 0000 0000 0000 0000 0000 000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 00000 0000 000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 00000 000 0000 0000 000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 000 0000 0000 0000 0000 0000 0000 000 0000 0000 0000 0000 0000 0000 000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 000 00000 0000 000 0000 000 0000 0000 0000 0000 0000 0000 0000 0000 0000 000 0000 0000 000 0000 0000 0000 0000 0000 0000 0000 000 0000 0000 0000 0000 0000 0000 000 0000 0000 000 0000 000 000 0000 0000 000 0000 0000 0000 000 0000 0000 0000 0000 0000 0000 000 000 0000 0000 0000 0000 0000 000 0000 0000 000 0000 0000 0000 0000 0000 0000 000 0000 000 0000 000 0000 000 0000 000 000 0000 000 0000 0000 000 0000 0000 000 0000 000 000 0000 000 0000 000 0000 000 0000 000 0000 0000 0000 0000 000 0000
entropy classification on frame-level (for SED) or clip-level (for audio tagging), or occasionally CTC [11] for "sequentially labeled data" [12].

**Animal sound classification** involves related research that addresses specifically at animal sounds, e.g., [13, 14] on the classification of general animal species and [15, 16, 17] on species, call types and caller identities of primates. While this line of research uses similar acoustic feature representation techniques as seen in SED tasks, the models tend to work on top of individual short audio units of animal vocalizations that are manually selected from raw audio recordings by human experts, and thus fall short on dealing with a continuous audio signal that contains many audio events as well as noise. Instead, our method detects and classifies great ape calls directly from raw recordings ranging from seconds to minutes (theoretically recurrent models also extend to hours, but it could pose challenges in model training time).

**Automatic Speech Recognition (ASR)** on humans - one of the extant five great apes - has seen significant progress [18] (while animal calls remain mysterious to fully decipher), including the recent work wav2vec 2.0 [1]. Wav2vec 2.0 is pretrained on Librispeech [2] in a self-supervised way, and it demonstrates the feasibility of ASR with limited amounts of labeled data. We are interested in whether the speech representation learned by wav2vec 2.0 can be applied successfully to great ape call detection and classification.

## 3 Data

Our data consists of recordings of chimpanzee pant-hoots, orangutan long calls, and bonobo high-hoots. Table 1 provides an overview of these data sets.

**Chimpanzee pant-hoots** are vocal sequences composed of up to four acoustically distinct phases, typically produced in this order: _introduction_, _build-up_, _climax_, and _let-down_[19]. Most pant-hoots contain two or more phases, although single phases can be produced in specific contexts [20]. The successive nature and well-balanced phase duration facilitate the training of a classifier (SS5.1). We use annotated recordings of isolated pant-hoots (i.e., no temporal overlap with others' calls) produced during feeding, traveling, and resting context.

**Orangutan long calls** are composed by a _full pulse_, which is sub-divided into a _sub-pulse transitory element_ and a _pulse body_, or into sequences of _bubble sub-pulse_ or _grumble sub-pulse_[21, 22]. The complex temporal interrelationship between phases and the overlapping and class-unbalanced characteristics (some phases are extremely short) pose challenges for training a multi-class classifier. On the other hand, the duration of calls and non-calls in the data set are well balanced, which opens the gate to a binary call detection model (SS5.2). Annotated recordings include spontaneous long calls and long calls produced in response to other males' vocal presence or environmental disturbances.

**Bonobo high-hoots** - which make up the plurality of our bonobo call data set - are loud, tonal vocalizations [23]. Unlike the chimpanzee and orangutan data sets, call types other than the homologous loud calls are also present in the data set but are rather minor and diverse (e.g., peep-yelp, soft barks), which could be challenging to find. The total call duration is relatively short.

## 4 Method

This section introduces our method, which is illustrated in Fig. 1. We describe each step in turn.

### Audio Preprocessing

First, all audio clips are converted to _.wav_ format and resampled to 16 kHz sample rate. We segment them into 20ms frames and pad with zeros if necessary.

### Feature and Label Extraction

Next, we extract three types of acoustic features with the _torchaudio_ Python package. For an audio clip of \(T\) frames, we get a sequence of frame-level features \(I_{1:T}\), in the shape of \(T\times feature\_dim\):

\begin{table}
\begin{tabular}{l r r r r} \hline \hline
**data set** & **\# audio clips** & **mean / call / total duration** & **\# indiv. (\(\sigma\)/ \(\sigma\))** & **\# call types (duration ratio) / units** \\ \hline chimpanzee & 235 & \(\sim\) 8s / 1,955s / 1,964s & 11 (11 / 0) & 4 (6:3:3:1) / 686 \\ orangutan & 65 & \(\sim\) 74s / 2,793s / 4,817s & 10 (10 / 0) & 7 (700:40:200:10:20:1:1) / 9016 \\ bonobo & 28 & \(\sim\) 24s / 62s / 677s & 7 (3 / 4) & 18 (20:40:10:20:...:200:...:5:1) / 356 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Data set overview and stats. Shown in the table are the number of audio clips, mean duration of each clip, total duration of calls of all clips, total duration of all clips, number of individuals (male and female), number of call types (approximate total duration ratio of them, partially omitted for bonobo due to space limit), and number of individually annotated call units.
* **Raw waveform**: the original amplitude values over time. \(feature\_dim=0.02\times 16000=320\).
* **Spectrogram**: calculated then from the raw waveform with \(feature\_dim=201\).
* **wav2vec 2.0**: inferred from the raw waveform by the _WAV2VEC2_BASE_ model on a CPU device with \(feature\_dim=768\).

We then extract frame-level labels from our annotations. For each annotated unit, we mark all frames inside the annotated time span with a positive class index indicating the call type (e.g., 1 for _intro_); unmarked frames are 0s by default, indicating non-calls. The labels' shape of a clip \(L_{1:T}\) is \(T\times 1\).

### Data Split

We shuffle the clips (features and labels) of each data set and split them into 80%, 10%, and 10% for training, validation, and test sets, respectively. We make three different splits by three different random seeds 0, 42, and 3407 [24] to allow multiple experiments and study the effect of randomness.

### Sequence Modeling

Since our goal is to learn a function \(f\) from \(I_{1:T}\) to \(L_{1:T}\), we first input \(I_{1:T}\) to a sequence modeling function \(f^{m}\) that outputs a hidden sequence \(H_{1:T}\) (\(T\times hidden\_dim\)), which captures inter-frame interaction. \(H_{1:T}\) then goes through a dense linear function \(f^{d}\) (output \(D_{1:T}\)) followed by a Softmax function \(f^{s}\) to produce the probability distribution over target classes, i.e., \(P_{1:T}\) (\(T\times num\_class\)).

Specifically, \(f^{m}\) takes the following forms:

* **RNN**: a bidirectional LSTM with hidden size 1024. \(hidden\_dim=1024*2=2048\).
* **Transformer encoder**: a Transformer encoder with 8 heads and 6 layers. \(hidden\_dim=1024\).
* **Autoregressive model**: we optionally add autoregressive connections between time steps to encourage consistent output labels. The output of \(f^{d}\) at time step \(t\), i.e., \(D_{t}\) is concatenated to the next time step's input \(I_{t+1}\).

Lastly, a cross-entropy loss is computed between the model output \(P_{1:T}\) and the gold labels \(L_{1:T}\), then backpropagated to train the models. To mitigate the class imbalance problem, we set class weights to the reciprocal of each class's occurrence times.

## 5 Experiments and Results

Our experiments were done in PyTorch [25], with Python 3.8 on an Nvidia Tesla V100 GPU (32GB ram). Most models have \(\sim\)40k million parameters and finish the training of up to 200 epochs with early stopping on validation F1-score within one hour (or a few hours because autoregressive recurrent models run slowly on PyTorch). Table 2 presents our results.

### Initial Exploration with Chimpanzee Data

We first test the viability of our approach on the chimpanzee data in light of the simplicity of the pant-hood annotation scheme, although phylogenetically orangutans are closer to humans.

We start from _E1_, a simple waveform + LSTM baseline, and observe in _E2_ that wav2vec 2.0 outperforms the raw waveform and spectrogram by a large margin, which demonstrates the power of transfer learning from pretraining on human speech.

In _E2.1_, we find that the Transformer encoder does not outperform LSTM. Hence, we infer that

Figure 1: Our method (bottom-up). The audio clip shown in the figure is from the chimpanzee data set. _(Non-)ar_ stands for (non-)autoregressive.
Transformer's ability to capture arbitrary long-range dependency is not beneficial to our task.

Next, we explore the hyper-parameters in the second experimental group and we find in _E3.5_ that balancing class weights (SS4.4) has a small impact.

Lastly, we show in _E4_ that the autoregressive connections are beneficial for consistent output, as illustrated by the tiny gaps in _non-ar_ output in Fig. 1.

### Extending to Orangutan and Bonobo Data

We successfully extend the model in _E4_ to as trained on the orangutan (_E5_) and bonobo (_E6_) data sets. We note that some minority classes perform less well due to data scarcity, in contrast to the well-balanced situation in the chimpanzee data set (see Table 1).

We further reduce the task to a binary (call vs. non-call) classification that resembles voice activity detection of human speech. This is a useful tool to automatically extract calls from raw recordings for further bioacoustic analysis (e.g., studying the repertoire of a given species).

Finally, to understand the generalizability of our models, we try zero-shot transferring the model trained in _E5.1_ on orangutans directly to unseen bonobo data. The results show that it is promising to build a potential general-purpose sound event detection model for all great ape calls.

## 6 Discussion

We have addressed a gap in the bioacoustics research of non-human great apes by developing an approach for automatically identifying sound events and classifying great ape calls using a neural network architecture. Our method successfully and accurately identifies and classifies calls in three species of non-human great apes, and provides a tool for primatologists to bootstrap call identification and analysis from raw unannotated audio recordings.

Our method also shows the general applicability of the wav2vec 2.0 model trained on human speech for identifying vocalizations and call types in other species. Future work may apply our approach to more animals, as part of the goal to decode the communication systems of great apes and other non-human animals more broadly.2

\begin{table}
\begin{tabular}{l l l c c c c c} \hline \hline
**ID** & **data** & **feature** & **model** & **dev acc.** & **dev f1** & **test acc.** & **test f1** & **aucpr** \\ \hline \multicolumn{8}{l}{_Explore the best feature and model combination_} \\ E1 & chimp & waveform & lstm (baseline) & \(51.7\pm 2.1\) & \(35.4\pm 2.5\) & \(51.0\pm 3.6\) & \(34.7\pm 4.3\) & - \\ E1.1 & chimp & spectrogram & lstm & \(60.3\pm 1.5\) & \(55.7\pm 2.3\) & \(58.7\pm 4.7\) & \(53.9\pm 5.4\) & - \\ E2 & chimp & wav2vec2 & lstm & \(81.0\pm 4.0\) & \(79.9\pm 4.6\) & \(79.3\pm 2.3\) & \(77.9\pm 3.6\) & - \\ E2.1 & chimp & wav2vec2 & transformer & \(71.3\pm 0.6\) & \(68.3\pm 0.2\) & \(75.3\pm 0.6\) & \(72.1\pm 0.5\) & - \\ \hline \multicolumn{8}{l}{_Explore the hyper-parameters_} \\ E3.1 & chimp & wav2vec2 & lstm (E2 + \(batch\_size=4\)) & \(69.7\pm 1.5\) & \(71.8\pm 2.6\) & \(67.7\pm 4.0\) & \(69.6\pm 4.0\) & - \\ E3.2 & chimp & wav2vec2 & lstm (E2 + \(batch\_size=8\)) & \(63.3\pm 0.6\) & \(62.6\pm 1.0\) & \(62.0\pm 4.4\) & \(61.5\pm 4.0\) & - \\ E3.3 & chimp & wav2vec2 & lstm (E2 + \(dropout=0.2\)) & \(80.7\pm 3.5\) & \(80.0\pm 4.4\) & \(78.0\pm 1.7\) & \(76.8\pm 2.7\) & - \\ E3.4 & chimp & wav2vec2 & lstm (E2 + \(dropout=0.1\)) & \(81.0\pm 4.0\) & \(80.2\pm 4.8\) & \(78.7\pm 2.9\) & \(77.3\pm 3.9\) & - \\ E3.5 & chimp & wav2vec2 & lstm (E2 - \(balance\_weights\)) & \(81.0\pm 3.6\) & \(79.6\pm 4.4\) & \(79.3\pm 2.3\) & \(78.3\pm 3.6\) & - \\ \hline \multicolumn{8}{l}{_Explore autoregressive modeling_} \\ E4 & chimp & wav2vec2 & lstm (E2 + autoregressive) & \(87.7\pm 1.2\) & \(87.1\pm 1.8\) & \(85.7\pm 2.1\) & \(85.6\pm 2.5\) & - \\ \hline \multicolumn{8}{l}{_Extend to orangutan long calls and a binary setting_} \\ E5 & orange & wav2vec2 & lstm (E4) & \(83.0\pm 1.0\) & \(82.7\pm 1.4\) & \(81.7\pm 3.1\) & \(82.0\pm 2.6\) & - \\ E5.1 & orange & wav2vec2 & lstm (E5 + binary target) & \(92.3\pm 2.5\) & \(92.1\pm 2.5\) & \(92.0\pm 1.0\) & \(91.9\pm 1.1\) & 0.96 \\ \hline \multicolumn{8}{l}{_Extend to bonobo calls and a binary setting_} \\ E6 & bonobo & wav2vec2 & lstm (E4) & \(87.0\pm 4.6\) & \(85.9\pm 6.3\) & \(83.7\pm 3.8\) & \(82.3\pm 2.2\) & - \\ E6.1 & bonobo & wav2vec2 & lstm (E6 + binary target) & \(92.0\pm 3.6\) & \(91.9\pm 3.4\) & \(87.7\pm 3.5\) & \(87.8\pm 2.9\) & 0.87 \\ \hline \multicolumn{8}{l}{_Zero-shot transferring from orangutan to bonobo_} \\ E7 & bonobo & wav2vec2 & lstm (E5.1) & \(63.0\pm 13\) & \(69.2\pm 10\) & \(72.0\pm 4.0\) & \(74.2\pm 3.1\) & 0.55 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Experimental results. We run all experiments three times based on different random seeds and report the mean and standard deviation. _acc._ stands for frame-level accuracy, _fl_ stands for the frame-level average F1-score weighted by the number of true instances per class, and _aucpr_ stands for the area under the precision-recall curve for the positive class in the binary case at test time when the random seed is set to 0. For hyper-parameters, we start _E1_ with \(batch\_size=1\), \(dropout=0.4\) and keep them by default, if not otherwise specified in the table.