# animal2vec and MeerKAT
A self-supervised transformer for rare-event raw audio input
and a large-scale reference dataset for bioacoustics

Julian C. Schäfer-Zimmermann<EMAIL_ADDRESS>Department for the Ecology
of Animal Societies, Max Planck Institute of Animal Behavior, Konstanz,
Germany  Department of Biology, University of Konstanz, Konstanz, Germany
Centre for the Advanced Study of Collective Behaviour, University of Konstanz,
Konstanz, Germany  Vlad Demartsev  Department for the Ecology of Animal
Societies, Max Planck Institute of Animal Behavior, Konstanz, Germany
Department of Biology, University of Konstanz, Konstanz, Germany  Centre for
the Advanced Study of Collective Behaviour, University of Konstanz, Konstanz,
Germany  Kalahari Research Centre, Van Zylsrus, Northern Cape, South Africa
Baptiste Averly  Department for the Ecology of Animal Societies, Max Planck
Institute of Animal Behavior, Konstanz, Germany  Department of Biology,
University of Konstanz, Konstanz, Germany  Centre for the Advanced Study of
Collective Behaviour, University of Konstanz, Konstanz, Germany  Kalahari
Research Centre, Van Zylsrus, Northern Cape, South Africa  Kiran Dhanjal-Adams
Department for the Ecology of Animal Societies, Max Planck Institute of Animal
Behavior, Konstanz, Germany  Department of Biology, University of Konstanz,
Konstanz, Germany  Centre for the Advanced Study of Collective Behaviour,
University of Konstanz, Konstanz, Germany  Mathieu Duteil  Department for the
Ecology of Animal Societies, Max Planck Institute of Animal Behavior,
Konstanz, Germany  Department of Biology, University of Konstanz, Konstanz,
Germany  Centre for the Advanced Study of Collective Behaviour, University of
Konstanz, Konstanz, Germany  Gabriella Gall  Department for the Ecology of
Animal Societies, Max Planck Institute of Animal Behavior, Konstanz, Germany
Department of Biology, University of Konstanz, Konstanz, Germany  Centre for
the Advanced Study of Collective Behaviour, University of Konstanz, Konstanz,
Germany  Zukunftskolleg, University of Konstanz, Konstanz, Germany  Marius
Faiß  Department for the Ecology of Animal Societies, Max Planck Institute of
Animal Behavior, Konstanz, Germany  Department of Biology, University of
Konstanz, Konstanz, Germany  Lily Johnson-Ulrich  Kalahari Research Centre,
Van Zylsrus, Northern Cape, South Africa  Department of Evolutionary Biology
and Environmental Studies, University of Zurich, Zurich, Switzerland  Dan
Stowell  Department of Cognitive Science and Artificial Intelligence, Tilburg
University, Tilburg, The Netherlands  Naturalis Biodiversity Center, Leiden,
The Netherlands  Marta B. Manser  Kalahari Research Centre, Van Zylsrus,
Northern Cape, South Africa  Department of Evolutionary Biology and
Environmental Studies, University of Zurich, Zurich, Switzerland
Interdisciplinary Center for the Evolution of Language, University of Zurich,
Zurich, Switzerland  Marie A. Roch  Shared senior authorship Department of
Computer Science, San Diego State University, San Diego, California, USA
Ariana Strandburg-Peshkin  Shared senior authorship Department for the Ecology
of Animal Societies, Max Planck Institute of Animal Behavior, Konstanz,
Germany  Department of Biology, University of Konstanz, Konstanz, Germany
Centre for the Advanced Study of Collective Behaviour, University of Konstanz,
Konstanz, Germany  Kalahari Research Centre, Van Zylsrus, Northern Cape, South
Africa

###### Abstract

Bioacoustic research provides invaluable insights into the behavior, ecology,
and conservation of animals. Most bioacoustic datasets consist of long
recordings where events of interest, such as vocalizations, are exceedingly
rare. Analyzing these datasets poses a monumental challenge to researchers,
where deep learning techniques have emerged as a standard method. Their
adaptation remains challenging, focusing on models conceived for computer
vision, where the audio waveforms are engineered into spectrographic
representations for training and inference. We improve the current state of
deep learning in bioacoustics in two ways: First, we present the _animal2vec_
framework: a fully interpretable transformer model and self-supervised
training scheme tailored for sparse and unbalanced bioacoustic data.
animal2vec learns general representations from audio waveforms without labels
and then uses a second training step to finetune the representations based on
labeled data. Second, we openly publish _MeerKAT_ : Meerkat Kalahari Audio
Transcripts, a large-scale dataset containing audio collected via biologgers
deployed on free-ranging meerkats with a length of over
$1068\text{\,}\mathrm{h}$, of which $184\text{\,}\mathrm{h}$ have twelve time-
resolved vocalization-type ground truth target classes, each with millisecond-
resolution, making it the largest publicly-available labeled dataset on non-
human terrestrial mammals to date. Further, we benchmark animal2vec against
the publicly available _NIPS4Bplus_ birdsong dataset. We report new state-of-
the-art classification performance on both datasets and evaluate the few-shot
capabilities of animal2vec of labeled training data. Finally, we perform
ablation studies to highlight the differences between our architecture and a
vanilla transformer baseline for human-produced sounds. animal2vec allows
researchers to classify massive amounts of sparse bioacoustic data even with
little ground truth information available. In addition, the MeerKAT dataset is
the first large-scale, millisecond-resolution corpus for benchmarking
bioacoustic models in the pretrain/finetune paradigm. We believe this sets the
stage for a new reference point for bioacoustics.

self-supervised deep learning; bioacoustics; animal vocalizations;
transformer; reference dataset

††preprint: v1.1

##  Introduction

Bioacoustics, the study of animal sounds, reveals invaluable insights into the
behavior [1, 2], ecology [3, 4], and conservation [5, 6] of animal species.
Automated analysis of acoustic recordings can greatly advance the types of
questions that can be asked by enabling annotation of long-duration recordings
covering multiple seasons and years. While existing datasets cover extensive
hours, events of interest such as vocalizations are often sparse, brief, and
in noisy conditions, making manual analysis challenging [7, 8, 9, 10, 11, 12,
13, 14].

Generally, deep learning is a common approach to tackle large and dense
datasets [15], where, recently, transformer-based models [16] have achieved
state-of-the-art results across many tasks and modalities [17]. However, there
is a lack of such large-scale datasets and training approaches for sparse data
using next-generation transformer-based models within bioacoustics [18]. To
fill this gap, we introduce the _animal2vec_ framework and the _MeerKAT_
dataset. animal2vec is an interpretable transformer model and training scheme
tailored for sparse and unbalanced bioacoustic data, achieving state-of-the-
art results, and _MeerKAT_ : Meerkat Kalahari Audio Transcripts, is the
largest publicly-available labeled dataset on non-human terrestrial mammals to
date.

Currently, in bioacoustics, the primary data (audio waveforms) are usually
feature-engineered into spectrograms for input to convolutional neural network
models (CNNs) originally designed for computer vision [18]. However, using
spectrograms and CNNs is justified more by empirical success than conceptual
fitness. Spectrograms challenge the notion of translational invariance in CNNs
[19], discard phase information or temporal fine structure [18], and the
commonly used Mel-scale biases the input toward human hearing [18]. Further,
in computer vision, attention-based encoder-only Visual transformers (ViTs)
[20, 21] have replaced CNNs, excelling through large-scale pretraining on
dense datasets. In this context, pretraining is a method of learning a general
model, which can then be finetuned on downstream tasks. This training paradigm
is referred to as pretraining/finetune. Current best-performing pretraining
strategies include either supervised [20] or contrastive learning (CLR) [22,
23, 24]. Pretraining in CLR enforces to pull similar data points (different
patches of the same image) together and pushing dissimilar ones (patches from
other images in the batch) apart in a latent representation space [24].

Supervised pretraining is not feasible in bioacoustics due to limited
datasets. The largest publicly available labeled bioacoustic dataset is
arguably the Animal subset of _Audio set_ [25], with
$112.6\text{\,}\mathrm{h}$ across $40\,758$ $10\text{\,}\mathrm{s}$ samples.
However, Audio set is weakly-labeled [26], is based on YouTube videos that do
not reflect realistic bioacoustic recording scenarios, is heavily dominated by
recordings of domestic animals and birds ($\approx\,$$75\text{\,}\mathrm{\char
37\relax}$), and is still significantly shorter than the smallest pretraining
corpus in human speech recognition, Librispeech (940 hours) [27]. Furthermore,
CLR presents conceptual challenges with sparse and unbalanced bioacoustic
data, as sparsity hinders effective negative sampling [28, 29]. Despite these
obstacles, ViTs have been introduced to bioacoustics [30, 31, 32, 33, 34, 35],
where approaches range from no pretraining [31, 32] to various pretraining
strategies, including web-scraped human speech [35], human language audio-
caption pairs [34], pretraining on ImageNet [36, 30], or repurposing
pretrained CNNs from Audio set [25] as a pre-transformer feature extraction
step [33]. As of now, pretraining a transformer model with bioacoustic data is
still an open problem.

Consequently, deep learning in bioacoustics is facing a double-edged
challenge: the inherent constraints of spectrograms and the conceptual
problems of prevailing pretraining strategies in computer vision.

We address both issues by releasing animal2vec and MeerKAT; animal2vec is a
framework for training animal call recognizers from raw waveforms containing
sparsely distributed calls with non-uniformly distributed call types. Features
are extracted from the pressure waveforms using a learned set of SincNet-style
filterbanks [37] and a transformer encoder [16, 38].

MeerKAT is a $1068\text{\,}\mathrm{h}$ large-scale dataset containing data
from audio-recording collars worn by free-ranging meerkats (_Suricata
suricatta_) at the Kalahari Research Centre, South Africa [39], of which
$184\text{\,}\mathrm{h}$ are labeled with twelve time-resolved vocalization-
type ground truth target classes, each with millisecond resolution. animal2vec
is conceptually simple, excels with noisy and sparse datasets, achieves state-
of-the-art performance, is capable of learning from limited training data
(few-shot learning), and provides temporal and spectral interpretability. The
labeled $184\text{\,}\mathrm{h}$ MeerKAT subset exhibits realistic sparsity
conditions ($96\text{\,}\mathrm{\char 37\relax}$ background-noise or other
signals and $4\text{\,}\mathrm{\char 37\relax}$ vocalizations), dispersed
across $66\,398$ $10$-second samples, spanning $251\,562$ labeled events and
showcasing significant spectral and temporal variability, making it the first
large scale reference point with real-world conditions for benchmarking
pretraining and finetune approaches in bioacoustics deep learning.

We release all the code and pretrained models under an MIT license [40] at our
GitHub [41], and the MeerKAT dataset under a Creative Commons BY-NC license
[42] at the Max-Planck data repository Edmond [43].

Our work: (i) paves the way to adapt and specialize next-generation
transformer models to the domain of bioacoustics using the unified animal2vec
framework, (ii) allows researchers with only little labeled data to classify
large amounts of challenging data, and (iii) introduces the first bioacoustic
benchmark to evaluate large-scale pretrain/finetune approaches under realistic
sparsity and class balancing conditions using MeerKAT.

##  Materials and Methods

###  The MeerKAT dataset

####  Recording and label availability

Recording and labeling a large-scale dataset is a large-scale endeavor  The
compilation of MeerKAT reflects an extensive collaborative effort by
researchers and students (see Acknowledgements) who diligently recorded,
labeled, and validated the dataset over an extended period.

The data were collected during two field seasons (Aug-Sep 2017 and Jul-Aug
2019) at the Kalahari Research Centre in South Africa. Meerkats are a social
mongoose species native to the arid parts of southern Africa. Meerkats forage
throughout the day by digging in the ground for prey, remaining cohesive with
their group mates while moving within their territory. They use vocalizations
to mediate a variety of social behaviors, and their vocal repertoire has been
extensively characterized [44, 45].

Overall, $2521\text{\,}\mathrm{h}$ of audio has been recorded in $1284$ files.
The majority of the audio ($2269\text{\,}\mathrm{h}$ in $756$ $3$-hour-long
files) originated from acoustic collars (Edic Mini Tiny+ A77, Zelenograd,
Russia, which sample at $8\text{\,}\mathrm{kHz}$ with $10$ bit quantization)
that were attached to the animals ($41$ individuals throughout both
campaigns), where each file corresponds to a recording for a single individual
and day. The remainder of the dataset ($252\text{\,}\mathrm{h}$ in $528$ files
of varying length) was recorded using Marantz PMD661 digital recorders
(Carlsbad, CA, U.S.) attached to directional Sennheiser ME66 microphones
(Wedemark, Germany) sampling at $48\text{\,}\mathrm{kHz}$ with $32$ bit
quantization. When recording, field researchers held the microphones close to
the animals (within $1\text{\,}\mathrm{m}$). The data were recorded during
times when meerkats typically forage for food by digging in the ground for
small prey.

Labeling was done to cover as many different files from as many days and
individual meerkats as possible. In total, $325$ files were partially labeled
(at least $1\text{\,}\mathrm{h}$, but not the full file to cover more files),
of which $278$ contain audio from collars and $47$ from directional
microphones.

Figure 1:  The statistics of the meerkat dataset and precision-recall curves
of the presented classifier. a) shows the temporal distributions of all
MeerKAT classes in $12$ violin plots. Each category shows kernel density
estimates of duration for the class (colored splits on the right). The global
distribution across all categories is shown in gray on the left of each plot
to make clear how the label durations of each category relate to the dataset
overall. All splits are scaled to full width, where the scaling multiplier is
shown at the top of each split, as the number of examples for each category
varies considerably. In each split, dashed lines show the $25$th, $50$th, and
$75$th percentile, where the $50$th percentile (median) value is written next
to its dashed line. In addition, the event-count, the total length in minutes,
and the percentage with respect to all counts/total length are displayed at
the top of each plot. b) shows four precision-recall curves for (i) the global
micro average, and the (ii) close call, (iii) short-note call, and (iv) alarm
call class. Results of animal2vec using $1\text{\,}\mathrm{\char 37\relax}$,
$25\text{\,}\mathrm{\char 37\relax}$, and $100\text{\,}\mathrm{\char
37\relax}$ of the training data are in red, yellow, and teal, respectively,
and the baseline results are in gray. Overlays within each subplot show
statistics about the occurrence-wise percentage share and the median duration
of all events in this class.

MeerKAT contains the full audio of these $325$ files, amounting to
$1068\text{\,}\mathrm{h}$, released as $384\,592$ $10$-second samples. The
labeled subset has a total length of $184\text{\,}\mathrm{h}$ spread across
$66\,398$ $10$-second samples. This subset is ground-truth-complete; all call
and recurring anthropogenic events in this $184\text{\,}\mathrm{h}$ are
labeled. All $10$-second samples have been standardized to a sample rate of
$8\text{\,}\mathrm{kHz}$ with 16-bit quantization, where downsampling was done
using Torchaudios resample method [46] with a Kaiser window [47] and a low-
pass filter width of $8\text{\,}\mathrm{ms}$. This yields a total dataset size
of just under $59\text{\,}\mathrm{GB}$ ($61\text{\,}\mathrm{GB}$ including the
label files). While the sampling rate of $8\text{\,}\mathrm{kHz}$ is
sufficient to capture the majority of meerkat vocalization frequencies (the
first two formants are below the Nyquist frequency of $4\text{\,}\mathrm{kHz}$
[48]), the total dataset size is comparatively small, making MeerKAT easily
accessible and portable despite its extensive length. By agreement with the
Kalahari Research Centre (KRC), we have made these data available in a way
that can further machine learning research without compromising the ability of
the KRC to continue conducting valuable ecological research on these data.
Consequently, the filenames of the $10$-second samples have been randomly
sampled, and their temporal order and individual identity cannot be recovered,
but can be requested from us. The order and identity of the files are not
needed in order to recreate the results of this work.

The overall structure of MeerKAT, having a huge unlabeled and a large fully-
labeled subset originating from the same pool of audio files, make it an ideal
test-bed for pretraining/finetune approaches in bioacoustics.

####  Label structure and statistics

The labels in MeerKAT reflect over twenty years of research  In total, eight
_vocalization_ classes and three _miscellaneous_ classes were identified. The
vocalization classes are: _close call_ [49], _short-note call_ [50, 51],
_social call_ [44], _alarm call_ [52], _aggressive call_ [50], _move call_
[53, 50], _lead call_ [53], and _other call_ (see also [44, 45] for a general
overview on meerkat vocalizations). Meerkats can produce some calls that do
not fit well into the set of described calls. These calls are frequently
hybrid calls that bear similarity to multiple call types, or are simply too
rare to have their own category. Such calls are labeled as _other call_ within
MeerKAT. The three miscellaneous classes are for non-call events. The _synch_
and _beep_ events are generated by a GPS clock that was used to synchronize
acoustic streams to one another across animals for the purposes of the
behavioral study for which the data were collected (see 3.1.3 in [2]). The
eating label indicates chewing noises from a successful foraging event. In
addition to the vocalization and miscellaneous classes, a superordinate class
called _focal_ is used to indicate when a call was produced by the focal
animal wearing the collar as opposed to a nearby conspecific. Trained analysts
made this decision based on relative intensity of calls, changes in the
frequency spectrum, and contextual information (see also supplemental
information in [54]). Each $10$-second file has an accompanying HDF5 label
file [55] that lists label categories, start and end time offsets (s), and a
focal indicator.

MeerKAT is _multi-class_ and _multi-label_ , which means that ground-truth
labels may overlap. The temporal resolution of all labels is in the few
$\mathrm{ms}$ range ($12(7)\text{\,}\mathrm{ms}$ inter-rater reliability
[56]). All classes, as well as the distribution of the call durations, are
shown in figure 1 a).

MeerKAT is intra- and inter-class unbalanced, sparse, spectrally rich, and
dirty  MeerKAT is a highly unbalanced and sparse dataset in terms of event-
occurrence, event-lengths, and class balance. While the labeled subset covers
almost $184\text{\,}\mathrm{h}$, the total duration of all labeled parts is
only $13.2\text{\,}\mathrm{h}$ ($7.2\text{\,}\mathrm{\char 37\relax}$).
Moreover, considering only the vocalization events, this reduces to
$7.8\text{\,}\mathrm{h}$ ($4.2\text{\,}\mathrm{\char 37\relax}$).

For example, while the Short-note class is overrepresented in terms of
occurrence ($27\text{\,}\mathrm{\char 37\relax}$ of all counts), the median
length is only $37\text{\,}\mathrm{ms}$, making it underrepresented in terms
of duration ($4\text{\,}\mathrm{\char 37\relax}$ of the total duration,
whereas $9\text{\,}\mathrm{\char 37\relax}$ would be expected in a balanced
dataset). On the other end, the synch signal is an artificial voice signal
from a GPS clock whose median duration is $1258\text{\,}\mathrm{ms}$, making
it account for $28.1\text{\,}\mathrm{\char 37\relax}$ of the total duration of
all events but only for $4\text{\,}\mathrm{\char 37\relax}$ of all
occurrences. Furthermore, the rarest five classes (other call, alarm call,
aggressive call, move call, lead call) added together account for only
$14\text{\,}\mathrm{\char 37\relax}$ of the occurrences and
$12\text{\,}\mathrm{\char 37\relax}$ of the total duration. Consequently,
MeerKAT is two-fold unbalanced: The significant differences in the temporal
and occurrence-wise distributions can make a single class over- and
underrepresented simultaneously (intra-class balance) and make reasoning about
a particular class’s abundance, compared to other classes, very complex
(inter-class balance). Therefore, there is no clear path to implement existing
approaches to handle imbalance in datasets [57, 58].

Figure 2 provides concrete examples plotted as Mel spectrograms in dB scale
for (a) a representative stream of audio and (b) the individual classes in
MeerKAT. a) shows four alarm call events covered by a varying amount of
spectrally broad, ultra-short, and non-stationary noise patterns originating
from the meerkats foraging for food by digging in the ground or bumping their
collars into obstacles. Noise patterns such as these permeate the majority of
MeerKAT. b) shows the spectral variability between classes, where the examples
shown do not represent the overall data quality but reflect ideal candidates.

Ultimately, MeerKAT makes a formidable benchmark for exploring sparsity,
noise-resistance, and imbalance in bioacoustics, containing events that are
rare or plentiful, long or short, artificial or natural, temporally and/or
occurrence-wise sparse, and spectrally rich, all while being covered in a
challenging amount of spectrally broad, ultra-short, and non-stationary noise
patterns.

Figure 2:  Example Mel spectrograms for a representative audio snippet and for each class.  Table 1:  Class-wise dataset statistics and results. a) shows the average precision scores (AP) [59] of each model, where the underscores indicate the training sample size during finetuning. _Transformer baseline_ uses $100\text{\,}\mathrm{\char 37\relax}$ of the training samples for finetuning. The strongest result per class is in bold letters. The two bottom rows show the micro- and macroaverage across all classes except _Focal_. b) shows the training and evaluation split sample sizes used for finetuning. The standard deviation across the stratified multilabel 5-fold cross validation routine [60] is given in smaller brackets next to each value.  a) |  | Average precision scores [59] | b) | Sample sizes
---|---|---|---|---
|  | Transformer baseline[38] | animal2vec |  | Evaluation | Training
| % Training labels | $100\text{\,}\mathrm{\char 37\relax}$ | $1\text{\,}\mathrm{\char 37\relax}$ | $25\text{\,}\mathrm{\char 37\relax}$ | $100\text{\,}\mathrm{\char 37\relax}$ |  | - | $1\text{\,}\mathrm{\char 37\relax}$ | $25\text{\,}\mathrm{\char 37\relax}$ | $100\text{\,}\mathrm{\char 37\relax}$
| Focal | 0.59 (2) | 0.86 (1) | 0.92 (1) |  0.94 (1) |  | 24594 (280) | 983 (28) | 24650 (135) | 98520 (280)
Vocalizations
| Close call | 0.49 (1) | 0.90 (2) | 0.93 (2) |  0.94 (1) |  | 22418 (153) | 907 (41) | 22342 (137) | 89310 (153)
| Short-note call | 0.14 (1) | 0.88 (1) | 0.91 (1) |  0.92 (1) |  | 13336 (158) | 522 (40) | 13505 (139) | 54051 (158)
| Social call | 0.30 (2) | 0.65 (1) | 0.79 (1) |  0.84 (1) |  | 4788 (81) | 207 (14) | 4847 (42) | 19431 (81)
| Other call | 0.07 (1) | 0.33 (2) | 0.43 (1) |  0.50 (2) |  | 2754 (67) | 114 (18) | 2799 (58) | 11192 (67)
| Alarm call | 0.03 (1) | 0.57 (1) | 0.77 (1) |  0.80 (1) |  | 1649 (118) | 71 (12) | 1704 (74) | 6684 (118)
| Aggressive call | 0.09 (1) | 0.54 (2) | 0.62 (1) |  0.71 (2) |  | 1214 (58) | 50 (15) | 1338 (28) | 5181 (58)
| Move call | 0.09 (2) | 0.59 (1) | 0.53 (2) |  0.61 (1) |  | 1080 (23) | 42 (6) | 1071 (34) | 4226 (23)
| Lead call | 0.01 (1) | 0.41 (1) | 0.39 (2) |  0.50 (1) |  | 165 (10) | 8 (1) | 174 (12) | 667 (10)
Miscellaneous
| Synch signal | 0.91 (1) | 0.89 (1) | 0.96 (1) |  0.98 (2) |  | 1999 (2) | 80 (0) | 1997 (1) | 7990 (2)
| Eating | 0.12 (1) | 0.59 (1) | 0.83 (2) |  0.87 (1) |  | 760 (10) | 31 (2) | 754 (11) | 3019 (10)
| Beep Signal | 0.26 (1) | 0.74 (1) | 0.77 (2) |  0.80 (1) |  | 430 (5) | 18 (1) | 431 (4) | 1725 (5)
| Macroaverage | 0.26 (1) | 0.66 (1) | 0.74 (1) |  0.78 (1) |  |  |  |  |
| Microaverage | 0.30 (1) | 0.83 (1) | 0.88 (1) |  0.91 (1) |  |  |  |  |

MeerKAT comes with 5-fold stratified multi-label cross-validation
finetune/evaluation splits  Along with the $10$-second audio files and their
labels, we release manifest files containing the filenames of the $10$-second
splits used to build each experiment’s finetune and evaluation splits. We
conducted four experiments: (i) A full finetuning using all training data,
(ii) two few-shot experiments, and (iii) one generalizability study with a
holdout file not used for pretraining or finetuning. All four experiments’
finetune and evaluation splits are produced using stratified 5-fold multi-
label cross-validation [60]. Final results are averaged and provided along
with their standard deviation.

Our strategy to produce (i) and (ii) was to produce the five folds of finetune
and evaluation splits in the first step using all available data. This
produces the scenario in (i). Then, for each of the five finetune splits, we
construct two stratified few-shot versions using $1\text{\,}\mathrm{\char
37\relax}$ and $25\text{\,}\mathrm{\char 37\relax}$ of the data, keeping the
evaluation split identical for all scenarios. This produces the scenario in
(ii) and yields a total of $15$ finetune and five evaluation splits: Five
folds for each of the three finetune splits (using $1\text{\,}\mathrm{\char
37\relax}$, $25\text{\,}\mathrm{\char 37\relax}$, and
$100\text{\,}\mathrm{\char 37\relax}$ of the training data) and one evaluation
split for each fold.

We publish this set of manifest files along with MeerKAT to ensure
reproducibility and comparability of different approaches when using MeerKAT
and to provide low-threshold access to this extensive evaluation routine.

Table 1 b) holds the average sample sizes for all experiments except the
generalizability study. When averaging across all five folds, the standard
deviation is given in smaller brackets next to each value.

For the holdout generalizability study (iii), we randomly selected $62$
($\approx$20\text{\,}\mathrm{\char 37\relax}$$) full files of the $325$ base
files in MeerKAT and used all labeled $10\text{\,}\mathrm{s}$-segments within
them to produce a $33.5\text{\,}\mathrm{h}$ evaluation split. The data used in
this evaluation split are not used during pretraining or finetuning. The
results for the generalizability study and their discussion are in section
3.1.2.

###  The NIPS4Bplus dataset

NIPS4Bplus is a diverse multi-label small-scale dataset  The NIPS4Bplus
dataset is an openly available and densely annotated multi-class and multi-
label birdsong audio dataset initially created for the NIPS4B 2013 challenge
[61]. It contains $687$ recordings of $51$ bird species categorized into $81$
classes (Multiple bird species produce a vocalization and a song signal). Song
Meter SM2BAT recorders using SMX-US microphones (both manufactured by Wildlife
Acoustics, Maynard, MA, U.S.) were placed in $39$ locations across France and
Spain. The total duration of all field recordings is $30\text{\,}\mathrm{h}$,
where the final dataset release has a duration of $3435\text{\,}\mathrm{s}$ in
$687$ $56$-second files($\approx$ $1\text{\,}\mathrm{h}$) [12], sampled at
$44.1\text{\,}\mathrm{kHz}$ with $16\text{\,}\mathrm{bit}$ quantization.
$1104\text{\,}\mathrm{s}$ ($33.7\text{\,}\mathrm{\char 37\relax}$) of the
total duration are bird vocalizations or songs, of which
$184\text{\,}\mathrm{s}$ contain overlapping signals. The selected recordings
are chosen to maximize diversity across bird species, recording location, and
signal type. A full description of the dataset is available in [12].

We include the NIPS4Bplus dataset in this work to be able to benchmark our
model with other approaches using the same dataset [62, 63] but resample the
entire dataset to $32\text{\,}\mathrm{kHz}$ with $16\text{\,}\mathrm{bit}$
quantization to strike a balance between the computational cost of our model
and the necessary sampling rate to capture all relevant features. The
computational cost of transformer models scales quadratically with longer
input sequences [16], which is directly proportional to the sampling rate.
However, we found a Nyquist frequency of $16\text{\,}\mathrm{kHz}$ sufficient
for our experiments as the fundamental frequencies of birds rarely exceed
$12\text{\,}\mathrm{kHz}$ [64]. Furthermore, we pad all recordings to
$6\text{\,}\mathrm{s}$ length and store them in the same format as MeerKAT. We
provide scripts in our repository to reproduce this processed version of
NIPS4Bplus; see _Code availability_ statement.

To finetune with NIPS4Bplus, we pretrain with unlabeled xeno-canto data
Transformer-based architectures lack the inductive bias found in
convolutional-based models; as such, learning from small-scale datasets is
prohibitively more difficult [65]. Consequently, and since NIPS4Bplus is a
birdsong dataset, we pretrained our model with data from the xeno-canto
database, which is a large citizen science project for recordings of bird
vocalizations and shares enough similarity with the NIPS4Bplus dataset to
provide a good starting point for downstream finetuning. We use the data
provided as part of the Cornell Birdcall Identification Kaggle challenge [66],
with additional xeno-canto data from [67] and [68]. This pretraining dataset
has no overlap with the NIPS4Bplus dataset and a total length of approximately
$700\text{\,}\mathrm{h}$, where we cropped/padded all files to have a maximal
length of $6\text{\,}\mathrm{s}$. Files with a duration longer than
$6\text{\,}\mathrm{s}$ were split into multiple samples. We resampled all
files to $32\text{\,}\mathrm{kHz}$ with $16\text{\,}\mathrm{bit}$
quantization. We provide scripts in our repository to reproduce this
pretraining corpus; see _Code availability_ statement. We do not use the label
information from this pretraining dataset at any time.

###  animal2vec pretraining

####  Concept

Pretraining is necessary, but the dominant schemes are not ideal for
bioacoustics  In the last decade, there has been no training scheme more
successful across all data modalities than the self-supervised
pretrain/finetune scheme [69, 70, 24]. In general, self-supervised learning is
a training paradigm in which a generalist model is trained using an artificial
supervisory task created from the data without using any ground truth [24].

Currently, contrastive-learning-based (CLR) pretraining is the dominant scheme
in computer vision [22, 23, 71, 72, 73] and audio processing [74, 75], whereas
generative methods (learning by reconstructing), either autoregressive [76] or
bidirectional mask-prediction [77, 78, 79], yield state-of-the-art results in
natural language processing.

However, these approaches are not conceptually well suited to handle
bioacoustic data. CLR-based methods suffer from inefficient negative sampling
with sparse and unbalanced datasets [28, 29, 80], and generative pretraining
diverges faster when faced with sparse and noisy data [81, 82].

animal2vec is a mean teacher self-distillation process for sparse data  An
alternative pretraining method is to use a distillation process [83] with no
labels, called mean teacher self-distillation [84, 85, 86, 87, 38], which is
known to be a noise-robust learning scheme [88]. Distillation, in general, is
the notion of transferring knowledge from a teacher to a student model,
whereas mean teacher self-distillation is to update only the student model via
gradient descent and let the teacher model track the student’s weights using
an exponentially moving average (EMA).

Grill and colleagues [85] provide intuition for why this process works. They
randomly initialized two models: A fixed non-trainable (teacher) and a
trainable model (student). The teacher received the input, and the student was
trained to regress the embeddings produced by the teacher. Evaluating the
student model, after training, on ImageNet [36] using the linear-evaluation
protocol [89, 90] resulted in $18.8\text{\,}\mathrm{\char 37\relax}$ top-1
accuracy, whereas the fixed teacher scored $1.4\text{\,}\mathrm{\char
37\relax}$.

Consequently, it is possible to obtain improved embeddings from the embeddings
of an inferior model. Intuitively, in the mean teacher self-distillation
regime, the student learns an improved representation by regressing the
teacher’s output, which in return improves the teacher, as its weights are
updated by tracking the student’s ones. This is a feedback loop and
sequentially increases the performance of both models over the course of the
pretraining. Afterward, only the student model is used for subsequent
finetuning.

In animal2vec, we use a self-distillation framework similar to _data2vec 2.0_
[38], where the model is treated as three components: A single feature
extractor and two contextualizing networks (student and teacher), see figure
3. The feature extractor is domain-specific, and the two contextualizing
networks are domain-agnostic transformer architectures [16]. The feature
extractor receives the batch of input samples and produces a fixed-size
initial representation that is fed to the two contextualizing networks. The
teacher receives the full initial representation from the feature extractor,
and the student receives the unmasked timesteps from a masked embedding (see
figure 3 and the following subsection). The teacher produces a target
embedding, and the student produces a prediction embedding. The loss function
during pretraining is then a mean-squared-error regression to match the
prediction and the target; see section 2.3.3 for more details.

####  Masking and regularization

Bioacoustic data requires strong regularization  Regularization alters the
model architecture, the input data, or the training process to increase
robustness against out-of-distribution samples, noise, label imbalance, and
overfitting [88]. Using strong regularization has a rich history in
bioacoustics since natural sounds inherently present immense variability
between species, individuals, and environments. Models in bioacoustics can
easily overfit to training data, struggling to generalize to new scenarios.
[91, 92, 93, 18].

For animal2vec pretraining we use decoupled weight decay [94], dropout [95],
and layer normalization [96] in all layers. We augment the input audio files
using the stochastic A-weighted input mixing in between-classes-learning [97]
(BCL), as shown in figure 3. We reduce the window length in BCL from
$100\text{\,}\mathrm{ms}$ to $50\text{\,}\mathrm{ms}$ to calculate the
A-weighted sound pressure levels to account for the fact that animal
vocalizations act on a shorter timescale than the generic sound input BCL was
designed for.

However, the most critical regularization technique during the animal2vec
pretraining stage is the stochastic strategy used to mask the embeddings input
to the student model, see figure 3, where we use the same ruleset as in [74,
87, 38]. First, the masking routine randomly selects a proportion (specified
by the probability parameter $p$) of the total timesteps in the embedding
space. These act as starting points for the masked spans. Then, starting from
each selected timestep, the model masks a consecutive span of steps (the
length of this span is determined by the mask length parameter $M$) by filling
the span with randomized values from a normal distribution. Masked spans can
overlap, where the union of the overlapping regions is then used for masking.
Higher values for $p$ select more start frames, and higher values for $M$ mask
longer spans, starting from these start frames.

Humans sound different than meerkats  Most research on pretraining using mean-
teacher self-distillation and raw audio as input is concerned with human
speech audio [87, 38]. However, reconstructing masked timesteps is easier from
their surroundings in human speech than in bioacoustics. Humans have rich
vocal expressions where information is correlated over longer stretches of
time compared to meerkats and most other non-human animals. Furthermore,
sparsity is much less pronounced in human speech data than in bioacoustic
datasets [18, 38].

The best set of parameters for human speech is $p=0.065$ and $M=10$ [74, 87,
38], which results in $49\text{\,}\mathrm{\char 37\relax}$ of all timesteps
masked, where the most frequent span length (the mode in the red mask length
distribution in figure 4) is $50\text{\,}\mathrm{ms}$. This set of parameters
would almost always mask a complete short-note call (median duration of
$35\text{\,}\mathrm{ms}$, see figure 1 a) when a starting point was selected
near such a call. It would expose long stretches of noise to the teacher as
only $49\text{\,}\mathrm{\char 37\relax}$ of all timesteps are masked, but
over $96\text{\,}\mathrm{\char 37\relax}$ of MeerKAT contains various
challenging noise patterns.

Therefore, the masking strategy in animal2vec selects more starting points
($p=0.150$) with shorter mask lengths ($M=2$). This way, almost
$96\text{\,}\mathrm{\char 37\relax}$ are masked, but the mode of the mask
length distribution is $22\text{\,}\mathrm{ms}$ (the yellow mask length
distribution in figure 4). We ablate our choice for $p$, $M$, and the BCL
window length in section 3.1.2. Ablation studies in machine learning test the
importance of different model components by systematically removing them and
observing the impact on performance.

Figure 3:  animal2vec pretraining schematic.

####  Architecture and technical details

animal2vec uses an efficient transformer architecture with a convolutional
decoder for pretraining  We follow [74, 87, 38] and use large transformers
[16] for the student and the teacher with $16$ layers ($N$ in figure 3) and
$16$ attention heads with an embedding dimensionality of $1024$. We use a
custom feature extractor tailored for bioacoustic data that employs SincNet-
style filterbanks [37] and a stack of 1D convolutional layers that learns a
downsampling to an effective sample rate of $f_{\text{sr,
eff}}=$200\text{\,}\mathrm{Hz}$$. The whole feature extractor module (see
yellow segment in figure 3) has a receptive field size of
$46\text{\,}\mathrm{ms}$. Our Sinc module differs from [37] in that it uses no
max pooling, a custom activation function, layer normalization, and smaller
Sinc filters ($8\text{\,}\mathrm{ms}$ receptive field size, kernel size $k=63$
over $31\text{\,}\mathrm{ms}$, $k=251$ in [37]). We choose a kernel size $k$
according to $k=\lfloor\nicefrac{{f_{sr}}}{{126}}\rfloor$, where $f_{sr}$ is
the sample rate of the input. This rule fixes the spectral resolution to
$126\text{\,}\mathrm{Hz}$ and results in a kernel size of $63$ for MeerKAT
$253$ for NIPS4Bplus. A spectral resolution of $126\text{\,}\mathrm{Hz}$ was
empirically found to work best for both datasets.

Research on activation functions has shown that early convolutional layers
benefit from near-linear parametric activation functions as these often act as
bandpass filters [98]. Therefore, we use a custom version of the Swish
activation function [99] (sometimes called SiLU with learnable $\beta$
parameter [100]) called PSwish with additional learnable parameter $\alpha$ of
the form: $h\left(x\right)\,=\,x\,\alpha\,\sigma\left(\beta\,x\right)$, where
$\sigma$ is the sigmoid function. We set the shape of $\alpha$ and $\beta$ to
the number of Sinc filters, initialize them with $\alpha=2$ and $\beta=0$, and
exclude $\alpha$ and $\beta$ from the weight decay regularization as smaller
activation parameters are not necessarily beneficial in parametric activations
[98]. This way, PSwish is a linear activation when training starts but becomes
an individual non-linearity for every filter throughout training.

As in [38], we embed only unmasked timesteps with the student network, use a
learned positional encoding [101], and implement a 1D convolutional decoder to
reconstruct the prediction embedding, where the masked timesteps are filled
with Gaussian noise prior to passing it to the decoder [101]. Furthermore, we
re-use the teacher representation as target for multiple masked versions of
the input sample, this is called multi-mask training [38]. Passing only
unmasked timesteps to the student and re-using the teacher for multi-masking
improves efficiency and reduces the computational cost significantly [38]
since the computational complexity of transformer networks scales
quadratically with their input length [16].

The teacher network tracks the student weights using an EMA update rule, where
we use the implementation and parameters in [87, 38].

The output of our model consists of likelihood estimates with a resolution of
$200\text{\,}\mathrm{Hz}$. A full schematic of the model in pretraining mode
is shown in figure 3. The final model has $315$M trainable parameters. We
train for $100$ epochs using the decoupled Adam optimizer (weight decay of
$0.01$) [94], a cosine learning rate schedule [102], linear warmup for
$10\,000$ steps, a final learning rate of $1\text{\times}{10}^{-4}$, gradient
clipping of $1$ [103], and a batch size of $1020\text{\,}\mathrm{s}$ on four
NVIDIA A100-SXM4-80GB GPUs for $20\text{\,}\mathrm{d}$. The code is written in
PyTorch [104] using the fairseq framework [105]. The pretraining parameters
for all settings can be found in table 5 in the appendix.

We estimate that our setup consumed
$3200\text{\,}\mathrm{kW}\text{\,}\mathrm{h}$ (the typical yearly consumption
of a German household [106]) with a carbon footprint of approximately
$1400\text{\,}\mathrm{kg}_{\mathrm{\ce{CO2}\textit{eq.}}}$ (average emission
factor of 2023 for Germany is
$400\text{\,}\mathrm{g}_{\mathrm{\ce{CO2}\textit{eq.}}}\text{\,}{\mathrm{kW}}^{-1}\text{\,}{\mathrm{h}}^{-1}$
[107]). While the resources to train such a model are expensive, leased
resources such as using Google Cloud tensor processing unit (TPU) v5p chips
are capable of pretraining our model for a more modest cost (about
$500\text{\,}\mathrm{[}$$] at the time of writing) and reduce the carbon
footprint significantly [108]. In addition, researchers can already use our
MeerKAT- or xeno-canto-pretrained animal2vec models to finetune on their
custom downstream tasks.

###  Finetuning

Strong regularization to tackle sparsity and a self-weighted loss to address
label imbalance are critical  For finetuning, we mostly follow the approach in
[74, 87, 38]. We discard the teacher and the 1D convolutional decoder network
(see figure 3) and freeze the weights of the feature extractor. We add a
classification head containing a sigmoid-activated linear projection layer and
perform a warm-up phase for $10\,000$ steps, where the student weights are
also frozen. The warm-up phase aligns the randomly initialized classification
head with the rest of the model. Prior to the classification head, we average
the embeddings from all transformer layers rather than just using the output
of the last layer; this has been shown to improve results in human speech
recognition [87].

After the warm-up phase, we unfreeze the student model and finetune the
classification head along with the student model for the remainder of the
training. To account for the label imbalance in MeerKAT, we use the focal
criterion [109] as opposed to cross entropy as loss function. Focal loss adds
a regularization term of the form $\left(1-p_{t}\right)^{\gamma}$ to the cross
entropy, where $p_{t}$ is the model’s estimated probability for observing a
particular class - the model’s likelihood. Setting $\gamma$ to a positive
number reduces the relative difference in loss between examples where the
model is confident (high likelihood) and where it is not (low likelihood).
This approach penalizes the effort to improve predictions with high likelihood
and forces the model to focus on the ones with low likelihood. As during
pretraining, we use between-classes-Learning [97] (BCL) with our modified
window length for augmenting the input audio, and we mask parts of the input
using the same stochastic masking strategy but with fewer masked spans,
depending on the finetuning setting (see table 5 in the appendix). For the
MeerKAT ($100\text{\,}\mathrm{\char 37\relax}$) setting, we use $p=0.0825$ and
$M=4$, which sets the mode of the mask distribution to
$22\text{\,}\mathrm{ms}$ while $60\text{\,}\mathrm{\char 37\relax}$ of all
timesteps are masked. Compared to the pretraining setting, we aim to maintain
a masking distribution with a mode duration below the shortest vocalizations
in the dataset but mask fewer of total timesteps. The idea of masking during
finetuning is not to create an artificial regression task, as during
pretraining, but to use masking as a regularization technique.

We benchmark our layer averaging strategy, the use of focal loss, BCL
augmentation, and masking strategy in an ablation study [110] in section
3.1.2. The finetuning parameters for each setting can be found in table 5 in
the appendix.

###  Calculating metrics

We report real-life relevant per-event metrics  While our model generates
likelihood estimates at a temporal resolution of $200\text{\,}\mathrm{Hz}$, we
evaluate performance using per-event scores since reporting metrics on a frame
level are biased with respect to event lengths. Longer events cover more
timesteps and need more predictions than shorter ones. This favors overly
positive or, likewise, disfavors overly negative models. We calculate event-
based scores where a single event is a single prediction, regardless of the
event length. This enables us to report how many calls were correctly
identified and how many were missed, rather than focusing on the likelihood of
each audio frame. We calculate this as follows:

  1. 1.

_Event boundary prediction_ : We slide a fixed-length average-pooling window
across the model’s likelihood output to predict event onsets and offsets
within a continuous audio stream. A fixed threshold is applied to binarize the
output, generating a step function representing our event boundary estimates.

  2. 2.

_Intersection-over-union (IOU) calculation_ : Using the IOU metric, we measure
the overlap between the ground truth event spans and our predictions.
Predicted spans without corresponding ground truth events are assigned an IOU
of zero.

  3. 3.

_Final likelihood assignment_ : If the IOU for a predicted event exceeds
$0.5$, the average model likelihood within the predicted span is used as the
final scalar likelihood. All reported metrics utilize these final likelihood
values.

  4. 4.

_Error identification_ : Ground truth events without predicted boundaries are
considered false negatives. Predicted spans lacking a ground truth
counterpart, or those with insufficient IOU, are considered false positives.

##  Results & Discussion

Figure 4:  Mask length distributions of the baseline (solid red line) and our
animal2vec model (solid yellow line) during pretraining. For comparison we
also show the distribution of the sample lengths (dashed teal line). Modeled
after figure 2 in the appendix in [74].

###  MeerKAT

We selected three classes to highlight the different challenges in
bioacoustics compared to human speech  We compare animal2vec to the
transformer baseline presented in Baevski et al. (2022) [38]. This baseline
introduced the pretraining scheme we have adapted to bioacoustics, and
achieved state-of-the-art results on the Librispeech speech recognition
corpus. It is optimized for human speech but is architecturally close to
animal2vec. We chose precision-recall (PR) curves to characterize model
performance with the MeerKAT dataset. A PR curve is constructed by plotting
precision values on the y-axis against recall values on the x-axis across
various likelihood thresholds. Precision describes how accurate the positive
predictions are (Out of all predictions, how many are correct), while Recall
describes how complete they are (How many events should have been labeled).
The likelihood threshold determines the likelihood above which an instance is
predicted as belonging to a class (whether a call was detected). By varying
this threshold, we can explore the trade-off between the model’s precision and
its ability to recall all positive instances.

We present PR curves for selected classes and for all three finetuning
settings with MeerKAT in figure 1 b) and for the remaining classes in the
appendix (see figures 10 to 19). Figure 1 b) provides results in four subplots
showing (i) the global micro average, (ii) the close call, (iii) the short-
note call, and (iv) the alarm call class. Each subplot holds four PR curves
that belong to (i) the transformer baseline (in gray) and (ii - iv) animal2vec
in the reduced ($1\text{\,}\mathrm{\char 37\relax}$, $25\text{\,}\mathrm{\char
37\relax}$), and full ($100\text{\,}\mathrm{\char 37\relax}$) training data
finetuning settings (in red, yellow, and teal, respectively).

While the global micro average provides a good overview of overall
performance, we selected the three vocalization classes to discuss relevant
use cases:

  1. 1.

_The easiest class_ : The close call class is the most abundant single
vocalization with a representative median length of $115\text{\,}\mathrm{ms}$
(compared to $105\text{\,}\mathrm{ms}$ in MeerKAT).

  2. 2.

_The shortest class_ : While it is the second most abundant vocalization
class, the short-note call class has the shortest median length of
$37\text{\,}\mathrm{ms}$. As such, it is an ideal candidate to assess the
model’s performance on very short signals. This is especially interesting as
it highlights conceptual differences between bioacoustic and human speech
datasets, where the typical phoneme length in the English language is $\approx
100\text{\,}\mathrm{ms}$ [111, 112].

  3. 3.

_The rare class_ : The alarm call class has a somewhat comparable duration
distribution to the close call class (_the easiest class_), see figure 1 a)
but is one of the rarest vocalization classes in MeerKAT, which makes it ideal
to assess a model’s ability to learn from underrepresented classes.

Furthermore, we use the AP [59] score to assess class-wise performance in the
main text in table 1 a). The AP score is a robust estimator for the area under
the PR curve [113] and is computed by averaging the interpolated precision
values at evenly spaced recall levels across the entire precision-recall
curve.

animal2vec is strong where the baseline is weak and very strong where the
baseline is ok  animal2vec consistently outperforms the transformer baseline
even in the $1\text{\,}\mathrm{\char 37\relax}$ setting (see global micro
average in figure 1 b) and AP scores in table 1 a)). The baseline achieves a
precision of $0.5$ for recall values below $0.2$ (Overall AP score is $0.30$),
whereas even the animal2vec $1\text{\,}\mathrm{\char 37\relax}$ setting never
falls below a precision of almost $0.7$ (at recall around $0.8$; Overall AP
score is $0.83$). The models trained using the $25\text{\,}\mathrm{\char
37\relax}$ and $100\text{\,}\mathrm{\char 37\relax}$ finetune splits
outperform the baseline and the $1\text{\,}\mathrm{\char 37\relax}$ result by
a wide margin (Overall AP scores of $0.88$ and $0.91$). Their AP scores and
precision-recall curves are comparable, indicating that, for the global micro
average, a saturation level may have been reached above which improvement is
not achievable by merely providing more labeled samples.

The results for the three selected classes in figure 1 b) show the transformer
baseline performing reasonably on the close call class (_the easiest class_ ;
AP score of $0.49$) but achieving low scores with the Short-note and alarm
call class (_the shortest and the rare class_ ; AP scores of $0.14$ and
$0.03$). animal2vec significantly outperforms this baseline, achieving AP
scores of $0.90$, $0.88$, and $0.57$ using $1\text{\,}\mathrm{\char 37\relax}$
of the data and $0.94$, $0.92$, and $0.80$ using $100\text{\,}\mathrm{\char
37\relax}$ of the data. As observed in the global average, results using
$25\text{\,}\mathrm{\char 37\relax}$ and $100\text{\,}\mathrm{\char 37\relax}$
of the labels are comparable, indicating a saturation effect, where further
improvements are not attainable through more labeling.

We observe this saturation effect for all but three of the four rarest classes
(Alarm, Aggressive, and lead calls), see table 1 a). The move call class is
the only rare class where animal2vec was able to achieve a comparable result
in the $25\text{\,}\mathrm{\char 37\relax}$ and $100\text{\,}\mathrm{\char
37\relax}$ setting. We attribute this to the move call class being somewhat
easier to predict, having a longer median length of $182\text{\,}\mathrm{ms}$
(the third longest of all vocalization classes), but being not so rare as the
comparably long lead call class, see figure 1 a).

The results for the miscellaneous classes (synch, beep, and eating) are
comparable to the vocalization classes, except for the synch signal class.
There, the transformer baseline performs almost on par with animal2vec,
achieving an AP score of $0.91$. This is expected as a synch signal is an
artificial human voice from a GPS clock saying loudly the current time, see
section 2.1.2. The baseline performs well on this class since it was derived
for human language [38]. However, even in this case, animal2vec
($1\text{\,}\mathrm{\char 37\relax}$) matches the baseline performance (AP
score of $0.89$), using only $80$ instances throughout finetuning, compared to
$7990$ in baseline finetuning.

Figure 5:  Cumulative frequency response (CFR) of the SincNet filters learned
by animal2vec after pretraining (solid yellow) and initialized with the Mel-
scale (solid red). For comparison, the $800\text{\,}\mathrm{Hz}$ cepstral-
smoothed linear predictive coding (LPC) spectrum of meerkat alarm calls [48]
(dashed grey) and the integrated spectral energies of the vocalization samples
in figure 2 b) (dashed teal) are shown. All lines are area-normalized.

####  Interpretability of animal2vec

As far as neural networks are concerned, animal2vec is not quite such a black
box  animal2vec’s decision-making can be understood in the spectral and
temporal domain. (i) Spectral importance can be deduced from the cumulative
frequency response of the learned sinc filters (much like the analysis in
[37]), and (ii) temporal importance can be inferred from the attention maps of
our transformer architecture. While there are many variants of attention
modules [16, 114, 115], a transformer typically uses the scaled matrix-matrix
dot product attention [16] as the dominant information extraction mechanism.
First, the matrix-matrix product between the input ($X\in\mathbb{R}^{L\times
d}$) and three learnable matrices ($\text{Q}_{\text{w}}$,
$\text{K}_{\text{w}}$, and $\text{V}_{\text{w}}$), where
$\\{\text{Q}_{\text{w}},\text{K}_{\text{w}},\text{V}_{\text{w}}\\}\in\mathbb{R}^{d\times
d}$ is calculated, where $L$ is the segment length of the input sequence and
$d$ is the embedding size of the model. Their output is then referred to as
Query ($\text{Q}=\text{X}\cdot\text{Q}_{\text{w}}$), Key
($\text{K}=\text{X}\text{K}_{\text{w}}$), and Value
($\text{V}=\text{X}\text{V}_{\text{w}}$), where
$\\{\text{Q},\text{K},\text{V}\\}\in\mathbb{R}^{L\times d}$. Then, the matrix-
matrix product between the query and the transposed key matrix is passed
through a softmax layer, and a final matrix-matrix product is calculated with
the value matrix. Concretely:

$\mathcal{H}\left(\text{Q},\text{K},\text{V}\right)=\underbrace{\text{softmax}\left(\text{Q}\text{K}^{\top}n^{-\frac{1}{2}}\right)}_{\text{Attention}}\text{V}=\text{A}\text{V}$
(1)

where
$\mathcal{H}\left(\text{Q},\text{K},\text{V}\right)\in\mathbb{R}^{L\times d}$
and $n$ is a scalar normalization constant [16]. The output of the softmax
layer is referred to as attention ($\text{A}\in\mathbb{R}^{L\times L}$), as it
acts as a normalized weighting matrix for the value matrix. So, the whole
attention module learns two things: (i) a softmax-normalized weighting matrix
and (ii) a projection of the input. The first is called attention matrix A,
and the second is the Value matrix. Therefore, the attention matrix (A)
provides relevance in terms of how the projection of the input (V) should be
passed to the next layer. In practice, a single transformer layer calculates
equation 1 M times $\left(\text{Q}_{\text{w}}\text{,
}\text{K}_{\text{w}}\text{, and }\text{V}_{\text{w}}\text{, with
}1\leq\text{w}\leq\text{M}\right)$, where M is referred to as the number of
heads (this is what is called Multi-head attention in figure 3), and a full
transformer model has N layers (see the N$\times$ on the left in figure 3).
Weights of Q, K, and V are not shared between heads or layers, so a full
transformer model has M times N attention matrices A.

For interpreting animal2vec’s decision-making in the temporal domain, we show
the global average of all A matrices as a heatmap. Concretely, we average
across $256$ attention maps ($16$ heads times $16$ layers; see table 5 in the
appendix).

Interpreting attention maps is a much-debated topic [116, 117, 118, 119, 120].
The consensus is that attention maps often provide excellent and intuitive
explanations but sometimes are entirely misleading. Therefore, scientists
still need to interpret them cautiously [119]. We follow the nomenclature in
[114] and interpret the attention scores not as explanations but as
importance. Furthermore, we provide scripts in our repository to extract all
attention maps from all heads and layers and the parameters of the sinc
module.

Figure 6:  Globally averaged attention map of a four second segment showing
four move calls. Blue stretches surrounded by dashed white lines correspond to
onset/offset predictions of animal2vec. An attention map shows the
_importance_ of every input frame with respect to every other frame. For
predicting, animal2vec attends to the immediate past and future of an event,
as well as to a previous instance in the case of the noisy Move vocalization.

animal2vec’s frequency response broadly aligns with the frequencies found in
meerkat calls  First, we discuss the spectral interpretability of animal2vec
via the cumulative frequency response (CFR) of the learned sinc filters (see
figure 5).

Both reference curves, the dashed lines in figure 5, show good agreement with
each other. The most notable features of the data from [48] are the bimodal
structures with nodes at around $1\text{\,}\mathrm{kHz}$ and
$2.2\text{\,}\mathrm{kHz}$. This is mostly mirrored by our reference data from
figure 2, where the spectral distribution shows a trimodal distribution, with
peaks at around $600\text{\,}\mathrm{Hz}$, $1.1\text{\,}\mathrm{kHz}$, and
$2\text{\,}\mathrm{kHz}$. Both reference lines decay fast to almost zero
amplitude between $\approx$2.53.0\text{\,}\mathrm{kHz}$$.

The CFR of animal2vec’s learned sinc filters show the strongest response to
signals at $355\text{\,}\mathrm{Hz}$, an increase by about
$115\text{\,}\mathrm{Hz}$ with respect to the filters initialized using the
Mel scale (maximum at $240\text{\,}\mathrm{Hz}$). This increase is expected as
the Mel scale is derived for human vocalizations which are overall lower in
frequency compared to meerkat vocalizations [48, 44]. For frequencies above
$500\text{\,}\mathrm{Hz}$ animal2vec’s CFR largely follows the trimodal
structure of the MeerKAT reference data, even mirroring the decline for
frequencies between $\approx$2.53.0\text{\,}\mathrm{kHz}$$, highlighting that
animal2vec learned the relevant frequencies to identify the MeerKAT
vocalizations. Since the feature extraction module (the yellow stack in figure
3, including the sinc module) is frozen throughout subsequent finetuning, the
presented CFR is learned in the fully self-supervised pretraining, without any
labels used.

Table 2:  Ablation studies for animal2vec compared to the transformer baseline [38], evaluated using the MeerKAT ($100\text{\,}\mathrm{\char 37\relax}$) setting. The microaveraged average precision scores (AP) [59] are obtained after finetuning on the MeerKAT dataset.  |  Average precision score [59]
---|---
Transformer baseline [38] | $0.30$
Changes to pretraining as described in [38]:
|  $\bullet$ Change feature extractor layout |
|  |  $\llcorner$ $f_{\text{sr, eff}}$ from $50200\text{\,}\mathrm{Hz}$ | $+\,0.16$
|  |  $\llcorner$ SincNet-style filterbanks [37] | $+\,0.03$
|  |  $\llcorner$ PSwish instead of LeakyRely | $+\,0.02$
|  $\bullet$ Change pretraining masking strategy |
|  |  $\llcorner$ Prob. $p$ from $0.06500.1500$ | $+\,0.05$
|  |  $\llcorner$ Length $M$ from $102$ | $+\,0.11$
|  $\bullet$ Between-classes learning [97] (No Targets) | $+\,0.04$
Changes to finetuning as described in [74, 38]:
|  $\bullet$ Average over all transformer layers | $+\,0.04$
|  $\bullet$ Focal loss [109] instead of cross entropy | $+\,0.05$
|  $\bullet$ Between-classes learning [97] | $+\,0.03$
|  $\bullet$ Change finetuning masking strategy |
|  |  $\llcorner$ Prob. $p$ from $0.06500.0825$ | $+\,0.03$
|  |  $\llcorner$ Length $M$ from $104$ | $+\,0.05$
animal2vec | $0.91$

animal2vec attends to the future, the past, and repeating patterns to predict
meerkat calls  We discuss temporal interpretability by showing globally
averaged attention maps [114] in figure 6. As expected, the dominant feature
is the diagonal, which is the importance given by the model to use the current
audio frame to make predictions about the current audio frame. Moreover, three
observations are striking in figure 6. (i) For the three move calls that are
not covered in regions of spectrally broad noise (first, second, and fourth
from the top), animal2vec attends to the future and the past of the current
audio frame (left and right from the diagonal) for the whole duration of the
call. (ii) For the one noisy move call (third from the top), animal2vec
attends to the previous move call while predicting the noisy one, and (iii)
animal2vec attends to almost all frames in the input sequence for most of the
ultra-short, spectrally broad noise patterns.

The most challenging prediction in this segment is the one on the noisy move
call. animal2vec did not miss it and achieved an IoU of $0.95$ with the ground
truth, meaning that the labeling expert estimated onset and offset almost
exactly as animal2vec did. Interestingly, animal2vec stops attending to the
previous move call when it stops predicting a move call event, although this
area is fully buried under noise and other background signals. We hypothesize
that the repetition of vocalizations within a sequence can be a proxy for
predicting them. This is corroborated by the somewhat extreme example in
figure 10 in the appendix. There, a repetition of $15$ alarm calls, closely
following each other, is shown. animal2vec attends to almost all previous and
future calls while predicting the current one. This is also true for the empty
stretches in figure 10, where no call was recorded. animal2vec attends to past
and future empty stretches when predicting them as negative.

Behavioral research confirms that meerkat vocalizations often appear in
repetition patterns within and among conspecific individuals. [121, 122].
Therefore, attending to previous or following repetitions is a proxy for
producing predictions.

Table 3:  Results on the evaluation split for the holdout generalizability study. The average precision scores (AP) [59] for the baseline and animal2vec. Both models use $100\text{\,}\mathrm{\char 37\relax}$ of the training samples for finetuning. The strongest result per class is in bold letters. The two bottom rows show the micro- and macroaverage across all classes except _Focal_. Arrows and values in a smaller font next to the results show the change of score with respect to table 1.  | Average precision scores [59]
---|---
| Transformer baseline[38] | animal2vec ($100\text{\,}\mathrm{\char 37\relax}$)
Focal | 0.54 $\pmb{\downarrow}$$\,0.05$ | 0.93 $\pmb{\downarrow}$$\,0.01$
Vocalizations
Close call | 0.47 $\pmb{\downarrow}$$\,0.02$ | 0.93 $\pmb{\updownarrow}$$\,0.01$
Short-note call | 0.14 $\pmb{\updownarrow}$$\,0.00$ | 0.90 $\pmb{\downarrow}$$\,0.01$
Social call | 0.27 $\pmb{\downarrow}$$\,0.03$ | 0.82 $\pmb{\downarrow}$$\,0.02$
Other call | 0.04 $\pmb{\downarrow}$$\,0.03$ | 0.46 $\pmb{\downarrow}$$\,0.04$
Alarm call | 0.03 $\pmb{\updownarrow}$$\,0.00$ | 0.80 $\pmb{\updownarrow}$$\,0.00$
Aggressive call | 0.10 $\pmb{\uparrow}$$\,0.01$ | 0.69 $\pmb{\downarrow}$$\,0.02$
Move call | 0.08 $\pmb{\downarrow}$$\,0.01$ | 0.61 $\pmb{\updownarrow}$$\,0.00$
Lead call | 0.01 $\pmb{\updownarrow}$$\,0.00$ | 0.51 $\pmb{\uparrow}$$\,0.01$
Miscellaneous
Synch signal | 0.89 $\pmb{\downarrow}$$\,0.02$ | 0.98 $\pmb{\updownarrow}$$\,0.00$
Eating | 0.11 $\pmb{\downarrow}$$\,0.01$ | 0.86 $\pmb{\downarrow}$$\,0.01$
Beep signal | 0.22 $\pmb{\downarrow}$$\,0.04$ | 0.80 $\pmb{\updownarrow}$$\,0.00$
Macroaverage | 0.24 $\pmb{\downarrow}$$\,0.02$ | 0.76 $\pmb{\downarrow}$$\,0.02$
Microaverage | 0.28 $\pmb{\downarrow}$$\,0.02$ | 0.90 $\pmb{\downarrow}$$\,0.01$

####  Ablation and generalizability studies

animal2vec is not more than the sum of its parts, but they are thought-through
and plenty  Here, we provide an ablation study [110] on every addition we did
compared to the baseline presented in [38]. We conducted $13$ full animal2vec
pretrainings along with their $5$-fold cross-validated finetunings
($100\text{\,}\mathrm{\char 37\relax}$), wherein each ablation, we added a
single component present in animal2vec but missing in the baseline. Table 2
holds the microaverage AP scores for these ablations. The table is organized
in the same order in which the components were added.

The changes that produced the most substantial increase in AP are the
alteration of the 1D convolutional layer stack to produce an effective
sampling rate $f_{\text{sr, eff}}$ of $200\text{\,}\mathrm{Hz}$ instead of
$50\text{\,}\mathrm{Hz}$ (an increase of $0.16$), and the changes to the
pretraining masking strategy. Changing the masking length $M$ from $102$
yielded an increase of $0.11$ AP, and changing the masking probability $p$
from $0.06500.1500$ increased AP by about $0.05$.

Importantly, while the change in $f_{\text{sr, eff}}$ produced a $16$-fold
increase in computational complexity, as four times higher $f_{\text{sr,
eff}}$, results in a $16$ times higher complexity in transformers [16], the
changes to the masking strategy reduced the complexity $66$-fold (the baseline
masks $49\text{\,}\mathrm{\char 37\relax}$, whereas we mask
$93.7\text{\,}\mathrm{\char 37\relax}$). Therefore, discussing just the change
in $f_{\text{sr, eff}}$ and pretraining masking strategy, we increase AP by
$0.32$ and reduce the computational cost to around $24\text{\,}\mathrm{\char
37\relax}$ of the baseline.

animal2vec generalizes well  For analyzing the generalizability of animal2vec,
we derived a leave-out generalizability study, where a number of files in
MeerKAT were excluded from pretraining and finetuning but used for evaluation,
see section 2.1.2. Table 3 holds the class-wise AP scores of this experiment.
We analyze them in terms of change with respect to the results from 1.
Observations that stand out are that (i) the baseline generalizes worse, which
can be seen from the global averages, and (ii) performance on the other call
class is strongly decreased in both models.

  1. (i)

The macroaverage in the baseline is reduced by about $8\text{\,}\mathrm{\char
37\relax}$ (a drop from $0.240.21$), and the microaverage by about
$7\text{\,}\mathrm{\char 37\relax}$ (a drop from $0.300.28$), compared to
$3\text{\,}\mathrm{\char 37\relax}$ (a drop from $0.780.76$) and
$1\text{\,}\mathrm{\char 37\relax}$ (a drop from $0.910.90$) in animal2vec.

  2. (ii)

The other call class is a collective class for vocalizations that did not fit
well into any of the other classes, see section 2.1.2, as such, performance
with this class was always significantly lower compared to classes with sample
counts on the same order (like the social call class in the
$100\text{\,}\mathrm{\char 37\relax}$ setting, or the short-note call class in
the $25\text{\,}\mathrm{\char 37\relax}$ setting), or having similar median
duration (like the aggressive call class that only contains half the amount of
samples). As such, performance is expected to deteriorate stronger than in
other classes. We observe a relative decrease by about
$43\text{\,}\mathrm{\char 37\relax}$ (a drop from $0.070.04$) in the baseline
and about $8\text{\,}\mathrm{\char 37\relax}$ (a drop from $0.500.46$) for
animal2vec - the largest decrease in any vocalization class for both models.

However, the overall performance of animal2vec remains competitive with data
that were included in the unlabeled pretraining, where the observed
$1\text{\,}\mathrm{\char 37\relax}$ decrease in microaverage performance could
also be explained by statistical fluctuations (The standard deviation of the
microaverage in table 1 is on the same order).

###  NIPS4Bplus

We provide Precision, Recall, and F1 scores of animal2vec to compare to
results reported in [63, 62] in table 4. While Morfi et al. (2018) [62]
reported results from an event detector calculated on a timestep level, Bravo
Sanchez et al. (2021) [63] report class-prediction scores for pre-segmented
sequences only containing the event to be classified.

We compare animal2vec class-wise event-based predictions, see section 2.5, to
the results of Bravo Sanchez et al. (2021) [63] in table 4 a). Event-based
predictions are arguably more challenging than pre-segmented sequences, as
onsets, offsets, and timesteps without a signal also have to be predicted.

Furthermore, to enable a comparison with Morfi et al. (2018) [62], we
calculate event detection scores from the likelihood output of animal2vec by
treating any prediction for any class for a given timestep as an event
prediction, which can be found in table 4 b). Consequently, we create many
false positives with this approach, which, presumably, reduces the reported
precision score of animal2vec.

With this in mind, the increase in F1 of about $0.06$ compared to Densenet121
(the strongest model in [63]), and $0.08$ compared to the WHEN (MMM) model in
[62], sets a new baseline on the NIPS4BPlus dataset. The increase is
noteworthy compared to the original SincNet model, which utilizes a similar
audio frontend to process raw audio. We would like to highlight that our
intention with this benchmark is to connect our work to published research and
to show that small-scale datasets can be used for finetuning if animal2vec is
pretrained with data from a comparable domain. If pretraining is not an
option, animal2vec is not recommended since models like the very small SincNet
($2.6$M parameters) provide a reasonable trade-off between model
interpretability, computational complexity, and classification performance.

Table 4:  Microaverage classification results on the NIPS4Bplus dataset [12]. The metrics for the models trained on pre-segmented sequences are taken from [63] and the one for binary timestep prediction is the best result from [62], called _WHEN model using MMM loss_. In addition, we provide results for animal2vec’s Onset/Offset/Overlap predictions, using the same methodology described in section 2.5.  Model | Precision | Recall | F1
---|---|---|---
a)
Predictions on pre-segmented sequences
Densenet121 | 0.76 | 0.75 | 0.76
Resnet50 | 0.76 | 0.74 | 0.75
SincNet | 0.75 | 0.73 | 0.74
VGG16 | 0.74 | 0.73 | 0.74
Waveform + CNN | 0.72 | 0.71 | 0.71
Onset/Offset/Overlap predictions
animal2vec | 0.81 | 0.88 | 0.82
b)
Binary predictions on timesteps
WHEN (MMM) | - | - | 0.74
animal2vec | 0.79 | 0.86 | 0.82

##  Conclusions

In this work, we present animal2vec and MeerKAT and make them openly
available. animal2vec is a self-supervised pretraining and finetuning scheme
that utilizes a custom transformer-based model tailored for bioacoustics, and
MeerKAT is the largest publicly-available dataset on non-human terrestrial
mammals ever released. The coming age in AI will primarily focus on
pretraining/finetuning training paradigms in combination with large-scale
generalist models. This transition has so far been challenging for the
bioacoustic research community. First, bioacoustics lacks datasets that are
large enough to train transformer models with hundreds of millions of
parameters and have enough fine-grain ground truth labels to enable effective
finetuning. Second, bioacoustics has so far lacked a pretraining/finetuning
training paradigm that takes advantage of the novelties in bioacoustics, like
the high sparsity, noise corruption, and having raw waveforms as primary data
format. This in combination with the focus on ultra-short events of interest
make bioacoustics in many regards a more demanding field than human speech
recognition.

animal2vec established a new baseline on the MeerKAT dataset, where we showed
that it outperforms a comparable transformer architecture devised for human
speech by a large margin and that animal2vec is a strong few-shot learner.
Using only $1\text{\,}\mathrm{\char 37\relax}$ of the finetune data,
animal2vec surpassed the transformer baseline by more than doubling the
baseline AP score ($0.30$ for the baseline, compared to $0.76$ for animal2vec
in the $1\text{\,}\mathrm{\char 37\relax}$ setting). We further showed that
our proposed model in animal2vec converges towards a saturation level in
performance when using $25\text{\,}\mathrm{\char 37\relax}$ or more of the
available labeled data, indicating that further improvements might not be
gained using more ground truth but new conceptual ideas.

To enable a comparison with already published datasets and results, we
evaluated a xeno-canto pretrained animal2vec using the small-scale NIPS4Bplus
dataset [61], where animal2vec sets a new baseline outperforming currently
published results.

The MeerKAT dataset we provide here is large-scale with over
$1000\text{\,}\mathrm{h}$ of audio recordings, but it is still portable with
just under $59\text{\,}\mathrm{GB}$ ($61\text{\,}\mathrm{GB}$ including the
label files) in size. It has $184\text{\,}\mathrm{h}$ of millisecond-resolved
strong labels of meerkat vocalizations and artificial audio sources buried in
a challenging amount of ultra-short and spectrally broad noise patterns. This
enables not only analyzing the event detection performance of a model but also
its resilience concerning sparsity and noise.

We designed animal2vec as a modular framework whose components can be used as
needed depending on the desired application. Our novel feature extraction
module can be used as a frontend for other models, and our transformer model
can be used with other frontends. Both can be used with other pretraining or
finetuning approaches on different datasets or jointly trained with MeerKAT.

The immediate future for animal2vec is (i) to incorporate more data from more
species (insects, birds, marine, and terrestrial animals), recording
environments (marine, avian), using a more diverse set of recorders (passive
acoustic monitoring, different portable recorders using different microphones,
audio from video traps, citizen science data) where challenges like the large
variability in different sampling rates need to be solved, and (ii) to include
more data modalities like accelerometer and GPS data from next-gen collar tags
[2], where animal2vec needs to be enabled to make use of such auxiliary data
streams but not to decrease in performance when they are missing.

Ultimately, our vision for animal2vec and MeerKAT is for them to be the first
stepping stone towards a next-gen reference work, where, in the future, we
envision a foundational-level pretrained animal2vec model that researchers can
directly use for finetuning on their data without the need for large-scale GPU
facilities.

There is plenty of work ahead of us before we get there. However, it is our
great hope that by making everything accessible, fully open, and portable, the
research community will strive with us towards this, we believe, shared goal
using animal2vec and MeerKAT.

## References

  * [1] Bradbury, J. W. & Vehrencamp, S. L.  _Principles of animal communication_ (Sinauer Associates Sunderland, MA, 1998).
  * [2] Demartsev, V. _et al._ Signalling in groups: New tools for the integration of animal communication and collective movement.  _Methods Ecol. Evol._ (2022).
  * [3] Penar, W., Magiera, A. & Klocek, C.  Applications of bioacoustics in animal ecology.  _Ecol. Complex._ 43, 100847 (2020).
  * [4] Fleishman, E. _et al._ Ecological inferences about marine mammals from passive acoustic data.  _Biol. Rev. Camb. Philos. Soc._ 98, 1633–1647 (2023).
  * [5] Laiolo, P.  The emerging significance of bioacoustics in animal species conservation.  _Biol. Conserv._ 143, 1635–1645 (2010).
  * [6] Mcloughlin, M. P., Stewart, R. & McElligott, A. G.  Automated bioacoustics: methods in ecology and conservation and their potential for animal welfare monitoring.  _J. R. Soc. Interface_ 16, 20190225 (2019).
  * [7] Sugai, L. S. M., Silva, T. S. F., Ribeiro, J. W., Jr & Llusia, D.  Terrestrial passive acoustic monitoring: Review and perspectives.  _Bioscience_ 69, 15–25 (2019).
  * [8] Lindseth, A. V. & Lobel, P. S.  Underwater soundscape monitoring and fish bioacoustics: A review.  _Fishes_ 3, 36–30 (2018).
  * [9] Madhusudhana, S. _et al._ Choosing equipment for animal bioacoustic research.  _Exploring Animal Behavior Through Sound: Volume_ 37 (2022).
  * [10] Allen, A. N. _et al._ A convolutional neural network for automated detection of humpback whale song in a diverse, long-term passive acoustic dataset.  _Front. Mar. Sci._ 8 (2021).
  * [11] Lostanlen, V., Salamon, J., Farnsworth, A., Kelling, S. & Bello, J. P.  Birdvox-full-night: A dataset and benchmark for avian flight call detection.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 266–270 (IEEE, 2018).
  * [12] Morfi, V., Bas, Y., Pamuła, H., Glotin, H. & Stowell, D.  NIPS4Bplus: a richly annotated birdsong audio dataset.  _PeerJ Comput. Sci._ 5, e223 (2019).
  * [13] Ness, S., Symonds, H., Spong, P. & Tzanetakis, G.  The orchive: Data mining a massive bioacoustic archive.  _Preprint athttps://arxiv.org/abs/1307.0589_ (2013).
  * [14] Wall, C. C. _et al._ The next wave of passive acoustic data management: How centralized access can enhance science.  _Front. Mar. Sci._ 8, 703682 (2021).
  * [15] LeCun, Y., Bengio, Y. & Hinton, G.  Deep learning.  _Nature_ 521, 436–444 (2015).
  * [16] Vaswani, A. _et al._ Transformer: Attention is all you need.  _Advances in Neural Information Processing Systems 30_ 5998–6008 (2017).
  * [17] Lin, T., Wang, Y., Liu, X. & Qiu, X.  A survey of transformers.  _AI Open_ 3, 111–132 (2022).
  * [18] Stowell, D.  Computational bioacoustics with deep learning: a review and roadmap.  _PeerJ_ 10, e13152 (2022).
  * [19] Wyse, L.  Audio spectrogram representations for processing with convolutional neural networks.  In _Proceedings of the First International Conference on Deep Learning and Music_ , 37–41 (2017).
  * [20] Dosovitskiy, A. _et al._ An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ (2020).
  * [21] Khan, S. _et al._ Transformers in vision: A survey.  _ACM Comput. Surv._ (2021).
  * [22] He, K., Fan, H., Wu, Y., Xie, S. & Girshick, R.  Momentum contrast for unsupervised visual representation learning.  _Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ 9726–9735 (2020).
  * [23] Chen, T., Kornblith, S., Norouzi, M. & Hinton, G.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , 1597–1607 (PMLR, 2020).
  * [24] Liu, X. _et al._ Self-supervised learning: Generative or contrastive.  _IEEE Trans. Knowl. Data Eng._ 1–1 (2021).
  * [25] Gemmeke, J. F. _et al._ Audio set: An ontology and human-labeled dataset for audio events.  In _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , 776–780 (IEEE, 2017).
  * [26] Zhou, Z.-H.  A brief introduction to weakly supervised learning.  _Natl. Sci. Rev._ 5, 44–53 (2018).
  * [27] Panayotov, V., Chen, G., Povey, D. & Khudanpur, S.  Librispeech: an ASR corpus based on public domain audio books.  In _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , 5206–5210 (IEEE, 2015).
  * [28] Jaiswal, A., Babu, A. R., Zadeh, M. Z., Banerjee, D. & Makedon, F.  A survey on contrastive self-supervised learning.  _Technologies (Basel)_ 9, 2 (2020).
  * [29] Chung, Y.-A. _et al._ W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training.  In _2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , 244–250 (IEEE, 2021).
  * [30] Gong, Y., Chung, Y.-A. & Glass, J.  AST: Audio Spectrogram Transformer.  In _Proc. Interspeech 2021_ , 571–575 (2021).
  * [31] Wyatt, S. _et al._ Environmental sound classification with tiny transformers in noisy edge environments.  In _2021 IEEE 7th World Forum on Internet of Things (WF-IoT)_ , 309–314 (IEEE, 2021).
  * [32] Wolters, P., Sizemore, L., Daw, C., Hutchinson, B. & Phillips, L.  Proposal-based few-shot sound event detection for speech and environmental sounds with perceivers.  _Preprint athttps://arxiv.org/abs/2107.13616_ (2021).
  * [33] You, L., Coyotl, E. P., Gunturu, S. & Van Segbroeck, M.  Transformer-based bioacoustic sound event detection on few-shot learning tasks.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 1–5 (IEEE, 2023).
  * [34] Robinson, D., Robinson, A. & Akrapongpisak, L.  Transferable models for bioacoustics with human language supervision.  In _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 1316–1320 (IEEE, 2024).
  * [35] Gu, N. _et al._ Positive transfer of the whisper speech transformer to human and animal voice activity detection.  In _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 7505–7509 (IEEE, 2024).
  * [36] Deng, J. _et al._ Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255 (IEEE, 2009).
  * [37] Ravanelli, M. & Bengio, Y.  Speaker recognition from raw waveform with sincnet.  In _2018 IEEE spoken language technology workshop (SLT)_ , 1021–1028 (IEEE, 2018).
  * [38] Baevski, A., Babu, A., Hsu, W.-N. & Auli, M.  Efficient self-supervised learning with contextualized target representations for vision, speech and language.  In _International Conference on Machine Learning_ , 1416–1429 (PMLR, 2023).
  * [39] The kalahari research centre KRC.  https://kalahariresearchcentre.org.  Accessed: 2024-04-25.
  * [40] The MIT license.  https://opensource.org/license/mit.  Accessed: 2024-04-25.
  * [41] Official GitHub repository for animal2vec.  https://github.com/livingingroups/animal2vec.  Accessed: 2024-04-25.
  * [42] Creative Commons Attribution-NonCommercial 4.0 International.  https://creativecommons.org/licenses/by-nc/4.0.  Accessed: 2024-04-25.
  * [43] Schäfer-Zimmermann, J. C. _et al._ MeerKAT: Meerkat Kalahari Audio Transcripts (2024).  URL https://doi.org/10.17617/3.0J0DYB.
  * [44] Manser, M. B.  _The evolution of auditory communication in suricates, Suricata suricatta_.  Ph.D. thesis, University of Cambridge (1998).
  * [45] Manser, M. B., Jansen, D. A. W. A., Graw, B. & le Roux, A.  Vocal complexity in meerkats and other mongoose species.  _Adv. Stud. Behav._ 46, 281–310 (2014).
  * [46] Resample method of PyTorch 2.2.0.  https://pytorch.org/audio/2.2.0/generated/torchaudio.transforms.Resample.html.  Accessed: 2024-04-25.
  * [47] Kaiser, J. F.  Digital filters.  Systems Analysis by Digital Computer, chap. 7, 218–285 (John Wiley and Sons, New York, 1966).
  * [48] Townsend, S. W., Charlton, B. D. & Manser, M. B.  Acoustic cues to identity and predator context in meerkat barks.  _Anim. Behav._ 94, 143–149 (2014).
  * [49] Townsend, S. W., Hollén, L. I. & Manser, M. B.  Meerkat close calls encode group-specific signatures, but receivers fail to discriminate.  _Anim. Behav._ 80, 133–138 (2010).
  * [50] Collier, K., Townsend, S. W. & Manser, M. B.  Call concatenation in wild meerkats.  _Anim. Behav._ 134, 257–269 (2017).
  * [51] Demartsev, V., Strandburg-Peshkin, A., Ruffner, M. & Manser, M.  Vocal turn-taking in meerkat group calling sessions.  _Curr. Biol._ 28, 3661–3666.e3 (2018).
  * [52] Manser, M. B.  The acoustic structure of suricates’ alarm calls varies with predator type and the level of response urgency.  _Proc. Biol. Sci._ 268, 2315–2324 (2001).
  * [53] Bousquet, C. A. H., Sumpter, D. J. T. & Manser, M. B.  Moving calls: a vocal mechanism underlying quorum decisions in cohesive groups.  _Proc. Biol. Sci._ 278, 1482–1488 (2011).
  * [54] Demartsev, V., Thomas, M., Manser, M. B. & Strandburg-Peshkin, A.  Mapping vocal interactions in space and time differentiates signal broadcast vs signal exchange in meerkat groups.  _Phil. Trans. R. Soc. B_ (2024).
  * [55] The HDF Group.  Hierarchical Data Format, version 5.  URL https://github.com/HDFGroup/hdf5.
  * [56] Hallgren, K. A.  Computing inter-rater reliability for observational data: an overview and tutorial.  _Tutorials in quantitative methods for psychology_ 8, 23 (2012).
  * [57] He, H. & Garcia, E. A.  Learning from imbalanced data.  _IEEE Trans. Knowl. Data Eng._ 21, 1263–1284 (2009).
  * [58] Chawla, N. V., Bowyer, K. W., Hall, L. O. & Kegelmeyer, W. P.  SMOTE: Synthetic minority over-sampling technique.  _J. Artif. Intell. Res._ 16, 321–357 (2002).
  * [59] Zhu, M.  Recall, precision and average precision.  _Department of Statistics and Actuarial Science, University of Waterloo, Waterloo_ 2, 6 (2004).
  * [60] Sechidis, K., Tsoumakas, G. & Vlahavas, I.  On the stratification of multi-label data.  In _Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011, Proceedings, Part III 22_ , 145–158 (Springer, 2011).
  * [61] Morfi, V., Stowell, D. & Pamula, H.  NIPS4Bplus: Transcriptions of NIPS4B 2013 Bird Challenge Training Dataset (2018).  URL https://doi.org/10.6084/m9.figshare.6798548.
  * [62] Morfi, V. & Stowell, D.  Data-efficient weakly supervised learning for low-resource audio event detection using deep learning.  In _Proceedings of the detection and classification of acoustic scenes and events 2018 workshop (DCASE)_ , Proceedings of the detection and classification of acoustic scenes and events 2018 workshop, 123–127 (2018).
  * [63] Bravo Sanchez, F. J., Hossain, M. R., English, N. B. & Moore, S. T.  Bioacoustic classification of avian calls from raw sound waveforms with an open-source deep learning architecture.  _Sci. Rep._ 11, 15733 (2021).
  * [64] Goller, F. & Riede, T.  Integrative physiology of fundamental frequency control in birds.  _J. Physiol. Paris_ 107, 230–242 (2013).
  * [65] Lee, S., Lee, S. & Song, B. C.  Improving vision transformers to learn small-size dataset from scratch.  _IEEE Access_ 10, 123212–123224 (2022).
  * [66] Howard, A., Klinck, H., Dane, S., Kahl, S. & Denton, T.  Cornell birdcall identification (2020).  URL https://kaggle.com/competitions/birdsong-recognition.  Accessed: 2023-05-24.
  * [67] Xeno-Canto Bird Recordings Extended (A-M).  https://www.kaggle.com/datasets/rohanrao/xeno-canto-bird-recordings-extended-a-m/versions/11.  Accessed: 2024-04-25.
  * [68] Xeno-Canto Bird Recordings Extended (N-Z).  https://www.kaggle.com/datasets/rohanrao/xeno-canto-bird-recordings-extended-n-z/versions/11.  Accessed: 2024-04-25.
  * [69] Longpre, S. _et al._ A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity.  _Preprint athttps://arxiv.org/abs/2305.13169_ (2023).
  * [70] Chen, H.-Y., Tu, C.-H., Li, Z.-H., Shen, H. & Chao, W.-L.  On the importance and applicability of pre-training for federated learning.  _Int Conf Learn Represent_ (2022).  eprint 2206.11488.
  * [71] Robinson, J. D. _et al._ Can contrastive learning avoid shortcut solutions?  Conference on Neural Information Processing Systems (2021).
  * [72] Tomasev, N. _et al._ Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels on imagenet?  In _First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022_ (2022).
  * [73] Chen, T., Kornblith, S., Swersky, K., Norouzi, M. & Hinton, G. E.  Big self-supervised models are strong semi-supervised learners.  vol. 33, 22243–22255 (2020).
  * [74] Baevski, A. _et al._ Data2vec: A general framework for self-supervised learning in speech, vision and language.  In _International Conference on Machine Learning_ , 1298–1312 (PMLR, 2022).
  * [75] Saeed, A., Grangier, D. & Zeghidour, N.  Contrastive learning of general-purpose audio representations.  In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 3875–3879 (2021).
  * [76] Brown, T. _et al._ Language models are few-shot learners.  _Advances in neural information processing systems_ 33, 1877–1901 (2020).
  * [77] Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K.  BERT: Pre-training of deep bidirectional transformers for language understanding.  _NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference_ 1, 4171–4186 (2018).
  * [78] Liu, Y. _et al._ Roberta: A robustly optimized bert pretraining approach.  _Preprint athttps://arxiv.org/abs/1907.11692_ (2019).
  * [79] Hsu, W.-N. _et al._ HuBERT: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE ACM Trans. Audio Speech Lang. Process._ 29, 3451–3460 (2021).
  * [80] Rethmeier, N. & Augenstein, I.  A primer on contrastive pretraining in language processing: Methods, lessons learned, and perspectives.  _ACM Computing Surveys_ 55, 1–17 (2023).
  * [81] Lin, C.-C., Jaech, A., Li, X., Gormley, M. R. & Eisner, J.  Limitations of autoregressive models and their alternatives.  In Toutanova, K. _et al._ (eds.) _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 5147–5173 (Association for Computational Linguistics, Online, 2021).
  * [82] Zhu, D., Hedderich, M. A., Zhai, F., Adelani, D. & Klakow, D.  Is bert robust to label noise? a study on learning with noisy labels in text classification.  In _Proceedings of the Third Workshop on Insights from Negative Results in NLP_ , 62–67 (2022).
  * [83] Hinton, G., Vinyals, O. & Dean, J.  Distilling the knowledge in a neural network.  _Preprint athttps://arxiv.org/abs/1503.02531_ (2015).
  * [84] Tarvainen, A. & Valpola, H.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  _Advances in neural information processing systems_ 30 (2017).
  * [85] Grill, J.-B. _et al._ Bootstrap your own latent-a new approach to self-supervised learning.  _Adv. Neural Inf. Process. Syst._ 33, 21271–21284 (2020).
  * [86] Caron, M. _et al._ Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 9650–9660 (2021).
  * [87] Baevski, A. _et al._ data2vec: A general framework for self-supervised learning in speech, vision and language.  In Chaudhuri, K. _et al._ (eds.) _Proceedings of the 39th International Conference on Machine Learning_ , vol. 162 of _Proceedings of Machine Learning Research_ , 1298–1312 (PMLR, 2022).
  * [88] Song, H., Kim, M., Park, D., Shin, Y. & Lee, J.-G.  Learning from noisy labels with deep neural networks: A survey.  _IEEE Trans. Neural Netw. Learn. Syst._ 34, 8135–8153 (2023).
  * [89] Oord, A. v. d., Li, Y. & Vinyals, O.  Representation learning with contrastive predictive coding.  _Preprint athttps://arxiv.org/abs/1807.03748_ (2018).
  * [90] Chen, X., Fan, H., Girshick, R. & He, K.  Improved baselines with momentum contrastive learning.  _Preprint athttps://arxiv.org/abs/2003.04297_ (2020).
  * [91] Lasseck, M.  Audio-based bird species identification with deep convolutional neural networks.  _CLEF (working notes)_ 2125 (2018).
  * [92] Li, L. _et al._ Automated classification of Tursiops aduncus whistles based on a depth-wise separable convolutional neural network and data augmentation.  _J. Acoust. Soc. Am._ 150, 3861 (2021).
  * [93] Padovese, B., Frazao, F., Kirsebom, O. S. & Matwin, S.  Data augmentation for the classification of north atlantic right whales upcalls.  _J. Acoust. Soc. Am._ 149, 2520 (2021).
  * [94] Loshchilov, I. & Hutter, F.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_ (2018).
  * [95] Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R.  Dropout : A simple way to prevent neural networks from overfitting.  _J. Mach. Learn. Res._ 15, 1929–1958 (2014).
  * [96] Ba, J. L., Kiros, J. R. & Hinton, G. E.  Layer normalization.  _Preprint athttps://arxiv.org/abs/1607.06450_ (2016).
  * [97] Tokozume, Y., Ushiku, Y. & Harada, T.  Learning from between-class examples for deep sound recognition.  In _International Conference on Learning Representations_ (2018).
  * [98] He, K., Zhang, X., Ren, S. & Sun, J.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  _Proceedings of the IEEE International Conference on Computer Vision_ 11-18-Dece, 1026–1034 (2016).
  * [99] Ramachandran, P., Zoph, B. & Le, Q. V.  Searching for activation functions.  _Preprint athttps://arxiv.org/abs/1710.05941_ (2017).
  * [100] Hendrycks, D. & Gimpel, K.  Gaussian error linear units (GELUs).  _Preprint athttps://arxiv.org/abs/1606.08415_ (2016).
  * [101] He, K. _et al._ Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022)_ , 16000–16009 (2022).
  * [102] Loshchilov, I. & Hutter, F.  SGDR: Stochastic gradient descent with warm restarts.  International Conference on Learning Representations (2017).
  * [103] Bengio, Y., Frasconi, P. & Simard, P.  The problem of learning long-term dependencies in recurrent networks.  In _IEEE International Conference on Neural Networks_ , 1183–1188 vol.3 (IEEE, 2002).
  * [104] Paszke, A. _et al._ PyTorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ 32 (2019).
  * [105] Ott, M. _et al._ fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of NAACL-HLT 2019: Demonstrations_ (2019).
  * [106] Federal Statistical Office of Germany (Statistisches Bundesamt - Destatis).  Electricity consumption of households by household size.  https://www.destatis.de/EN/Themes/Society-Environment/Environment/Environmental-Economic-Accounting/private-households/Tables/electricity-consumption-private-households.html.  Accessed: 2024-05-06.
  * [107] Electricity Maps ApS.  Yearly averaged emission factor for 2023.  https://app.electricitymaps.com/zone/DE.  Accessed: 2023-05-24.
  * [108] Jouppi, Norm and Patterson, David.  Google’s Cloud TPU v4 provides exaFLOPS-scale ML with industry-leading efficiency.  https://shorturl.at/jlmW4.  Accessed: 2024-05-08.
  * [109] Lin, T.-Y., Goyal, P., Girshick, R., He, K. & Dollár, P.  Focal loss for dense object detection.  In _2017 IEEE International Conference on Computer Vision (ICCV)_ , 2999–3007 (2017).
  * [110] Meyes, R., Lu, M., de Puiseau, C. W. & Meisen, T.  Ablation studies in artificial neural networks.  _Preprint athttps://arxiv.org/abs/1901.08644_ (2019).
  * [111] Gwilliams, L., King, J., Marantz, A. & Poeppel, D.  Neural dynamics of phoneme sequences reveal position-invariant code for content and order.  _Nat. Commun._ 13 (2022).
  * [112] Ma, G., Hu, P., Kang, J., Huang, S. & Huang, H.  Leveraging phone mask training for phonetic-reduction-robust e2e uyghur speech recognition.  _Preprint athttps://arxiv.org/abs/2204.00819_ (2022).
  * [113] Boyd, K., Eng, K. H. & Page, C. D.  Area under the precision-recall curve: Point estimates and confidence intervals.  In _Advanced Information Systems Engineering_ , Lecture notes in computer science, 451–466 (Springer Berlin Heidelberg, Berlin, Heidelberg, 2013).
  * [114] Bahdanau, D., Cho, K. & Bengio, Y.  Neural machine translation by jointly learning to align and translate.  _Preprint athttps://arxiv.org/abs/1409.0473_ (2014).
  * [115] Luong, M.-T., Pham, H. & Manning, C. D.  Effective approaches to attention-based neural machine translation.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , 1412–1421 (2015).
  * [116] Jain, S. & Wallace, B. C.  Attention is not explanation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 3543–3556 (2019).
  * [117] Wiegreffe, S. & Pinter, Y.  Attention is not not explanation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 11–20 (2019).
  * [118] Akula, A. R. & Zhu, S.-C.  Attention cannot be an explanation.  _Preprint athttps://arxiv.org/abs/2201.11194_ (2022).
  * [119] Bibal, A. _et al._ Is attention explanation? an introduction to the debate.  In Muresan, S., Nakov, P. & Villavicencio, A. (eds.) _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 3889–3900 (Association for Computational Linguistics, Stroudsburg, PA, USA, 2022).
  * [120] Yeh, C. _et al._ AttentionViz: A global view of transformer attention.  _IEEE Trans. Vis. Comput. Graph._ 30, 262–272 (2024).  eprint 2305.03210.
  * [121] Engesser, S. & Manser, M. B.  Collective close calling mediates group cohesion in foraging meerkats via spatially determined differences in call rates.  _Anim. Behav._ 185, 73–82 (2022).
  * [122] Demartsev, V. _et al._ Mapping vocal interactions in space and time differentiates signal broadcast versus signal exchange in meerkat groups.  _Philos. Trans. R. Soc. Lond. B Biol. Sci._ 379 (2024).
  * [123] Fan, A., Grave, E. & Joulin, A.  Reducing transformer depth on demand with structured dropout.  In _International Conference on Learning Representations_ (2019).

### Data availability

The MeerKAT dataset is openly available at the Max-Planck data repository
Edmond [43] under a CC BY-NC license [42].

### Code availability

The code for animal2vec and the weights for our pretrained and finetuned
transformer models are available at our GitHub [41] under an MIT license [40].

### Acknowledgments

We thank Rebecca Schäfer for their immense help during the field data
collection. We are indebted to many student research assistants for their help
with processing and labeling of the audio data (L. Leonardos, C. Maier, J.
Denger, L. Batke, S. Eleonori, F. Raabe, H. Brønnvik, J. Ruff, B. Ehrmann and
S. Knab). We are grateful to the Kalahari Research Trust and Northern Cape
Department of Environment and Nature Conversation for research permission at
the Kalahari Research Centre, as well as the support of the Universities of
Cambridge and Zurich, and MAVA foundation on the maintenance of the habituated
meerkat population. We thank T. Vink and W. Jubber for organizing the field
site, the managers and volunteers of the Kalahari Meerkat Project for
maintaining habituation and long-term data collection of the meerkats.

### This research was supported by

Max Planck Society; Alexander von Humboldt-Stiftung; Centre for the Advanced
Study of Collective Behaviour: EXC 2117-422037984; Human Frontier Science
Program: RGP0051/2019; Minerva Foundation; Gips-Schüle Foundation; Young
Scholars Fund at the University of Konstanz; Alexander von Humboldt Foundation
post-doctoral fellowships; The long-term research on meerkats is currently
supported by funding from the European Research Council (ERC) under the
European Union’s Horizon 2020 research and innovation program (No. 742808 and
No. 294494) and a Grant from the Natural Environment Research Council (Grant
NE/G006822/1) to Tim Clutton-Brock, as well as by Grants from the University
of Zurich to M.B.M. and the MAVA Foundation.

### Ethics

All procedures were approved by ethical committees of University of Pretoria,
South Africa (permit: EC031-17) and the Northern Cape Department of
Environment and Nature Conservation (permit: FAUNA 1020/2016).

### Authors’ contributions

The field studies at the KRC in 2017 and 2019 were performed by V.D., B.A.,
G.G., L.J.U., M.B.M., and A.S.P.. Data labeling was performed by many student
research assistants (see Acknowledgments) under the supervision of V.D., B.A.,
and A.S.P.. Data cleaning and post-processing were performed by V.D. and
J.C.S.Z.. Early studies using different deep learning architectures in
combination with the MeerKAT dataset were performed by K.D.A., M.D., and D.S..
The conceptual idea for animal2vec was conceived by J.C.S.Z., where M.A.R.,
A.S.P., and J.C.S.Z. jointly analyzed the results of animal2vec. J.C.S.Z.
wrote the codebase for animal2vec, with guidance from M.A.R., where M.F. and
M.A.R. conducted a code review before publication. The figures were created by
J.C.S.Z.. The initial manuscript draft was written by J.C.S.Z. All authors
provided edits and feedback on the final manuscript.

### Conflict of Interest

The authors declare no competing financial or non-financial interests.

Appendix

Table 5:  Training parameter for pretraining the animal2vec framework on MeerKAT and xeno canto.  | Pretraining setting
---|---
Parameter | MeerKAT | Xeno canto | Transformer baseline [38]
Learning rate | $1\text{\times}{10}^{-4}$ | $2.5\text{\times}{10}^{-4}$ | $1\text{\times}{10}^{-4}$
Adam $\beta_{1}$ / $\beta_{2}$ |  $0.9$ / $0.98$ |  $0.9$ / $0.98$ |  $0.9$ / $0.98$
Weight decay | $0.01$ | $0.01$ | $0.01$
Clip norm | $1$ | $1$ | $1$
Learning rate schedule | cosine | cosine | cosine
Warmup steps | $10\,000$ | $8000$ | $10\,000$
GPUs | 4 A100-SXM4-80GB | 4 3090Ti-24GB | 4 A100-SXM4-80GB
Batch size (sec. p. GPU / total) |  $255$ / $1020$ |  $250$ / $1000$ |  $255$ / $1020$
Transformer layers | $16$ | $12$ | $16$
Attention heads | $16$ | $12$ | $16$
Embedding dimensions | $1024$ | $768$ | $1024$
Updates |  $408\,000$ ($100$ epochs) |  $264\,600$ ($100$ epochs) |  $408\,000$ ($100$ epochs)
Decoder dim. | $768$ | $384$ | $768$
Decoder conv. groups | $16$ | $16$ | $16$
Decoder kernel width | $7$ | $7$ | $7$
Decoder layers | $4$ | $4$ | $4$
Mask probability $p$ | $0.1500$ | $0.1500$ | $0.0650$
Mask length $M$ | $2$ | $2$ | $10$
Nr. of Sinc filters [37] | $127$ | $127$ | -
Sinc filter kernel width [37] | $63$ | $125$ | -
BCL mixing strength (target) [97] | $0$ | $0$ | $0$
BCL mixing strength (input) [97] | $0.5$ | $0.5$ | $0$
BCL token prob. [97] | $1.0$ | $1.0$ | $0$
BCL window length [97] | $0.05$ | $0.05$ | $0$
PSwish initial $\alpha$ | $2$ | $2$ | -
PSwish initial $\beta$ | $0$ | $0$ | -
Feature Extractor layout (Nr. of filter, width, stride) |  (512, 10, 5) (512, 3, 2) (512, 3, 2) (512, 3, 2) (512, 3, 1) (512, 2, 1) (512, 2, 1) |  (512, 10, 5) (512, 3, 2) (512, 3, 2) (512, 3, 2) (512, 3, 2) (512, 2, 1) (512, 2, 1) |  (512, 10, 5) (512, 3, 2) (512, 3, 2) (512, 3, 2) (512, 3, 2) (512, 2, 2) (512, 2, 2)
Nr. of trainable parameters |  $315$M |  $94$M |  $315$M

Table 6:  Training parameter for finetuning the animal2vec framework on MeerKAT and NIPS4Bplus.  | Finetune setting |
---|---|---
Parameter | MeerKAT ($1\text{\,}\mathrm{\char 37\relax}$) | MeerKAT ($25\text{\,}\mathrm{\char 37\relax}$) | MeerKAT ($100\text{\,}\mathrm{\char 37\relax}$) | Baseline ($100\text{\,}\mathrm{\char 37\relax}$) | NIPS4Bplus
Learning rate | $1\text{\times}{10}^{-4}$ | $7.5\text{\times}{10}^{-5}$ | $3\text{\times}{10}^{-5}$ | $3\text{\times}{10}^{-5}$ | $1\text{\times}{10}^{-4}$
Adam $\beta_{1}$ / $\beta_{2}$ |  $0.9$ / $0.98$ |  $0.9$ / $0.98$ |  $0.9$ / $0.98$ |  $0.9$ / $0.98$ |  $0.9$ / $0.98$
Learning rate schedule | cosine | cosine | cosine | cosine | cosine
GPUs | 4 A100-SXM4-80GB | 4 3090Ti-24GB
Batch size (sec. p. GPU / total) |  $480$ / $1920$ |  $480$ / $1920$ |  $480$ / $1920$ |  $480$ / $1920$ |  $392$ / $1568$
Warmup steps | $2000$ | $2000$ | $2000$ | $2000$ | $1000$
Steps with fixed transformer | $10\,000$ | $10\,000$ | $10\,000$ | $10\,000$ | $3000$
Steps with transformer |  $3000$ ($1000$ epochs) |  $8000$ ($107$ epochs) |  $20\,000$ ($67$ epochs) |  $20\,000$ ($67$ epochs) |  $2000$ ($913$ epochs)
Total steps |  $13\,000$ ($4333$ epochs) |  $18\,000$ ($240$ epochs) |  $30\,000$ ($100$ epochs) |  $30\,000$ ($100$ epochs) |  $5000$ ($2283$ epochs)
Mask probability $p$ | $0.1100$ | $0.1000$ | $0.0825$ | $0.0650$ | $0.1100$
Mask length $M$ | $2$ | $3$ | $4$ | $10$ | $2$
BCL mixing strength (target) [97] | $0.5$ | $0.5$ | $0.5$ | $0$ | $0.5$
BCL mixing strength (input) [97] | $0.5$ | $0.5$ | $0.5$ | $0$ | $0.5$
BCL token prob. [97] | $1.0$ | $1.0$ | $1.0$ | $0$ | $1.0$
BCL window length [97] | $0.05$ | $0.05$ | $0.05$ | $0$ | $0.05$
$\gamma$ in Focal loss [109] | $2$ | $2$ | $2$ | $0$ | $2$
Averaging over K transformer layers | $16$ | $16$ | $16$ | $8$ | $12$
Dropout [95] | $0.1$ | $0.1$ | $0.1$ | $0.1$ | $0.1$
Layerdrop [123] | $0.1$ | $0.1$ | $0.1$ | $0.1$ | $0.1$

Figure 7: Globally averaged attention map of a four second segment showing
four move calls. An attention map shows the _importance_ of every input frame
with respect to every other frame during finetuning. Blue stretches surrounded
by dashed white lines correspond to the predictions of animal2vec. For
predicting, animal2vec attends to the immediate past and future of an event,
as well as to previous and future alarm calls.

Figure 8:  The precision-recall curve for the focal class. Results of
animal2vec using $1\text{\,}\mathrm{\char 37\relax}$,
$25\text{\,}\mathrm{\char 37\relax}$, and $100\text{\,}\mathrm{\char
37\relax}$ of the training data are in red, yellow, and teal, respectively,
and the baseline results are in gray.

Figure 9:  The precision-recall curve for the close call class. Results of
animal2vec using $1\text{\,}\mathrm{\char 37\relax}$,
$25\text{\,}\mathrm{\char 37\relax}$, and $100\text{\,}\mathrm{\char
37\relax}$ of the training data are in red, yellow, and teal, respectively,
and the baseline results are in gray.

Figure 10:  The precision-recall curve for the short-note call class. Results
of animal2vec using $1\text{\,}\mathrm{\char 37\relax}$,
$25\text{\,}\mathrm{\char 37\relax}$, and $100\text{\,}\mathrm{\char
37\relax}$ of the training data are in red, yellow, and teal, respectively,
and the baseline results are in gray.

Figure 11:  The precision-recall curve for the social call class. Results of
animal2vec using $1\text{\,}\mathrm{\char 37\relax}$,
$25\text{\,}\mathrm{\char 37\relax}$, and $100\text{\,}\mathrm{\char
37\relax}$ of the training data are in red, yellow, and teal, respectively,
and the baseline results are in gray.

Figure 12:  The precision-recall curve for the other call class. Results of
animal2vec using $1\text{\,}\mathrm{\char 37\relax}$,
$25\text{\,}\mathrm{\char 37\relax}$, and $100\text{\,}\mathrm{\char
37\relax}$ of the training data are in red, yellow, and teal, respectively,
and the baseline results are in gray.

Figure 13:  The precision-recall curve for the alarm call class. Results of
animal2vec using $1\text{\,}\mathrm{\char 37\relax}$,
$25\text{\,}\mathrm{\char 37\relax}$, and $100\text{\,}\mathrm{\char
37\relax}$ of the training data are in red, yellow, and teal, respectively,
and the baseline results are in gray.

Figure 14:  The precision-recall curve for the aggressive call class. Results
of animal2vec using $1\text{\,}\mathrm{\char 37\relax}$,
$25\text{\,}\mathrm{\char 37\relax}$, and $100\text{\,}\mathrm{\char
37\relax}$ of the training data are in red, yellow, and teal, respectively,
and the baseline results are in gray.

Figure 15:  The precision-recall curve for the move call class. Results of
animal2vec using $1\text{\,}\mathrm{\char 37\relax}$,
$25\text{\,}\mathrm{\char 37\relax}$, and $100\text{\,}\mathrm{\char
37\relax}$ of the training data are in red, yellow, and teal, respectively,
and the baseline results are in gray.

Figure 16:  The precision-recall curve for the lead call class. Results of
animal2vec using $1\text{\,}\mathrm{\char 37\relax}$,
$25\text{\,}\mathrm{\char 37\relax}$, and $100\text{\,}\mathrm{\char
37\relax}$ of the training data are in red, yellow, and teal, respectively,
and the baseline results are in gray.

Figure 17:  The precision-recall curve for the synch signal class. Results of
animal2vec using $1\text{\,}\mathrm{\char 37\relax}$,
$25\text{\,}\mathrm{\char 37\relax}$, and $100\text{\,}\mathrm{\char
37\relax}$ of the training data are in red, yellow, and teal, respectively,
and the baseline results are in gray.

Figure 18:  The precision-recall curve for the eating class. Results of
animal2vec using $1\text{\,}\mathrm{\char 37\relax}$,
$25\text{\,}\mathrm{\char 37\relax}$, and $100\text{\,}\mathrm{\char
37\relax}$ of the training data are in red, yellow, and teal, respectively,
and the baseline results are in gray.

Figure 19:  The precision-recall curve for the beep signal class. Results of
animal2vec using $1\text{\,}\mathrm{\char 37\relax}$,
$25\text{\,}\mathrm{\char 37\relax}$, and $100\text{\,}\mathrm{\char
37\relax}$ of the training data are in red, yellow, and teal, respectively,
and the baseline results are in gray.