# Arboretum: A Large Multimodal Dataset
Enabling AI for Biodiversity

Chih-Hsuan Yang  Iowa State University, Ames, IA 50011, USA  Joint first
author  Ben Feuer  New York University, New York, NY 10003, USA  Joint first
author  Zaki Jubery  Iowa State University, Ames, IA 50011, USA  Zi K. Deng
University of Arizona, Tucson, AZ 85721, USA  Andre Nakkab  New York
University, New York, NY 10003, USA  Md Zahid Hasan  Iowa State University,
Ames, IA 50011, USA  Shivani Chiranjeevi  Iowa State University, Ames, IA
50011, USA  Kelly Marshall  New York University, New York, NY 10003, USA
Nirmal Baishnab  Iowa State University, Ames, IA 50011, USA  Asheesh K Singh
Iowa State University, Ames, IA 50011, USA  Arti Singh  Iowa State University,
Ames, IA 50011, USA  Soumik Sarkar  Iowa State University, Ames, IA 50011, USA
Nirav Merchant  University of Arizona, Tucson, AZ 85721, USA  Chinmay Hegde
New York University, New York, NY 10003, USA  Baskar Ganapathysubramanian
Iowa State University, Ames, IA 50011, USA

###### Abstract

We introduce Arboretum, the largest publicly accessible dataset designed to
advance AI for biodiversity applications. This dataset, curated from the
iNaturalist community science platform and vetted by domain experts to ensure
accuracy, includes 134.6 million images, surpassing existing datasets in scale
by an order of magnitude. The dataset encompasses image-language paired data
for a diverse set of species from birds (Aves), spiders/ticks/mites
(Arachnida), insects (Insecta), plants (Plantae), fungus/mushrooms (Fungi),
snails (Mollusca), and snakes/lizards (Reptilia), making it a valuable
resource for multimodal vision-language AI models for biodiversity assessment
and agriculture research. Each image is annotated with scientific names,
taxonomic details, and common names, enhancing the robustness of AI model
training.

We showcase the value of Arboretum by releasing a suite of CLIP models trained
using a subset of 40 million captioned images. We introduce several new
benchmarks for rigorous assessment, report accuracy for zero-shot learning,
and evaluations across life stages, rare species, confounding species, and
various levels of the taxonomic hierarchy.

We anticipate that Arboretum will spur the development of AI models that can
enable a variety of digital tools ranging from pest control strategies, crop
monitoring, and worldwide biodiversity assessment and environmental
conservation. These advancements are critical for ensuring food security,
preserving ecosystems, and mitigating the impacts of climate change. Arboretum
is publicly available, easily accessible, and ready for immediate use. Please
see the project website for links to our data, models, and code.

##  1 Introduction

AI advances are poised to play a crucial role in biodiversity conservation,
ecology management, and agriculture. Already, AI tools have been shown to
enable automated species identification, monitoring of ecological changes, and
optimization of crop management [32, 5]. However, standard AI approaches for
biodiversity applications persistently face major challenges. Training
datasets are labor-intensive and costly to create; they cover only a narrow
set of visual concepts; standard vision models excel at single tasks, but
require extensive retraining for new tasks; models often struggle with
generalizing to unseen labels and new environments, limiting their
effectiveness in real-world applications [30, 13]. Models that perform well on
benchmarks often fail in the wild [11, 1]. Standard computer vision datasets
(ImageNet and its successors) have significant limitations, including
incorrectly labeled images, geographical and cultural biases, and overlapping
or ill-defined labels, all of which impair the development of high-performant
AI models [20]. Consequently, there is a critical need for large, diverse,
accurately annotated datasets that are specific to biodiversity, ecology, and
agricultural research [23, 19].

In response to this need, several datasets have been introduced. Perhaps the
most well-known (raw) pool of biodiversity images on the Web is iNaturalist
[38], from which several curated datasets have been sourced, among them being
iNat2021 [37] with 2.7M images of over 10,000 species of plants, animals, and
fungi. However, insects (which comprise a very large fraction of extant
species) are under-represented in this dataset. IP102 [40], Insecta [9], and
the more recent BioScan-1M [12], are alternative datasets that focus on the
Insecta Class. Perhaps the latest advance in such research is TreeOfLife-10M
[35], which is currently the state-of-the-art dataset of text-annotated
biological images, comprising 10M images with approximately 450K unique
taxonomic classes.

Figure 1: The seven taxonomic classes in the Arboretum dataset. Example images
of the top 5 most frequent species in each category, including their counts,
common names, and scientific names. For clarity, the word ’class’ used in this
context is denoted ’taxonomic class’ in the rest of the paper to distinguish
it from the other common ML definitions.

In this paper, we make significant contributions to this body of work by
curating and releasing Arboretum. This dataset includes 134.6 million
captioned images of approximately 326.9K species. The dataset surpasses all
existing datasets in scale by an order of magnitude, constituting the largest
public, “AI-ready" dataset of curated biodiversity images. The dataset
encompasses image-language paired data for a diverse set of species from birds
(Aves), spiders/ticks/mites (Arachnida), insects (Insecta), plants (Plantae),
fungus/mushrooms (Fungi), snails (Mollusca), and snakes/lizards (Reptilia).
See Figure 1 for representative examples, and the project website for details.

Each image in Arboretum is sourced from the iNaturalist community science
platform [38] and is annotated with the corresponding common (English) name,
scientific (Latin) name, and taxonomic hierarchy. The metadata for each
species has been vetted by domain experts, ensuring the accuracy of the text
annotations for robust AI model training. We open-source the tooling pipeline
used to curate Arboretum, and hope that the community will use this dataset as
a valuable resource for further development of multimodal (vision-language) AI
models for biodiversity and agriculture research.

To showcase the potential of Arboretum, we further make two technical
contributions. First, we train and release ArborCLIP, a suite of vision-
language foundation models trained on a subset of approximately 40M Arboretum
samples, comprising approximately 33K species. This subset was constructed
with standard filtering criteria in order to reduce skewness in species
counts. Our models show excellent generalization performance and can support
zero-shot (or few-shot) classification with either common or scientific names
of unseen taxa. We also anticipate that these models can be fruitfully fine-
tuned in the future on datasets for specific biodiversity-related
applications.

Second, we rigorously quantify the performance of our foundation models on
five existing fine-grained image classification benchmarks, as well on three
newly curated test datasets. We find that ArborCLIP models comfortably achieve
the state-of-the-art in certain settings, while both the original (OpenAI)
CLIP model as well as BioCLIP [35] excel in certain other settings. We analyze
these findings in further detail below, but overall we hope that our dataset
can be used by the AI community as a testbed for further algorithmic and
scaling research in fine-grained image recognition.

Figure 2: Distribution of the Arboretum dataset. (a) Size of each taxonomic
class in the Arboretum dataset. (b) Species counts for each taxonomic class.
(c) The 40 highest occuring species.

The remainder of this paper is organized as follows. Section 2 introduces the
Arboretum dataset, the dataset’s salient characteristics, and a comparison
with previous work. Section 3 details our curation methodology. Section 4
introduces our newly proposed test datasets and their characteristics. Section
5 details our new ArborCLIP models and their benchmark performance relative to
previous work. Section 6 concludes with a discussion of limitations and
potential future directions.

##  2 The Arboretum Dataset

Figure 3: Treemap diagram of the Arboretum dataset, starting from _Kingdom_.
The nested boxes represent phyla, (taxonomic) classes, orders, and families.
Box size represents the relative number of samples.

#### Characteristics.

Arboretum comprises over 134.6M images across seven taxonomic classes —Aves,
Arachnida, Insecta, Plantae, Fungi, Mollusca, and Reptilia. These taxonomic
classes were chosen to represent the span of species — outside of charismatic
megafauna — which critically impact biodiversity. The images in Arboretum span
326,888 species. Overall, this dataset nearly matches the state-of-the-art
curated dataset (TreeOfLife-10M) in terms of _species diversity_ , while
comfortably exceeding it in terms of _scale_ by a factor of nearly
$13.5\times$. Figure 1 shows representative image samples, and Figure 2
displays the distribution of samples according to the seven categories (and
the most frequently occurring species). Figure 3 displays phyla, taxonomic
classes, orders, and families represented in the dataset.

Each image sample in Arboretum is annotated with a rich amount of curated
metadata. This metadata design facilitates easy filtering of species by image
count and taxonomic information. The metadata integrates common names,
scientific names, and detailed taxonomic hierarchies. For the full list of
fields, see Table 1.

Along with the dataset we also release our data curation tooling pipeline,
that enables users to easily access and manipulate the dataset. The pipeline
allows researchers to select specific categories, visualize data
distributions, and manage class imbalance effectively based on their needs. It
facilitates the downloading of specific images by their URLs and provides
image-text pairs and user-defined chunks to support various AI applications.
This pipeline enables users to define different subsets of Arboretum in an
easy manner, overall making the dataset AI-ready and easy to use, and reducing
the barriers to follow-up research in AI for biodiversity.

#### Dual-language text descriptions.

We adopt both common and scientific names, since Latin is a low-resource
language and current vision-language backbones do not perform well on
scientific names alone in a zero-shot manner. We found that a well-structured
text description that integrate common names, scientific names, and detailed
taxonomic hierarchies facilitates the learning of relationships between Latin
and English terms, thereby improving the models’ applicability in scientific
contexts [6, 34, 39]. Moreover, incorporating the taxonomic hierarchy enables
models to more effectively associate visual data with taxonomic terminology
[21, 2]. This matches the guidelines suggested by BioCLIP [35] to enhance
model performance and generalization.

Table 1: Annotations provided in the Arboretum Dataset. Text Type | Description
---|---
Common Name | Vernacular name (e.g., Western honey bee)
Scientific Name | Genus and species (e.g., Apis mellifera)
Taxonomic Name | Flattened taxonomy concatenated into a single string
Taxonomic Rank | Specific level in the hierarchy (e.g., subspecies, species)

Privacy Measures: The images of Arboretum were sourced from the iNaturalist
Open Dataset, whose metadata included Personally Identifiable Information
(PII). This included information about observers, such as their usernames and
sometimes their real names if they have chosen to share that information
publicly. We removed all such fields to ensure that no PII is present in the
metadata associated with Arboretum samples, ensuring the privacy of all
contributors.

License: During curation, we took care to include only images from iNaturalist
Open Data, which are all licensed under either the CC0, or CC-BY, or CC-BY-NC
licenses. This ensures that all our images are available for public research
purposes.

Offensive Content: Some of our URLs may point to images that users could find
disturbing or harmful, such as photos of dead or dismembered animals. We
retained these types of images since they sometimes can provide valuable
scientific data about wildlife, including information on predation events,
roadkill, and other occurrences relevant to conservation and biodiversity
studies. Although iNaturalist relies on user contributions and community
moderation to maintain the quality and appropriateness of the data, we
acknowledge that the vast and diverse nature of the data means that some
offensive or inappropriate content might be present.

Our closest comparisons are with BioScan-1M (which appeared in NeurIPS 2023
Datasets and Benchmarks) and TreeOfLife-10M (which will appear in CVPR 2024).
BioScan-1M focuses solely on the Insecta Class and provides scientific names,
taxonomic ranks, as well as DNA barcodes. The TreeOfLife-10M dataset comprises
10.4 million images, integrating data from iNat2021 [37], BioScan-1M, and a
fresh set of image samples sourced from the Encyclopedia of Life (EOL). It
also supports dual-language labels and detailed taxonomic hierarchies, and was
used to train the BioCLIP vision-language model. See Table 2 for essential
differences.

Table 2: Comparison of Arboretum with existing biodiversity datasets.

Feature |  Arboretum |  TreeOfLife |  BioScan
---|---|---|---
Size |  134.6 million images |  10.4 million images |  1.1 million images
Diversity |  326.9K species |  454.1K species |  8.3K
Labels Provided |  Dual language (common and scientific names), detailed taxonomic hierarchies |  Dual language (common and scientific names), detailed taxonomic hierarchies |  Single language (scientific names), taxonomic ranks (family to species), DNA barcodes
Data Source |  iNaturalist Open Dataset |  iNaturalist, Encyclopedia of Life (EOL), BioScan-1M |  Specimens from Malaise traps, DNA barcodes matched to BOLD
Key Features |  Ready-to-use pipeline, reduce class imbalance, high-quality annotations, supports ArborCLIP |  Rich hierarchical representations, comprehensive metadata, supports BioCLIP |  Focus on insects, high-resolution images, detailed taxonomic annotation, DNA codes

##  3 Data Collection and Curation Methodology

#### Challenges with iNaturalist Open Data.

All of Arboretum is sourced from the iNaturalist Open Data community science
platform, which (in all) comprises over 190M biodiversity-relevant
observations shared by users. However, there are still significant gaps in
usability for AI research. The photos and metadata, although easily
downloadable, are provided in four separate metadata sheets that are not ready
to use. Taxa information is encoded as numerical IDs, requiring additional API
calls and non-trivial lookups to convert these into common or scientific
names. The multiple metadata sheets structure is fragmented across four
separate files—photos, taxa, observations, and observers—adding complexity to
data integration. Managing data balance and filtering out species with too few
images can lead to biases towards common (charismatic) species and an
imbalanced training process.

#### Curation of Arboretum.

The iNaturalist Open Dataset comprises a collection of 250.0M images stored on
an AWS S3 bucket as of May 27, 2024, with associated metadata in the form of
four separate CSV files (indexed as photos, observations, taxa, and
observers). Details of each of these files are provided in Section A.5 in the
Appendix. While these files contain a plethora of interesitng information,
they are designed for rapid lookup, and therefore not “AI-ready". To resolve
this, we curate the metadata into a streamlined format for easy usage by AI
practitioners.

We first use the corresponding CSV files to populate an SQL database with each
CSV file as its own SQL table. We create a new aggregate SQL table by joining
photos, observations, and taxa on its relational columns, and discarding
columns not relevant to us. We also construct a new column in the aggregated
table and populate it with the Amazon S3 URL where the image file is hosted.

We then query this SQL table and extract rows corresponding to the following
seven Arboretum categories: Aves, Arachnida, Insecta, Plantae, Fungi,
Mollusca, and Reptilia.111As mentioned above, we choose these 7 classes since
they represent non-megafauna that criticially impact biodiversity. Typically,
megafauna species are well suited to standard image recognition models, but
there remains a pressing need for similar efforts for these particular
classes. The iNaturalist Open Data metadata files do not contain common name
information; to reconstruct this information, we cross-match species names
from the iNaturalist Taxonomy DarwinCore Archive, which is compiled monthly.
Once the common name is reconstructed we append this to the SQL table.
Finally, we export this table as a CSV file, which we release for public use
on HuggingFace.

#### Data filtering and preprocessing.

As described above, Arboretum comprises well-processed metadata with full taxa
information and URLs pointing to image files. The metadata can be used to
filter specific categories, visualize data distribution, and manage imbalance
effectively. We provide a collection of software tools that enable users to
easily download, access, and manipulate the dataset. This makes it
straightforward for researchers to explore different data subsets.

The citizen science aspect of iNaturalist data leads to a wide variation in
the number of observations per species, with some species having only a few
records while others have thousands. To mitigate this, our tools can be used
to apply user-defined filtering criteria to exclude species with fewer than a
specified number of images. Additionally, we imposed an upper limit on the
number of images per species to mitigate overfitting.

To address imbalances during training (described further below in our
experiments), we download the data and organize into chunked tar files using a
semi-global shuffling strategy. Initially, each tar file is shuffled and
divided into smaller groups. These groups were then randomly merged to form
larger batches, ensuring a balanced distribution of species within each batch.
This approach significantly improves dataset integrity by preventing any
single species from spanning full batches.

##  4 Models and Benchmarks

We now showcase demonstrate the utility of the Arboretum dataset by creating
and benchmarking ArborCLIP, a new suite of vision-language foundation models
for biodiversity.

###  4.1 Arboretum-40M

To start, we first construct Arboretum-40M, a subset comprising approximately
40M samples and 33K species, which we constructed by applying the data
filtering tools described above.

The data included in this subset include all samples in the seven Arboretum
categories posted on iNaturalist prior to January 27, 2024. We applied
filtering criteria to exclude species with fewer than 30 images and imposed a
maximum limit of 50,000 samples per species. We conducted semi-global
shuffling and divided the data into mini-batches of approximately 50,000
samples each. From these mini-batches, 95% were randomly selected for training
and validation, while the remaining 5% were reserved for testing. Detailed
information can be found in Table 3.

Table 3: Training data sources used in Arboretum-40M and Diversity in
Different Taxonomy Levels. We integrate taxonomic labels into the images.

Dataset Description Images Unique Classes TreeOfLife-10M Dataset combines a subset of iNaturalist, Encyclopedia of Life (EOL), BioScan-1M. 10.4M 454,103 Arboretum-40M One subset of Arboretum with size 40M. 39.9M 33,364 |  Level Uniques kingdom 3 phylum 14 class 50 order 311 family 1692 genus 11506 species 33364
---|---

###  4.2 New Benchmarks

From Arboretum, we create three new benchmark datasets for fine-grained image
classification. In addition, we report results on several established
benchmarks in the literature; see Table 4.

Arboretum-Balanced. To enforce a balanced species distribution across the
seven categories, we curate Arboretum-Balanced. Each category includes up to
500 species, with 50 images per species. Table 6 in the Appendix shows each
category’s exact number of species.

Arboretum-Unseen. To provide a robust benchmark for evaluating the
generalization capability of models on unseen species, we curate Arboretum-
Unseen. The test dataset was constructed by identifying species with fewer
than 30 instances in Arboretum, ensuring that the dataset contains species
that were unseen by ArborCLIP. Each species contained 10 images.

Arboretum-LifeStages. To assess the model’s ability to recognize species
across various developmental stages, we curate Arboretum-LifeStages (see
Figure 4). This dataset has 20 labels in total and focuses on insects, since
these species often exhibits significant visual differences across their
lifespan. Arboretum-LifeStages contains five insect species and utilized the
observation export feature on the iNaturalist platform to collect data from
2/1/2024 to 5/20/2024 to ensure no overlap with the training dataset. For each
species, life stage filters (egg, larva, pupa, or adult) were applied.

Figure 4: (a) Example images from Arboretum-Unseen. (b) Arboretum-Life-Stages
with 20 class labels: four life stages (egg, larva, pupa, and adult) for five
distinct insect species.

###  4.3 ArborCLIP: New vision-language foundation models for biodiversity

We use Arboretum-40M to train new CLIP-style foundation models, and then
evaluate them on zero-shot image classification tasks. Following the
implementation of Stevens et al. [35], we utilize a ViT-B/16 architecture
initialized from the OpenAI CLIP weights [29], and train for 40 epochs. We
also train a ViT-L/14 model from the MetaCLIP [41] checkpoint for 12 epochs,
and a ViT-B/16 from the BioCLIP checkpoint for 8 epochs. All training
hyperparameters are included in the Appendix (Section B). We compare with
OpenAI’s ViT-B/16 CLIP model, the BioCLIP ViT-B/16 checkpoint, and MetaCLIP-CC
ViT-L/14. We publicly release all code needed to reproduce our results here.

##  5 Experimental Results

Table 4:  Existing benchmark datasets; our novel datasets are described
separately in section 4.2.

| Name |  Description | Examples | Classes | Labels
---|---|---|---|---|---
Anim | Birds 525 |  Scraped dataset of bird images from web search. [27] | $89\,885$ | $525$ | Taxonomic
BioCLIP-Rare |  Subset of species in the IUCN Red List categories: Near Threatened through Extinct in the Wild (iucnredlist.org). | $12\,000$ | $400$ | Taxonomic
Plt & Fun | Fungi |  Expert-labeled images of Danish fungi [26]. | $1000$ | $25$ | Scientific
DeepWeeds |  Weed images collected in situ from eight rangelands across northern Australia [25] | $17\,509$ | $9$ | Common
Inse | Confounding Species |  Dataset evaluating models on challenging visually similar species pairs [4]. | $100$ | $10$ | Mixed
Insects-2 |  Mixed common and scientific name classification for insect pests [40]. | $4080$ | $102$ | Mixed

Metrics. For each of the benchmark datasets, we report top-1 zero-shot accuracy. For datasets where taxonomic information is available, we report accuracy over scientific names. For datasets where no taxonomic information is available, we utilize category names provided by the benchmark’s authors. For our Arboretum-Life-Stages benchmark, we consider a class label to be a unique (species | life-stage) tuple, resulting in a 20-class benchmark. Finally, we report an aggregate metric which is the weighted average over unique class labels across all benchmarks in our suite.

Overview of results. In Table 5, we report the results of our core benchmark
suite. At a high level, we observe that ArborCLIP variants achieve the best
accuracy averaged over benchmarks. In particular, they perform extremely well
on Arboretum-Balanced (a remarkable 91.1 top-1 accuracy over 2250+ class
labels). ArborCLIP also does very well on the Fungi dataset (even though the
Fungi class is not central to Arboretum-40M), and the DeepWeeds dataset.
Therefore, ArborCLIP exhibits strong generalization capabilities across
diverse datasets.

We also observe that BioCLIP performs very well on Arboretum-Unseen and
BioCLIP-Rare. The reasons might be that BioCLIP has seen approximately 450K
species, and there might be nontrivial overlap with the species set in
Arboretum-Unseen. On the other hand, it could be that ArborCLIP suffers from
forgetting issues while training on Arboretum-40M. For BioCLIP-Rare, the
dataset is a subset from EOL which BioCLIP did not see before, but TreeofLife
contains the majority of the EOL dataset.

#### Limitations.

We also evaluated all models on the challenging Confounding-species benchmark
introduced in [4], but find that all models perform at or below random chance,
and do not report results here; this could be an interesting avenue for
follow-up work.

In Table 7 in the Appendix, we report model performance at different levels of
the taxonomic hierarchy. Generally, we find that models trained on web-scraped
data perform better with common names, whereas models trained on specialist
datasets perform better when using scientific names. Additionally, models
trained on web-scraped data excel at classifying at the highest taxonomic
level (kingdom), while models begin to benefit from specialist datasets like
Arboretum-40M and Tree-of-Life-10M at the lower taxonomic levels (order and
species). From a practical standpoint, this is not problematic: ArborCLIP is
highly accurate at the species level, and higher-level taxa can be
deterministically derived from lower ones.

Table 5: ArborCLIP performs well on a range of benchmarks. ArborCLIP-O was
pretrained from an OpenAI model checkpoint, ArborCLIP-B from the BioCLIP
checkpoint, and ArborCLIP-M from a MetaCLIP-cc checkpoint. AU stands for
Arboretum-Unseen, using Scientific Names (n-classes=300). AB stands for
Arboretum-Balanced, using Scientific Names (n-classes=2253). BCR stands for
the BioCLIP Rare species benchmark (n-classes=400). F stands for Fungi
(n-classes=25). I2 stands for Insects2 (n-classes=102). B stands for Birds525
(n-classes=525), LS for the Life Stages benchmark (n-classes=20), and DW for
DeepWeeds (n-classes=9).

Dataset | Arch | AU | AB | BCR | F | I2 | B | LS | DW | Weighted Average
---|---|---|---|---|---|---|---|---|---|---
OpenAI | ViT-B-16 | 12.9 | 7.3 | 10.9 | 11.5 | 10.2 | 50 | 56.5 | 10.3 | 14.7
BioCLIP | ViT-B-16 | 68.2 | 62.2 | 30.2 | 45.1 | 20.8 | 68.7 | 18 | 19.9 | 58.5
MetaCLIP-cc | ViT-L-14 | 24.9 | 15.4 | 20.5 | 24.6 | 16.1 | 70.1 | 64.3 | 14.7 | 25.0
ArborCLIP-O | ViT-B-16 | 47.1 | 91.1 | 22.9 | 43.2 | 16.5 | 47.8 | 28 | 17 | 70.8
ArborCLIP-B | ViT-B-16 | 53.8 | 82.2 | 23.7 | 53.9 | 16.9 | 57.1 | 15 | 18.4 | 67.2
ArborCLIP-M | ViT-L-14 | 44.3 | 91.1 | 21.8 | 54.7 | 5.1 | 42.5 | 26.3 | 49.9 | 69.5

Addressing these limitations will further enhance the applicability of models
like ArborCLIP in real-world biodiversity monitoring tasks.

##  6 Concluding Discussion

We introduce Arboretum, the largest publicly accessible dataset designed to
advance AI for biodiversity applications. This dataset, curated from the
iNaturalist community science platform, includes 134.6 million images,
surpassing existing datasets in scale by an order of magnitude. We anticipate
that Arboretum will enable the development of AI models that can enable
various digital tools ranging from pest control strategies, crop monitoring,
and worldwide biodiversity assessment and environmental conservation.

We also believe that Arboretum can be used as a unique testbed for measuring
progress on fine-grained image recognition. The success of ArborCLIP on
Arboretum-Unseen underscores the importance of scaling up per-category sample
size, or vertical scaling [8], in achieving high accuracy on long-tailed
extreme-imbalance classification. However, BioCLIP continues to exhibit
superior performance on several datasets, and we believe that this is because
TreeofLife-10M contains an order-of-magnitude more _classes_ (species) than
Arboretum-40M. We invite the AI community to create new subsets of Arboretum
with varying degrees of balance and species diversity, and use our tooling to
measure model performance against current benchmarks.

## Acknowledgements

This work was supported by the AI Research Institutes program supported by the
NSF and USDA-NIFA under AI Institute: for Resilient Agriculture, Award No.
2021-67021-35329. This was also partly supported by the NSF under CPS Frontier
grant CNS-1954556. CH, BF, AN, and KM gratefully acknowledge the support of
NYU IT High Performance Computing resources, services, and staff expertise.

## References

  * Alcorn et al. [2019] Michael A Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen.  Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4845–4854, 2019.
  * Chen et al. [2021] Siqi Chen, Yijie Pei, Zunwang Ke, and Wushour Silamu.  Low-resource named entity recognition via the pre-training model.  _Symmetry_ , 13(5):786, 2021.
  * Chiavassa and Kraft [2024] Juan A Chiavassa and Martin Kraft.  The fair-device–an ai image recognition-based non-lethal and generalist monitoring system for insect biodiversity in agriculture.  In _44\. GIL-Jahrestagung, Biodiversität fördern durch digitale Landwirtschaft_ , pages 209–214. Gesellschaft für Informatik eV, 2024\.
  * Chiranjeevi et al. [2023] Shivani Chiranjeevi, Mojdeh Sadaati, Zi K Deng, Jayanth Koushik, Talukder Z Jubery, Daren Mueller, Matthew E O Neal, Nirav Merchant, Aarti Singh, Asheesh K Singh, Soumik Sarkar, Arti Singh, and Baskar Ganapathysubramanian.  Deep learning powered real-time identification of insects using citizen science data, 2023.
  * Chiu et al. [2020] Mang Tik Chiu, Xingqian Xu, Yunchao Wei, Zilong Huang, Alexander G Schwing, Robert Brunner, Hrant Khachatrian, Hovnatan Karapetyan, Ivan Dozier, Greg Rose, et al.  Agriculture-vision: A large aerial image database for agricultural pattern analysis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2828–2838, 2020.
  * De Langhe et al. [2024] Loic De Langhe, Orphée De Clercq, and Veronique Hoste.  Unsupervised authorship attribution for medieval latin using transformer-based embeddings.  In _Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA)@ LREC-COLING-2024_ , pages 57–64, 2024.
  * Di Cecco et al. [2021] Grace J Di Cecco, Vijay Barve, Michael W Belitz, Brian J Stucky, Robert P Guralnick, and Allen H Hurlbert.  Observing the observers: how participants contribute data to inaturalist and implications for biodiversity science.  _BioScience_ , 71(11):1179–1188, 2021.
  * Feuer and Hegde [2023] Ben Feuer and Chinmay Hegde.  Exploring dataset-scale indicators of data quality.  _ArXiv_ , abs/2311.04016, 2023.  URL https://api.semanticscholar.org/CorpusID:265042924.
  * Feuer et al. [2024] Benjamin Feuer, Ameya Joshi, Minsu Cho, Shivani Chiranjeevi, Zi Kang Deng, Aditya Balu, Asheesh K. Singh, Soumik Sarkar, Nirav Merchant, Arti Singh, Baskar Ganapathysubramanian, and Chinmay Hegde.  Zero-shot insect detection via weak language supervision.  _The Plant Phenome Journal_ , 7(1):e20107, 2024\.  URL https://acsess.onlinelibrary.wiley.com/doi/abs/10.1002/ppj2.20107.
  * Gabeff et al. [2024] Valentin Gabeff, Marc Rußwurm, Devis Tuia, and Alexander Mathis.  Wildclip: Scene and animal attribute retrieval from camera trap data with domain-adapted vision-language models.  _International Journal of Computer Vision_ , pages 1–17, 2024.
  * Geirhos et al. [2018] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel.  Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness.  _arXiv preprint arXiv:1811.12231_ , 2018.
  * Gharaee et al. [2024] Zahra Gharaee, ZeMing Gong, Nicholas Pellegrino, Iuliia Zarubiieva, Joakim Bruslund Haurum, Scott Lowe, Jaclyn McKeown, Chris Ho, Joschka McLeod, Yi-Yun Wei, et al.  A step towards worldwide biodiversity assessment: The bioscan-1m insect dataset.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Güldenring and Nalpantidis [2021] Ronja Güldenring and Lazaros Nalpantidis.  Self-supervised contrastive learning on agricultural images.  _Computers and Electronics in Agriculture_ , 191:106510, 2021.
  * Kalamkar et al. [2019] Dhiraj D. Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey.  A study of BFLOAT16 for deep learning training.  _CoRR_ , abs/1905.12322, 2019.  URL http://arxiv.org/abs/1905.12322.
  * Kommineni et al. [2023] Vamsi Krishna Kommineni, Jens Kattge, Jitendra Gaikwad, Susanne Tautenhahn, and Birgitta Koenig-ries.  The role of the clip model in analysing herbarium specimen images.  _Biodiversity Information Science and Standards_ , 7, 2023.
  * Li et al. [2023] Xiang Li, Congcong Wen, Yuan Hu, and Nan Zhou.  Rs-clip: Zero shot remote sensing scene classification via contrastive vision-language supervision.  _International Journal of Applied Earth Observation and Geoinformation_ , 124:103497, 2023.
  * Liaw et al. [2018] Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E. Gonzalez, and Ion Stoica.  Tune: A research platform for distributed model selection and training.  _CoRR_ , abs/1807.05118, 2018.  URL http://arxiv.org/abs/1807.05118.
  * Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization, 2019.
  * Lu et al. [2022] Yuzhen Lu, Dong Chen, Ebenezer Olaniyi, and Yanbo Huang.  Generative adversarial networks (gans) for image augmentation in agriculture: A systematic review.  _Computers and Electronics in Agriculture_ , 200:107208, 2022.
  * Luccioni and Rolnick [2023] Alexandra Sasha Luccioni and David Rolnick.  Bugs in the data: How imagenet misrepresents biodiversity.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 14382–14390, 2023.
  * Mars [2022] Mourad Mars.  From word embeddings to pre-trained language models: A state-of-the-art walkthrough.  _Applied Sciences_ , 12(17):8805, 2022.
  * Mou et al. [2023] Chao Mou, Aokang Liang, Chunying Hu, Fanyu Meng, Baixun Han, and Fu Xu.  Monitoring endangered and rare wildlife in the field: A foundation deep learning model integrating human knowledge for incremental recognition with few data and low cost.  _Animals_ , 13(20):3168, 2023.
  * Müller et al. [2023] Jörg Müller, Oliver Mitesser, H Martin Schaefer, Sebastian Seibold, Annika Busse, Peter Kriegel, Dominik Rabl, Rudy Gelis, Alejandro Arteaga, Juan Freile, et al.  Soundscapes and deep learning enable tracking biodiversity recovery in tropical forests.  _Nature communications_ , 14(1):6191, 2023.
  * Niemiller et al. [2021] K Denise Kendall Niemiller, Mark A Davis, and Matthew L Niemiller.  Addressing ‘biodiversity naivety’through project-based learning using inaturalist.  _Journal for Nature Conservation_ , 64:126070, 2021.
  * Olsen et al. [2019] Alex Olsen, Dmitry Konovalov, Bronson Philippa, Peter Ridd, Jake Wood, Jamie Johns, Wesley Banks, Benjamin Girgenti, O.P. Kenny, James Whinney, Brendan Calvert, Mostafa Rahimi Azghadi, and Ron White.  Deepweeds: A multiclass weed species image dataset for deep learning.  _Scientific Reports_ , 9, 02 2019.  doi: 10.1038/s41598-018-38343-3.
  * Picek et al. [2022] Lukáš Picek, Milan Šulc, Jiří Matas, Thomas S Jeppesen, Jacob Heilmann-Clausen, Thomas Læssøe, and Tobias Frøslev.  Danish fungi 2020-not just another image recognition dataset.  In _Proceedings of the IEEE Winter Conference on Applications of Computer Vision_ , pages 1525–1535, 2022.
  * Piosenka [2023] Gerald Piosenka.  Birds 525 species - image classification, 05 2023.  URL https://www.kaggle.com/datasets/gpiosenka/100-bird-species.
  * Radford et al. [2021a] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  _CoRR_ , abs/2103.00020, 2021a.  URL https://arxiv.org/abs/2103.00020.
  * Radford et al. [2021b] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021b.
  * Roy et al. [2024] DB Roy, J Alison, TA August, M Bélisle, K Bjerge, JJ Bowden, MJ Bunsen, F Cunha, Q Geissmann, K Goldmann, et al.  Towards a standardized framework for ai-assisted, image-based monitoring of nocturnal insects.  _Philosophical Transactions of the Royal Society B_ , 379(1904):20230108, 2024.
  * Sen et al. [2021] Atriya Sen, Beckett Sterner, Nico Franz, Caleb Powel, and Nathan Upham.  Combining machine learning & reasoning for biodiversity data intelligence.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 14911–14919, 2021.
  * Shivaprakash et al. [2022] Kadukothanahally Nagaraju Shivaprakash, Niraj Swami, Sagar Mysorekar, Roshni Arora, Aditya Gangadharan, Karishma Vohra, Madegowda Jadeyegowda, and Joseph M Kiesecker.  Potential for artificial intelligence (ai) and machine learning (ml) applications in biodiversity conservation, managing forests, and related services in india.  _Sustainability_ , 14(12):7154, 2022.
  * Silvestro et al. [2022] Daniele Silvestro, Stefano Goria, Thomas Sterner, and Alexandre Antonelli.  Improving biodiversity protection through artificial intelligence.  _Nature sustainability_ , 5(5):415–424, 2022\.
  * Sprugnoli et al. [2020] Rachele Sprugnoli, Giovanni Moretti, and Marco Passarotti.  Building and comparing lemma embeddings for latin. classical latin versus thomas aquinas.  _IJCoL. Italian Journal of Computational Linguistics_ , 6(6-1):29–45, 2020.
  * Stevens et al. [2023] Samuel Stevens, Jiaman Wu, Matthew J Thompson, Elizabeth G Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M Dahdul, Charles Stewart, Tanya Berger-Wolf, et al.  Bioclip: A vision foundation model for the tree of life.  _arXiv preprint arXiv:2311.18803_ , 2023.
  * Stringham and Izbicki [2020] Nathan Stringham and Mike Izbicki.  Evaluating word embeddings on low-resource languages.  In _Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems_ , pages 176–186, 2020.
  * Unger et al. [2021] Shem Unger, Mark Rollins, Allison Tietz, and Hailey Dumais.  inaturalist as an engaging tool for identifying organisms in outdoor activities.  _Journal of Biological Education_ , 55(5):537–547, 2021.
  * Van Horn et al. [2018] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8769–8778, 2018.
  * Wijayanti et al. [2023] Rini Wijayanti, Masayu Leylia Khodra, Kridanto Surendro, and Dwi H Widyantoro.  Learning bilingual word embedding for automatic text summarization in low resource language.  _Journal of King Saud University-Computer and Information Sciences_ , 35(4):224–235, 2023.
  * Wu et al. [2019] Xiaoping Wu, Chi Zhan, Yukun Lai, Ming-Ming Cheng, and Jufeng Yang.  IP102: A large-scale benchmark dataset for insect pest recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 8787–8796, 2019.
  * Xu et al. [2024] Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer.  Demystifying clip data, 2024.

##  Appendix A Appendix

###  A.1 Background on CLIP and zero-shot classification

Unlike traditional vision models, CLIP jointly trains an image encoder and a
text encoder to predict the correct pairings of a batch of (image, text)
examples, leveraging natural language supervision to enhance generalization
[29]. CLIP’s approach allows it to learn from a wide variety of images and
their associated textual descriptions, making it more flexible and general
compared to standard vision models. This flexibility is crucial for in various
domains, including biodiversity monitoring and agriculture. For instance, CLIP
models analyze digital plant specimen images, aiding in pre-processing and
filtering for further analysis for agriculture purposes [15, 16]. As for
biodiversity, WildCLIP and KI-CLIP facilitate wildlife observation and
monitoring with high accuracy and effectiveness in data-sparse settings [10,
22]. These examples underscore the importance of developing and utilizing
comprehensive datasets to fully leverage the capabilities of CLIP models in
advancing biodiversity and agricultural research.

###  A.2 The value of taxonomic information

Taxonomic classification, the hierarchical arrangement of organisms into
categories based on shared characteristics, is foundational in biological
sciences. Taxonomy underpins various scientific, ecological, and agricultural
applications. It allows for precise identification and classification of
species, which is fundamental for understanding biodiversity and monitoring
ecosystems. For instance, accurate species identification can aid in tracking
invasive species, as noted in studies such as [33]. In agriculture, detailed
taxonomic information helps in identifying pests and beneficial species,
thereby improving pest control strategies and crop management; supports
ecological research by providing insights into species interactions,
distribution patterns, and evolutionary relationships [12]; and is essential
for policy-making and conservation planning [31].

###  A.3 Scientific versus common names

Although we identify the importance and need to include taxonomic information
in the dataset for biodiversity, one potential challenge is the fact that this
information is mostly in Latin for which text embedding models often exhibit
suboptimal performance due to its status as a low-resource language [36].
Nonetheless, Latin remains indispensable as it is the standard for
representing scientific names and taxonomic classifications. We therefore
integrate common names, scientific names, and detailed taxonomic hierarchies.
We believe that such an “all-encompassing” approach facilitates the learning
of relationships between Latin and English terms, thereby improving the
models’ applicability in scientific contexts [6, 34, 39]. Furthermore,
incorporating taxonomic data into the training process significantly enhances
the multimodal capabilities of the models, enabling them to associate visual
data with taxonomic terminology [21, 2].

###  A.4 iNaturalist, iNaturalist Open Data

iNaturalist is an online social network for sharing biodiversity information
and learning about nature. It serves as a crowdsourced species identification
system and organism occurrence recording tool. Users from around the world
upload images, making the continuously updated dataset valuable for AI
applications in biodiversity and research. Each photo includes detailed
metadata: copyright status, location, uploader, time, and taxonomic
classification. This diversity in image sources makes iNaturalist an excellent
dataset for training AI models intended for real-world applications [37, 24,
7, 3]. Despite its vast and diverse data, iNaturalist is not directly
optimized for AI researchers: arranging this data for use in AI models like
CLIP is not straightforward. Each photo has its own page on the iNaturalist
website, making it difficult to download images along with all the necessary
information in a streamlined manner.

The iNaturalist Open Dataset aims to address some of these challenges. It is
one of the world’s largest public datasets of photos of living organisms,
structured as a "bucket" of images stored using Amazon Web Service’s Simple
Storage Service (S3). The dataset includes multiple resized versions of each
photo, allowing users to download the size most useful to their research.

Additionally, the dataset provides four tab-separated CSV files representing
observations, observers, photos, and taxa_id. These files are generated
monthly, capturing a snapshot of the continually changing iNaturalist data.
The images in the iNaturalist Open Dataset are licensed under either CC0, CC-
BY, or CC-BY-NC and are open for public research. Photos with a CC0 license
can be attributed as "[observer name or login], no rights reserved (CC0)".
Photos with other Creative Commons licenses can be attributed as "© [observer
name or login], some rights reserved ([license abbreviation])".

###  A.5 iNaturalist Details

Each image in the iNaturalist Open Dataset can be associated with its
appropriate metadata through a group of four metadata CSV files,

representing photos, observations, taxa, and observers.

The photos metadata file contain nine distinct columns of metadata information
of each photo. Of these columns, only photo_id and observation_uuid are
relevant for us. The value of photo_id is a identifier number used to access
individual photos, the photo’s iNaturalist page can be found by constructing a
URL in this format: https://www.inaturalist.org/photos/[photo_id]. The value
of observation_uuid indicates which observation the photo is associated with,
it is used to map the photos metadata to the observations metadata.

An observation represents one user submission of a species encounter to the
iNaturalist website. One observation can have multiple photos of the same
species but never multiple species. The observation metedata file contains
eight distinct columns of metadata information on each observation. The
columns relevant to us are observation_uuid, quality grade, and taxon_id. Each
observation is given a unique number identifier indicated by its
observation_uuid. iNaturalist has its own system to determining the quality of
an observation and its associated photos, quality_grade represents this and
can range from "Casual", "Research Grade", or "Needs ID". The value taxon_id
indicates the species is represented in the observation, it is used to map the
observations metadata to the taxa metadata.

The taxa metadata file contains information about each specific taxon in
iNaturalist, it has has six distinct metadata columns. The columns relevant to
us are taxon_id, name, ancestry, and active. Each specific taxon in
iNaturalist has a unique identifier number associated with it, this is its
taxon_id. This taxon_id will map to the scientific name of the taxon which is
represented in the name metadata column. Each taxon also has associated with
it a taxonomic ancestry, this is represented as a string of taxon_ids
concatenated together with "$\backslash$" like so
"48460/1/47115/47584/1051154". The active column indicated whether the taxon
is currently in use in iNaturalist.

The observer metadata file comtains information about each user within the
iNaturalist site. For the purpose of machine learning research none of its
three metadata columns are relevant.

While the iNaturalist Open Dataset metadata files provide a plethora of
interesting information, its structure makes it inherently cumbersome to use
for research. To solve this, we aggregate and process the iNaturalist metadata
into a concise and streamlined format for easy query and usage.

First, the respective CSV files are used to populate a SQL database with each
CSV file as its own SQL table. A new aggregate SQL table is created that joins
the photos, observations, and taxa tables on its relational columns. Only the
metadata columns we deemed relevant are kept and the extraneous non-useful
metadata columns are discarded.

One of the difficulties working with the base iNaturalist metadata files is
that it does not contain the image URL, information that is critical in image
downloads. We include a new column in the aggregated metadata table that
explicitly links to the Amazon S3 URL in which the image is hosted.

The Arboretum metadata file used for model training contains the metadata
columns phylum, class, order, family, genus, species, scientific_name,
common_name for the seven Arboretum categories Aves, Arachnida, Insecta,
Plantae, Fungi, Mollusca, and Reptilia. To ensure that only images and
metadata from the seven Arboretum categories appear in our final dataset we
use the taxa table to find the taxon in our categories then use it in a SQL
query on the ancestry column of our aggregated metadata table.

The taxonomic rank columns are also found utilizing the ancestry metadata
column. A difficulty in working with the ancestry metadata is present in that
there is not a clear indication of what taxonomic rank a taxon id represents
the ancestry string. This problem is exacerbated due to the presence of
taxonomic ranks and dsub ranks whose presence is variable across different
species. As such, a custom function is applied to each row to dynamically find
the rank of each taxon id in the ancestry and then appropriately populate the
taxon id to a metadata column of that rank. This process results in all
taxonomies rank represented as metadata columns; only phyllum, class, order,
family, genus and species are kept in the Arboretum metadata file.

The scientific name of a species is found using the name metadata column of
our aggregated metadata table. The common name of a species is also useful
metadata information. Unfortunately, the iNaturalist Open Data metadata files
do not contain the common name information of a species. To address this, we
curate a lookup table of the common names in our dataset. This is obtained
from the iNaturalist Taxonomy DarwinCore Archive, Having obtained the common
names for each species, we append it to the Arboretum-specific metadata.

###  A.6 Composition of Arboretum-40M

See Figure 5 and Table 6.

Figure 5: Arboretum-40M Dataset Analysis: a) Consistent category distribution across Arboretum-40M and Arboretum-116M datasets. b) Species exhibit a long-tailed distribution. c) Impact of local vs. semi-global shuffling on species representation within training minibatches. Table 6: Number of Unique Species in Each Category in Arboretum-Balanced.  Category | Number of Unique Species
---|---
Kingdom: Fungi | 281
Kingdom: Plantae | 500
Phylum: Mollusca | 147
Class: Insecta | 500
Class: Arachnida | 136
Class: Reptilia | 189
Class: Aves | 500

##  Appendix B ArborCLIP training details

We use Arboretum-40M to train new CLIP-style foundation models, and then
evaluate them on zero-shot image classification tasks. Following the
implementation of Stevens et al. [35], we utilize a ViT-B/16 architecture
initialized from the OpenAI pretrained weights for our main model, and train
for 40 epochs.

In addition, we also train a ViT-L/14 model from the MetaCLIP [41] checkpoint
for 12 epochs, and a ViT-B/16 from the BioCLIP checkpoint for 8 epochs. We
select the AdamW optimizer from Loshchilov and Hutter [18] along with a cosine
learning rate scheduler, as this has previously been shown to perform well for
CLIP pretraining [28]. We conduct twenty rounds of hyperparameter optimization
using Ray Tune [17] to determine the optimal learning rate, $\beta_{1}$,
$\beta_{2}$ and weight decay settings.

We train our models for a combined 10 days on 8xH100 nodes in bfloat16
precision [14] with gradient checkpointing, computing loss with local
features, and utilizing static graph optimization for DDP.

##  Appendix C Additional ArborCLIP results

In Table 7, we report model performance at different levels of the taxonomic
hierarchy. Generally, we find that models trained on web-scraped data perform
better with common names, whereas models trained on specialist datasets
perform better when using scientific names. Additionally, models trained on
web-scraped data excel at classifying at the highest taxonomic level
(kingdom), while models begin to benefit from specialist datasets like
Arboretum-40M and Tree-of-Life-10M at the lower taxonomic levels (order and
species).

However, ArborCLIP shows a performance decline at taxonomic levels below the
species level. This is likely because our training metadata structure allows
for classifications solely by referring to species information. From a
practical standpoint, this is not problematic for the species in our test set
since ArborCLIP is highly accurate at the species level, and higher-level taxa
can be deterministically derived from the lower ones.

Furthermore, the OpenCLIP and MetaCLIP baselines outperform ArborCLIP on the
life stages benchmark. This highlights the importance of retaining the general
linguistic capabilities of the pretrained CLIP models for hybrid tasks.

Table 7: Different levels of taxonomic classification reveal the strengths and
weaknesses of different models. DS stands for dataset, Arch for model
architecture AU stands for Arboretum-Unseen, AB stands for Arboretum-Balanced,
BCR for BioCLIP-Rare, K for Kingdom taxonomy, O for Order taxonomy, and CN for
species taxonomy using common names.

DS | Arch | AU-K | AU-O | AU-CN | AB-K | AB-O | AB-CN | BCR-CN
---|---|---|---|---|---|---|---|---
OpenAI | ViT-B-16 | 78 | 26.7 | 18.6 | 79.9 | 26.7 | 13.1 | 31.6
BioCLIP | ViT-B-16 | 75.5 | 38.4 | 41.8 | 73.2 | 34.5 | 42.7 | 32.1
MetaCLIP-cc | ViT-L-14 | 90.3 | 33.2 | 27.9 | 87.1 | 34.5 | 19.9 | 42.4
ArborCLIP from OpenAI | ViT-B-16 | 62.9 | 13.5 | 29.7 | 56.5 | 12.4 | 70.8 | 22.2
ArborCLIP from BioCLIP | ViT-B-16 | 62.8 | 17.8 | 24 | 65.7 | 14.5 | 52.6 | 23.6
ArborCLIP from MetaCLIP | ViT-L-14 | 68.3 | 9.6 | 4.3 | 58.1 | 3.8 | 15.7 | 8.8