Discovering Novel Biological Traits From Images Using Phylogeny-Guided
  Neural Networks
Discovering evolutionary traits that are heritable across species on the tree
of life (also referred to as a phylogenetic tree) is of great interest to
biologists to understand how organisms diversify and evolve. However, the
measurement of traits is often a subjective and labor-intensive process, making
trait discovery a highly label-scarce problem. We present a novel approach for
discovering evolutionary traits directly from images without relying on trait
labels. Our proposed approach, Phylo-NN, encodes the image of an organism into
a sequence of quantized feature vectors -- or codes -- where different segments
of the sequence capture evolutionary signals at varying ancestry levels in the
phylogeny. We demonstrate the effectiveness of our approach in producing
biologically meaningful results in a number of downstream tasks including
species image generation and species-to-species image translation, using fish
species as a target example.
Mohannad Elhamod, Mridul Khurana, Harish Babu Manogaran, Josef C. Uyeda, Meghan A. Balk, Wasila Dahdul, Yasin Bakış, Henry L. Bart Jr., Paula M. Mabee, Hilmar Lapp, James P. Balhoff, Caleb Charpentier, David Carlyn, Wei-Lun Chao, Charles V. Stewart, Daniel I. Rubenstein, Tanya Berger-Wolf, Anuj Karpatne
# Discovering Novel Biological Traits From Images Using Phylogeny-Guided Neural Networks

###### Abstract.

Discovering evolutionary traits that are heritable across species on the tree of life (also referred to as a phylogenetic tree) is of great interest to biologists to understand how organisms diversify and evolve. However, the measurement of traits is often a subjective and labor-intensive process, making _trait discovery_ a highly label-scarce problem. We present a novel approach for discovering evolutionary traits directly from images without relying on trait labels. Our proposed approach, _Phylo-NN_, encodes the image of an organism into a sequence of quantized feature vectors-or codes-where different segments of the sequence capture evolutionary signals at varying ancestry levels in the phylogeny. We demonstrate the effectiveness of our approach in producing biologically meaningful results in a number of downstream tasks including species image generation and species-to-species image translation, using fish species as a target example.1

computer vision, neural networks, phylogeny, morphology, knowledge-guided machine learning +
Footnote †: journal: Computer vision, neural networks, “Regularization”,

## 1. Introduction

One of the grand challenges in biology is to find features of organisms- or _traits_-that define groups of organisms, their genetic and developmental underpinnings, and their interactions with environmental selection pressures (Battelle, 2016). Traits can be physiological, morphological, and/or behavioral (e.g., beak color, stripe pattern, and fin curvature) and are integrated products of genes and the environment. The analysis of traits is critical for predicting the effects of environmental change or genetic manipulation, and to understand the process of evolution. For example, discovering traits that are heritable across species on the tree of life (also referred to as the _phylogenetic tree_), can serve as a starting point for linking traits to underlying genetic factors. Traits with such genetic or phylogenetic signal, termed _evolutionary traits_, are of great interest to biologists, as the history of genetic ancestry captured by such traits can guide our understanding of how organisms diversify and evolve. This understanding enables tasks such as estimating the morphological features of ancestors, understanding how species have responded to environmental changes, and even predicting the potential future course of trait changes (Battelle, 2016; Mabee, 2016). However, the measurement of traits is not straightforward and often relies on subjective and labor-intensive human expertise and definitions (Battelle, 2016). Hence, _trait
_discovery_ has remained a highly label-scarce problem, hindering rapid scientific advancement (Krishnan et al., 2017).

With the recent availability of large-scale image repositories containing millions of images of biological specimens (Zhu et al., 2017; Zhang et al., 2018; Zhang et al., 2018), there is a great opportunity for machine learning (ML) to contribute to the problem of trait discovery (Krishnan et al., 2017). In particular, advances in deep learning have enabled us to extract useful information from images and to map them to structured feature spaces where they can be manipulated in a number of ways. We ask the question: _how can we develop deep learning models to discover novel evolutionary traits automatically from images without using trait labels?_

Despite the biological relevance of this question, answering it is challenging for two main reasons. First, not all image features extracted by a deep learning model for ML tasks such as image reconstruction or species classification will exhibit evolutionary signals. Hence, it is important to disentangle the image features of an organism that preserve evolutionary information, from remaining features influenced by unrelated factors (Beng et al., 2016). Second, information about evolutionary signals is not available as a set of known attributes (or trait labels) but rather in the form of structured knowledge of how species are related to each other in the phylogenetic tree (see Figure 1). Without access to trait labels, current methods for feature disentanglement in deep learning (Beng et al., 2016; Krishnan et al., 2017) are unfit for discovering evolutionary traits. Furthermore, current standards in deep learning for generative modeling (Zhu et al., 2017; Zhang et al., 2018) or interpretable ML (Beng et al., 2016; Krishnan et al., 2017) are unable to leverage structured forms of biological knowledge (e.g., phylogenetic trees) in the learning of image features, and hence are unable to analyze and manipulate learned features in biologically meaningful ways.

We propose a novel approach for discovering evolutionary traits automatically from images, termed _phylogeny-guided neural networks_ (_Phylo-NN_), which encodes the image of an organism into a sequence of quantized feature vectors or "codes" (see Figure 1). A unique feature of the image-derived sequences learned by _Phylo-NN_ is that different segments of the sequence capture evolutionary information at varying ancestry levels in the phylogeny, where every level corresponds to a certain point of time in evolutionary history. Analogous to how the genome of an organism encodes all its genetic information and structures it as a set of genes, our image-derived sequences encodes all of the visual information contained in the organism's image and structures it as a set of evolutionary traits shared with ancestor nodes within its lineage (at different levels of phylogeny). We thus refer to the image-derived sequences of _Phylo-NN_ as _Imageomes_, a brand-new concept in evolutionary biology. By analyzing and manipulating the codes in the Imageomes of organisms, we can perform a number of biologically meaningful downstream tasks such as species image generation, species-to-species image translation, and visualization of evolutionary traits. We demonstrate the effectiveness of _Phylo-NN_ in solving these tasks using fish species as a target example.

Our work, for the first time, provides a bridge between the "language of evolution" represented as phylogeny and the "language of images" extracted by _Phylo-NN_ as Imageomes. This work is part of a larger-scale effort to establish a new field of research in "Imageomics" (Zhu et al., 2017), where images are used as the source of information to accelerate biological understanding of traits, ranging from their selective consequences to their causation. Our work also provides a novel methodological advance in the emerging field of knowledge-guided machine learning (KGML) (Krishnan et al., 2017; Krishnan et al., 2017; Krishnan et al., 2017) by using tree-based knowledge to structure the embedding space of neural networks and produce scientifically meaningful image generation and translation results.

## 2. Background and Related Work

**What is a Phylogenetic Tree?** A phylogenetic tree visually characterizes the evolutionary distances among a set of species and their common ancestors represented as nodes of the tree. In this tree, the length of every edge is a value that represents the evolutionary distance between two nodes (measured in time intervals representing thousands or millions of years), which is estimated from living species and time-calibrated ages using dated fossil ancestors or molecular methods. While rates of change along different edges may vary substantially, on average we expect that longer edges will accumulate higher levels of evolutionary trait change than shorter edges. In our work, we consider discretized versions of the phylogenetic tree with \(n_{\text{l}}=4\) ancestry levels, such that every species class (leaf node in the tree) has exactly \(m_{\text{l}}-1\) ancestors. Every ancestry level corresponds to a certain point of time in evolutionary history. See Appendix B for details on phylogeny preprocessing.

**Generative Modeling for Images:** There exists a large body of work in deep learning for image generation, including methods based on Variational Autoencoders (VAEs) (Krishnan et al., 2017), Generative Adversarial Networks (GANs) (Krishnan et al., 2017; Krishnan et al., 2017; Zhang et al., 2018; Zhang et al., 2018), Transformer networks (Zhu et al., 2017; Zhang et al., 2018), and Diffusion models (Beng et al., 2016). While some recent advances in this field (e.g., DALL-E 2) have been shown to produce images with very high visual quality, they involve large and complex embedding spaces that are difficult to structure and analyze using tree-based knowledge (e.g., phylogeny). Instead, we build upon a recent line of work in generative modeling using vector-quantized feature representations of images (Zhu et al., 2017; Zhang et al., 2018) that are easier to manipulate than continuous features. In particular, a recent variant of the VAE, termed Vector-Quantized VAE (VQVAE) (Zhang et al., 2018), uses discrete feature spaces quantized using a learned codebook of feature vectors and employs a PixelCNN (Zhu et al., 2017) model for sampling in the discrete feature

Figure 1. _Phylo-NN_ converts images to discrete sequences of features (called Imageomes) where different sequence segments (shown in distinct colors) capture evolutionary information at varying ancestry levels of phylogeny (L1 to L4).
space. This work was extended in [14] to produce VQGAN, which is different from VQVAE in two aspects. First, it adds a discriminator to its framework to improve the quality of the generated images. Second, it uses a Transformer model, namely the GPT architecture [39], to generate images from the quantized feature space instead of a PixelCNN. VQGAN is a state-of-the-art method that generates images of better quality efficiently at higher resolutions than other counterparts such as StyleGAN [23] and Vision Transformers [11; 17]. Our work draws inspiration from VQGAN to embed images in discrete feature spaces (analogous to the discrete nature of symbols used in genome sequences) but with the grounding of biological knowledge available as phylogenetic trees.

**Interpretable ML:** There is a growing trend in the ML community to focus on the interpretability of deep learning features [12]. Some of the earliest works in this direction include the use of saliency scores [44] and Class Activation Maps (CAMs) [42] that reveal sensitive regions of an image influencing classification decisions. However, these methods are known to be noisy and often imprecise [1]. Recent work includes the ProtoPNet model [4], which first learns a set of template image patches (or prototypes) for each class during training, and then uses those templates to both predict and explain the class label of a test image. These methods suffer from two drawbacks. First, they do not allow for structured knowledge to guide the learning of interpretable features and hence are not designed to produce results that are _biologically meaningful_. Second, they are mostly developed for classification problems and cannot be directly applied to image generation or translation problems.

**Disentangling ML Features:** Another related line of research involves disentangling the feature space of deep learning models to align the disentangled features with target "concepts." This includes the approach of "Concept whitening" (CW) [5], where the latent space of a classification model is whitened (i.e., normalized and decorrelated) such that the features along every axis of the latent space corresponds to a separate class. Another approach in this area is that of Latent Space Factorization (LSF) [28], where the latent space of an autoencoder is linearly transformed using matrix subspace projections to partition it into features aligned with target concepts (or attributes) and features that do not capture attribute information. Note that our proposed _Phylo-NN_ model can also be viewed as a latent space disentanglement technique, where the disentangled segments of the learned Imageome correspond to different ancestry levels of the phylogeny. We thus use CW and LSF as baselines in our experiments to test if it is possible to discover evolutionary traits just by disentangling the latent space using species classes as orthogonal concepts, without using the structured knowledge of how species are related to one another in the phylogeny.

**Knowledge-Guided ML:** KGML is an emerging area of research that aims to integrate scientific knowledge in the design and learning of ML models to produce generalizable and scientifically valid solutions [22]. Some examples of previous research in KGML include modifying the architecture of deep learning models to capture known forms of symmetries and invariances [2; 51], and adding loss functions that constrain the model outputs to be scientifically consistent even on unlabeled data [9; 40]. In biology, KGML methods have been developed for species classification that leverage the knowledge of taxonomic grouping of species [10; 13]. KGML methods have also been developed for generative modeling of images using domain knowledge available as knowledge graphs or ontologies [15; 37]. In contrast to these prior works, we focus on structuring the embedding space of neural networks using tree-based knowledge (i.e., phylogeny) to enable the discovery and analysis of novel evolutionary traits automatically from images.

## 3. Proposed Approach: Phylo-Nn

We consider the problem of discovering novel (or "unknown") evolutionary traits automatically from images without using any trait labels or knowledge of how the unknown traits correspond to known concepts in a knowledge graph or ontology. We only use the "distant" supervision of how these unknown traits have evolved over time and are shared across species, available in the form of the phylogenetic tree. Figure 2 provides an overview of our proposed _Phylo-NN_ model. Our method can operate on the latent space of any backbone encoder model \(E\) that takes in images as input and produces continuous feature maps \(\mathbf{x}\) as output. There are three computing blocks in _Phylo-NN_ as shown in Figure 2. The first block, Phylo-Encoder (\(PE\)), takes continuous feature maps \(\mathbf{x}\) as input and generates quantized feature sequences (or Imageomes) as output. Imageome sequences \(\mathbf{z}^{Q}\) comprise of two _disentangled_ segments: \(\mathbf{z}^{Q}_{\text{p}}\), which captures phylogenetic information (p) at varying ancestry levels, and \(\mathbf{z}^{Q}_{\text{np}}\), which captures non-phylogenetic information (np) that is still important for image reconstruction but is unrelated to the phylogeny. The second block, Phylo-Decoder (\(PD\)), maps the Imageone sequences back to the space of feature maps \(\mathbf{\hat{x}}\), such that \(\mathbf{\hat{x}}\) is a good reconstruction of \(\mathbf{x}\). We then feed \(\mathbf{\hat{x}}\) into a backbone decoder model \(D\) that reconstructs the original image. Note that in the training of \(PE\) and \(PD\) models, both the backbone models \(E\) and \(D\) are kept frozen, thus requiring low training time. _Phylo-NN_ can thus be plugged into the latent space of any powerful encoder-decoder framework. The third block of _Phylo-NN_ is a transformer model \(T\) that takes in the species class variable as input, and generates a distribution of plausible Imageome sequences corresponding to the class as output. These sequences can be fed to the \(PD\) model to generate a distribution of synthetic images. In the following, we provide details on each of the three blocks of _Phylo-NN_.

### Phylo-Encoder (PE) Block

Figure 3 shows the sequence of operations that we perform inside the \(PE\) block. We first apply a convolutional layer on \(\mathbf{x}\) to produce feature maps of size (\(H\times W\times C\)), where \(C\) is the number of channels. We split these \(C\) feature maps into two sets. The first \(C_{\text{p}}\) maps are fed into an MLP layer to learn a global set of feature vectors \(\mathbf{z}_{\text{p}}\) capturing phylogenetic information. The size of \(\mathbf{z}_{\text{p}}\) is kept equal to

Figure 2. Overview of proposed _Phylo-NN_ model architecture.
(\(n_{\eta}n_{\mathrm{p}}\times d\)), where \(n_{\eta}\) is the number of phylogeny levels, \(n_{\mathrm{p}}\) is the number of feature vectors we intend to learn at every phylogeny level, and \(d\) is the dimensionality of feature vectors. Similarly, the remaining \(C-C_{\mathrm{p}}\) maps are fed into an MLP layer to produce a set of feature vectors \(\mathbf{z}_{\mathrm{np}}\) capturing non-phylogenetic information of size (\(n_{\mathrm{np}}\times d\)).

**Vector Quantization:** Both \(\mathbf{z}_{\mathrm{p}}\) and \(\mathbf{z}_{\mathrm{np}}\) are converted to _quantized_ sequences of feature vectors, \(\mathbf{z}_{\mathrm{p}}^{Q}\) and \(\mathbf{z}_{\mathrm{np}}^{Q}\), respectively, using the approach developed in VQVAE (Sutton et al., 2017). The basic idea of this quantization approach is to learn a set (or codebook) of \(n_{\mathrm{q}}\) distinct feature vectors (or codes), such that every feature vector in \(\mathbf{z}_{\mathrm{p}}\) and \(\mathbf{z}_{\mathrm{np}}\) is replaced by its nearest counterpart in the codebook. This is achieved by minimizing the _quantization loss_, \(L_{\mathrm{q}}=|\mathbf{z}-\mathbf{z}^{Q}|\). The advantage of working with quantized vectors is that every feature vector in \(\mathbf{z}_{\mathrm{p}}^{Q}\) and \(\mathbf{z}_{\mathrm{np}}^{Q}\) can be referenced just by its location (or index) in the codebook. This allows for faster feature manipulations in the space of discrete code positions than continuous feature vectors.

**Using phylogenetic knowledge in \(\mathbf{z}_{\mathrm{p}}^{Q}\)**: Here, we describe our approach to ensure that the quantized feature sequence \(\mathbf{z}_{\mathrm{p}}^{Q}\) contains phylogenetic information. Note that \(\mathbf{z}_{\mathrm{p}}^{Q}\) contains \(n_{\eta}\) sub-sequences of length \(n_{\eta}\), where every sub-sequence corresponds to a different ancestry level in the phylogeny. While the first sub-sequence \(S_{1}\) should ideally capture information contained in \(\mathbf{x}\) that is necessary for identifying ancestor nodes at level 1 of the phylogeny, \(S_{2}\) should contain additional information that when combined with \(S_{1}\) is sufficient to identify the correct ancestor node of \(\mathbf{x}\) at level 2. In general, we define the concept of a _Phylo-descriptor_\(D_{i}=\{S_{1},S_{2},\ldots,S_{i}\}\) of \(\mathbf{x}\) that contains the necessary information for identifying nodes at level \(i\) (see Figure 3). We feed \(D_{i}\) to an MLP layer that predicts the class probabilities of nodes at level \(i\), which are then matched with the correct node class of \(\mathbf{x}\) at level \(i\), \(c_{i}(\mathbf{x})\), by minimizing the following _phylogeny-guided loss_, \(L_{\mathrm{p}}\):

\[L_{\mathrm{p}}=\sum_{i=0}^{m}\beta_{i}\mathrm{CE}(\mathrm{MLP}_{i}(D_{i}( \mathbf{x})),c_{i}(\mathbf{x})), \tag{1}\]

where \(\mathrm{CE}\) is the cross-entropy loss and \(\beta_{i}\) is the weighting hyperparameter for level \(i\).

**Disentangling \(\mathbf{z}_{\mathrm{p}}^{Q}\) and \(\mathbf{z}_{\mathrm{np}}^{Q}\)**: While minimizing \(L_{\mathrm{p}}\) guides the learning of \(\mathbf{z}_{\mathrm{p}}^{Q}\) to contain phylogenetic information, we still need a way to ensure that \(\mathbf{z}_{\mathrm{np}}^{Q}\) focuses on complementary features and does not contain phylogenetic information. To achieve this, we first apply an orthogonal convolution loss \(L_{\mathrm{o}}\) (originally proposed in (Sutton et al., 2017)) to the convolutional layer of Phylo-Encoder, to constrain the \(C\) convolutional kernels to be orthogonal to each other. To further ensure that \(\mathbf{z}_{\mathrm{np}}^{Q}\) has no phylogenetic information, we also employ an adversarial training procedure to incrementally remove phylogenetic information from \(\mathbf{z}_{\mathrm{np}}^{Q}\). In particular, we apply an MLP layer _MLP_adv on \(\mathbf{z}_{\mathrm{np}}^{Q}\), and then train the parameters of _MLP_adv to minimize the following _adversarial loss_:

\[L_{\mathrm{adv}}=\sum_{i=0}^{n_{\eta}}\beta_{i}\mathrm{CE}(\mathrm{MLP}_{i}( \mathbf{z}_{\mathrm{np}}^{Q}(\mathbf{x}))),c_{i}(\mathbf{x})), \tag{2}\]

This is aimed at training _MLP_adv to detect any phylogenetic information contained in \(\mathbf{z}_{\mathrm{np}}^{Q}\). Simultaneously, we train the rest of _Phylo-NN_'s parameters to maximize \(L_{\mathrm{adv}}\), such that \(\mathbf{z}_{\mathrm{np}}^{Q}\) becomes irrelevant for the task of identifying nodes in the phylogeny and only contains non-phylogenetic information.

## 4. Phylo-Decoder (PD) Block

The goal of the PD block is to convert the space of Imageome sequences, \(\mathbf{z}^{Q}=\{\mathbf{z}_{\mathrm{p}}^{Q},\mathbf{z}_{\mathrm{np}}^{Q}\}\), back to the space of original feature maps, \(\mathbf{x}\). The sequence of operations in \(PD\) is almost a mirror image of those used in _PE_. We first pass \(\mathbf{z}_{\mathrm{p}}^{Q}\) and \(\mathbf{z}_{\mathrm{np}}^{Q}\) through two MLPs, and then concatenate their outputs to create feature maps of size (\(H\times W\times C\)). These feature maps are then fed into a convolutional layer to produce \(\hat{\mathbf{x}}\). Minimizing the reconstruction loss, \(L_{\mathrm{rec}}=|\hat{\mathbf{x}}-\mathbf{x}|\), ensures that \(\hat{\mathbf{x}}\) is a good approximation of \(\mathbf{x}\). Finally, _PE_ and _PD_ are jointly trained using a weighted summation of all the losses mentioned above.

### Transformer (T) Block

Once _PE_ and _PD_ are trained, we can extract Imageome sequences \(\mathbf{z}^{Q}\) for every image in the training set. The goal of the Transformer block is to learn the patterns of codes in the extracted Imageome sequences of different classes (e.g., species class or ancestor node class), and use these patterns to generate synthetic Imageome sequences for every class. To achieve this task, we follow the approach used by VQGAN (Krizhevsky et al., 2014) and train a GPT transformer model (Zhu et al., 2017)\(T_{i}\) to generate plausible sequences of \(\mathbf{z}^{Q}\) for every node class at level \(i\). The generated Imageome sequences can then be converted into synthesized specimen images using _PD_ and \(D\).

## 5. Evaluation Setup

### Data

We used a curated dataset of teleost fish images from five ichthytological research collections that participated in the Great Lakes Invasics Network Project (GLIN). After obtaining the raw images from these collections, we handpicked a subset of about \(11,000\) images and pre-processed them by resizing and appropriately padding each image to be of a \(256\times 256\) pixel resolution. Finally, we partitioned the images into a training set and a validation set using an \(80-20\) split. See Appendix A for details on data pre-processing.

Our dataset includes images from \(38\) species of teleost fishes with an average number of \(200\) images per species. We discretized the phylogenetic tree to have \(n_{\eta}=4\) ancestry levels, where the last

Figure 3. Detailed view of the Phylo-Encoder block.
level is the species class. See Appendix B for details on phylogeny selection and discretization.

### Backbone Encoder and Decoder

Since _Phylo-NN_ can operate on the feature space \(\mathbf{x}\) of any backbone encoder \(E\) and produce reconstructed feature maps \(\mathbf{\hat{x}}\) that can be decoded back to images by a corresponding backbone decoder \(D\), we tried different encoder-decoder choices including pix2pix (Dordes and Torr, 2017), ALAE (Srivastava et al., 2017), and StyleGAN (Srivastava et al., 2017). However, we found VQGAN (Srivastava et al., 2017) feature maps to produce images of better visual quality than other encoder-decoder models. Hence, we used the embeddings of a base VQGAN encoder \(E\) as inputs in _Phylo-NN_ for all our experiments. The reconstructed feature maps of _Phylo-NN_ were then fed into a base VQGAN quantizer serving as the backbone decoder \(D\). Note that while training _Phylo-NN_, we kept the parameters of the backbone models fixed, thus saving training time and resources.

### Baseline Methods

Since no direct baselines exist for structuring the embedding space of neural networks using tree-based knowledge or discovering novel evolutionary traits from images, we considered the following baselines that are closest in motivation to _Phylo-NN_:

**Vanilla VQGAN (Srivastava et al., 2017):** The first baseline that we consider is a vanilla VQGAN model trained to generate and reconstruct images on the fish dataset. By comparing the learned embeddings and generated images of _Phylo-NN_ with vanilla VQGAN, we aim to demonstrate the importance of using biological knowledge to structure the embedding space of neural networks for trait discovery, rather than solely relying on information contained in data.

**Concept whitening (CW) (Beng et al., 2015):** For this second baseline, we replaced the last normalization layer in the encoder block of vanilla VQGAN with the concept whitening (CW) module, where we used species class labels as concept definitions. This is intended to evaluate if CW is capable of disentangling the evolutionary traits of species automatically from images without using the phylogeny. The whitened embeddings \(z_{\text{CW}}\) produced by the CW module are fed into the quantizer module of vanilla VQGAN for converting the embeddings to images. While training the CW module, we optimized the whitening and rotation matrices for all concepts every 30 batches. We used the VQGAN's transformer to generate plausible feature sequences \(z_{\text{cw}}\) conditioned on the species label, which are then decoded into specimen images using the VQGAN's decoder.

**Latent Space Factorization (LSF) (Krizhevsky et al., 2014):** The third baseline that we considered is the LSF method, which is another approach for feature disentanglement given concept attribute labels. Specifically, we introduced a variational autoencoder (VAE) model between the encoder and the quantization layer of the base VQGAN model. Similar to CW, we used the species class of each image as the concept attribute for factorizing the latent space in LSF. The LSF module was trained to optimize VAE's KL-divergence loss and recreation loss along with the attribute and non-attribute losses, as originally defined in the LSF method (Krizhevsky et al., 2014).

## 6. Results

In the following, we analyze the results of _Phylo-NN_ from multiple angles to assess the quality of its learned embeddings and generated images in comparison with baseline methods.

### Validating Species Distances in the Embedding Space

In order to evaluate the ability of _Phylo-NN_ to extract novel (or unknown) evolutionary traits from images without using trait labels, we show that distances between species pairs in the embedding space of _Phylo-NN_ are biologically meaningful and are correlated with ground-truth values better than baseline methods. In the following, we describe the two types of ground-truths used, the approach used for computing distances in the embedding space of comparative methods, and the comparison of correlations with ground-truth values.

**Phylogenetic Ground-truth (GT):** The first ground-truth distance between pairs of species is the _evolutionary distance_ between their corresponding nodes in the phylogenetic tree. In particular, for any two species, we can calculate the total sum of edge lengths in the path between their nodes in the phylogenetic tree. The longer the path, the more distant the species are on the evolutionary scale. Hence, if _Phylo-NN_ indeed captures evolutionary traits in its embedding space, we would expect it to show higher correlations with evolutionary distances computed from the phylogeny as compared to baselines. We applied min-max scaling of evolutionary distances so that they range from 0 to 1.

**Morphological Ground-truth (GT):** Another type of ground-truth distance between species was computed based on measurements of known morphological traits obtained from the FishShapes v1.0 dataset (Srivastava et al., 2017), which contains expert-measured traits known to carry evolutionary signals, defined and collected using traditional methods that are subjective and labor-intensive. We specifically used 8 functionally relevant traits from this dataset for every fish species. Some species were not available in this dataset, so when possible, either the closest relative was substituted or the species was dropped. The species were then matched to a time-calibrated phylogeny of fishes (Beng et al., 2015; Wang et al., 2016) and the log-transformed measurements were rotated with phylogenetically-aligned components analysis (PACA) (Beng et al., 2015), which rotates the traits to the axis with the highest level of phylogenetic signal. After correcting for overall size and allometry, the principal components of PACA were used to compute the Mahalonobis distance between every species-pair, using a covariance matrix proportional to the evolutionary rate matrix. See Appendix D for details on PACA calculations.

**Computing Embedding Distances:** To compute pair-wise distances in the embedding space of _Phylo-NN_, we first compute the probability distributions (or histograms) of quantized codes at every position of the Imageome sequence (i.e., \(\mathbf{z}_{\text{p}}^{Q}\) and \(\mathbf{z}_{\text{np}}^{Q}\)) in the test images for every species. We then compute the _Jensen-Shannon (JS) divergence_(Srivastava et al., 2017) between the probability distributions of codes at a pair of species to measure the dissimilarity of their learned embeddings. We adopt a similar approach for computing the JS-divergence of species-pairs in the quantized feature space of vanilla VQGAN. For baseline methods that operate in continuous feature spaces (CW and LSF), we first calculate the mean feature vector for every
species and then compute the _cosine distance_ (\(1-\text{cosine similarity}\)) of vectors for a pair of species. For both metrics, JS-divergence and cosine distance, a value closer to \(0\) represents higher similarity.

**Comparing Correlations with Ground Truth:** Figure 4(a) and Figure 4(b) show the pair-wise species distance matrices for morphological and phylogenetic GTs, respectively. Note that the rows and columns of all matrices in Figure 4 are species ordered according to their position in the phylogeny (see Appendix B for details) and the diagonal values (correlation with self) are removed so that they do not affect the colormap scale. We can see that both ground-truths show a similar clustering structure of species. However, there are differences too; while phylogenetic GT is solely based on phylogeny, the morphological GT uses both the phylogeny and information about "known" traits. Figure 4(c) and Figure 4(d) show the JS-divergences among species computed separately for the two disentangled parts of PhyloNN's embeddings (\(\mathbf{z}_{\text{p}}^{Q}\) and \(\mathbf{z}_{\text{mp}}^{Q}\)). We can see that the embeddings containing phylogenetic information show a similar clustering structure of distances as the GT matrices, in contrast to the non-phylogenetic embeddings. This shows the ability of _Phylo-NN_ to disentangle features related to phylogeny from other unrelated features. Figure 4 also shows the embedding distance matrices of the baseline methods, which are not as visually clean as _Phylo-NN_ in terms of matching with the GT matrices.

To quantitatively evaluate the ability of _Phylo-NN_ to match with GT distances, we compute the Spearman correlation between the GT distance matrices and embedding distance matrices for different methods as shown in Table 1. We can see that _Phylo-NN_ shows higher correlations at the species level with both GTs. Furthermore, since _Phylo-NN_ learns a different descriptor for every ancestry level in contrast to baseline methods that learn a flat representation, we can also compute _Phylo-NN_'s distance matrix at any ancestry level and compare it with GT matrices at the same level. Table 1 shows that _Phylo-NN_ shows significantly higher correlations with GT matrices at higher ancestry levels than the species level.

### Evaluating Species-to-species Image Translations

To further assess how well _Phylo-NN_'s embeddings capture evolutionary traits, we investigate how altering the learned Imageome sequence of an image specimen incrementally in a phylogenetically meaningful ordering affects the observed traits when the altered embeddings are decoded back as an image. To do that, we set up the following experiment. We pick two specimen images from a pair of species. By encoding the two images using _Phylo-NN_, we obtain their corresponding Imageome encodings, \(\mathbf{z}_{1}^{Q}\) and \(\mathbf{z}_{2}^{Q}\). We then start to replace the codes in the Imageome sequence \(\mathbf{z}_{1}^{Q}\) with the corresponding codes in \(\mathbf{z}_{2}^{Q}\) iteratively, until \(\mathbf{z}_{1}^{Q}\) transforms completely into \(\mathbf{z}_{2}^{Q}\). The order of this iterative replacement is by first replacing the codes representing the non-phylogenetic part of the embedding \(\mathbf{z}_{\text{mp}}^{Q}\), then the part capturing evolutionary information at the earliest ancestry level (level 0), to the next ancestry level (level 1), till we eventually reach the last level of the phylogeny, which is the species level. At the final point, the entire Imageome sequence \(\mathbf{z}_{1}^{Q}\) has been replaced with \(\mathbf{z}_{2}^{Q}\). This phylogeny-driven ordering of code replacements helps us capture key "snapshots" of the species-to-species translation process that are biologically

\begin{table}
\begin{tabular}{l c|c c} \hline \hline  & & Morphological & Phylogenetic \\ \hline \multirow{4}{*}{PhyloNN} & level0 & 0.86 & 0.83 \\  & level1 & 0.87 & 0.85 \\  & level2 & 0.78 & 0.83 \\  & species & 0.70 & 0.78 \\ \hline \hline \multirow{2}{*}{LSF} & 0.38 & 0.55 \\  & 0.70 & 0.67 \\ \cline{1-1}  & vanilla VQGAN & 0.31 & 0.24 \\ \hline \hline \end{tabular}
\end{table}
Table 1. Correlations between GT and embedding distances

Figure 4. Comparing embedding distance matrices of methods with morphological and phylogenetic ground-truths.
meaningful. In particular, by observing the traits that appear or disappear at every ancestry level of code replacement, we can infer and generate novel hypotheses about the biological timing of trait changes as they may have happened in evolutionary history.

Figure 5 shows an example of such a translation process between a specimen of the species _Carassius auratus_ to a specimen of the species _Lepomis cyanellus_. We can see that although the two specimens look similar on the surface, there are several subtle traits that are different in the two species that are biologically interesting. For example, the source species has a V-shaped tail fin (termed caudal fin), while the target species has a rounded caudal fin. By looking at their place of occurrence in the translation process of _Phylo-NN_, we can generate novel biological hypotheses of whether they are driven by phylogeny or not, and whether they appeared earlier or later in the target species in the course of evolution. For example, we can see that the rounded tail feature of the target species appears right after replacing the non-phylogenetic part of the embedding (see blue circle), indicating that this feature may not be capturing evolutionary signals and instead maybe affected by unrelated factors (e.g., environment). On the other hand, if we observe another fin (termed pectoral fin) that appears on the side of the body just behind the gill cover, compared to the fin's lower position closer to the underside of the specimen in the source image, we can see that it seems to get sharper and compact only in the later levels (it is faintly visible in level 1 but shows up prominently as a white region in level 2, see green circle). This suggests that the change in the position and shape of the pectoral fin occurred later in fish evolution, which is supported by the phylogeny and is in fact the case. Our work opens novel opportunities for generating such biological hypotheses, which can be further investigated by biologists to potentially accelerate scientific discoveries. Figure 5 also shows the translations obtained by baseline methods for the same pair of species specimens. We can see that the baselines are mostly performing a smooth interpolation between the source and target images. This is in contrast to the discrete and non-smooth nature of changes observed in the translation of _Phylo-NN_, which is indeed the desired behavior since the appearance or disappearance of traits at every ancestry level are expected to be orthogonal to those at other levels. Furthermore, the transition points in the translation process of baseline methods do not correspond to biologically meaningful events as opposed to _Phylo-NN_.

### Generalization to Unseen Species

As _Phylo-NN_ aims to encode specimen images into their corresponding phylogenetic and non-phylogenetic sequences of codes, we expect specimens that belong to the same species to largely share the same phylogenetic code in terms of the species descriptor \(D_{\text{fn}}\), while varying in terms of the non-phylogenetic codes. More generally, specimens belonging to species sharing a common ancestor at phylogenetic level \(i\) should largely share the codes with the descriptor at level \(i\), \(D_{i}\), while varying at the rest of codes. This should also apply for specimens of _unseen_ (or newly discovered) species that we have not yet observed in the training set. We posit that by looking at the similarity of codes generated for an unseen species, we should be able to infer its ancestral lineage in terms of the species sampled during training. In other words, by analyzing the distribution of codes generated from the image of an unseen species, we should be able to locate it on the phylogenetic tree and position it to next to the subset of known species (seen during training) that share a common ancestor.

To quantify this phenomenon, for a given species or ancestor node, we construct two sets of histograms, \(H_{\text{p}}\) and \(H_{\text{np}}\), of sizes \([n\times n_{\text{p}}]\) and \([n_{\text{mp}}]\), respectively. Each value in the histograms, \(H_{\text{p}}^{i,j}\) and \(H_{\text{np}}^{k}\), describes the distribution of codes at a certain location in the Imageomes across all specimens belonging to the species or ancestor node. See Appendix F for an example. We then compute the entropy of each sequence location in \(H_{\text{p}}\) and \(H_{\text{np}}\) to measure the "purity" of codes used at every location. If the entropy is low for a certain location, it means only a few possible codes occur at that location, suggesting that those specific codes are key at characterizing the species or ancestor node in question. On the other hand, higher entropy means a variety of codes occur at that location, implying that such a location is not discriminative to the species or ancestor node. Finally, to compare the code distributions for two species, we use the JS-divergence metric for calculating the difference between two histograms of a sequence location. Similar to Section 6.1, such a metric can be aggregated to quantify the coding differences between species-pairs.

To assess _Phylo-NN_'s ability to generalize to unseen species, we train it on a subset of the species and then evaluate the quality of the embedding space when the model is introduced to species it has never seen before during training. In our experiment, we chose to train on the same dataset as before while only excluding three species. Once the model is trained, we look at the average JS-divergence distance between these missing species and three other species in the tree. These three other species were selected such that each missing species has one seen species that is close to it phylogenetically (i.e., both species share the same ancestor at the immediate ancestry level) while others are relatively far from it.

Table 2 shows the average distance of the phylogenetic codes among the six aforementioned species. We can see that the distance is smallest for each unseen species and its counterpart that shares the same immediate ancestor (shown as the diagonal in the table). This confirms that even though the model has not seen the former species, it was able to characterize it using an Imageome sequence that is significantly closer to that of its seen counterpart than the other species' Imageomes.

While Table 2 highlights the phylogenetic matching in the embedding space at the species descriptor level, \(D_{\text{fn}}\), Table 3 does the same but for the descriptor at a distant ancestry level (level 0), i.e., \(D_{0}\). Based on the phylogenetic tree we have used in this example, both the _Notropis_ and _Noturus_ species share the same distant ancestor at that descriptor level. On the other hand, _Lepomis_ species does not share that ancestor. Hence, we find that the JS-divergences increase for the _Lepomis_ unseen species with seen species that are not _Lepomis_ as compared to Table 2. On the other hand, the JS-divergences decrease for the other two unseen species w.r.t. seen species that are on the off-diagonals of the table. This confirms that \(D_{0}\) specifically captures the phylogenetic information of that distant ancestor that is common across _Notropis_ and _Noturus_ seen and unseen species. Finally, to confirm that this phylogenetic correlation is mainly constrained only to the phylo-descriptors, we
calculate the same distances but using the non-phylogenetic part of the sequences. The result is shown in Table 4. We can see that the distances are much closer to each other, implying that the non-phylogenetic embedding is not specialized at differentiating among different species, and hence cannot be used to phylogenetically categorize the unseen species.

### Assessing the Clustering Quality of the Embedding Space Using t-SNE Plots

In this section, we qualitatively assess the quality of generated images by visualizing their embedding space. Visualization tools such as loss landscape visualizations (Kang et al., 2017) and t-SNE plots (Zhu et al., 2018), have been frequently used as investigative tools in deep learning in recent years as they help gauge a model's generalization power. To that end, we are interested in understanding how _Phylo-NN_ clusters the embedding space compared to other baselines by analyzing these models' t-SNE plots. To construct the t-SNE plot for each model,

\begin{table}
\begin{tabular}{l l|l l l} \hline \hline  & & & \multicolumn{3}{c}{Seen species} \\ \hline  & & \multicolumn{1}{c}{_Notropis_} & \multicolumn{1}{c}{_Lepomis_} & \multicolumn{1}{c}{_Noturus_} \\  & & \multicolumn{1}{c}{_nublilus_} & \multicolumn{1}{c}{_macrochirus_} & \multicolumn{1}{c}{_flavus_} \\ \hline \multirow{4}{*}{_Notoris_} & \multicolumn{1}{c}{_Notropis pero-bromus_} & 0.39 & 0.45 & 0.39 \\  & \multicolumn{1}{c}{_Lepomis mega-lotiis_} & 0.46 & 0.36 & 0.48 \\ \cline{1-1}  & \multicolumn{1}{c}{_Noturus mu-rus_} & 0.40 & 0.42 & 0.36 \\ \hline \hline \end{tabular}
\end{table}
Table 4. JS-diveergence of the non-phylogenetic codes between unseen and seen species

Figure 5. Comparing species-to-species image translations from a _Carassius auratus_ specimen to a _Lepomis cyanellus_ specimen.

\begin{table}
\begin{tabular}{l|l|l l l} \hline \hline  & & & \multicolumn{3}{c}{Seen species} \\ \hline  & & \multicolumn{1}{c}{_Notropis_} & \multicolumn{1}{c}{_Lepomis_} & \multicolumn{1}{c}{_Noturus_} \\  & & \multicolumn{1}{c}{_nublilus_} & \multicolumn{1}{c}{_macrochirus_} & \multicolumn{1}{c}{_flavus_} \\ \hline \multirow{4}{*}{_Notoris_} & \multicolumn{1}{c}{_Notropis pero-bromus_} & 0.47 & 0.71 & 0.62 \\  & \multicolumn{1}{c}{_Lepomis mega-lotiis_} & 0.73 & 0.43 & 0.72 \\ \cline{1-1}  & \multicolumn{1}{c}{_Noturus mu-rus_} & 0.62 & 0.71 & 0.48 \\ \hline \hline \end{tabular}
\end{table}
Table 2. JS-diveergence of the phylogenetic codes at the seriesci level between unseen and seen species

\begin{table}
\begin{tabular}{l l|l l l l} \hline \hline  & & & \multicolumn{3}{c}{Seen species} \\ \hline  & & \multicolumn{1}{c}{_Notropis_} & \multicolumn{1}{c}{_Lepomis_} & \multicolumn{1}{c}{_Noturus_} \\  & & \multicolumn{1}{c}{_nublilus_} & \multicolumn{1}{c}{_macrochirus_} & \multicolumn{1}{c}{_flavus_} \\ \hline \multirow{4}{*}{_Notoris_} & \multicolumn{1}{c}{_Notropis pero-bromus_} & 0.26 & 0.81 & 0.50 \\  & \multicolumn{1}{c}{_Lepomis mega-lotiis_} & 0.81 & 0.27 & 0.81 \\ \cline{1-1}  & \multicolumn{1}{c}{_Noturus mu-rus_} & 0.52 & 0.80 & 0.31 \\ \hline \hline \end{tabular}
\end{table}
Table 3. JS-diveergence of the phylogenetic codes at the earliest ancestral level between unseen and seen species
we iterate through its generated images, encode them, obtain the quantized embedding vector for each image (\(\mathbf{z}_{\text{up}}^{Q}\) and \(\mathbf{z}^{Q}\) for _Phylo-NN_ and vanilla VQGAN, respectively), and finally create the t-SNE plots. For CW, we use the whitened embeddings \(z_{\text{cw}}\) instead.

Figure 6 shows these constructed t-SNE plots with two different color-coding schemes. The first one (left column) color-codes the data-points based on the grouping of species at the second phylogenetic level (i.e., the direct ancestor of the specimen's species). Using this color-coding scheme allows us to inspect how different species cluster in the embedding space. The second color-coding (right column) is the average phylogenetic distance between the data-point and its \(k\)-nearest neighbors (KNN), where \(k=5\) in this setup. The higher the average distance (i.e., the darker the data-point's color), the more distant the specimen is from those \(k\) specimen's that are closest to it in the quantized embedding space. This color-coding helps us spot how well the different species are separated from each other in the embedding space, which generally characterizes the quality of the encoding and its propensity for downstream tasks, such as classification.

From Figure 6, we can see that _Phylo-NN_ (top row) clusters the generated images better than vanilla VQGAN and CW as evident from its hierarchical clustering where the specimens belonging to the same species clump into small clusters and these clusters in turn clump into larger clusters (representing ancestor nodes) that have a singular color. This demonstrates that _Phylo-NN_ is able to learn a phylogenetically-meaningful encoding, whereas the other base models' clustering is quite fuzzy and poorly characterizes any biological knowledge. Also, by looking at the right column, we can see that _Phylo-NN_ commits very little clustering error in terms of its phylogenetic constraints because the average phylogenetic distance is low (almost zero) for the majority of points. This is in contrast to the other baselines where there is quite a high clustering error as seen from the "heat" of its scatter plot.

## 7. Conclusions and Future Work

In this work, we presented a novel approach of _Phylo-NN_ for discovering biological traits related to evolution automatically from images in an unsupervised manner without requiring any trait labels. The key novelty of our approach is to leverage the biological knowledge of phylogeny to structure the quantized embedding space of _Phylo-NN_, where different parts of the embedding capture phylogenetic information at different ancestry levels of the phylogeny. This enables our method to perform a variety of tasks in a biologically meaningful way such as species-to-species image translation and identifying the ancestral lineage of newly discovered unseen species.

In the future, our work can be extended to include a larger number of embedding dimensions to improve the visual quality of generated images and can be applied to other image datasets beyond the fish dataset. Future work can explore extensions of _Phylo-NN_ to generate images of ancestor species or to predict images of species that are yet to be evolved. Future work can also focus on making the discovered Imageome sequences more explainable by understanding the correspondence of each quantized code with a region in the image space. Our work opens a novel area of research in grounding image representations using tree-based knowledge, which can lead to new research paradigms in other fields of science where images are abundant but labels are scarce.

###### Acknowledgements.

 This work was supported, in part, by NSF awards for the HDR Imageomics Institute (Award # 2118240) and the Biology-guided Neural Network (BGNN) projects (Award # 1940247, # 1940322, # 1940233, # 2022042, # 1939505). Access to computing facilities was provided by the Advanced Research Computing (ARC) Center at Virginia Tech.
