What Do You See in Common? Learning Hierarchical Prototypes over Tree-of-Life to Discover Evolutionary Traits
Harish Babu Manogaran M. Maruf Arka Daw Kazi Sajeed Mehrab Caleb Patrick Charpentier Josef C. Uyeda Wasila Dahdul Matthew J Thompson Elizabeth G Campolongo Kaiya L Provost Paula M. Mabee Hilmar Lapp Anuj Karpatne



Coaching a Robotic Sonographer: Learning Robotic Ultrasound with Sparse Expert's Feedback
    Deepak Raina^*, Mythra V. Balakuntala^*, Byung Wook Kim, Juan Wachs  Senior Member, IEEE, 
Richard Voyles Fellow, IEEE


This work was supported by National Science Foundation (NSF) USA under Grant #2140612 and Purdue University seed grants for West Lafayette-Indianapolis campuses collaboration. (Corresponding author: Deepak Raina)

Deepak Raina is with the Malone Center for Engineering in Healthcare, Johns Hopkins University, Baltimore, MD 21218 USA (e-mail: draina1@jh.edu)
Mythra V. Balakuntala is with Nikon Research Corporation of America, Belmont, CA 94002 USA (e-mail: mythra.balakuntala@nikon.com)

Byung Wook Kim is with the Department of Computer Engineering, Purdue University, West Lafayette, IN 47907 USA (e-mail: kim2986@purdue.edu)
Juan Wachs is with the School of Industrial Technology, Purdue University, West Lafayette, IN 47907 USA (e-mail: jpwachs@purdue.edu).

Richard Voyles is with the School of Engineering Technology, Purdue University, West Lafayette, IN 47907 USA (e-mail: rvoyles@purdue.edu)
*DR and MVB primarily conducted this research at Purdue University
    
======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================





§ ABSTRACT 

  A grand challenge in biology is to discover evolutionary traits—features of organisms common to a group of species with a shared ancestor in the tree of life (also referred to as phylogenetic tree). With the growing availability of  image repositories in biology, there is a tremendous opportunity to discover evolutionary traits directly from images in the form of a hierarchy of prototypes. However, current prototype-based methods are mostly designed to operate over a flat structure of classes and face several challenges in discovering hierarchical prototypes, including the issue of learning over-specific features at internal nodes. To overcome these challenges, we introduce the framework of Hierarchy aligned Commonality through Prototypical Networks ().
  We empirically show that  learns prototypes that are accurate, semantically consistent, and generalizable to unseen species in comparison to baselines on birds, butterflies, and fishes datasets. 
  
  
  [The code and datasets are available at <https://github.com/Imageomics/HComPNet>.]





toc



§ INTRODUCTION



A central goal in biology is to discover the observable characteristics of organisms, or traits (e.g., beak color, stripe pattern, and fin curvature), that help in discriminating between species and understanding how organisms evolve and adapt to their environment  <cit.>. For example,
discovering traits inherited by a group of species that share a common ancestor on the tree of life (also referred to as the phylogenetic tree, see Figure <ref>) is of great interest to biologists to understand how organisms diversify and evolve <cit.>. 
The measurement of such traits with evolutionary signals, termed evolutionary traits, is not straightforward and often relies on subjective and labor-intensive human expertise and definitions <cit.>, hindering rapid scientific advancement <cit.>. 

With the growing availability of large-scale image repositories in biology containing millions of images of organisms  <cit.>, there is an opportunity for machine learning (ML) methods to discover evolutionary traits automatically from images <cit.>. This is especially true in light of recent advances in the field of explainable ML, such as the seminal work of ProtoPNet <cit.> and its variants <cit.> which find representative patches in  training images (termed prototypes) capturing discriminatory features for every class. We can thus cast the problem of discovering evolutionary traits into asking the following question: what image features or prototypes are common across a group of species with a shared ancestor in the tree of life that are absent in species with a different shared ancestor? 

For example, in Figure <ref>, we can see that the four species of birds on the left descending from the blue node show the common feature of having “long tails”, unlike any of the descendant species of the red node. 
Learning such common features at every internal node as a hierarchy of prototypes can help biologists generate novel hypotheses of species diversification (e.g., the splitting of blue and red nodes) and accumulation of evolutionary trait changes.





Despite the success of ProtoPNet <cit.> and its variants in learning prototypes over a flat structure of classes, applying them to discover a hierarchy of prototypes is challenging for three main reasons. First,  existing methods that learn multiple prototypes for every class are prone to learning “over-specific” prototypes at internal nodes of a tree, which cover only one (or a few) of its descendant species. Figure <ref> shows a few examples to illustrate the concept of over-specific prototypes. Consider the problem of learning prototypes common to descendant species of the Felidae family: Lion and Bobcat. If we learn one prototype focusing on the feature of the mane (specific only to Lion) and another prototype focusing on the feature of spotted back (specific only to Bobcat), then these two prototypes taken together can classify all images from the Felidae family. However, they do not represent common features shared between Lion and Bobcat and hence are not useful for discovering evolutionary traits. 
Such over-specific prototypes should be instead pushed down to be learned at lower levels of the tree (e.g., the species leaf nodes of Lion and Bobcat). 

Second, while existing methods such as ProtoPShare <cit.>, ProtoPool <cit.>, and ProtoTree <cit.> allow prototypes to be shared across classes for re-usability and sparsity, in the problem of discovering evolutionary traits,  we want to learn prototypes at an internal node n that are not just shared across all it descendant species but are also absent in the contrasting set of species (i.e., species descending from sibling nodes of n representing alternate paths of diversification). 
Third, at higher levels of the tree, finding features that are common across a large number of diverse species is challenging <cit.>. In such cases, we should be able to abstain from finding common prototypes  without hampering  accuracy at the leaf nodes—a feature missing in existing methods. 


To address these challenges, we present Hierarchy aligned Commonality through Prototypical Networks (), a framework to learn hierarchical prototypes over the tree of life for discovering evolutionary traits. Here are the main contributions of our work:

    
  *  learns common traits shared by all descendant species of an internal node and avoids the learning of over-specific prototypes in contrast to baseline methods using a novel over-specificity loss.
    
  *  uses a novel discriminative loss to ensure that the prototypes learned at an internal node are absent in the contrasting set of species with different ancestry.
    
    
  *      includes a novel masking module to allow for the exclusion of over-specific prototypes at higher levels of the tree without hampering classification performance.
    
  * We empirically show that  learns prototypes that are accurate, semantically consistent, and generalizable to unseen species compared to baselines on data from 190 species of birds (CUB-200-2011 dataset) <cit.>, 38 species of fishes <cit.>, and 30 species of butterflies <cit.>.
    We show the ability of  to generate novel hypotheses about evolutionary traits at different levels of the phylogenetic tree of organisms.
    

 


§ RELATED WORKS

One of the seminal lines of work in the field of prototype-based interpretability methods is the framework of ProtoPNet <cit.> that learns a set of “prototypical patches” from training images of every class to enable case-based reasoning. Following this work, several variants have been developed, such as ProtoPShare <cit.>, ProtoPool <cit.>, ProtoTree <cit.>, and HPnet <cit.> suiting to different interpretability requirements.  
Among all these approaches, our work is closely related to HPnet <cit.>, the hierarchical extension of ProtoPNet that learns a prototype layer for every parent node in the tree. Despite sharing a similar motivation as our work, HPnet is not designed to avoid the learning of over-specific prototypes or to abstain from learning common prototypes at higher levels of the tree.


Another related line of work is the framework of PIPNet <cit.>, which uses self-supervised learning to reduce the “semantic gap” <cit.> between the latent space of prototypes and the space of images, 
such that the prototypes in latent space correspond to the same visual concept in the image space. In , we build upon the idea of self-supervised learning introduced in PIPNet to learn a semantically consistent hierarchy of prototypes.
Our work is also related to ProtoTree <cit.>, which structures the prototypes as nodes in a decision tree to offer more granular interpretability. However, ProtoTree differs from our work in that it learns the tree-based structure of prototypes automatically from data and cannot handle a known hierarchy. Moreover, the prototypes learned in ProtoTree are purely discriminative and allow for negative reasoning, which is not aligned with our objective of finding common traits of descendant species. 

Other related works that focus on finding shared features are ProtoPShare <cit.> and ProtoPool <cit.>. 
Both approaches aim to find common features among classes, but their primary goal is to reduce the prototype count by exploiting similarities among classes, leading to a sparser network. This is different from our goal of finding a hierarchy of prototypes to find evolutionary traits common to a group of species (that are absent from other species). 

Outside the realm of prototype-based methods, the framework of Phylogeny-guided Neural Networks (PhyloNN) <cit.> shares a similar motivation as our work to discover evolutionary traits by representing biological images in feature spaces structured by tree-based knowledge (i.e., phylogeny). However, PhyloNN primarily focuses on the tasks of image generation and translation rather than interpretability. Additionally, PhyloNN can only work with discretized trees with fixed number of ancestor levels per leaf node, unlike our work that does not require any discretization of the tree.




§ PROPOSED METHODOLOGY





 §.§  Model Architecture
 

Given a phylogenetic tree with N internal nodes, the goal of  is to jointly learn a set of prototype vectors 𝐏_𝐧 
for every internal node n∈{1,…,N}. 
Our architecture as shown in Figure <ref> begins with a CNN that acts as a common feature extractor f(x;θ) for all nodes, where θ represents the learnable parameters of f. f converts an image x into a latent representation Z ∈ℝ^H × W × C, where each “patch” at location (h,w) is, 𝐳_𝐡,𝐰∈ℝ^C. Following the feature extractor, for every node n, we initialize a set of K_n prototype vectors 𝐏_𝐧 = {𝐩_𝐢}^K_n_i=1, where 𝐩_𝐢∈ℝ^C. Here, the number of prototypes K_n learned at node n  varies in proportion to the number of children of node n, with β as the proportionality constant, i.e., at each node n we assign β prototypes for every child node. 
To simplify notations, we drop the subscript n in 𝐏_𝐧 and K_n while discussing the operations occurring in node n.

We consider the following sequence of operations at every node n. We first compute the similarity score between every prototype in 𝐏 and every patch in Z. This results in a matrix Ẑ∈ℝ^H × W × K, where every element represents a similarity score between image patches and prototype vectors. We apply a softmax operation across the K channels of Ẑ such that the vector ẑ_𝐡,𝐰∈ℝ^K at spatial location (h,w) in Ẑ represents the probability that the corresponding patch 𝐳_𝐡,𝐰 is similar to the K prototypes.
Furthermore, the i^th channel of Ẑ serves as a prototype score map for the prototype vector 𝐩_𝐢, indicating the presence of 𝐩_𝐢 in the image. We perform global max-pooling across the spatial dimensions H× W of Ẑ to obtain a vector 𝐠∈ℝ^K, where the i^th element represents the highest similarity score of the prototype vector 𝐩_𝐢 across the entire image.
𝐠 is then fed to a linear classification layer with weights ϕ to produce the final classification scores for every child node of node n. We restrict the connections in the classification layer so that every child node n_c is connected to a distinct set of β prototypes, to ensure that every prototype uniquely maps to a child node. ϕ is restricted to be non-negative to ensure that the classification is done solely through positive reasoning, similar to the approach used in PIP-Net <cit.>. We borrow the regularization scheme of PIP-Net to induce sparsity in ϕ by computing the logit of child node n_c as log((𝐠ϕ)^2 + 1). 𝐠 and ϕ here are again unique to each node.




 §.§ Loss Functions Used to Train 


Contrastive Losses for Learning Hierarchical Prototypes: 
PIP-Net <cit.> introduced the idea of using self-supervised contrastive learning to learn semantically meaningful prototypes.
We build upon this idea in our work to learn semantically meaningful hierarchical prototypes at every node in the tree as follows. 
For every input image 𝐱, we pass in two augmentations of the image, 𝐱' and 𝐱” to our framework. The prototype score maps for the two augmentations, Ẑ^' and Ẑ^”, are then considered as positive pairs.
Since ẑ_𝐡,𝐰∈ℝ^K represents the probabilities of patch 𝐳_𝐡,𝐰 being similar to the prototypes from 𝐏, we align the probabilities from the two augmentations ẑ^'_𝐡,𝐰 and ẑ^”_𝐡,𝐰 to be similar using the following alignment loss:

    ℒ_A = -1/HW∑_(h,w) ∈ H × Wlog(ẑ_𝐡,𝐰^'·ẑ_𝐡,𝐰^”)

Since ∑_i=1^K ẑ_𝐡,𝐰,𝐢 = 1 due to softmax operation, ℒ_A is minimum  (i.e., ℒ_A = 0) when both ẑ_𝐡,𝐰^' and ẑ_𝐡,𝐰^” are identical one-hot encoded vectors. A trivial solution that minimizes ℒ_A is when all patches across all images are similar to the same prototype. To avoid such representation collapse, 
we use the following tanh-loss ℒ_T of PIP-Net <cit.>, which  serves the same purpose as uniformity losses in <cit.> and <cit.>: 

    ℒ_T = -1/K∑_i=1^Klog(tanh(∑_b=1^B𝐠_𝐛,𝐢)),

where 𝐠_𝐛,𝐢 is the prototype score for prototype i with respect to image b of mini-batch. ℒ_T encourages each prototype 𝐩_𝐢 to be activated at least once in a given mini-batch of B images, thereby helping to avoid the possibility of 
representation collapse. The use of tanh ensures that only the presence of a prototype is taken into account and not its frequency. 

 Over-specificity Loss: 
 To achieve the goal of learning prototypes common to all descendant species of an internal node, we introduce a novel loss, termed over-specificity loss ℒ_ovsp that avoids learning over-specific prototypes at any node n.
 ℒ_ovsp is formulated as a modification of the tanh-loss such that  prototype 𝐩_𝐢 is encouraged to be activated at least once in every one of the descendant species d ∈{1,…,D_i} of its corresponding child node in the mini-batch of images fed to the model, as follows:
 

 
    ℒ_ovsp = -1/K∑_i=1^K∑_d=1^D_ilog(tanh(∑_b ∈ B_d𝐠_𝐛,𝐢)),

where B_d is the subset of images in the mini-batch that belong to species d.

Discriminative loss: 
In order to ensure that a learned prototype for a child node n_c is not activated by any of its contrasting set of species (i.e., species that are descendants of child nodes of n other than n_c), we introduce another novel loss function, ℒ_disc, defined as follows:

    ℒ_disc = 1/K∑_i=1^K∑_d ∈D_imax_b ∈ B_d (𝐠_𝐛,𝐢),

where D_i is the contrasting set of all descendant species of child nodes of n other than n_c. This is similar to the separation loss used in other prototype-based methods such as <cit.>, <cit.>, and <cit.>.


Orthogonality loss: We also apply kernel orthogonality as introduced in <cit.> to the prototype vectors at every node n, so that the learned prototypes are orthogonal and capture diverse features:

    ℒ_orth = 𝐏̂𝐏̂^⊤ - I _F^2

where 𝐏̂ is the matrix of normalized prototype vectors of size C × K,  I is an identity matrix, and ._F^2 is the Frobenius norm. Each prototype 𝐩̂_𝐢 in 𝐏̂ is normalized as, 𝐩̂_𝐢 = 𝐩_𝐢/𝐩_𝐢.




Classification loss: Finally, we apply cross-entropy loss for classification at each internal node as follows: 



    ℒ_CE = -∑_b^B y_b log(ŷ_b)

where y is ground truth label and ŷ is the prediction at every node of the tree.


 §.§ Masking Module to Identify Over-specific Prototypes

We employ an additional masking module at every node n to identify over-specific prototypes without hampering their training. The learned mask for prototype 𝐩_𝐢 simply serves as an indicator of whether 𝐩_𝐢 is over-specific or not, enabling our approach to abstain from finding common prototypes if there are none, especially at higher levels of the tree.
To obtain the mask values, we first calculate the over-specificity score for  prototype 𝐩_𝐢 as the product of the maximum prototype scores obtained across all images in the mini-batch belonging to every descendant species d as:



    𝒪_i = -∏_d=1^D_imax_(b ∈ B_d) (𝐠_𝐛,𝐢)

where 𝐠_𝐛,𝐢 is the prototype score for prototype 𝐩_𝐢 with respect to image b of mini-batch and B_d is the subset of images in the mini-batch that belong to descendant species d. Since 𝐠_𝐛,𝐢 takes a value between 0 to 1 due to the softmax operation, 𝒪_i ranges from -1 to 0, where -1 denotes least over-specificity and 0 denotes the most over-specificity. The multiplication of the prototype scores ensures that even when the score is less with respect to only one descendant species, the prototype will be assigned a high over-specificity score (close to 0).

As shown in Figure <ref>, 𝒪_i is then fed into the masking module, which includes a learned mask value M_i for every prototype 𝐩_𝐢. We generate M_i from a Gumbel-softmax distribution <cit.> so that the values are skewed to be very close to either 0 or 1, i.e., M_i = Gumbel-Softmax(γ_i, τ), where γ_i are the learnable parameters of the distribution  and τ is temperature. We then compute the masking loss, ℒ_mask, as:

    ℒ_mask =  ∑_i=1^K ( λ_mask M_i∘(𝒪_i) + λ_L_1 M_i _1)

where λ_mask and λ_L_1 are trade-off coefficients, ._1 is the L_1 norm added to induce sparsity in the masks, and  represents the stop gradient operation applied over 𝒪_i to ensure that the gradient of ℒ_mask does not flow back to the learning of prototype vectors and impact their training.
Note that the learned masks are not used for pruning the prototypes during training, they are only used during inference to determine which of the learned prototypes are over-specific and likely to not represent evolutionary traits. Therefore, even if all the prototypes are identified as over-specific by the masking module at an internal node, it will not affect the classification performance at that node. 



 §.§ Training 
 
We first pre-train the prototypes
at every internal node in a self-supervised learning manner using alignment and tanh-losses as
    ℒ_SS = λ_Aℒ_A + λ_Tℒ_T.
We then fine-tune the model using the following combined loss: (λ_CEℒ_CE + ℒ_SS + λ_ovspℒ_ovsp + λ_discℒ_disc + λ_orthℒ_orth + ℒ_mask),
where λ's  are trade-off parameters. Note that the loss is applied over every node in the tree. We show an ablation of key loss terms in our framework in Table <ref> in the Supplementary Section.




§ EXPERIMENTAL SETUP
 
Dataset:
In our experiments, we primarily focus on the 190 species of birds (Bird) from the CUB-200-2011 <cit.> dataset for which the phylogenetic relationship <cit.> is known. The tree is quite large with a total of 184 internal nodes. 
We removed the background from the images to avoid the possibility of learning prototypes corresponding to background information, such as the bird's habitat, as we are only interested in the traits corresponding to the body of the organism. 
We also apply our method on a fish dataset with 38 species (Fish) <cit.> along with its associated phylogeny <cit.> and 30 subspecies of Heliconius butterflies (Butterfly) from the Jiggins Heliconius Collection dataset <cit.> collected from various sources [Sources: <cit.>]

along with its phylogeny <cit.>. The qualitative results of Butterfly and Fish datasets are provided in the supplementary materials.
The complete details of hyper-parameter settings and training strategy are also provided in the Supplementary Section <ref>.


Baselines: We compare  to ResNet-50 <cit.>, INTR (Interpretable Transformer) <cit.> and HPnet <cit.>. For HPnet, we used the same hyperparameter settings and training strategy as used by ProtoPNet for the CUB-200-2011 dataset. For a fair comparison, we also set the number of prototypes for each child in HPnet to be equal to 10 similar to our implementation. We follow the same training strategy as provided by ProtoPNet for the CUB-200-2011 dataset.




§ RESULTS




 §.§ Fine-grained Accuracy


Similar to HPnet <cit.>, we calculate the fine-grained accuracy for each leaf node by calculating the path probability over every image. During inference, the final probability for leaf class Y given an image X is calculated as,
P(Y|X) = P(Y^(1), Y^(2), ..., Y^(L)|X) =
        ∏_l=1^L P(Y^(l)|X),
where P(Y^(l)|X) is the probability of assigning image X to a node at level l, and L is the depth of the leaf node. Every image is assigned to the leaf class with maximum path probability, which is used to compute the fine-grained accuracy. The comparison of the fine-grained accuracy calculated for  and the baselines are given in Table <ref>.
We can see that  performs better than the other interpretable methods, such as INTR and HPNet, and is also able to nearly match the performance of non-interpretable models, such as ResNet-50, even outperforming it for the Fish and Butterfly dataset.
This shows the ability of our proposed framework to achieve competitive classification accuracy along with serving the goal of discovering evolutionary traits.





 §.§ Generalizing to Unseen Species in the Phylogeny


We analyze the performance of  in generalizing to unseen species that the model has not seen during training. 

The biological motivation for this experiment is to evaluate if  can situate newly discovered species at its appropriate position in the phylogeny by identifying its common ancestors shared with the known species. An added advantage of our work is that along with identifying the ancestor of an unseen species, 
we can also identify the common traits shared by the novel species with known species in the phylogeny.

 Since unseen species cannot be classified to the finest levels (i.e., up to the leaf node corresponding to the unseen species), we analyze the ability of  to classify unseen species accurately up to one level above the leaf level in the hierarchy. With this consideration, the final probability of an unseen species for a given image is calculated as,
P(Y|X_unseen) = P(Y^(1), Y^(2), ..., Y^(L-1)|X) =
        ∏_l=1^L-1 P(Y^(l)|X).
Note that we leave out the class probability at the L^th level, since we do not take into account the class probability of the leaf level. We leave four species from the Bird training set and calculate their accuracy during inference in  Table <ref>. We can see that  is able to generalize better than HPnet for all four species. 






 §.§ Analyzing the Semantic Quality of Prototypes
 


Following the method introduced in PIPNet <cit.>, we assess the semantic quality of our learned prototypes by evaluating their part purity. A prototype with high part purity (close to 1) is one that consistently highlights the same image region in the score maps (corresponding to consistent local features such as the eye or wing of a bird) across images belonging to the same class. 
The part 
r0.53
  Part purity of prototypes on Bird dataset.
  
  

  
   
  
    Model     ℒ_ovsp     Masking     Part purity     % masked

    HPnet     -     -     0.14 ± 0.09     -

         -      -      0.68 ± 0.22     -

         -         0.75 ± 0.17     21.42%

             -     0.72 ± 0.19     -

                 0.77 ± 0.16     16.53%

  
  


purity is calculated using the part locations of 15 parts that are provided in the CUB dataset. For each prototype, we take the top-10 images from each leaf descendant. We consider the 32 × 32 image patch that is centered around the max activation location of the prototype from the top-10 images. With these top-10 image patches, we calculate how frequently each part is present inside the image patch. For example, a part that is found inside the image patch 8 out of 10 times is given a score of 0.8. In PIP-Net, the highest value among the values calculated for each part is given as the part purity of the prototype. In our approach, since we are dealing with a hierarchy and taking the top-10 from each leaf descendant, a particular part, let's say the eye, might have a score of 0.5 for one leaf descendant and 0.7 for a different leaf descendant. Since we want the prototype to represent the same part for all the leaf descendants, we take the lowest score (the weakest link) among all the leaf descendants as the score of the part. By following this method, for a given prototype we can arrive at a value for each part and finally take the maximum among the values as the purity of the prototype. We take the mean of the part purity across all the prototypes and report the results in Table <ref> for different ablations of  and also HPnet, which is the only baseline method that can learn hierarchical prototypes. 

We can see that , even without the use of over-specificity loss, performs much better than HPnet due to the contrastive learning approach we have adopted from PIPNet <cit.>. The addition of over-specificity loss improves the part purity because over-specific prototypes tend to have poor part purity for some of the leaf descendants, which will affect their overall part purity score. Further, for both ablations with and without over-specificity loss, we apply the masking module and remove masked (over-specific) prototypes during the calculation of part purity. We see that the part purity goes higher by applying the masking module, demonstrating its effectiveness in identifying over-specific prototypes. We further compute the purity of masked-out prototypes and notice that the masked-out prototypes have drastically lower part purity (0.29 ± 0.17) compared to non-masked prototypes (0.77 ± 0.16). We also provide a visual comparison of a masked (over-specific) prototype and an unmasked (non-over-specific) prototype in the Supplementary Section <ref>.
An alternative approach to learning the masking module is to identify over-specific prototypes using a fixed global threshold over 𝒪_i. We show in Table <ref> of Supplementary Section <ref>, that given the right choice of such a threshold, we can identify over-specific prototypes. However, selecting the ideal threshold can be non-trivial. On the other hand, our masking module learns the appropriate threshold dynamically as part of the training process.

Figure <ref> visualizes the part consistency of prototypes discovered by  in comparison to HPnet for the bird dataset. We can see that  is finding a consistent region in the image (corresponding to the head region) across all three descendant species and all images of a species, in contrast to HPnet. Furthermore, thanks to the alignment loss, every patch ẑ_𝐡,𝐰 is encoded as nearly a one-hot encoding with respect to the K prototypes which causes the prototype score maps to be highly localized. The concise and focused nature of the prototype score maps makes the interpretation much more effective compared to baselines.



 §.§ Analyzing Evolutionary Traits Discovered by 
 





We now qualitatively analyze some of the hypothesized evolutionary traits discovered in the hierarchy of prototypes learned by . 
Figure <ref> shows the hierarchy of prototypes discovered over a small subtree of the phylogeny from Bird (four species) and Fish (three species) dataset.
In the visualization of bird prototypes, we can see that the two Pelican species share a consistent region in the learned Prototype labeled 2, which corresponds to the head region of the birds. We can hypothesize this prototype is capturing the white-colored crown common to the two species. On the other hand, Prototype 1 finds the shared trait of similar beak morphology (e.g., sharpness of beaks) across the two Cormorant species. We can see that  avoids the learning of over-specific prototypes at internal nodes, which are pushed down to individual leaf nodes, as shown in visualizations of Prototype 3, 4, 5, and 6. 
Similarly, in the visualization of the fish prototypes, we can see that  Prototype 1
is highlighting a specific fin (dorsal fin) of the Carassius auratus and Notropis hudsonius  species, possibly representing their pigmentation and structure, which is noticeably different compared to the contrasting species of Alosa chrysochloris.
Note that while  identifies the common regions corresponding to each prototype (shown as heatmaps), the textual descriptions of the traits provided in Figure <ref> are based on human interpretation.

Figure <ref> shows another visualization of the sequence of prototypes learned by  for the Western Grebe species at different levels of the phylogeny. We can see that at level 0, we are capturing features closer to the neck region, indicating the likely difference between the length of necks between Grebe species and other species (Cuckoo, Albatross, and Fulmar) that diversify at an earlier time in the process of evolution. At level 1, the prototype is focusing on the eye region, potentially indicating a difference in the color of red and black patterns around the eyes. At level 2, we are differentiating Western Grebe from Horned Grebe based on the feature of bills. We also validate our prototypes by comparing them with the multi-head cross-attention maps learned by INTR <cit.>. We can see that some of the prototypes discovered by  can be mapped to equivalent attention heads of INTR. However, while INTR is designed to produce a flat structure of attention maps, we are able to place these maps on the tree of life. This shows the power of  in generating novel hypotheses about how trait changes may have evolved and accumulated across different branches of the phylogeny. 
Additional visualizations of discovered evolutionary traits for butterfly species and fish species are provided in the supplementary section in Figures <ref> to <ref>.






§ CONCLUSION

We introduce a novel approach for learning hierarchy-aligned prototypes while avoiding the learning of over-specific features at internal nodes of the phylogenetic tree, enabling the discovery of novel evolutionary traits. 
Our empirical analysis on birds, fishes, and butterflies, demonstrates the efficacy of  over baseline methods. Furthermore,  demonstrates a unique ability to generate novel hypotheses about evolutionary traits, showcasing its potential in advancing our understanding of evolution. We discuss the limitations of our work in Supplementary Section <ref>.
While we focus on the biological problem of discovering evolutionary traits, our work can be applied in general to domains involving a hierarchy of classes, which can be explored in future research.


This research is supported by National Science Foundation (NSF) awards for the HDR Imageomics Institute (OAC-2118240). We are thankful for the support of computational resources provided by the Advanced Research Computing (ARC) Center at Virginia Tech. This manuscript has been authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains, and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript or allow others to do so for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (<https://www.energy.gov/doe-public-access-plan>).






unsrt














    
























 




























































































































    

























    

































































































































    







toc 





§ : SUPPLEMENTARY MATERIAL











§ ADDITIONAL BIOLOGICAL BACKGROUND

One of the first steps in any study of evolutionary morphology is character construction - the process of deciding which measurements will be taken of organismal variation that are replicable and meaningful for the underlying biology, and how these traits should be represented numerically <cit.>. For phylogenetic studies, researchers typically attempt to identify synapomorphies – versions of the traits that are shared by two or more species, are inherited from their most recent common ancestor, and may have evolved along the phylogeny branch. The difficulty with the traditional character construction process is that humans often measure traits in a way that is inconsistent and difficult to reproduce, and can neglect shared features that may represent synapomorphies, but defy easy quantification. To address the problem of human inconsistency, PhyloNN <cit.> and Phylo-Diffusion <cit.> took a knowledge-guided machine learning (KGML) <cit.> approach to character construction, by giving their neural networks knowledge about the biological process they were interested in studying (in their case, phylogenetic history), and specifically optimizing their models to find embedded features (analogous to biological traits) that are predictive of that process. To address the problem of visual irreproducibility, Ramirez et al. <cit.> suggested photographing the local structures where the empirical traits vary and linking the images to written descriptions of the traits. In this paper, we take influence from both approaches. We extend the hierarchical prototype approach from Hase et al. <cit.> to better reflect phylogeny, similar in theory to the way PhyloNN <cit.> and Phylo-Diffusion <cit.> learned embeddings that reflect phylogeny. Using prototypes, however, we enforce local visual interpretability similar to how researchers may use “type-specimens” to define prototypical definitions of particular character states.

Specifically, our method is about finding synapomorphies–shared derived features unique to a particular group of species that share a common ancestor in the phylogeny (referred to as clade). While such features may bear similarities to convergent phenotypes in other clades, our goal is not to identify features that exhibit convergence. It is typical for phylogenetic studies to specifically avoid features that exhibit high levels of convergence, as they can lend support for erroneous phylogenetic relationships. Identifying convergence required additional information such as shared habitat, niche, diet, or behavior, which is not incorporated in our work.





§ ABLATION OF OVER-SPECIFICITY LOSS TRADE-OFF HYPERPARAMETER

We have provided an ablation for the over-specificity loss trade-off hyperparameter (λ_ovsp) in Table <ref>. 

We can observe that increasing the weight of over-specificity loss reduces the model's classification performance, as the model struggles to find any commonality, especially at internal nodes where the number of leaf descendant species is large and quite diverse. It is natural that species that are diverse and distantly related may share fewer characteristics with each other, in comparison to a set of species that diverged more recently from a common ancestor <cit.>. Therefore, forcing the model to learn common traits with a strong ℒ_ovsp constraint can cause the model to perform badly in terms of accuracy. 





§ ABLATION OF NUMBER OF PROTOTYPES

In Table <ref>, we vary the number of prototypes per child β for a node to see the impact on the model's performance.
We note that while the accuracy increases marginally with increasing the number of prototypes per child (β) from 10 to 15, it does not affect the performance of the model significantly.

Therefore, we continue to work with β=10 for all of our experiments.





§ ABLATION OF INDIVIDUAL LOSSES

In Table <ref>, we perform an ablation of the various loss terms used in our methodology. As it can be observed, the removal of ℒ_ovsp and ℒ_disc degrades performance in terms of both semantic consistency (part purity) and accuracy. On the other hand, the removal of self-supervised contrastive loss ℒ_SS improves accuracy but at the cost of drastically decreasing the semantic consistency.






§ CONSISTENCY OF CLASSIFICATION PERFORMANCE OVER MULTIPLE RUNS
 

We trained the model using five distinct random weight initializations. The results showed that the model's fine-grained accuracy averaged 70.63% with a standard deviation of 0.18%.



§ IMPLEMENTATION DETAILS
 


We have included all the source code and dataset, along with the comprehensive instructions to reproduce the results at <https://github.com/Imageomics/HComPNet>.

Model hyper-parameters:
We build  on top of a ConvNeXt-tiny architecture as the backbone feature extractor. We have modified the stride of the max pooling layers of later stages of the backbone from 2 to 1, similar to PIP-Net, such that the backbone produces feature maps of increased height and width, in order to get more fine-grained prototype score maps. We implement and experiment with our method on ConvNeXt-tiny backbones with 26 × 26 feature maps. The length of prototype vectors C is 768. The weights ϕ at every node n of  are constrained to be non-negative by the use of the ReLU activation function <cit.>. Further, the prototype activation nodes are connected with non-negative weights only to their respective child classes in W while their weights to other classes are made zero and non-trainable. 


Training details:
All models were trained with images resized and appropriately padded to 224 × 224 pixel resolution and augmented using TrivialAugment <cit.> for contrastive learning. The prototypes are pretrained with self-supervised learning similar to PIP-Net for 10 epochs, following which the model is trained with the entire set of loss functions for 60 epochs. We use a batch size of 256 for the Bird dataset and 64 for the Butterfly and Fish dataset.

The masking module is trained in parallel, and its training is continued for 15 additional epochs after the training of the rest of the model is completed. The trade-off hyper-parameters for the loss functions are set to be λ_CE = 2; λ_A = 5; λ_T = 2; λ_ovsp = 0.05; λ_disc = 0.1; λ_orth = 0.1; λ_mask = 2.0; λ_L1 = 0.5. λ_CE, λ_T and λ_A were borrowed from PIP-Net <cit.>. Ablations to arrive at suitable λ_ovsp is provided in Table <ref>. λ_disc and λ_orth were chosen empirically and found to work well on all three datasets. Experiment on unseen species was done by leaving out certain classes from the datasets, so that they are not considered during training. 

Dataset and Phylogeny Details: Dataset statistics and phylogeny statistics are provided in Table <ref> and Table <ref> respectively. Bird dataset is created by choosing 190 species from CUB-200-2011 [License: https://creativecommons.org/licenses/by/4.0/CC BY] <cit.> dataset, which were part of the phylogeny. Background from all images was filtered using the associated segmentation metadata <cit.>. For Butterfly dataset we considered each subspecies as an individual class and considered only the subspecies of genus Heliconius from the Heliconius Collection (Cambridge Butterfly)[Note that this dataset is a compilation of images from 25 Zenodo records by the Butterfly Genetics Group at Cambridge University, licensed under https://creativecommons.org/licenses/by/4.0/Creative Commons Attribution 4.0 International (<cit.>).] <cit.>. There is substantial variation among subspecies of Heliconius species. Furthermore, we balanced the dataset by filtering out the subspecies that did not have 20 or more images. We also sampled a subset of 100 images from each subspecies that had more than 100 images. For Fish [License: https://creativecommons.org/licenses/by-nc/4.0/CC BY-NC] dataset, we followed the exact same preprocessing steps as outlined in PhyloNN <cit.>.




Compute Resources: The models for the Bird dataset were trained on two NVIDIA A100 GPUs with 80GB of RAM each. Butterfly and Fish models were trained on a single A100 GPU. As a rough estimate, the execution time for the training model on the Bird dataset is around 2.5 hours. For Butterfly and Fish datasets, the training is completed in under 1 hour. We used a single A100 GPU during the inference stage for all other analyses.













§ POST-HOC THRESHOLDING TO IDENTIFY OVER-SPECIFIC PROTOTYPES
 
An alternative approach to learning the masking module is to calculate the over-specificity score for each prototype on the test set after training the model. We calculate the over-specificity scores for the prototypes of a trained model as follows,

    𝒪_i = -∏_d=1^D_i1/top_k∑_i=1^top_k (𝐠_𝐢)


For a given prototype, we choose the top_k images with the highest prototype scores from each leaf descendant. After taking mean of the top_k prototype score, we multiply the values from each descendant to arrive at the over-specificity score for the particular prototype. Subsequently, we choose a threshold to determine which prototypes are over-specific. We provide the results of post-hoc thresholding approach that can also be used to identify over-specific prototypes in Table <ref>. While we can note that this approach can also be effective, validating the threshold particularly in scenarios where there is no part annotations available (such as part location annotation of CUB-200-2011) can be an arduous task. In such cases, directly identifying over-specific prototypes as part of the training through the masking module can be the more feasible option.



§ VISUAL COMPARISON OF A OVER-SPECIFIC AND A NON-OVER-SPECIFIC PROTOTYPE
 

We do a visual comparison of a prototype that has been identified as over-specific by the masking module and a prototype that is not identified as over-specific in Figure <ref>. As it can be observed in Figure <ref>(a), the Red-Legged Kittiwake has legs that are shorter in comparison to other species of its clade - Heerman Gull and Western Gull. Therefore, the prototype is identified as over-specific, as long legs are not common to all three species. On the other hand, in Figure <ref>(b), the prototype has been identified as non-over-specific because all three species share white-colored crowns. Prototype from Figure <ref>(a) has very low activation for Red Legged Kittiwake and also has poor part purity since it does not highlight the same part of the bird in the images of Red-legged Kittiwake.






§ PERFORMANCE OF HPNET WITH HIGH RESOLUTION FEATURE MAPS
 
We analyze the performance of HPnet with high-resolution feature maps in Table <ref>. We modified the backbone by removing the max pooling layers at the final stages of the model to produce a 28x28 feature map instead of the original 7x7 feature map. It can be observed that the accuracy and part purity do not improve with high-resolution feature maps. We also make a qualitative comparison between an HPnet and  prototype with a higher resolution feature map in Figure <ref>, showing that part purity does not improve with high-resolution feature maps for HPnet.









§ ADDITIONAL VISUALIZATIONS OF THE HIERARCHICAL PROTOTYPES DISCOVERED BY 

We provide more visualizations of the hierarchical prototypes discovered by  for Butterfly (Figures <ref> and <ref>) and Fish (Figure <ref>) datasets in this section. For ease of visualization, in each figure we visualize the prototypes learned over a small sub-tree from the phylogeny. The prototypes at the lowest level capture traits that are species-specific, whereas the prototypes at internal nodes capture the commonality between its descendant species. For Fish dataset, we have provided textual descriptions purely based on human interpretation for the traits that are captured by prototypes at different levels. For Butterfly dataset, since the prototypes are capturing different wing patterns, assigning textual descriptions for them is not straightforward. Therefore, we refrain from providing any text description for the highlighted regions of the learned prototypes and leave it to the reader's interpretation.














§ ADDITIONAL TOP-K VISUALIZATIONS OF  PROTOTYPES

We provide additional top-K visualizations of the prototypes from Butterfly (Figures <ref> to <ref>) and Fish (Figures <ref> to <ref>) datasets, where every row corresponds to a descendant species and the columns corresponds to the top-K images from the species with the largest prototype activation scores. A requirement of a semantically meaningful prototype is that it should consistently highlight the same part of the organisms in various images, provided that the part is visible. We can see in the figures that the prototypes learned by  consistently highlight the same part across all top-K images of a species, and across all descendant species. We additionally show that  can find common traits at internal nodes with a varying number of descendant species, including 4 species (Figure <ref>), 5 species (Figures <ref> and <ref>), and 10 species (Figure <ref>) of butterflies, and 5 species (Figure <ref>), 8 species (Figure <ref>) and 18 species (Figure <ref>) for fish. We also provide several top-k visualizations of prototypes learned for bird species in Figures <ref> to <ref>. This shows the ability of  to discover common prototypes at internal nodes of the phylogenetic tree that consistently highlight the same regions in the descendant species images even when the number of descendants is large.







































§ LIMITATIONS OF OUR WORK
 
A fundamental challenge of every prototype-based interpretability method (including ours) is the difficulty in associating a semantic interpretation with the underlying visual concept of a prototype. While some prototypes can be interpreted easily based on visual inspection of prototype activation maps, other prototypes are harder to interpret and require additional domain expertise of biologists. Also, while we have considered large phylogenies as that of the 190 species from the CUB dataset, it may still not be representative of all bird species. This limited scope may cause our method to identify apparent homologous evolutionary traits that could differ with the inclusion of more species into the phylogeny. Therefore, our method can be seen as a system that generates potential hypotheses about evolutionary traits discovered in the form of hierarchical prototypes.









