# Learning Hierarchical Semantic Classification by
Grounding on Consistent Image Segmentations

Seulki Park1 Youren Zhang1 Stella X. Yu1 Sara Beery2 Jonathan Huang3
1University of Michigan, Ann Arbor 2MIT 3Google
{seulki, yourenz<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Hierarchical semantic classification requires the prediction of a taxonomy
tree instead of a single flat level of the tree, where both accuracies at
individual levels and consistency across levels matter. We can train
classifiers for individual levels, which has accuracy but not consistency, or
we can train only the finest level classification and infer higher levels,
which has consistency but not accuracy.

Our key insight is that hierarchical recognition should not be treated as
multi-task classification, as each level is essentially a different task and
they would have to compromise with each other, but be grounded on image
segmentations that are consistent across semantic granularities. Consistency
can in fact improve accuracy.

We build upon recent work on learning hierarchical segmentation for flat-level
recognition, and extend it to hierarchical recognition. It naturally captures
the intuition that fine-grained recognition requires fine image segmentation
whereas coarse-grained recognition requires coarse segmentation; they can all
be integrated into one recognition model that drives fine-to-coarse internal
visual parsing. Additionally, we introduce a Tree-path KL Divergence loss to
enforce consistent accurate predictions across levels. Our extensive
experimentation and analysis demonstrate our significant gains on predicting
an accurate and consistent taxonomy tree.

##  1 Introduction

The goal of hierarchical classification is to predict all classes at each
hierarchical level simultaneously (e.g., Ruby-throated Hummingbird \- Bird).
Figure 1 provides an example of various challenges in the hierarchical
classification problem. For our analysis, we conduct training on the
iNaturalist 2018 dataset [19], with a two-level hierarchy. Following the most
commonly used approach for hierarchical classification [8, 6, 7], we extract
features from a shared backbone (ImageNet [34]-pretrained ResNet-50 [18]) and
train separate classifiers to predict classes at each level.

Unlike flat-level classification, hierarchical classification has multiple
possible outputs. The challenges arise when predictions at the fine and coarse
levels differ (2nd and 3rd rows in Figure 1). Even if predictions are correct
at one level, it is difficult to trust the prediction if predictions at other
levels are incorrect (e.g., Ruby-throated Hummingbird \- Plant), especially
when we do not have ground truth labels at test time. Therefore, our focus
lies in achieving consistent and accurate predictions across all levels
simultaneously.

However, predicting multiple levels of hierarchy is more challenging than
predicting a single flat-level category, leading to a confrontation between
coarse and fine level learning [6]. That is, models trained to predict all
hierarchical levels show degraded performance compared to the model trained to
predict a single flat level of class. This is because hierarchical
classification entails addressing multiple classification tasks concurrently.
For instance, distinguishing between similar-looking bird species at the
finest level (e.g., Anna’s Hummingbird \- Ruby-throated Hummingbird) and
differentiating between birds and plants at the coarse level represent
distinct tasks, each requiring attention to different regions in the image.
This phenomenon is illustrated in Figure 2, where we observe that classifiers
at each level often internally attend to different parts of the image. From
this observation, we attribute performance degradation in hierarchical
recognition to the inconsistency among hierarchy classifiers. We seek to
enhance consistency by guiding classifiers to consistently focus on the same
areas, where “bird” (for the coarse classifier) and “Ruby-throated
Hummingbird” (for the fine-level classifier) are located.

Figure 1: Consistency is not only necessary for hierarchical classification
but also helps improve accuracy. In hierarchical classification, categorizing
all levels of classes accurately is important to establish the model’s
reliability. However, training on multiple levels can lead to conflicts
between classifiers, hindering performance compared to single-level training
[6]. We address this challenge by focusing on inconsistent cases (like rows 2
& 3), where some classifiers correctly identify relevant image regions. Our
method encourages classifiers at all levels to consistently focus on the same
image regions, potentially leveraging the correctly identified parts from one
of the classifiers. Compared to the baseline (separate classifiers with shared
ResNet-50 [18] backbone pre-trained on ImageNet [34]), by promoting
consistency, our method significantly boosts accuracy on the iNaturalist 2018
dataset [19]. (Green border indicates correct predictions after applying our
method, and red indicates incorrect predictions. The outer lines represent
coarse-level predictions, while the inner lines represent fine-level
predictions.)

Our insight is that although the fine-level classifier and the coarse
classifier may have different focal points, they are inherently connected. For
instance, while the fine-level classifier examines aspects like the beak,
wings, and feathers to classify bird species, the coarse classifier can
discern between birds and flowers by integrating these details into the
overall body of the bird. In other words, the coarse classifier should focus
on the larger part formed by grouping the features attended to by the fine
classifier. Therefore, we hypothesize that by consistently aligning the areas
of focus for the fine classifier and coarse classifier, we could improve both
consistency and accuracy.

To this end, we employ the recent unsupervised image segmentation method, CAST
[23]. CAST exhibited hierarchical segmentation, transitioning from finer to
coarser segments. CAST has demonstrated the capability to group related pixels
consistently through internal parsing within images, without segmentation
labels. Inspired by this, we propose Hierarchical-CAST (H-CAST), which
utilizes CAST to align the focus of different levels of classifiers on the
same areas for hierarchical recognition. Unlike previous hierarchical
classification approaches that train separate models for each level [6, 26,
7], H-CAST is an integrated model where features learned at the fine level are
grouped and passed to the coarse level (Figure 2), ensuring spatial
consistency. Since our method is an integrated model, if details initially
captured at the fine level are incorrect, it will receive negative signals
(errors) during the learning process toward coarser levels. As training
processes, the model is encouraged to capture accurate fine-level details to
improve learning at subsequent levels.

Additionally, we propose Tree-path KL Divergence loss to further enhance
semantic consistency by considering label relationships across levels so that
predictions at the fine-level and coarse-level align within the taxonomy. In
Figure 1, we can observe that our method of improving consistency surprisingly
leads to an enhancement in accuracy as well. Many previously inconsistent and
incorrect cases are transformed into correct classifications.

To assess both accuracy and consistency, we evaluate our method with a new
metric called Full-Path Accuracy (FPA), which measures the proportion of
samples in the dataset correctly predicted at all hierarchy levels. Validated
on common benchmarks for hierarchical recognition, our method consistently
outperforms the ViT-based baseline by more than 17-25%p in FPA metric. We
empirically demonstrate its effectiveness through extensive experimentation
and analysis. Furthermore, compared to the common segmentation learned through
a flat semantic taxonomy, we demonstrate that a hierarchical semantic taxonomy
also in turn improves image segmentation.

Our key contributions can be summarized as follows.

  * •

Hierarchical-CAST (H-CAST): We propose H-CAST to consistently align the focus
of hierarchy classifiers within the image. Our model enables the utilization
of various granular segments, ensuring that focus remains on the same region
while fine-detailed features transitioning to broader features at the coarse
level (spatial consistency). This expansion of the focus region fosters a
natural and consistent learning process for hierarchical recognition.

  * •

Tree-path KL Divergence loss: We introduce a novel loss function that
considers label relationships across hierarchy levels (semantic consistency).
Our loss enforces predictions to be correct at different levels, demanding
accuracy and consistency at the same time.

  * •

These contributions combined lead to significant improvements in both accuracy
and consistency, which we demonstrate via a new metric: Full-Path Accuracy
(FPA).

Figure 2: While prior works train separate branches for different levels, our
integrated model encourages internal consistency for consistent prediction
among classifiers. Commonly used separate classifiers often lead to
inconsistency between hierarchy levels due to different attending regions
between classifiers. Our Hierarchical-CAST (H-CAST) enables the utilization of
various granular segments, ensuring that focus remains on the same region
while fine-detailed features transitioning to broader features at the coarse
level. In the segmentation images inside H-CAST, consistent groupings are
represented by identical colors. The segmentation images are the results of
32-way and 8-way, respectively. For instance, when observing the red bird
segments, we notice finer details such as wings, body, head, and tail in the
fine level (32-way), while in the subsequent level (8-way), it is grouped as
the entire bird. By leveraging this consistent internal parsing, we encourage
the model to focus on the coherent regions within images.

##  2 Related Work

Hierarchical Classification (HC) is a problem that has been extensively
studied. Before the advent of deep learning, works in this area mainly focused
on tree-based [37, 13, 32] and DAG-based [21, 1] methods. For a more thorough
study on this topic, we refer the reader to survey [36, 10]. Recent deep
learning-based methods can be broadly categorized into three main types: 1)
label-embedding methods, 2) hierarchical losses, and 3) hierarchical
architectures.

1) Label-embedding methods focus on incorporating hierarchical information
into labels to replace flat one-hot encodings. Label-embedding space can be
derived by hierarchical relationships [4, 2], class characteristics [43], or
learned by a language model [14]. The label-embedding can be constructed on
the hypersphere [14, 2] or in the hyperbolic space [30, 24] as well to enhance
the expressiveness. Our approach diverges from these methods in two
fundamental ways: 1) our tree path encoding is designed to encourage semantic
consistency across granularities. 2) Instead of only encoding the finest
level, our method provides the complete label hierarchy.

2) Hierarchical losses modify objective functions to integrate hierarchical
information as a form of supervision. One straightforward method involves
utilizing tree structures, either based on tree-metrics [47], or the lowest
common ancestors [16, 11]. Another approach directly calculates losses at
different levels according to the hierarchy [4, 42]. However, recent work
showed that sharing information between coarse and fine-grained levels may
hamper performance [6]. This necessitates a more ingenious design for
combining information across granularities. Our work utilizes hierarchical
features in a cascade manner, leveraging the connection between coarse and
fine-grained levels.

3) Hierarchical architectures enable models to output predictions directly at
different levels of hierarchy. A common approach designs models with separate
branches, each responsible for a specific level [49, 44, 26, 7, 8, 40], or by
structuring the feature space according to the hierarchy [6, 39]. Other works
construct the prediction tree in a _coarse-to-fine_ manner based on the
conditional likelihoods of the ancestors [33, 4, 5]. In contrast, our method
establishes a _fine-to-coarse_ internal parsing and ensures that different
hierarchical granularities contribute consistently to learning at the next
level.

Hierarchical Fine-grained Recognition (HFR) aims to assign hierarchical labels
to the object within images, it is harder than traditional HC due to smaller
inter-class differences and emphasis on outputting the complete label
hierarchy. Typical HFR model designs adhere to a hierarchical architecture,
allowing the model to produce multiple outputs at varying levels of
granularity [6, 8, 26, 39, 7]. While branching is effective, it is prone to
inconsistent predictions.

Inconsistency occurs when the coarse and fine-grained prediction of an object
violates the hierarchy structure, i.e., the coarse prediction is not the
parent category of the fine-grained prediction. Several prior works have
attempted to solve the inconsistency by post-processing [1, 13, 29] or adding
violation penalization to the loss function [17, 7, 40]. More recently, Wang
et al. [39] proposed a consistency-aware approach by explicitly adjusting
prediction scores through coarse-to-fine deduction and fine-to-coarse
induction. In contrast, our method implicitly encourages consistency through
hierarchical label encoding without any post-processing. In addition, our
framework improves semantic consistency and spatial consistency
simultaneously.

Unsupervised Image Segmentation learns to segment the input images according
to the extracted features without human-annotated segmentation [20, 31, 22].
Recently, CAST [23] adopted the Vision Transformer (ViT) [12] with two
significant modifications. One change involved using superpixels instead of
fixed-shape patches to capture more complex visual contours for segmentation.
Secondly, graph pooling was applied for hierarchical segmentation. Our
framework takes a step further by utilizing the information from hierarchical
labels via cross-granularity supervision to enhance the consistency and
accuracy of hierarchical recognition.

##  3 Hierarchical Classification based on Internally Consistent Segmentations

Our goal is to enhance the consistency of hierarchical recognition, thereby
concurrently improving the accuracy of the model. The inconsistent predictions
in hierarchical recognition indicate that, in other words, labels at some
levels are correctly predicted. Therefore, our motivation is that fostering
consistent predictions across hierarchical levels, leveraging information from
such accurately predicted levels, would further enhance overall accuracy.
Based on this motivation, we design a progressive learning scheme for
hierarchical recognition, where the learning of each level contributes to the
learning of the next level, instead of training separate models focusing on
each individual level.

Specifically, we address two types of inconsistency in hierarchical
recognition. One is spatial inconsistency, where classifiers at different
levels attend to different regions (Figure 2). To address this, we propose
H-CAST in Section 3.1. The other is semantic inconsistency, where predictions
at different levels are not aligned within the taxonomy (e.g., “Hummingbird” -
“Plant”). We propose a new Tree-path KL Divergence loss that encodes parent-
child relations to handle semantic inconsistency in Section 3.2. Figure 3
provides an overview of our method.

Figure 3: Our method consists of two parts: Spatial Consistency and Semantic
Consistency module. In the Spatial Consistency module, the parsed images using
superpixels are grouped based on related parts as they transition from fine to
coarse levels. This ensures that each hierarchical classifier internally
focuses on the same corresponding regions. In the Semantic Consistency module,
we incorporate hierarchical relationships between labels. This approach allows
us to achieve consistent learning across the entire hierarchy. By promoting
consistency, our method encourages classifiers at different levels to enhance
overall performance, rather than conflicting with each other.

###  3.1 H-CAST for Spatial Consistency

The areas of focus within the image differ when conducting classification at
the fine-grained level compared to the coarse level. When distinguishing
between similar-looking species (e.g., Anna’s Hummingbird vs. Ruby-throated
Hummingbird), the fine-grained recognition requires attention to fine details
like the bird’s beak and wings; meanwhile, at the coarse level (e.g., bird vs.
plant), the attention shifts to larger parts such as the overall body of the
bird. However, this shift in focus towards larger objects does not imply a
sudden disregard for the previously focused details and a search for new
larger objects. Rather, a natural approach involves combining detailed
features such as the bird’s beak, belly, and wings for accurate bird
recognition. Therefore, to maintain a coherent focus during hierarchical
recognition and ensure spatial consistency, we design a model where the
details learned at the fine level (e.g., bird’s beak and wings) are
transferred to the coarse level as broader parts (e.g., bird’s body) through
consistent feature grouping.

For internally consistent feature grouping, we build upon recent work CAST
[23]. CAST develops a hierarchical segmentation from fine to coarse, an
internal part of the recognition process. However, their segmentation is
driven by a flat recognition objective at the very end of visual parsing. We
extend it by imposing fine-to-coarse semantic classification losses at
different stages of segmentations throughout the visual parsing process. Our
design reflects the intuition that finer segments can be helpful in capturing
fine-grained details (e.g., beaks and wings) required for fine-grained
recognition, whereas coarser segments can be effective in representing broader
features (e.g., the body of a bird) needed for coarse-grained recognition. We
have a single hierarchical recognition grounded on internally consistent
segmentations, each driven by a classification objective at a certain
granuality. We refer to our method as Hierarchical-CAST (H-CAST).

Consider a hierarchical recognition task where $x$ denotes an image associated
with hierarchical labels $y_{1},\ldots y_{L}$, encompassing a total of $L$
levels in the hierarchy. Level $L$ is the finest level (_i.e.,_ leaf node),
and Level 1 is the coarsest level (_i.e.,_ root node). Then, given an image
$x$, the hierarchical image recognition task is to predict labels at all
levels across the hierarchy.

Let $Z_{l}$ and $S_{l}$ denote the feature and segments at $l$-th hierarchical
level, respectively, Then, we obtain superpixels for image $x$ by using the
off-the-shelf algorithm SEEDS [3] to divide the image into regions with
similar colors and local connectivity. These superpixels serve as input for
the Vision Transformer (ViT) instead of fixed-size patches and simultaneously
become the finest (initial) segments, $S_{L+1}$. $l$-th feature tokens $Z_{l}$
is the concatenation of class tokens ($Z_{l}^{class}$) and segment tokens
($Z_{l}^{seg}$). Then, Graph pooling [23] aggregates segments with high
feature similarity, allowing feature $Z_{l}$ to progressively learn a more
global visual context as it transitions from $Z_{L}$ to $Z_{1}$.

For hierarchical recognition, we add a classification head ($f_{l}$)
consisting of a single linear layer at each level. Then, we define the
Hierarchical Spatial-consistency loss as the sum of $L$ cross-entropy losses
($L_{CE}$), denoted as

$L_{HS}=\sum_{l=1}^{L}L_{CE}(f_{l}(Z_{l}^{class}),y_{l}).$ (1)

Our approach differs from CAST in that while CAST uses the class token as the
final objective, we design our model to incorporate hierarchical supervision
during the training process. This ensures that labels from different levels
progressively contribute to each other. In the Experimental section 4.3, we
will demonstrate the effectiveness of our design, compared to alternative
designs, including hierarchy supervision in the coarse-to-fine direction.

###  3.2 Tree-path KL Divergence loss for Semantic Consistency

To improve semantic consistency, we propose a new loss function called Tree-
path KL Divergence loss, which directly incorporates hierarchical
relationships between labels. Our idea is to encode the entire hierarchical
structure so that a model can learn the hierarchy by outputting the tree of
hierarchy. To this end, we first concatenate labels from all levels to create
a distribution, as $Y=\frac{1}{L}[1_{y_{L}};\ldots;1_{y_{1}}]$, where
$1_{y_{l}}$ represents the one-hot encoding for level $l$. Next, we
concatenate the outputs of each classification head and then apply the log
softmax function (LogSoftmax). We use Kullback–Leibler divergence loss ($KL$)
to align this output with the ground truth distribution $Y$. Then, TK loss is
calculated as follows.

$L_{TK}=KL(\text{LogSoftmax}([f_{L}(Z_{L}^{class});\ldots;f_{1}(Z_{1}^{class})]),Y)$
(2)

This loss penalizes predictions that do not align with the taxonomy by
simultaneously training on multiple labels within the hierarchy. Therefore,
despite the simplicity, TK loss enables the model to enhance semantic
consistency through this vertical encoding from the root (parent) node of the
hierarchy level to the leaf (children) node. Our final loss becomes as
follows, where $\alpha$ is a hyperparameter to control the weight of $L_{TK}$,

$L=L_{HS}+\alpha L_{TK}.$ (3)

##  4 Experiments

###  4.1 Experimental Settings

Datasets. We use three widely used benchmarks in hierarchical recognition:
FGVC-Aircraft [27], CUB-200-2011 [41], and BREEDS [35]. FGVC-Aircraft consists
of a 3-level hierarchy including maker, family, and model (_e.g._ , Boeing -
Boeing 707 - 707-320 ); CUB-200-2011 comprises a 3-level hierarchy with order,
family, and species; BREEDS, a subset of ImageNet [34], includes four 2-level
hierarchy datasets with different depths/parts based on the WordNet [28]
hierarchy: Living-17, Non-Living-26, Entity-13, Entity-30. For BREEDS, we
conduct training and validation using their source splits. We include detailed
dataset descriptions in Table 2.

Table 1: FPA takes both correctness and consistency into consideration. The table illustrates the difference between TICE [39] and FPA. The first row lists all possible predictions. The predictions at different levels are represented as highlighted circles. The borders of the nodes that are correctly predicted are marked in green, and vice versa. The second and third rows show whether TICE and FPA mark the prediction results as positive under different circumstances. While TICE only measures consistency, FPA will only mark the prediction that is correct and consistent as positive.  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
TICE | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓
FPA | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗
Table 2: Benchmark Datasets.

Datasets | Aircraft | CUB | L-17 | NL-26 | E-13 | E-30
---|---|---|---|---|---|---
# Levels | 3 | 3 | 2 | 2 | 2 | 2
# of classes | 30-70-100 | 13-38-200 | 17-34 | 26-52 | 13-130 | 30-120
# Train images | 6,667 | 5,944 | 44.2K | 65.7K | 167K | 154K
# Test images | 3,333 | 5,794 | 1.7K | 2.6K | 6.5K | 6K

Evaluation Metrics. We evaluate our models using metrics for both accuracy and
consistency. First, we present level-accuracy, following community convention
[6], which is the proportion of correctly classified instances at each level.
Additionally, we evaluate the weighted average precision (wAP) [26] which
considers the classification difficulty across different hierarchies, as
$\text{wAP}=\sum_{l=1}^{L}\frac{N_{l}}{\sum_{k=1}^{L}N_{k}}P_{l}$, where
$N_{l}$ and $P_{l}$ denote the number of classes and Top-1 classification
accuracy at level $l$, respectively. This metric gives more weight to the
accuracy at the fine-grained levels, which is more challenging. To evaluate
consistency we calculate Tree-based InConsistency Error rate (TICE) [39],
which tests whether the prediction path exists in the tree.
$\text{TICE}={n_{\text{i}c}}/{N}$, where $n_{\text{i}c}$ denotes the number of
samples with inconsistent prediction paths, and $N$ refers to the number of
all test samples. Finally, we propose Full-Path Accuracy (FPA) as a metric to
assess both accuracy and consistency: $\text{FPA}={n_{\text{a}c}}/{N}$, where
$n_{\text{a}c}$ refers to the number of samples with all level of labels
correctly predicted. This metric straightforwardly calculates the proportion
of correctly classified samples at all levels, ultimately representing our
primary metric of interest. The difference between FPA and TICE is illustrated
in Table 1.

Comparison methods. Our architecture is based on the ViT architecture [12], so
we use the ViT model as our baseline. As a ceiling-performance baseline, we
compare with “Single” models which are trained with a single flat level in the
hierarchy. ViT-Single is trained to classify one hierarchy level using the
class token of the ViT model. Therefore, for the Aircraft Dataset, ViT-Single
entails training three independent models. Likewise, CAST-Single trains
independent models, each dedicated to learning a single hierarchy level with
the CAST architecture [23]. ViT-Hier is similar to our approach in that it
trains each hierarchy level using the class token from the last $l$ blocks.

Architecture and Training. For a fair comparison, we use ViT-S and CAST-S
models of corresponding sizes. As in CAST, segmentation granularity is set to
64, 32, 16, 8 after 3, 3, 3, 2 encoder blocks, respectively. Our training
progresses from fine to coarse levels, with each segment corresponding
accordingly. The initial number of superpixels is set to 196, and all data is
trained with a batch size of 256 for 100 epochs. Following the literature, we
use ImageNet pre-trained models for the Aircraft and CUB datasets. For the
ImageNet subset BREEDS dataset, we train the models from scratch. Detailed
hyperparameter settings can be found in the Appendix A.1.

###  4.2 Hierarchical Recognition Benchmarks

Table 3: Comparison of Methods in terms of Accuracy(%) and Consistency
Metrics(%) on FGVC-Aircraft and CUB-200-2011 benchmarks. Single-level serves
as a celing-performance baseline, with ViT-Hier showing decreased accuracy
compared to individually trained Single-level models. However, Single-level
models’ consistency performance suffers due to separate models. In contrast,
our model (H-CAST) demonstrates comparable or higher accuracy than Single-
level models while significantly enhancing Consistency performance. (Higher
the metric is the best, except TICE.)

Method | FGVC-Aircraft (30-70-100) | CUB-200-2011 (13-38-200)
---|---|---
FPA | maker | family | model | wAP | TICE | FPA | order | family | species | wAP | TICE
ViT-Single | 76.99 | 94.27 | 91.93 | 80.14 | 86.39 | 10.98 | 82.30 | 98.50 | 94.84 | 84.78 | 87.01 | 5.76
Cast-Single | 78.22 | 92.95 | 88.93 | 82.39 | 86.26 | 10.77 | 81.50 | 98.38 | 94.82 | 83.78 | 86.21 | 6.14
ViT-Hier | 72.10 | 92.35 | 86.26 | 75.94 | 82.01 | 15.75 | 77.03 | 98.40 | 92.94 | 79.43 | 82.46 | 8.72
Ours | 83.72 | 94.96 | 91.39 | 85.33 | 88.90 | 5.01 | 83.28 | 98.65 | 95.12 | 84.86 | 87.13 | 4.12
_Our Gains_ | +11.62 | +2.61 | +5.13 | +9.39 | +6.89 | +10.74 | +6.25 | +0.25 | +2.18 | +5.53 | +4.67 | +4.60

FGVC-Aircraft and CUB datasets. Table 3 presents results on Aircraft and CUB
datasets. In our experimental results, we first observe a significant
performance drop of ViT-Hier compared to ViT-Flat. This highlights a common
issue in hierarchical recognition using shared backbones, where a coarse-fine
tradeoff leads to performance degradation, as seen in prior ResNet-based
hierarchical recognition [6]. Our experiments reveal that this problem also
exists in ViT architectures. This indicates that hierarchical recognition is a
challenging problem that cannot be solely addressed by providing hierarchy
supervision to class tokens.

On the other hand, our method consistently outperforms most Flat models.
Notably, we achieve substantial improvements in consistency performance as
well. While Flat-level models exhibit high accuracy, they require $L$ times
the training time and memory to train each model. Moreover, since they are
separate models, consistency suffers. In contrast, our approach excels in both
accuracy and consistency. Specifically, from the FPA metric viewpoint, which
emphasizes both accurate and consistent predictions across all levels, we
observe an approximately +11.6%p improvement in the Aircraft dataset and a
+6.3%p improvement in the CUB dataset compared to ViT-Hier. These results
validate our approach of utilizing segments in progressive learning to enhance
both accuracy and consistency in hierarchical recognition.

###  4.3 Analysis

BREEDS datasets. Next, we compare model performance on the BREEDS dataset,
which has a larger number of training data compared to Aircraft and CUB. As
shown in Table 4, we observe a similar pattern to the results from Aircraft
and CUB datasets. When the models are trained from scratch without using
ImageNet-pretrained models, our model surpasses ViT-Hier by a large margin in
both accuracy and consistency, by more than 10.63-11.82%p in the FPA metric.
H-CAST even surpasses CAST-Single models by 2.2-7.1%p in the FPA metric. This
demonstrates that utilizing hierarchical information results in a significant
performance improvement in recognition tasks when pre-trained weights are not
employed.

| Fine to coarse segments |  | Fine to coarse segments
---|---|---|---
.. |  . |  . |  . |  .. |  . |  . |  .
.. |  . |  . |  . |  .. |  . |  . |  .
.. |  . |  . |  . |  .. |  . |  . |  .
a) Full-Path Correct Predictions | b) Full-Path Incorrect Predictions
Figure 4: H-CAST enhances the interpretability of the model’s predictions. We
visualize the feature grouping from fine to coarse for full-path correct and
incorrect predictions of our approach on Entity-30 dataset. The hue for all
segments is consistent, while saturation and value vary to represent different
granularities. For full-path correct predictions (_i.e._ , predictions at all
levels are correct), visual details are effectively captured and grouped to
identify larger objects at coarser levels. In contrast, the model’s segments
fail to recognize the object accurately for full-path incorrect predictions
(_i.e._ , predictions of all levels are incorrect). This highlights an
additional advantage of utilizing segments in hierarchical recognition models,
as it not only provides guidance on where to focus but also enhances
interpretability of the model’s predictions.  Table 4: Comparison of Methods
in terms of Accuracy(%) and Consistency Metrics(%) on BREEDS benchmarks. Our
model (H-CAST) significantly outperforms ViT-Hier in both Accuracy and
Consistency metrics (Our Gains).

Method | Living-17 (17-34) | Non-Living-26 (26-52)
---|---|---
FPA | coarse | fine | wAP | TICE | FPA | coarse | fine | wAP | TICE
ViT-Single | 66.24 | 75.71 | 72.06 | 73.28 | 17.11 | 57.46 | 65.73 | 66.32 | 57.46 | 23.27
Cast-Single | 79.82 | 88.06 | 82.88 | 84.61 | 8.82 | 76.77 | 84.77 | 81.08 | 82.31 | 11.77
ViT-Hier | 74.06 | 80.94 | 74.88 | 76.90 | 10.50 | 72.04 | 73.31 | 68.39 | 70.03 | 12.45
Ours | 85.12 | 90.82 | 85.24 | 87.10 | 3.19 | 82.67 | 87.89 | 83.15 | 84.73 | 5.26
Our Gains | +11.06 | +9.88 | +10.36 | +10.20 | +7.31 | +10.63 | +14.58 | +14.76 | +14.70 | +7.19
| Entity-13 (13-130) | Entity-30 (30-120)
ViT-Single | 64.22 | 76.28 | 76.06 | 76.08 | 21.33 | 66.93 | 76.47 | 74.35 | 74.77 | 18.75
Cast-Single | 78.63 | 87.80 | 83.72 | 84.09 | 10.65 | 82.67 | 87.89 | 83.15 | 84.73 | 5.26
ViT-Hier | 74.63 | 86.95 | 75.39 | 77.70 | 5.19 | 73.01 | 81.38 | 74.10 | 74.76 | 11.61
Ours | 85.68 | 93.42 | 86.15 | 87.60 | 1.69 | 84.83 | 90.23 | 85.45 | 85.88 | 2.57
Our Gains | +11.05 | +6.47 | +10.76 | +9.90 | +3.50 | +11.82 | +8.85 | +11.35 | +11.12 | +9.04

Fine-to-Coarse vs Coarse-to-Fine learning. Our model adopts a Fine-to-Coarse
learning strategy, wherein it initially focuses on learning fine labels in the
lower block and progressively integrates to learn coarse labels. This is
contrary to conventional ResNet-based models, which typically learn coarser
features in the lower block [49, 44, 46]. Thus, to demonstrate the
effectiveness of this Fine-to-Coarse learning architecture, we compare our
model with two baselines. The first baseline, referred to as the Coarse-to-
Fine approach, initiates training by focusing on coarse labels in the lower
block and progresses towards fine labels, which is contrary to our approach.
The second baseline, the Fine-Coarse Merging method, concatenates the class
token from the lower block with coarser segments from the upper block to learn
coarse labels, and vice versa concatenates the class token from the upper
block with finer segments from the lower block to learn fine labels. This
approach leverages class tokens and segments of varying granularities
intuitively. For the sake of evaluating architecture effectiveness, we do not
use Tree-path KL Divergence loss. The comparison results of these three
architectures on FGVC-Aircraft dataset are summarized in Table 5.

Table 5: Coarse-to-Fine learning scheme achieves best overall performance. We
report the impact of hierarchical learning direction on FGVC-Aircraft.

Learning Direction | FPA | maker | family | model | wAP
---|---|---|---|---|---
Coarse-to-Fine | 82.01 | 93.16 | 89.92 | 84.10 | 87.50
Fine-Coarse merging | 81.76 | 93.52 | 90.31 | 84.58 | 87.93
Fine-to-Coarse | 82.66 | 94.27 | 90.19 | 84.40 | 87.91

We find that Coarse-to-Fine learning shows the lowest performance, contrary to
convention. Fine-Coarse Merging exhibits slightly better accuracy at the fine
level compared to our model. However, the performance improvement is marginal
considering the significant increase in the number of parameters in the
classification head when using segment features as input. Our Coarse-to-Fine
learning scheme not only demonstrates simplicity in implementation but also
showcases good performance, making it a favorable choice for our study.

Table 6: Utilizing both losses yields best performance. We report the effect
of proposed losses on FGVC-Aircraft.

$L_{HS}$ | $L_{TK}$ | FPA | maker | family | model | wAP
---|---|---|---|---|---|---
✗ | ✓ | 82.48 | 94.30 | 90.37 | 84.04 | 87.80
✓ | ✗ | 82.66 | 94.27 | 90.19 | 84.40 | 87.91
✓ | ✓ | 83.72 | 94.96 | 91.39 | 85.33 | 88.90

Effect of proposed losses. To validate the efficacy of the two losses we
proposed, we conduct an ablation study on FGVC-Aircraft dataset. Table 6
reveals that each loss term, Hierarchical Spatial-consistency loss, and Tree-
path KL Divergence loss, significantly contributes to performance enhancement.
Ultimately, utilizing both losses concurrently yields the best performance in
terms of accuracy and consistency.

H-CAST can guide consistent spatial focus and improve the interpretability of
the model’s predictions. To explore how our model learns segments with varying
granularity from fine to coarse levels, we visualize segments based on
hierarchy levels on BREEDS Entity-30 dataset in Figure 4. In full-path correct
prediction cases, where predictions at all levels are correct (Figure 4,
Left), visual details are effectively captured at the fine level and
consistently grouped to identify larger objects at the coarse level. However,
in full-path incorrect prediction cases, where predictions at all levels are
incorrect (Figure 4, Right), the model’s segments fail to recognize the object
accurately. This underscores an added benefit of incorporating segments in
hierarchical recognition models, as it not only contributes to consistent
predictions but also enhances the interpretability of the model’s predictions.

Hierarchical semantic recognition enhances internal segmentation. While our
primary task is hierarchical recognition, we further examine whether
incorporating additional hierarchical label information can be beneficial for
image segmentation. To investigate this, we compare CAST, which employs flat-
level recognition, with segmentation results. In Figure 5, it can be observed
that utilizing additional taxonomy benefits segmentation. For example in the
first ‘bird’ image, H-CAST is able to segment meaningful parts such as the
face, belly, and a branch, with less fractured compared to the CAST.

To quantitatively evaluate the method, we use the ImageNet segmentation
dataset, ImageNet-S [15], to obtain the ground-truth segmentation data for
BREEDS dataset. The number of samples in the BREEDS validation data for which
ground-truth segmentation data can be obtained from ImageNet-S is 381 for
Living-17, 510 for Non-Living-26, 1,336 for Entity-30, and 1,463 for
Entity-13. To calculate the region mIOU for fine-level objects, we use the
last-level segments (8-way) for segmentation. Following CAST, we name the
8-way segmentations using OvSEG [25].

Table in Figure 6 presents a comparison of region mIOU. Since H-CAST uses
coarse-level supervision for the last-level segments, H-CAST can underperform
compared to CAST, which employs fine-level supervision. Nevertheless, H-CAST
outperforms CAST in most datasets, such as Living-17, Non-Living-26, and
Entity-13. Additionally, Figure 6 illustrates the segmentation results. In the
visualized images, we can observe that H-CAST better captures the overall
shape in a more coherent manner compared to CAST. In the Entity-30 example of
“llama” in the second row, CAST demonstrates better results than H-CAST.
H-CAST, in this case, incorrectly segments the face and body into different
groups. We believe that finetuning the model, which was trained with
hierarchical recognition, using segmentation data can further improve its
performance. These quantitative and qualitative results demonstrate that
utilizing hierarchical information is effective not only for recognition but
also for segmentation.

| CAST | H-CAST |  | CAST | H-CAST |  | CAST | H-CAST
---|---|---|---|---|---|---|---|---
. |  . |  |  . |  . |  |  . |  . |
. |  . |  |  . |  . |  |  . |  . |
Figure 5: Additional taxonomy information improves segmentation. We compare our segmentation results on Entity-30 dataset with CAST at the coarsest level to explore the potential benefits of incorporating additional hierarchical label information for image segmentation. For example in the first ‘bird’ image, H-CAST is able to segment meaningful parts such as the face, belly, and a branch, with less fractured compared to the CAST. Thus, H-CAST delivers an improvement in segmentation with the benefits of hierarchy. More segmentation results are shown in Appendix A.2.  |  |  CAST |  H-CAST |  GT |  |  CAST |  H-CAST |  GT
---|---|---|---|---|---|---|---|---
Living-17 |  .. |  . |  . |  . |  .. |  . |  . |  .
Entity-30 |  .. |  . |  . |  . |  .. |  . |  . |  .
Entity-13 |  .. |  . |  . |  . |
Figure 6: Using hierarchical information helps in segmenting object parts on
BREEDS. To further verify the effectiveness of using hierarchy information in
segmentation, we visualize segmentation results and measure the region mIOU of
Fine-level objects for samples whose segmentation ground truth (GT) data can
be obtained from ImageNet-S [15]. Following CAST, we use the last-level
segments and name the 8-way segmentations using OvSEG [25]. In the visualized
images, we can observe that H-CAST better captures the overall shape in a more
coherent manner compared to CAST. In the Entity-30 example of “llama” in the
second row, CAST demonstrates better results than H-CAST. H-CAST, in this
case, incorrectly segments the face and body into different groups. We believe
that finetuning the model, which was trained with hierarchical recognition,
using segmentation data can further improve its performance. In quantitative
evaluation, H-CAST outperforms CAST in most datasets despite using coarse-
level supervision for the last-level segments whereas CAST employs fine-level
supervision. These qualitative and quantitative results demonstrate that
utilizing additional taxonomy is effective not only for recognition but also
for segmentation.

## References

  * [1] Barutcuoglu, Z., DeCoro, C.: Hierarchical shape classification using bayesian aggregation. In: IEEE International Conference on Shape Modeling and Applications 2006 (SMI’06). pp. 44–44. IEEE (2006)
  * [2] Barz, B., Denzler, J.: Hierarchy-based image embeddings for semantic image retrieval. In: 2019 IEEE winter conference on applications of computer vision (WACV). IEEE (2019)
  * [3] Van den Bergh, M., Boix, X., Roig, G., De Capitani, B., Van Gool, L.: Seeds: Superpixels extracted via energy-driven sampling. In: Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VII 12 (2012)
  * [4] Bertinetto, L., Mueller, R., Tertikas, K., Samangooei, S., Lord, N.A.: Making better mistakes: Leveraging class hierarchies with deep networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2020)
  * [5] Brust, C.A., Denzler, J.: Integrating domain knowledge: using hierarchies to improve deep classifiers. In: Asian conference on pattern recognition. pp. 3–16. Springer (2019)
  * [6] Chang, D., Pang, K., Zheng, Y., Ma, Z., Song, Y.Z., Guo, J.: Your" flamingo" is my" bird": fine-grained, or not. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021)
  * [7] Chen, J., Wang, P., Liu, J., Qian, Y.: Label relation graphs enhanced hierarchical residual network for hierarchical multi-granularity classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022)
  * [8] Chen, T., Wu, W., Gao, Y., Dong, L., Luo, X., Lin, L.: Fine-grained representation learning and recognition by exploiting hierarchical semantic embedding. In: Proceedings of the 26th ACM international conference on Multimedia (2018)
  * [9] Cubuk, E.D., Zoph, B., Shlens, J., Le, Q.V.: Randaugment: Practical automated data augmentation with a reduced search space. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (2020)
  * [10] Defiyanti, S., Winarko, E., Priyanta, S.: A survey of hierarchical classification algorithms with big-bang approach. In: 2019 5th International Conference on Science and Technology (ICST). vol. 1, pp. 1–6. IEEE (2019)
  * [11] Deng, J., Berg, A.C., Li, K., Fei-Fei, L.: What does classifying more than 10,000 image categories tell us? In: Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part V 11. pp. 71–84. Springer (2010)
  * [12] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [13] Dumais, S., Chen, H.: Hierarchical classification of web content. In: Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval. pp. 256–263 (2000)
  * [14] Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., Mikolov, T.: Devise: A deep visual-semantic embedding model. Advances in neural information processing systems (2013)
  * [15] Gao, S., Li, Z.Y., Yang, M.H., Cheng, M.M., Han, J., Torr, P.: Large-scale unsupervised semantic segmentation. TPAMI (2022)
  * [16] Garnot, V.S.F., Landrieu, L.: Leveraging class hierarchies with metric-guided prototype learning. arXiv preprint arXiv:2007.03047 (2020)
  * [17] Giunchiglia, E., Lukasiewicz, T.: Coherent hierarchical multi-label classification networks. Advances in neural information processing systems (2020)
  * [18] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
  * [19] Horn, G.V., Aodha, O.M., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., Belongie, S.J.: The inaturalist species classification and detection dataset. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 (2018)
  * [20] Hwang, J.J., Yu, S.X., Shi, J., Collins, M.D., Yang, T.J., Zhang, X., Chen, L.C.: Segsort: Segmentation by discriminative sorting of segments. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7334–7344 (2019)
  * [21] Jin, B., Muller, B., Zhai, C., Lu, X.: Multi-label literature classification based on the gene ontology graph. BMC bioinformatics 9, 1–15 (2008)
  * [22] Ke, T.W., Hwang, J.J., Guo, Y., Wang, X., Yu, S.X.: Unsupervised hierarchical semantic segmentation with multiview cosegmentation and clustering transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2571–2581 (2022)
  * [23] Ke, T.W., Mo, S., Yu, S.X.: Learning hierarchical image segmentation for recognition and by recognition. In: The Twelfth International Conference on Learning Representations (2024)
  * [24] Khrulkov, V., Mirvakhabova, L., Ustinova, E., Oseledets, I., Lempitsky, V.: Hyperbolic image embeddings. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2020)
  * [25] Liang, F., Wu, B., Dai, X., Li, K., Zhao, Y., Zhang, H., Zhang, P., Vajda, P., Marculescu, D.: Open-vocabulary semantic segmentation with mask-adapted clip. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)
  * [26] Liu, Y., Zhou, L., Zhang, P., Bai, X., Gu, L., Yu, X., Zhou, J., Hancock, E.R.: Where to focus: Investigating hierarchical attention relationship for fine-grained visual classification. In: European Conference on Computer Vision. Springer (2022)
  * [27] Maji, S., Kannala, J., Rahtu, E., Blaschko, M., Vedaldi, A.: Fine-grained visual classification of aircraft. Tech. rep. (2013)
  * [28] Miller, G.A.: Wordnet: a lexical database for english. Communications of the ACM 38(11), 39–41 (1995)
  * [29] Naik, A., Rangwala, H.: Inconsistent node flattening for improving top-down hierarchical classification. In: 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA). pp. 379–388. IEEE (2016)
  * [30] Nickel, M., Kiela, D.: Poincaré embeddings for learning hierarchical representations. Advances in neural information processing systems (2017)
  * [31] Ouali, Y., Hudelot, C., Tami, M.: Autoregressive unsupervised image segmentation. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16. pp. 142–158. Springer (2020)
  * [32] Punera, K., Ghosh, J.: Enhanced hierarchical classification via isotonic smoothing. In: Proceedings of the 17th international conference on World Wide Web. pp. 151–160 (2008)
  * [33] Redmon, J., Farhadi, A.: Yolo9000: better, faster, stronger. In: Proceedings of the IEEE conference on computer vision and pattern recognition (2017)
  * [34] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) (2015)
  * [35] Santurkar, S., Tsipras, D., Madry, A.: {BREEDS}: Benchmarks for subpopulation shift. In: International Conference on Learning Representations (2021)
  * [36] Silla, C.N., Freitas, A.A.: A survey of hierarchical classification across different application domains. Data mining and knowledge discovery 22, 31–72 (2011)
  * [37] Sun, A., Lim, E.P.: Hierarchical text classification and evaluation. In: Proceedings 2001 IEEE International Conference on Data Mining. pp. 521–528. IEEE (2001)
  * [38] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition (2016)
  * [39] Wang, R., Zou, C., Zhang, W., Zhu, Z., Jing, L.: Consistency-aware feature learning for hierarchical fine-grained visual classification. In: Proceedings of the 31st ACM International Conference on Multimedia (2023)
  * [40] Wehrmann, J., Cerri, R., Barros, R.: Hierarchical multi-label classification networks. In: International conference on machine learning. PMLR (2018)
  * [41] Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., Perona, P.: Caltech-UCSD Birds 200. Tech. Rep. CNS-TR-2010-001, California Institute of Technology (2010)
  * [42] Wu, H., Merler, M., Uceda-Sosa, R., Smith, J.R.: Learning to make better mistakes: Semantics-aware visual food recognition. In: Proceedings of the 24th ACM international conference on Multimedia. pp. 172–176 (2016)
  * [43] Xian, Y., Akata, Z., Sharma, G., Nguyen, Q., Hein, M., Schiele, B.: Latent embeddings for zero-shot classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 69–77 (2016)
  * [44] Yan, Z., Zhang, H., Piramuthu, R., Jagadeesh, V., DeCoste, D., Di, W., Yu, Y.: Hd-cnn: hierarchical deep convolutional neural networks for large scale visual recognition. In: Proceedings of the IEEE international conference on computer vision (2015)
  * [45] Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: Cutmix: Regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6023–6032 (2019)
  * [46] Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13 (2014)
  * [47] Zeng, S., des Combes, R.T., Zhao, H.: Learning structured representations by embedding class hierarchy. In: The Eleventh International Conference on Learning Representations (2022)
  * [48] Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017)
  * [49] Zhu, X., Bain, M.: B-cnn: branch convolutional neural network for hierarchical classification. arXiv preprint arXiv:1709.09890 (2017)

##  Appendix A Appendix

###  A.1 Hyperparameters for Training.

Table 7: Hyper-parameters for training H-CAST and ViT on FGVC-Aircraft, CUB-200-2011, and BREEDS. We follow mostly the same set up as CAST [23].  Parameter | Aircraft | CUB | BREEDS
---|---|---|---
batch_size | 256 | 256 | 256
crop_size | 224 | 224 | 224
learning_rate | $1e^{-3}$ | $5^{e-4}$ | $5^{e-4}$
weight_decay | 0.05 | 0.05 | 0.05
momentum | 0.9 | 0.9 | 0.9
total_epochs | 100 | 100 | 100
warmup_epochs | 5 | 5 | 5
warmup_learning_rate | $1e^{-4}$ | $1e^{-6}$ | $1^{e-6}$
optimizer | Adam | Adam | Adam
learning_rate_policy | Cosine decay | Cosine decay | Cosine decay
augmentation [9] | RandAug(9, 0.5) | RandAug(9, 0.5) | RandAug(9, 0.5)
label_smoothing [38] | 0.1 | 0.1 | 0.1
mixup [48] | 0.8 | 0.8 | 0.8
cutmix [45] | 1.0 | 1.0 | 1.0
ViT: # Tokens | $[196]_{\times 11}$
CAST-S: # Tokens | $[196]_{\times 3},[64]_{\times 3},[32]_{\times 3},[16]_{\times 2}$

###  A.2 Additional Visulizations

image | CAST: Fine to coarse segments | H-CAST: Fine to coarse segments
---|---|---
.. |  . |  . |  .. |  . |  . |
.. |  . |  . |  .. |  . |  . |
.. |  . |  . |  .. |  . |  . |
.. |  . |  . |  .. |  . |  . |
.. |  . |  . |  .. |  . |  . |
.. |  . |  . |  .. |  . |  . |
.. |  . |  . |  .. |  . |  . |
.. |  . |  . |  .. |  . |  . |
.. |  . |  . |  .. |  . |  . |
.. |  . |  . |  .. |  . |  . |
Figure 7: Additional visual results on segmentation show that H-CAST with
additional taxonomy information improves segmentation.

###  A.3 Limitations

A limitation of our work, similar to CAST, is that the performance of segment
discovery relies on the superpixel algorithm. However, our work aims to
increase consistency at the hierarchical level, ensuring classifiers focus on
the same regions, rather than achieving precise segmentation. Therefore, the
inability to find accurate contours is not a significant issue.

## NeurIPS Paper Checklist

  1. 1.

Claims

  2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

  3. Answer: [Yes]

  4. Justification: The abstract and introduction clearly state our claims and contributions.

  5. Guidelines:

     * •

The answer NA means that the abstract and introduction do not include the
claims made in the paper.

     * •

The abstract and/or introduction should clearly state the claims made,
including the contributions made in the paper and important assumptions and
limitations. A No or NA answer to this question will not be perceived well by
the reviewers.

     * •

The claims made should match theoretical and experimental results, and reflect
how much the results can be expected to generalize to other settings.

     * •

It is fine to include aspirational goals as motivation as long as it is clear
that these goals are not attained by the paper.

  6. 2.

Limitations

  7. Question: Does the paper discuss the limitations of the work performed by the authors?

  8. Answer: [Yes]

  9. Justification: Limitation is described in Appendix.

  10. Guidelines:

     * •

The answer NA means that the paper has no limitation while the answer No means
that the paper has limitations, but those are not discussed in the paper.

     * •

The authors are encouraged to create a separate "Limitations" section in their
paper.

     * •

The paper should point out any strong assumptions and how robust the results
are to violations of these assumptions (e.g., independence assumptions,
noiseless settings, model well-specification, asymptotic approximations only
holding locally). The authors should reflect on how these assumptions might be
violated in practice and what the implications would be.

     * •

The authors should reflect on the scope of the claims made, e.g., if the
approach was only tested on a few datasets or with a few runs. In general,
empirical results often depend on implicit assumptions, which should be
articulated.

     * •

The authors should reflect on the factors that influence the performance of
the approach. For example, a facial recognition algorithm may perform poorly
when image resolution is low or images are taken in low lighting. Or a speech-
to-text system might not be used reliably to provide closed captions for
online lectures because it fails to handle technical jargon.

     * •

The authors should discuss the computational efficiency of the proposed
algorithms and how they scale with dataset size.

     * •

If applicable, the authors should discuss possible limitations of their
approach to address problems of privacy and fairness.

     * •

While the authors might fear that complete honesty about limitations might be
used by reviewers as grounds for rejection, a worse outcome might be that
reviewers discover limitations that aren’t acknowledged in the paper. The
authors should use their best judgment and recognize that individual actions
in favor of transparency play an important role in developing norms that
preserve the integrity of the community. Reviewers will be specifically
instructed to not penalize honesty concerning limitations.

  11. 3.

Theory Assumptions and Proofs

  12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

  13. Answer: [N/A]

  14. Justification: The paper does not include theoretical results.

  15. Guidelines:

     * •

The answer NA means that the paper does not include theoretical results.

     * •

All the theorems, formulas, and proofs in the paper should be numbered and
cross-referenced.

     * •

All assumptions should be clearly stated or referenced in the statement of any
theorems.

     * •

The proofs can either appear in the main paper or the supplemental material,
but if they appear in the supplemental material, the authors are encouraged to
provide a short proof sketch to provide intuition.

     * •

Inversely, any informal proof provided in the core of the paper should be
complemented by formal proofs provided in appendix or supplemental material.

     * •

Theorems and Lemmas that the proof relies upon should be properly referenced.

  16. 4.

Experimental Result Reproducibility

  17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

  18. Answer: [Yes]

  19. Justification: We provide details of our experiments in Section 4.1 and training hyperparameters in Appendix A.1.

  20. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

If the paper includes experiments, a No answer to this question will not be
perceived well by the reviewers: Making the paper reproducible is important,
regardless of whether the code and data are provided or not.

     * •

If the contribution is a dataset and/or model, the authors should describe the
steps taken to make their results reproducible or verifiable.

     * •

Depending on the contribution, reproducibility can be accomplished in various
ways. For example, if the contribution is a novel architecture, describing the
architecture fully might suffice, or if the contribution is a specific model
and empirical evaluation, it may be necessary to either make it possible for
others to replicate the model with the same dataset, or provide access to the
model. In general. releasing code and data is often one good way to accomplish
this, but reproducibility can also be provided via detailed instructions for
how to replicate the results, access to a hosted model (e.g., in the case of a
large language model), releasing of a model checkpoint, or other means that
are appropriate to the research performed.

     * •

While NeurIPS does not require releasing code, the conference does require all
submissions to provide some reasonable avenue for reproducibility, which may
depend on the nature of the contribution. For example

       1. (a)

If the contribution is primarily a new algorithm, the paper should make it
clear how to reproduce that algorithm.

       2. (b)

If the contribution is primarily a new model architecture, the paper should
describe the architecture clearly and fully.

       3. (c)

If the contribution is a new model (e.g., a large language model), then there
should either be a way to access this model for reproducing the results or a
way to reproduce the model (e.g., with an open-source dataset or instructions
for how to construct the dataset).

       4. (d)

We recognize that reproducibility may be tricky in some cases, in which case
authors are welcome to describe the particular way they provide for
reproducibility. In the case of closed-source models, it may be that access to
the model is limited in some way (e.g., to registered users), but it should be
possible for other researchers to have some path to reproducing or verifying
the results.

  21. 5.

Open access to data and code

  22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

  23. Answer: [No]

  24. Justification: We will provide the code and checkpoints of the model upon publication.

  25. Guidelines:

     * •

The answer NA means that paper does not include experiments requiring code.

     * •

Please see the NeurIPS code and data submission guidelines
(https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.

     * •

While we encourage the release of code and data, we understand that this might
not be possible, so “No” is an acceptable answer. Papers cannot be rejected
simply for not including code, unless this is central to the contribution
(e.g., for a new open-source benchmark).

     * •

The instructions should contain the exact command and environment needed to
run to reproduce the results. See the NeurIPS code and data submission
guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more
details.

     * •

The authors should provide instructions on data access and preparation,
including how to access the raw data, preprocessed data, intermediate data,
and generated data, etc.

     * •

The authors should provide scripts to reproduce all experimental results for
the new proposed method and baselines. If only a subset of experiments are
reproducible, they should state which ones are omitted from the script and
why.

     * •

At submission time, to preserve anonymity, the authors should release
anonymized versions (if applicable).

     * •

Providing as much information as possible in supplemental material (appended
to the paper) is recommended, but including URLs to data and code is
permitted.

  26. 6.

Experimental Setting/Details

  27. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

  28. Answer: [Yes]

  29. Justification: We provide details of our experiments in Section 4.1 and training hyperparameters in Appendix A.1.

  30. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The experimental setting should be presented in the core of the paper to a
level of detail that is necessary to appreciate the results and make sense of
them.

     * •

The full details can be provided either with the code, in appendix, or as
supplemental material.

  31. 7.

Experiment Statistical Significance

  32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

  33. Answer: [No]

  34. Justification: Due to computational resource limitations, we were unable to conduct a sufficient number of experiments to generate error bars.

  35. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The authors should answer "Yes" if the results are accompanied by error bars,
confidence intervals, or statistical significance tests, at least for the
experiments that support the main claims of the paper.

     * •

The factors of variability that the error bars are capturing should be clearly
stated (for example, train/test split, initialization, random drawing of some
parameter, or overall run with given experimental conditions).

     * •

The method for calculating the error bars should be explained (closed form
formula, call to a library function, bootstrap, etc.)

     * •

The assumptions made should be given (e.g., Normally distributed errors).

     * •

It should be clear whether the error bar is the standard deviation or the
standard error of the mean.

     * •

It is OK to report 1-sigma error bars, but one should state it. The authors
should preferably report a 2-sigma error bar than state that they have a 96%
CI, if the hypothesis of Normality of errors is not verified.

     * •

For asymmetric distributions, the authors should be careful not to show in
tables or figures symmetric error bars that would yield results that are out
of range (e.g. negative error rates).

     * •

If error bars are reported in tables or plots, The authors should explain in
the text how they were calculated and reference the corresponding figures or
tables in the text.

  36. 8.

Experiments Compute Resources

  37. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

  38. Answer: [Yes]

  39. Justification: We use four 40GB NVIDIA A40 GPUs for our experiments. We provide details of training hyperparameters in Appendix A.1.

  40. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The paper should indicate the type of compute workers CPU or GPU, internal
cluster, or cloud provider, including relevant memory and storage.

     * •

The paper should provide the amount of compute required for each of the
individual experimental runs as well as estimate the total compute.

     * •

The paper should disclose whether the full research project required more
compute than the experiments reported in the paper (e.g., preliminary or
failed experiments that didn’t make it into the paper).

  41. 9.

Code Of Ethics

  42. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

  43. Answer: [Yes]

  44. Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.

  45. Guidelines:

     * •

The answer NA means that the authors have not reviewed the NeurIPS Code of
Ethics.

     * •

If the authors answer No, they should explain the special circumstances that
require a deviation from the Code of Ethics.

     * •

The authors should make sure to preserve anonymity (e.g., if there is a
special consideration due to laws or regulations in their jurisdiction).

  46. 10.

Broader Impacts

  47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

  48. Answer: [N/A]

  49. Justification: There is no societal impact of the work performed.

  50. Guidelines:

     * •

The answer NA means that there is no societal impact of the work performed.

     * •

If the authors answer NA or No, they should explain why their work has no
societal impact or why the paper does not address societal impact.

     * •

Examples of negative societal impacts include potential malicious or
unintended uses (e.g., disinformation, generating fake profiles,
surveillance), fairness considerations (e.g., deployment of technologies that
could make decisions that unfairly impact specific groups), privacy
considerations, and security considerations.

     * •

The conference expects that many papers will be foundational research and not
tied to particular applications, let alone deployments. However, if there is a
direct path to any negative applications, the authors should point it out. For
example, it is legitimate to point out that an improvement in the quality of
generative models could be used to generate deepfakes for disinformation. On
the other hand, it is not needed to point out that a generic algorithm for
optimizing neural networks could enable people to train models that generate
Deepfakes faster.

     * •

The authors should consider possible harms that could arise when the
technology is being used as intended and functioning correctly, harms that
could arise when the technology is being used as intended but gives incorrect
results, and harms following from (intentional or unintentional) misuse of the
technology.

     * •

If there are negative societal impacts, the authors could also discuss
possible mitigation strategies (e.g., gated release of models, providing
defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms
to monitor how a system learns from feedback over time, improving the
efficiency and accessibility of ML).

  51. 11.

Safeguards

  52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

  53. Answer: [N/A]

  54. Justification: The paper poses no such risks.

  55. Guidelines:

     * •

The answer NA means that the paper poses no such risks.

     * •

Released models that have a high risk for misuse or dual-use should be
released with necessary safeguards to allow for controlled use of the model,
for example by requiring that users adhere to usage guidelines or restrictions
to access the model or implementing safety filters.

     * •

Datasets that have been scraped from the Internet could pose safety risks. The
authors should describe how they avoided releasing unsafe images.

     * •

We recognize that providing effective safeguards is challenging, and many
papers do not require this, but we encourage authors to take this into account
and make a best faith effort.

  56. 12.

Licenses for existing assets

  57. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

  58. Answer: [Yes]

  59. Justification: Our code is based on CAST [23], and CAST follows MIT license.

  60. Guidelines:

     * •

The answer NA means that the paper does not use existing assets.

     * •

The authors should cite the original paper that produced the code package or
dataset.

     * •

The authors should state which version of the asset is used and, if possible,
include a URL.

     * •

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

     * •

For scraped data from a particular source (e.g., website), the copyright and
terms of service of that source should be provided.

     * •

If assets are released, the license, copyright information, and terms of use
in the package should be provided. For popular datasets,
paperswithcode.com/datasets has curated licenses for some datasets. Their
licensing guide can help determine the license of a dataset.

     * •

For existing datasets that are re-packaged, both the original license and the
license of the derived asset (if it has changed) should be provided.

     * •

If this information is not available online, the authors are encouraged to
reach out to the asset’s creators.

  61. 13.

New Assets

  62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

  63. Answer: [N/A]

  64. Justification: Our code is based on CAST. We will make our codes publicly available.

  65. Guidelines:

     * •

The answer NA means that the paper does not release new assets.

     * •

Researchers should communicate the details of the dataset/code/model as part
of their submissions via structured templates. This includes details about
training, license, limitations, etc.

     * •

The paper should discuss whether and how consent was obtained from people
whose asset is used.

     * •

At submission time, remember to anonymize your assets (if applicable). You can
either create an anonymized URL or include an anonymized zip file.

  66. 14.

Crowdsourcing and Research with Human Subjects

  67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

  68. Answer: [N/A]

  69. Justification: The paper does not involve crowdsourcing nor research with human subjects.

  70. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Including this information in the supplemental material is fine, but if the
main contribution of the paper involves human subjects, then as much detail as
possible should be included in the main paper.

     * •

According to the NeurIPS Code of Ethics, workers involved in data collection,
curation, or other labor should be paid at least the minimum wage in the
country of the data collector.

  71. 15.

Institutional Review Board (IRB) Approvals or Equivalent for Research with
Human Subjects

  72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

  73. Answer: [N/A]

  74. Justification: The paper does not involve crowdsourcing nor research with human subjects.

  75. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Depending on the country in which research is conducted, IRB approval (or
equivalent) may be required for any human subjects research. If you obtained
IRB approval, you should clearly state this in the paper.

     * •

We recognize that the procedures for this may vary significantly between
institutions and locations, and we expect authors to adhere to the NeurIPS
Code of Ethics and the guidelines for their institution.

     * •

For initial submissions, do not include any information that would break
anonymity (if applicable), such as the institution conducting the review.