Convolutional Neural Networks Rarely Learn Shape for Semantic
  Segmentation
Shape learning, or the ability to leverage shape information, could be a
desirable property of convolutional neural networks (CNNs) when target objects
have specific shapes. While some research on the topic is emerging, there is no
systematic study to conclusively determine whether and under what circumstances
CNNs learn shape. Here, we present such a study in the context of segmentation
networks where shapes are particularly important. We define shape and propose a
new behavioral metric to measure the extent to which a CNN utilizes shape
information. We then execute a set of experiments with synthetic and real-world
data to progressively uncover under which circumstances CNNs learn shape and
what can be done to encourage such behavior. We conclude that (i) CNNs do not
learn shape in typical settings but rather rely on other features available to
identify the objects of interest, (ii) CNNs can learn shape, but only if the
shape is the only feature available to identify the object, (iii) sufficiently
large receptive field size relative to the size of target objects is necessary
for shape learning; (iv) a limited set of augmentations can encourage shape
learning; (v) learning shape is indeed useful in the presence of
out-of-distribution data.
Yixin Zhang, Maciej A. Mazurowski
# Convolutional Neural Networks Rarely Learn Shape for Semantic Segmentation

###### Abstract

Shape learning, or the ability to leverage shape information, could be a desirable property of convolutional neural networks (CNNs) when target objects have specific shapes. While some research on the topic is emerging, there is no systematic study to conclusively determine whether and under what circumstances CNNs learn shape. Here, we present such a study in the context of segmentation networks where shapes are particularly important. We define shape and propose a new behavioral metric to measure the extent to which a CNN utilizes shape information. We then execute a set of experiments with synthetic and real-world data to progressively uncover under which circumstances CNNs learn shape and what can be done to encourage such behavior. We conclude that (i) CNNs do not learn shape in typical settings but rather rely on other features available to identify the objects of interest, (ii) CNNs can learn shape, but only if the shape is the only feature available to identify the object, (iii) sufficiently large receptive field size relative to the size of target objects is necessary for shape learning; (iv) a limited set of augmentations can encourage shape learning; (v) learning shape is indeed useful in the presence of out-of-distribution data.

Segmentation Feature Measurement Machine Learning Computer Vision Since the breakthrough made by AlexNet on ImageNet [19], CNNs have achieved promising results on multiple image classification, segmentation, and object detection tasks [9, 11, 29, 30, 32]. Early research attributed CNNs' successes to the similarity between CNNs and a human's visual system. It was presumed that CNNs learn simple low-level features (i.e., edges) in shallow layers and combine them to form complex global shapes until objects can be recognized [18, 23]. Recent data suggest that CNNs utilize only limited shape information in a typical setting[7, 13, 14]. However, a systematic study of the issue is missing. If CNNs indeed do not learn shape, this property of CNNs is likely to lead to their limited ability to generalize in settings where noise distribution, brightness, or other appearance change.

Learning shape by CNNs could be of benefit. If we encourage CNNs to use shape information, they may learn and utilize a feature set more robust against common appearance disparities between training and testing images[26]. An example of an application area where this might be of particular benefit is medical. Magnetic resonance imaging (MRI) and computed tomography (CT) images maintain the appearance of shapes for a given patient because they are typically filmed at a fixed position relative to body parts with similar 3D geometry. However, each image's texture, noise distribution, and intensity/color can highly vary depending on a device's hardware specifications and imaging parameters. CNNs that learn shape could rely less on inconsistent characteristics but focus on the more invariant object shape.

Shape-learning CNNs may also be more robust against adversarial attacks. Recent research perceives adversarial vulnerability as a property of the data set rather than the network. Specifically, adversarial vulnerability arises from the presence of highly predictive but non-robust features uninterpretable to humans in the training set [16]. If we consider adversarial vulnerability arising from uninterpretable features, CNNs trained to utilize human-interpretable shape features may show higher robustness against adversarial attacks.
A systematic study on data factors that affect the shape learning of CNNs and the circumstances under which shapes can reliably be learned in real-life scenarios is lacking. However, there are some studies that while offering limited general interpretation of their results, provide some information on specific aspects of the question. Kubilius et al. [20] created a validation set entirely of object silhouettes. They found that CNNs still showed predictive capability on the silhouettes, although their performance dropped by 30%. Additionally, they found that objects in the same super-ordinate category have similar representations but offered little interpretation of this observation. Baker et al. [1, 2] show that texture plays a more significant role than global shape in ImageNet-trained CNN's perception of objects, where CNNs barely learn the abstract relations of elements. The authors made no further comments on the cause of this phenomenon. Geirhos et al. [7] also showed that ImageNet-trained CNNs rely heavily on textures for classification, instead of using a uses a conflict-cue method. The authors suggest the potential of applying neural style transfer (NST) as an augmentation to encourage shape learning for better testing accuracy and OOD robustness. Ritter et al.[31] demonstrated that Inception and Matching Nets[35] pre-trained on ImageNet tend to generate latent representations with higher similarity for objects with similar shapes than those with similar colors. However, it would be premature to conclude CNNs are shape-learning solely through their insensitivity to colors. Hermann et al.[13, 14] show that CNNs with the same model architecture may have vastly different shape learning behaviors for different training datasets and objectives. Their configurations have direct impacts on the feature extraction process of the SN. Though innovative in its nature, their research ended with a conclusion that the "most decodable feature from untrained mode" will be learned, without a further account of what it practically means by "decodable". Hosseini et al.[15] state that proper augmentation, initialization, and the use of batch normalization are essential for networks to learn shape. Li et al. [24] combined training with stylized images, soft labeling, and weighted loss as a heuristic to gain more control over the inductive bias towards texture or shape during the model training. Their approach achieved improved performance on the ImageNet-C benchmark [12]. Islam et al. [17] proposed a dimensionality estimation metric and used the metric to assess a CNN's capacity to represent shape in latent space. Their study answers the question "whether a CNN is capable of learning shape" and "where are those representations stored in the CNN" but did not address which factors encourage shape learning in a general setting.

While the prior work has limitations, we learn the following. First, neural networks might not learn shape as often as one intuitively expected. Second, whether the network learns shape may depend on various factors, but it is not entirely clear which factors those are.

In this paper, we devise a progressively complex systematic experimental study to answer the general questions related to learning shape. We do so in the context of segmentation networks, where shape plays a particularly important role. Specifically, using synthetic datasets, we explore a model's ability to learn shapes by controlling data parameters, model receptive fields, and object target sizes. We also observe the behavioral changes of shape-learning networks when they encounter target objects with variations in brightness and shape. It allows us to better understand shape-learning SNs' responses to different OOD data. Then, we validate our conclusions on three collections of real datasets to ensure our conclusions apply outside of synthetic scenarios. Though we conduct experiments with the setting of UNet-like networks on segmentation tasks, our conclusions may also apply to image classification and object detection tasks. Segmentation is a special case of image classification (namely pixel-wise classification); it thus shares a similar workflow to that of classification. (3) For object detection, a fully convolutional network (a superclass of UNet) is a major building block in the feature extraction head adapted by models specialized for object detection such as Mask R-CNN and YOLO. The behavior of the fully convolutional network should resemble that of UNet.

Our contributions are briefly summarized as follows:

* We present the first systematic study on the role shape plays in the feature-learning of CNNs based on a clear definition of shape.
* Toward this goal, we propose a metric that quantifies the behavior of a network to assess whether it considers shape during inference.
* We comprehensively analyze the properties of shape-learning segmentation networks.
* We propose and evaluate data augmentation techniques to promote learning of shape in neural networks.

In the rest of this paper, we list our definitions relevant to shape learning and a metric to measure shape learning in Section 1. We then briefly introduce our data in Section 2. We describe our experimental design and results in Section 3. Finally, we conclude our findings in Section 4.

## 1 Definitions

An _object_ in the context of images is a set of contiguous pixels. Three attributes of objects are of particular interest here: _shape, texture, and structure_. For an object \(\mathbf{O}\),
**Definition 1**.: _"Shape" is the spatial relationship between all boundaries separating pixels in \(\mathbf{O}\) from pixels that are not._

The definition of _texture_ has been well-established in the literature.[10, 34]. We adopt it as:

**Definition 2**.: _"Texture" is a distribution from which pixels forming similar visual impressions may be sampled._

Intuitively, each pixel can be associated with only one texture. We use the definition of texture to establish definitions of a _simple object_ and a _complex object_, which will be of importance in this study.

**Definition 3**.: _A "Simple object" is a set containing contiguous pixels with the same texture \(\mathbf{A}\) bounded by pixels with texture \(\mathbf{B}\neq\mathbf{A}\) or by the edge of the image._

**Definition 4**.: _A "Complex object" is an object formed by grouping multiple simple objects following certain spatial relationships._

**Definition 5**.: _"Structure" is the spatial relationships between the simple object(s) in each complex object. Simple objects all have the same null structure._

Given these definitions of shape, texture, and structure, we can rigorously describe the meaning of "shape learning". Conceptually, a shape-learning network should recognize objects with desired shapes even if the other features (e.g., textures) of the objects change. On the other hand, they ignore objects with irrelevant shapes but desirable non-shape features. To quantify the extent to which a Network network's performance matches this behavior, we propose a metric named Shape Bias Index (SBI). This metric is named after the concept of Shape Bias in psychology [5, 22], which is referred to as a model's ability to recognize objects by their shape in computer vision.

SBI is assessed empirically by isolating and comparing the effects of different discriminative features. Given a raw dataset, we split it to obtain a training set and a validation set such that the samples in the validation set contain a similar set of possible discriminative features to the training sets. We denote this validation set as \(D_{val}=[X,y]\). Based on \(D_{val}\), we then generate probing sets by removing certain discriminative features from \(D_{val}\)

* \(D_{rm}=[X_{rm},\ y_{rm}]\) by altering one or multiple presented non-shape discriminative features (e.g., texture changes, noise, inserting uninterested objects...). The specific alternation of non-shape features may differ by experiment
* \(D_{aff}=[X_{aff},y_{aff}]\) by rotating or flipping \([X,y]\) so that the output image contains objects with different shapes from before (represent affine transformation of shape).
* \(D_{shuf}=[X_{shuf},y_{shuf}]\) by dividing \([X,y]\) into 16 \((4\times 4)\) equally sized patches and shuffle their positions; (represent non-affine transformation of shape).

As shown in Fig. 1, with these four datasets, we can compute a model's SBI following the equations below. Let \(IOU(D_{foo})\) denote the segmentation performance of the analyzed model on some dataset \(D_{foo}\) (\(D_{foo}\) is a placeholder). The relative performance drop caused by the digression of non-shape discriminative features in \(D_{foo}\) from those in \(D_{foo}\) can be quantified as:

\[PD_{foo}=1-\frac{IOU(D_{foo})}{IOU(D_{val})}\]

Figure 1: Pipeline for (left) generating probing images and (right) computing SBI
We may obtain \(PD_{rm},PD_{aff}\) and \(PD_{shuf}\) in this manners by plugging in the place of \(D_{foo}\) with \(D_{rm},D_{aff}\) and \(D_{shuf}\). Then, we smoothen their values through a Soft-Max function as:

\[[\Delta_{rm},\Delta_{aff},\Delta_{shuf}]=SoftMax[PD_{rm},PD_{aff},PD_{shuf}]\]

Our SBI (Shape Bias Index) for the model to be assessed can now be computed as

\[SBI=\frac{\Delta_{aff}+\Delta_{shuf}}{\Delta_{rm}} \tag{1}\]

If the performance drop caused by the changes to the shape (\(\Delta_{aff}\) and \(\Delta_{shuf}\)) are higher than that caused by changes in non-shape features (\(\Delta_{rm}\)), it reflects that the network is more sensitive to changes in shape than non-shape features. The value of SBI would then be larger than one and increases with higher sensitivity to shape changes and lower sensitivity to non-shape changes. The larger the SBI is, the more weight to shape information a network weighs in its inference process.

Since the framework for computing SBI requires a task-specific validation set and has broad room for alternative implementation of probing sets, it is an unnormalized metric. Direct comparisons between SBIs for different models require the same validation set and probing sets. An explicitly defined SBI threshold may be helpful for a qualitative answer to "whether a specific network is shape-learning."

## 2 Data

To explore and verify factors that affect the shape learning of segmentation networks, we conducted an extensive set of experiments on a combination of synthetic and real images. We first use synthetic data to explore how the presence of different discriminative features, the size of the model receptive field, and the size of target objects influence shape learning. We then demonstrate the response of shape-learning segmentation networks to an increasing intensity of domain shift (e.g., brightness changes) and shape variations in target objects. In our synthetic dataset, each image contains one or two polygons sized 100-150 pixels in their width and height as potential (target or non-target) objects. The objects can be simple or complex. Object(s) in each image is/are placed on some background with one random texture. The specific textures in each image, both for those associated with objects and backgrounds, are drawn from:

1. the Colored Brodatz database (112 textures) for the training set, validation set \(D_{val}\), and two of the three probing sets \(D_{aff}\) and \(D_{shuf}\).
2. the CURET database (61 textures) for images in probing set \(D_{rm}\) so that no seen textures would exist in (\(D_{rm}\))

After the experiments and analysis on our synthetic datasets, we further validate our findings on three of real datasets. The three datasets are:

1. BFGT: Breast and FGT MRI Dataset [4]
2. FISH: A Large-Scale Dataset for Fish Segmentation and Classification[33]
3. LUNG: OSIC Pulmonary Fibrosis Progression[21, 27] + COVID-19 CT scans[8, 25, 28]

BFGT uses a subset of "Breast and FGT MRI Dataset"[4] containing slices parallel to axial planes of breast tomosynthesis. The masks for breast regions across different images are of similar shapes. FISH uses a subset of "A Large-Scale Dataset for Fish Segmentation and Classification" [33] containing four species of fish in two genera with their photos pasted on a wood or marble surface. Each fish is positioned with its longest dimension positioned horizontally, second longest dimension positioned vertically, and shortest dimension positioned perpendicular to the surface. Images and masks for LUNG are extracted from a combination of two different repositories. Specifically, "OSIC Pulmonary Fibrosis Progression" [21, 27] and "COVID-19 CT scans"[8, 25, 28]. Both repositories contain images extracted from the axial view of lung X-ray images. The shapes of lungs across different images resemble each other.

We divided each dataset into four partitions: one as the training set, one as the validation set, one noisy dataset created by adding six different types of noise to validation sets independently, and one with naturally occurring OOD data (e.g., images of fish from a different species but from similar family). These three versions of the datasets will help approximate the IOU(\(D_{rm}\)) used for computing the SBIs for the respective models.

## 3 Experiments

We conduct all our experiments on a UNet-like segmentation network. Unless otherwise specified, the implementation of our network is adapted from[3], with all BatchNorm layers using per-batch statistics. We use cross entropy as the
objective function to train on every synthetic dataset, which consists of RGB images with dimension \(320\times 320\). With a combination of synthetic and real data, we performed a comprehensive set of experiments to answer the following questions in the setting of semantic segmentation:

1. Under what circumstances does a network learn shape?
2. Do object size and model receptive field size affect shape learning?
3. How do input perturbations affect the performance of shape-learning networks?
4. Can augmentations encourage shape-learning with existing data?

Question 1 is the principal question that we thoroughly examine in this paper. We answer this question using synthetic data representing simple objects (section 3.1) and complex objects (section 3.2). Questions 2 and 3 are answered using synthetic data in Sections 3.3 and 3.4, respectively. Finally, question 4 investigates how our systematic findings can be applied to encourage shape-learning and improve generalization in real data (Section 3.5).

### Under what circumstances does a network learn shape? (simple object)

We first answer the question on the capability of networks to learn simple objects' shapes. In order to do so, we consider four different scenarios and four corresponding simulated datasets. In these scenarios, different features will be available based on which segmentation of the target object can be conducted. The first scenario is a simple one, where each image contains two simple objects in the shape of polygons: one with a discriminative shape (i.e., a shape that can be used to determine that this is the object of interest) throughout the dataset (target object) and the other with random shape (non-target object). An example training image is shown in Figure 2. The presence of other discriminative features such as texture in this initial training set is deliberately avoided. To ensure the model cannot use textures to classify, each object's texture is randomly selected.

In real-life scenarios, however, multiple discriminative features with similar predictiveness are usually present simultaneously (e.g., the shape of elephants and the color/texture of elephants' skin). To explore how shape-learning behaviors are affected by discriminative features co-occurring with shape, we design the following additional discriminative features through which target objects in an image can be identified:

1. Texture: The texture of each pixel associated with target objects is drawn from a set containing five selected textures. In other words, a pixel is associated with the target object if and only if it has one of the selected textures. Non-target objects and the background have disjoint sets of textures from target objects.
2. Singular: Instead of two polygons in each image, there is only one polygon of size 100-150 pixels in both dimensions, which is always the target object. Textures are randomly selected for both target and background unless co-occurred with texture.

Based on this definition of features, the remaining three scenarios, and corresponding simulated datasets are as follows. The second scenario is the same as the first one with the exception that texture is also a discriminative feature as described above. The third scenario is the same as the first with the addition of the "singular" discriminative feature as described above. In the fourth scenario, both texture and singular features are discriminative. We train a segmentation network on each training set. The performance of those models on \(D_{val},D_{rm},D_{aff}\)\(and\)\(D_{shuf}\) (the probing datasets used to compute SBI), together with SBIs, is reported in Table 1.

From the last column of Table 1, one can clearly see that only the model from the first scenario, where the shape is the only discriminative feature of the target object, was able to show considerable shape learning behavior (high SBI). Whenever another feature was available to aid the segmentation, the models did not show the tendency to learn shape (low SBI).

Figure 2: Image of simple objects with only shape being discriminative
Analysis of performance on the specific datasets (columns 2-5) allows for more detailed insights. First, the model trained on the dataset where shape was the only discriminative feature achieved high segmentation performance on the validation set \(D_{val}\). It demonstrates that a segmentation network can recognize shapes because objects' shapes are the only feature allowed for identifying the object of interest. Recall that \(D_{rm}\) perturbs non-shape features while \(D_{aff}\) and \(D_{shuf}\) perturbs target objects' shapes. This model experiences a lower performance drop when tested on \(D_{rm}\) than on \(D_{aff}\) and \(D_{shuf}\). The model's robustness against significant texture changes (also brightness changes and noise) and brittleness against shape changes aligns with our intuition about the properties of a shape-learning network. We hence conclude segmentation networks are capable of learning discriminative shape information in the target objects for simple objects.

We then observe the behavior of models trained on datasets containing other discriminative features in addition to shape (i.e., shape+texture, shape+singular, and shape+texture+singular). The discriminative features in addition to shape unanimously lower the SBI of each model to a significant extent, despite their similar validation IOUs. This phenomenon indicates that shape-learning is not a default behavior of segmentation networks; rather, these findings suggest that they default to learning other discriminative features if present. Practically, the suppression or removal of features unrelated to shape may be vital to encouraging the shape learning of networks in a typical object segmentation task.

It is worth noting that we observed that the model in scenario 3 (shape+singular) generally segmented both objects in the image. We believe this is because shape+singular was trained on a single foreground object, which may have encouraged the model to identify "any foreground objects of similar size as the targets seen in the training set", even if they have completely different shapes.

Based on the results from the experiments in this section, we draw the following conclusions:

1. A UNet-like network does not prioritize learning shape (by default) in the presence of other discriminative features of similar predictive capability (e.g., a unique texture (texture), or being the only object in the image (singular)).
2. A segmentation network learns shape only if the target objects' shapes are the sole discriminative feature.

In the next section, we will show that these conclusions extend to complex objects. We will also explore the interactions between shape and other discriminative features.

### Under what circumstances does a network learn shape? (complex object)

In the previous section, we investigated shape learning in simple objects. We now extend our experiments to the case where targets are complex objects, a setting more closely resembling real-life scenarios. Like in the previous section, we first explore whether there exists a scenario in which a network exhibits a shape-learning behavior for complex

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|} \hline Discriminative features & IOU(\(D_{val}\)) & IOU(\(D_{rm}\)) & IOU(\(D_{aff}\)) & IOU(\(D_{shuf}\)) & SBI \\ \hline Shape only (simp. obj.) & 0.993 & 0.967 & 0.033 & 0.146 & 4.490 \\ \hline shape+texture & 0.990 & 0.288 & 0.900 & 0.641 & 1.238 \\ \hline shape+singular & 0.993 & 0.589 & 0.897 & 0.434 & 1.902 \\ \hline shape+texture+singular & 0.999 & 0.398 & 0.976 & 0.812 & 1.222 \\ \hline \end{tabular}
\end{table}
Table 1: Performance of UNets trained on datasets containing simple objects with different discriminative features on the probing sets and their SBIs. The raw dataset and \(D_{val}\) are assumed to contain shape+texture+singular as discriminative features.

Figure 3: Image of simple obj. with non-shape features (bounded in red)
objects. Mimicking the previous experiment, we train a segmentation network on a dataset containing only shape as the discriminative feature.

We then generate different combinations of discriminative features and add them to the dataset and observe how they affect SN's ability to learn and leverage shape information. Since we assume targets are complex objects, each image can now contain more sophisticated features. In addition to "singular" and "texture" modifications in the simple objects described in Section 3.1, we also introduce "structure" and "semi-singular" as potential discriminative features:

1. Structure: All simple objects within the target have the same shape and are arranged in a fixed relative position to each other. Their respective textures, unless otherwise specified, are randomly sampled.
2. Semi-singular: There are two polygons sized 100-150 pixels in width and height per image. The target object is the only complex object in each image. The structure and texture of target objects are random. Note that semi-singular and singular are two features that cannot appear simultaneously, as semi-singular assumes the presence of at least two objects of similar sizes to the target object in the image.

As the combination of discriminative features becomes more complicated in this experiment, we adjusted the data generation process for \(D_{val},D_{rm},D_{aff}\) and \(D_{shuf}\) to maintain the efficacy of SBI. To avoid confusion, datasets in this experiment to compute SBI are denoted as \(D^{c}_{val},D^{c}_{rm},D^{c}_{aff}\) and \(D^{c}_{shuf}\), distinguished from those in the prior section by superscript 'c' (stands for "complex"). \(D^{c}_{val}\) is generated to simultaneously contain all listed discriminative features in each image, namely singular, texture, and structure. \(D^{c}_{rm}\) is generated to contain both target and non-target complex objects with random structures in textures randomly drawn from CURET database; shape is the only discriminative feature through which target objects may be identified. \(D^{c}_{aff}\) is obtained through rotating/flipping the target objects in \(D^{c}_{val}\) while keeping the structure, texture, and approximate size of the rotated object unaltered. A sample of the three datasets each is listed in Fig. 6. \(D^{c}_{shuf}\) divides \(D^{c}_{val}\) into patches and shuffled them.

Table 2 reports the performance of each network trained on the baseline with only shape being discriminative, and on datasets with different combinations of additional discriminative features.

According to Table 2, the networks continue to show a good capability to utilize discriminative shape information for segmentation tasks, as indicated by the high SBI and \(D^{c}_{val}\). It also has a significantly smaller performance drop on \(D^{c}_{rm}\) than on \(D^{c}_{aff}\) or \(D^{c}_{shuf}\). We thus conclude that segmentation networks are capable of segmenting through shape, regardless of the targets being simple or complex objects.

Figure 4: Image of complex objects with only shape being discriminative

Figure 5: Image of complex objects with non-shape features (bounded in red)
Similar to in experiments with simple objects, we again see that the features "texture" and "singular" in the training set distract a network from recognizing complex objects by shape. "semi-singular" appears to have similar effects as "singular" based on its performance on each sub-task. As we consider shape+singular teaches a network to recognize any object surrounded by pixels with only one texture representing the background, shape+semi-singular may result in a network identifying any complex object surrounded by pixels with only one texture representing the background.

The image on the left of Fig. 7 contains two polygons: one with the desired shape and another with an irrelevant shape. The two polygons form two complex objects in the background, with only the one in the lower-right corner being the target. However, the network trained on "+semi-singular" tends to also consider irrelevant objects as targets. As we include more types of discriminative features in this group of experiments, we also found the following patterns. Let \(\{Shape,F_{0},\dots,F_{i}\}\) be all possible discriminative features presenting in the training set, where \(\{F_{0},\dots,F_{i}\}\) are some non-shape features. In most cases, including \(\{Shape,\ F_{0},\dots,F_{j}\}\cup\{Shape,F_{j+1},\dots,F_{i}\}\) in the training of network (e.g., shape+singular+texture) leads to a network with SBI in-between those of networks trained on \(\{Shape,F_{0},\dots,F_{j}\}\) and \(\{Shape,F_{j+1},\dots,F_{i}\}\) respectively (e.g., shape+singular and shape+texture). In less common cases, such as shape+singular+structure, the multiple non-shape features may collaboratively cause more severe distraction of shape-learning either of it can do alone. However, in no circumstance does a network trained on \(\{Shape,F_{0},\dots,F_{i}\}\) have higher SBI than on both \(\{Shape,\ F_{0},\dots,F_{j}\}\) and \(\{Shape,F_{j+1},\dots,F_{i}\}\). This observation reflects that though shape is generally ignored in the presence of other discriminative features, other non-shape features may be learned collaboratively, rather than one preferred over another. This finding amends the hypothesis made in the study of Hermann et al. [14] that neural networks primarily utilize only one discriminative feature at a time, but only in rare cases multiple features simultaneously. Instead, our findings suggest the simultaneous learning of multiple features is prevalent.

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|} \hline Discriminative features & IOU(\(D^{c}_{val}\)) & IOU(\(D^{c}_{rm}\)) & IOU(\(D^{c}_{off}\)) & IOU(\(D^{c}_{shuf}\)) & SBI \\ \hline Shape only (complex obj.) & 0.996 & 0.933 & 0.266 & 0.278 & 3.940 \\ \hline shape+singular & 0.987 & 0.522 & 0.857 & 0.614 & 1.628 \\ \hline shape+semi-singular & 0.983 & 0.637 & 0.758 & 0.572 & 1.952 \\ \hline shape+texture & 0.985 & 0.254 & 0.958 & 0.756 & 1.098 \\ \hline shape+structure & 0.998 & 0.474 & 0.589 & 0.466 & 1.896 \\ \hline shape+singular+texture & 0.979 & 0.456 & 0.948 & 0.740 & 1.358 \\ \hline shape+singular+structure & 0.999 & 0.463 & 0.825 & 0.6456 & 1.526 \\ \hline shape+texture+structure & 0.999 & 0.270 & 0.905 & 0.690 & 1.180 \\ \hline shape+texture+singular+structure & 1.000 & 0.458 & 0.928 & 0.761 & 1.356 \\ \hline \end{tabular}
\end{table}
Table 2: Performance of UNets trained on datasets containing complex objects with different discriminative features on the probing sets and their SBIs

Figure 6: Image of complex objects with only shape being discriminative

Figure 7: Network trained on shape+singular shows significant failure to exclude objects with undesired shape.
Based on the results from experiments on complex objects, we extend our arguments made in Section 3.1 to both the case of simple and complex objects. We also draw additional conclusions:

1. A segmentation network does not prioritize learning shape in the presence of other discriminative features with similar prediction capabilities. This statement holds for both simple and complex objects.
2. Though shape learning is distracted in the presence of other discriminative features, other discriminative features tend to be learned simultaneously if multiple of them are present.

### How do object size and model receptive field affect shape learning?

In the previous sections, we learned that networks are capable of learning shape, though doing so requires the absence of undesired discriminative features. We now explore how the size of target objects and model receptive fields may affect shape learning. Geirhos et al.[7] reported that restricting the receptive field size of ResNet would prevent it from converging on stylized ImageNet. Model variants with larger receptive fields seem to suffer less from this phenomenon. This observation suggests a potential correlation between receptive field size and the tendency or capability of CNNs to leverage shape information.

We further expand on this topic by systematically analyzing how the size of an object of interest and the model receptive field affect shape learning. By receptive field, we mean the maximum possible number of pixels that can be involved in the computing of a pixel at the bottleneck of the network.

We resized the training set for model shape only (simple obj.) mentioned in Fig. 2 to 11 different sizes ranging from \(160\times 160\) to \(480\times 480\) with an interval of 32 pixels. We pair these 11 differently sized training sets each with two UNets of slightly different model architecture. Specifically, the CNN layers in the encoder of one UNet (denoted as UNet_140) all have dilation=1, while the other UNet (denoted as UNet_210) set the CNN layers in encoder block 2,3,4 to have dilations=2. This allows an expansion of receptive field size without changing the total number of parameters for the models. The formula for computing the receptive field size is:

* UNet_140: ((((1+4)*2+4)*2+4)*2+4)*2+4 = 140
* UNet_210: (((1+4)*2+9)*2+9)*2+9)*2+4 = 210

We train each [model variant, input size] pair with SGD optimizer (lr = 1e-2, weight_decay= 1e-5, momentum=0.9) until the models fully converge. As shown in Fig. 8, for all input sizes, UNet_210 has a larger SBI than UNet_140. Plus, a clear and sudden drop in SBI as the input size goes larger than 382 is observed for UNet_140, and 420 for UNet_210, when the object size inside each image becomes larger than the models' receptive field sizes.

Figure 8: SBIs for UNets with different receptive field sizes when trained on images containing target objects of different sizes. The p-value listed is the p-value for the f-test on the linear regression.
A statistically significant (p-value<0.05 for F-test) trend of decrease is observed for both UNet_140 and UNet_210 as the input image size increases. We hence conclude that models with small receptive fields may have difficulty learning shape. Increasing the model receptive field and/or reducing ROI size may alleviate this phenomenon. This increase in SBI would diminish once the receptive field size versus object size reaches a certain threshold. Despite increasing model receptive field size and/or reducing ROI size may encourage shape-learning, there are drawbacks to both operations. Increasing a model's receptive field usually introduces more parameters to it. More parameters may lead to longer training time and a higher risk of overfitting. Using dilated convolutional layers is a potential solution for increasing the model receptive field size without changing parameter sizes. Reducing ROI size reduces image resolution, resulting in the loss of potentially useful shape information. In practice, the hyperparameter turning is an essential step for obtaining a shape-learning segmentation network with high segmentation performance (e.g., IOU, DSC) on its designated task.

### How do input perturbations affect a shape-learning segmentation network?

In the previous sections, we explored how the presence of non-shape discriminative features, varying receptive field size, and varying object size can affect a network's shape learning. We now investigate how a shape-learning network responds to feature variations that may be common in a real-life scenario. We still use synthetic data to approach this question and control the degree of feature variation. Specifically, we observe how a shape-learning network responds to brightness changes and shape variations.

#### 3.4.1 Perturbations on brightness

Brightness change is one of the most prevalent domain shifts in real life that may significantly alter the textures of objects in images. Zhe et al. [36] demonstrated that models trained on data with limited variability in object brightness have poor generalizability to data with other brightness levels. Following common intuition, if a network learns shape, it should be insensitive to the changes in textures, hence be robust to changes in brightness.

#### 3.4.2 Perturbations on shape and textures

In Section 3.1, we show a segmentation network capable of utilizing unvaried shape information in the absence of other discriminative features. However, in real-life scenarios, two different objects rarely have identical shapes even if they

Figure 9: IOU of shape-learning UNet on dataset each with different combinations of foreground and background brightness change.
are in the same category. If a shape-learning network shows reasonable performance only when an identical shape appears, it has little advantage over symbolic methods such as template matching. We hence explore how a network trained on identical shapes responds to increasing degrees of shape variation. We also analyze the effects of including shape variation in the training set on SN's shape learning behavior.

In doing so, we take \(D_{val}\) used in for simple objects and apply increasing degrees of elastic deformation to the original polygon masks to create different levels of shape variations in the target objects, as shown in Fig. 11. We then denote the dataset derived from \(D_{val}\) as \(\{D_{val}^{1},\dots,D_{val}^{10}\}\), where the superscript denotes the degree of shape variation. We test how models trained on shape only (simple obj.), presented in Section 3.1 respond to such variations. Then, we train three additional models on three modified versions of shape only (simple obj.), each with different degrees of shape variation injected. After training, all four networks with different extents of shape variation (from no variation to some variation) show strictly decreasing IOUs as the degree of elastic deformation increases.

According to Fig. 10, models trained with little shape variation have much steeper decreases in performance when tested on perturbed input. These results show that shape-learning networks develop tolerance to shape variations according to variations in target objects' shape during training. On the other hand, the tolerance of shape-learning networks to shape variation may be affected by the extent of shape variation contained in the training set.

### Validation of finds on real data and exploration of augmentations

In the previous subsections, we explored factors affecting the shape learning of networks with synthetic data. In this section, we demonstrate how those findings can be applied to real-life tasks. Specially, we assess how well our conclusions on synthetic data align with real-world datasets combined with data augmentation, a common technique used in machine learning to improve the generalizability of models on OOD data.

Additionally, we evaluate augmentation methods to encourage shape learning when shape is a good discriminative feature in the dataset. Our selection of potential augmentation methods is inspired by the conclusions drawn from the systematic study with the synthetic data regarding the conditions under which segmentation networks learn shapes. We conduct our experiments on three real datasets that contain targets of similar shape, as listed in Section 2. Each dataset is split into four components:

1. Training set
2. Validation set (serve as \(D_{val}\)), which has a similar appearance to the training set. The performance of model on this dataset is denoted as IOU\({}_{val}\)
3. Noisy data (serves as part of \(D_{rm}\)) obtained by separately adding six different types of noise to \(D_{val}\). The six types of noise are Gaussian noise, shot noise, impulse noise, defocus blur, pixelate, and motion blur. Note that though adding noise may also be a common technique used in data augmentation, we did not inject noise into training data to prevent data leaking. The performance of model on this dataset is denoted as IOU\({}_{noisy}\)

Figure 10: Performance of shape-learning UNets trained with training sets containing different degrees of shape variation on shape variations in validation sets
4. OOD data (serves as part of \(D_{rm}\)) with naturally occurring domain shifts, which contains targets of similar shape as those in the training set, but with different visual appearance. The performance of model on this dataset is denoted as IOU\({}_{OOD}\)

Recall that three probing sets in addition to the validation set are needed to computer SBI. The other probing datasets \(D_{aff}\) and \(D_{shuf}\) are obtained following the same workflow as shown in Section 1.

For each training set, we apply the same group of augmentations, which may be perceived as perturbations to certain discriminative features as below:

1. Color Jitter
2. Separate Color Jitter
3. Neural Style Transfer (NST)
4. Negative Insertion
5. Random Resized Crop
6. Random Crop Reflect

Most of the augmentations are used in an attempt to suppress learning texture features. Specifically, Color Jitter changes the brightness, contrast, and saturation of images. Separate Color Jitter is an extension of Color Jitter which uses different parameters for targets and the rest of the image.

Neural Style Transfer (NST) [6] changes the Gram matrix of feature vectors. Visually, they significantly alter the texture of images but barely change objects' shape.

Negative Insert is designed to suppress singular; it inserts objects with similar textures but different shapes from the targets. In our case, this operation is implemented by cropping the targets into multiple patches and then shuffling and pasting them back to the original image. The inserted objects are not labeled as targets.

Random Resized Crop may have complicated effects, as it suppresses both shape and texture - the cropping changes the targets' shapes while the resizing changes texture density. Random Crop Reflect changes only object shape while preserving the texture on each object by padding the cropped patch reflectively. If target objects are mirrored, their masks are also mirrored.

Following our conclusions in Section 3.1 and Section 3.2, we expect all augmentations to produce higher model SBI than the baseline (no augmentation), except for Random Resized Crop and Random Crop Reflect, as these two augmentations alter the shape of target objects.

#### 3.5.1 Validation with BFGT

The complete BFGT dataset contains 922 volumes of breast MRI images produced by GE and SIEMENS machines. We obtain 219 slices (up to 3 slices per volume) produced by the GE machine as our training and validation set. We then select 81 slices (up to 3 slices per volume) produced by SIEMENS as our out-of-domain dataset. Each image is resized to have the same dimension of \(256\times 256\). We report below the mean IOUs and SBI of models trained on data with different augmentations.

In experiments with BFGT, augmentations with no modifications on targets' shapes achieved higher SBIs than the baseline model, while those that altered targets' shapes resulted in models with smaller SBIs. This phenomenon matched our prediction. Color Jitter (brightness shifts for grayscale images), Sep. Color Jitter, and NST all lead to models with higher performance on OOD and Noisy data. In contrast, Negative Insertion showed no improvement in performance on OOD nor on Noisy data compared to the baseline, despite its higher SBI. This is because SBI also takes an SN's performance on objects with dissimilar shapes into consideration. Random Resized Crop and Random Crop Reflect alter shape information in the training set. Both models have smaller SBIs than the baseline, plus similar or

Figure 11: Ground-truth masks of target objects after applying different intensities of elastic deformation
lower performance on OOD and noisy data. The results confirm that augmentation not altering shape improves models' SBIs; Color Jitter and NST are apparently the most practical augmentation method to be applied to encourage shape learning and improve models' generalizability.

#### 3.5.2 Validation with FISH

The fish dataset contains fish of various species photographed from similar perspectives. We extract fish of 4 species in two families, where species in the same family tend to have similar shapes. Two of the species are gilt-head bream and red sea bream from _Sparidae_ family, while the other two are striped-read mullet and red mullet from _Mullidae_ family. Images of gilt-head bream and striped-red mullet on two types of backgrounds are used as the training and validation set. Afterward, we pick images of red sea bream and red mullet as OOD data. The Noisy FISH dataset is obtained by adding Gaussian noise to the validation set. The results for each augmentation on this dataset are listed in Table 4.

In experiments with FISH, Random Crop Reflect and Random Crop Resize result in a further decrease in the already low SBI. The rest of the augmentations unanimously caused a big increase in their respective models' SBIs as predicted. Color Jitter, NST, and Sep. Color Jitter continues to demonstrate improved performance on Noisy and OOD data compared with the baseline. Unlike the other augmentations which increase models' SBI, Negative Insertion shows inferior prediction accuracy even on the validation set. Though there are different augmentation methods that may theoretically encourage shape learning, not all of them will encourage the training of a segmentation network with higher performance and robustness.

\begin{table}
\begin{tabular}{|l|l|l|l|l|} \hline Training set & IOU\({}_{val}\) & IOU\({}_{noisy}\) & IOU\({}_{OOD}\) & SBI \\ \hline No Augmentation & 0.853 & 0.718 & 0.761 & 3.396 \\ \hline Color Jitter & 0.830 & 0.744 & 0.820 & 3.726 \\ \hline NST & 0.850 & 0.802 & 0.823 & 4.102 \\ \hline Sep. Color Jitter & 0.872 & 0.730 & 0.780 & 3.696 \\ \hline Negative Insertion & 0.867 & 0.717 & 0.748 & 3.996 \\ \hline Random Resized Crop & 0.858 & 0.623 & 0.725 & 2.538 \\ \hline Random Crop Reflect & 0.863 & 0.619 & 0.774 & 2.518 \\ \hline \end{tabular}.

\end{table}
Table 3: Performance of models trained on datasets with different augmentations when applied to the validation set, dataset with noise injected, and dataset with naturally occurred domain shifts for BFGT

Figure 12: Samples from the three partitions of data domain in BFGT
The effects of different augmentations on the training of segmentation networks on BFGT and FISH confirm the statement that when non-shape features are suppressed, shape-learning will be encouraged. It also reflects the nature of SBI as a metric measuring how much a network weighs shape in its decision process. Models with higher SBI, or tendency to learn shape, do not necessarily have higher performance on the Noisy and OOD datasets. However, models with higher performance on the Noisy and OOD datasets tend to have high SBIs.

#### 3.5.3 Validation with LUNG

The LUNG dataset is a collection of axial view lung CTs from three sources. The training and validation set consists of approximately 4000 slices extracted from 111 patient CT volumes from Pulmonary Fibrosis Progression (PFP), with their ground truth lung masks available in "CT Lung & Heart & Trachea segmentation" in Kaggle. We adopt another 90 axial view slices of lung CT volumes from "COVID-19 CT scans" showing lung with pneumonia as naturally occurring OOD data. (Access details in Section 2) The difference in the appearance of the COVID slices from the training set results not only from different imaging parameters but also from different infections. Each image is processed to have dimension \(256\times 256\). We report below the mean IOUs and SBI of models trained on different models. Unlike in BFGT and FISH, as shown in Table 5, augmentations have much smaller effects on the performance of LUNG models. All augmentations that preserve targets' shape yield models with very close performance on Validation, Noisy, and OOD data. Despite this, the effects of augmentations on SBI still align with our assumptions. Again, Random Resized Crop and Random Crop Reflect yield models with smaller SBI than the baseline, while the remaining augmentations increase each model's SBI.

\begin{table}
\begin{tabular}{|l|l|l|l|l|} \hline Training set & IOU\({}_{val}\) & IOU\({}_{noisy}\) & IOU\({}_{OOD}\) & SBI \\ \hline No Augmentation & 0.998 & 0.847 & 0.978 & 2.294 \\ \hline Color Jitter & 0.996 & 0.884 & 0.990 & 4.014 \\ \hline NST & 0.988 & 0.932 & 0.986 & 4.378 \\ \hline Sep. Color Jitter & 0.997 & 0.889 & 0.994 & 3.882 \\ \hline Negative Insertion & 0.865 & 0.644 & 0.895 & 3.924 \\ \hline Random Resized Crop & 0.991 & 0.768 & 0.710 & 2.188 \\ \hline Random Corp Reflect & 0.949 & 0.677 & 0.667 & 2.068 \\ \hline \end{tabular}.

\end{table}
Table 4: Performance of models trained on datasets with different augmentations when applied to the validation set, dataset with noise injected, and dataset with naturally occurred domain shifts for FISH

Figure 13: Samples from the three partitions of data domain in FISH
Augmentations, however, have negligible effects on this dataset. We explain this phenomenon by arguing that non-shape features may persist across multiple domains for LUNG. In LUNG, pixels associated with the lung have a texture style easily distinguishable from those associated with the rest of the image. The cooccurrence of high-intensity contours formed by cartilages also helps locate the lung. These features persist even across different image domains. In this case, the shape may not be the best discriminative feature in prediction capability. Non-shape features may be the more desirable feature sets to learn when optimizing the Cross-Entropy objective function. To avoid overstretching on interpreting the results, we will stop here with the conclusion that though Color Jitter and NST may suppress non-shape features, they barely hurt a model's generalizability on Noisy and OOD data.

On all three datasets, we show that suppressing non-shape features encourages shape learning. Our explorations made on synthetic data are hence generalizable to many real datasets. Color Jitter and NST provide superior augmentation performance on both noisy and OOD data but are computationally intensive. Compared to NST, Color Jitter is a faster option but yields a smaller performance boost.

## 4 Conclusion & Discussion

In this paper, we defined shape and designed a behavioral metric that measures a segmentation network's tendency to consider shape by measuring its response to variations in input features. With this metric, we explored whether, and under what circumstances, a network can utilize shape as a discriminative feature.

Figure 14: Samples from the three partitions of data domain in FISH

\begin{table}
\begin{tabular}{|l|l|l|l|l|} \hline Training set & IOU\({}_{val}\) & IOU\({}_{noisy}\) & IOU\({}_{OOD}\) & SBI \\ \hline No Augmentation & 0.982 & 0.938 & 0.962 & 2.650 \\ \hline Color Jitter & 0.978 & 0.935 & 0.962 & 2.954 \\ \hline NST & 0.978 & 0.941 & 0.962 & 3.014 \\ \hline Sep. Color Jitter & 0.977 & 0.917 & 0.956 & 2.650 \\ \hline Negative Insertion & 0.981 & 0.931 & 0.953 & 3.022 \\ \hline Random Resized Crop & 0.942 & 0.793 & 0.853 & 2.534 \\ \hline Random Crop Reflect & 0.979 & 0.672 & 0.943 & 2.128 \\ \hline \end{tabular}
\end{table}
Table 5: Performance of models trained on datasets with different augmentations when applied to the validation set, dataset with noise injected, and dataset with naturally occurred domain shifts for LUNG
Our experiments demonstrate that shape is the least prioritized feature to be learned. Learning shape information requires the absence of alternative correlated features. Furthermore, a sufficiently large receptive field size relative to object size is vital for a segmentation network to learn shape. We also concluded that shape learning is indeed a useful property of a network as it helps with out-of-domain generalization. We confirm our results from synthetic data on reproducible real-life scenarios. We showed that Color Jitter and NST as augmentations that encourage shape learning. In addition, our experiments on real data also show that one needs to be careful about when such augmentations should be applied since shape learning does not guarantee better performance on OOD data.

We believe that experimental evaluation of neural networks using clear behavioral metrics, which measure the response of the network to systematically designed stimuli, can provide useful practical information on what should be expected from them in real-life scenarios including their notable drawbacks. This said, the specific metric devised in this paper has its limitations. The functionality of this metric relies on the perturbations of chosen discriminative features. We admit that it may potentially ignore discriminative features uninterpretable to humans. However, to the best of our ability, we include multiple representative discriminative features and systematically show their effects on an SN's shape-learning tendency. One should remember that encouraging shape learning is not always beneficial. In some settings, it is the color or the texture that is the most reliable feature. An example of such a scenario is the segmentation of lakes in satellite images. While lakes can have dramatically different shapes, there are similarities in the color and texture of their surface. In such situations, encouraging shape learning through data augmentation may lower a model's performance on OOD data.

## Acknowledgments

We would like to express our gratitude to Gregory Szumel, Nicholas Konz, and Hanxue Gu from Mazurowski Lab for their suggestions on the readability of this paper.
