# Which Backbone to Use: A Resource-efficient Domain Specific Comparison for
Computer Vision

Pranav Jeevan P Amit Sethi
Department of Electrical Engineering
Indian Institute of Technology Bombay
Mumbai, India
{194070025<EMAIL_ADDRESS>

###### Abstract

In contemporary computer vision applications, particularly image
classification, architectural backbones pre-trained on large datasets like
ImageNet are commonly employed as feature extractors. Despite the widespread
use of these pre-trained convolutional neural networks (CNNs), there remains a
gap in understanding the performance of various resource-efficient backbones
across diverse domains and dataset sizes. Our study systematically evaluates
multiple lightweight, pre-trained CNN backbones under consistent training
settings across a variety of datasets, including natural images, medical
images, galaxy images, and remote sensing images. This comprehensive analysis
aims to aid machine learning practitioners in selecting the most suitable
backbone for their specific problem, especially in scenarios involving small
datasets where fine-tuning a pre-trained network is crucial. Even though
attention-based architectures are gaining popularity, we observed that they
tend to perform poorly under low data fine-tuning tasks compared to CNNs. We
also observed that some CNN architectures such as ConvNeXt, RegNet and
EfficientNet performs well compared to others on a diverse set of domains
consistently. Our findings provide actionable insights into the performance
trade-offs and effectiveness of different backbones, facilitating informed
decision-making in model selection for a broad spectrum of computer vision
domains. Our code is available here:
https://github.com/pranavphoenix/Backbones

_K_ eywords Image Classification $\cdot$ Backbone $\cdot$ Fine tuning $\cdot$
Domains

Domains
---
Ranking | Natural | Texture | Remote Sensing | Plant | Astronomy | Medical | Overall
Best | ConvNeXt-Tiny | ConvNeXt-Tiny | ResNeXt-50 $32\times 4$d | RegNetY-3.2GF | WaveMix | WaveMix | ConvNeXt-Tiny
Better | EfficientNetV2-S | ResNeXt-50 $32\times 4$d | EfficientNetV2-S | ConvNeXt-Tiny | ConvNeXt-Tiny | EfficientNetV2-S | RegNetY-3.2GF
Good | RegNetY-3.2GF | RegNetY-3.2GF | ConvNeXt-Tiny | ShuffleNetV2 $2.0\times$ | DenseNet-161 | RegNetY-3.2GF | EfficientNetV2-S

Table 1: A summary of our observations showing the top 3 backbones for fine-
tuning on multiple domains for image classification

##  1 Introduction

In most computer vision applications, particularly in image classification,
practitioners commonly use a backbone feature extraction network paired with a
classification head. This approach involves using the backbone to output
features from the input image, which are then fed into a task-specific head
for further computations. A prevalent practice in the field is to leverage
backbones trained on large datasets, such as ImageNet-1k [1] and then fine-
tune them on smaller datasets. This strategy is particularly useful due to the
scarcity of large, domain-specific datasets, such as those in the medical
applications.

Practitioners often resort to using off-the-shelf models from libraries like
Torchvision [2], which offers a plethora of backbones with pre-trained
ImageNet weights. These pre-trained backbone architectures, when fine-tuned on
smaller, domain-specific datasets, consistently outperform models trained from
scratch. This method not only enhances performance but also reduces the
computational resources and time required for training, making it a dominant
procedure in the field of computer vision.

However, it is important to note that while Torchvision lists the performance
metrics (top-1 accuracy) of various backbone architectures on the ImageNet
dataset, these metrics might not necessarily translate to similar performance
rankings in fine-tuning datasets. For instance, a backbone with higher
ImageNet performance might not guarantee superior performance on a particular
fine-tuning dataset. Conversely, a backbone with weaker ImageNet performance
might outperform others when fine-tuned on a dataset which belongs to a
different domain. This possible discrepancy highlights the need for careful
selection and evaluation of backbone models in various fine-tuning datasets
belonging to different domains, underscoring the complexity and variability of
transfer learning in computer vision.

Moreover, one of the major challenges for practitioners is the limitation in
terms of resources. These limitations include GPU availability, training and
inference time, and even model size. To address these challenges, our
experiments focus on comparing only the light-weight (model size less than 100
MB) backbone architectures available on Torchvision, which are designed to be
resource-efficient while maintaining high inference speed.

The lack of benchmarks that fit these backbones across multiple domains
presents a significant challenge for practitioners in deciding which backbone
to use for specific datasets with limited data. Additionally, the size of the
fine-tuning dataset can also impact the performance of these backbone
architectures. Our study also aims to investigate whether the dataset size
influences the selection of backbone models and to provide insights on
choosing the most suitable backbone based on the specific domain and the
amount of available fine-tuning data.

We also analyse the architectures which perform well in each of the domain-
specific datasets and try to understand their properties. This will help the
research community to realise their fundamental limitations and design better
architectures.

##  2 Related Works

Most of the research on architectures uses ImageNet benchmark to compare image
classification performance of backbones. Pytorch Image Models [3] library
benchmarks ImageNet classification performance across different backbone
architectures. Visual Task Adaptation Benchmark [4] evaluated the performance
of computer vision models on 19 tasks that span different domains. Brigato et
al. [5] designed a benchmark for data-efficient image classification spanning
various domains and showed that tuning hyper-parameters like learning rate,
weight decay, and batch size results in a competitive baseline outperforming
most specialized methods. Taher et al. [6] benchmarked various self-supervised
pre-training methods, 14 pre-trained ImageNet models on 7 diverse medical
tasks. Battle of backbones [7] benchmarks few pre-trained ImageNet backbones,
including those pre-trained by self-supervised learning and Stable Diffusion,
across a diverse set of computer vision tasks ranging from image
classification, Out-of-distribution generalisation, image retrieval and object
detection. There is a lack of bench-marking on transfer learning performance
of popular lightweight backbones across multiple domain-specific datasets.

##  3 Backbone Architectures

Since most of the practitioners use out of the shelf ImageNet pre-trained
models from the popular Torchvision library, we selected models from this
library for our experiments. We used certain criteria to filter models keeping
in mind our resource-efficiency constraint. So, we decided to choose models
with number of parameters less than 30 million, which amounts to a storage
space of around 100 MB. WaveMix [8] model was added to our list of backbones
because WaveMix was shown to provide state-of-the-art (SOTA) performance in
multiple image classification datasets belonging to many diverse domains such
as EMNIST [9] and galaxy morphology datasets.

Torchvision has 20 backbone families with models of multiple sizes within each
family making the total available models 115. After we apply our constraint of
lightweight models (less than 30 M parameters), we have 53 models belonging to
14 backbone families. We decided to take the best model (highest ImageNet
top-1 accuracy) from each of these families for our experiments. We also
discarded models whose ImageNet top-1 accuracy was less than 75%.

The final list of selected 11 backbones for our experiments are shown in table
2. We added both Swin-Tiny [10] and SwinV2-Tiny [11] in our list because there
was an under-representation of attention-based transformer architectures in
our list and the only other attention based model in Torchvision was vision-
transformer (ViT) whose smallest model has 86 million parameters. We wanted to
also understand how attention-based models perform in resource-efficient fine-
tuning under low data regime since there has been a recent trend amoung
computer vision practitioners to use transformers for all tasks.

Architectures |  | # Params
---
(M)
| ImageNet-1k
---
Top-1 Accuracy
(%)
ResNet-50 [12] | 25.6 | 76.13
WaveMix [8] | 27.9 | 75.32
ConvNeXt-Tiny [13] | 28.6 | 82.52
Swin-Tiny [10] | 28.3 | 81.47
SwinV2-Tiny [11] | 28.4 | 82.07
EfficientNetV2-S [14] | 21.5 | 84.23
DenseNet-161 [15] | 28.7 | 77.14
MobileNetV3-Large [16] | 5.5 | 75.27
RegNetY-3.2GF [17] | 19.4 | 81.98
ResNeXt-50 [18] $32\times 4$d | 25.0 | 81.20
ShuffleNetV2 [19] $2.0\times$ | 7.4 | 76.23
Table 2: List of popular computer vision backbones we used for our experiments
along with the model size and ImageNet accuracy for which model weights are
available. All model weights are taken from Torchvision library except for
WaveMix which was taken from GitHub.

Below are the architectures which was selected for our experiments.

ResNet [12]: ResNets are the most popular and successful backbone
architectures currently in use today since its arrival almost a decade ago.
ResNet uses residual connections to allow for the training of very deep
convolutional neural networks (CNN) by mitigating the vanishing gradient
problem. We use ResNet-50 for our experiments.

WaveMix [8]: WaveMix is a token-mixing architecture that uses 2-dimensional
discrete wavelet transform for spacial token-mixing and has been shown to
provide SOTA performance in multiple image classification datasets. We use
WaveMix-192/16 (level 3) for our experiments.

ConvNeXt [13]: ConvNeXt is a recent CNN architecture designed to improve upon
traditional CNNs by incorporating elements from transformer models, resulting
in enhanced performance and scalability. It uses depth-wise convolutions,
inverted bottleneck blocks and large kernels. We use ConvNeXt-Tiny in our
experiments.

Swin Transformer [10]: Swin transformer was an improvement over conventional
ViT which overcame the massive data requirements for training. It incorporated
efficiency using hierarchical representations, limiting the attention window
and merging them stage by stage. We use Swin-Tiny and SwinV2-Tiny [11] for our
experiments.

EfficientNet [14]: EfficientNet is a family of CNNS that optimize both model
size and speed by utilizing a compound scaling method that uniformly scales
network depth, width, and resolution. It incorporates advanced techniques such
as progressive learning and a mix of regular and mobile convolutions. We use
EfficientNetV2-S in our experiments.

Densenet [15]: DenseNet is a CNN architecture that connects each layer to
every other layer in a feed-forward fashion, promoting feature reuse and
reduction in number of parameters. This dense connectivity pattern helps
alleviate the vanishing gradient problem and leads to improved training
efficiency and accuracy. We use Densenet-161 in our experiments.

MobileNet [16]: MobileNetV3 is a CNN architecture designed for on-device and
resource-constrained environments, which combines lightweight depth-wise
separable convolutions with squeeze-and-excitation modules. It is a highly
efficient model with improved accuracy and reduced computational complexity.
We use MobileNetV3-Large in our experiments.

RegNet [17]: RegNet is a family of CNN architectures that utilize a regular
design space to systematically generate a diverse range of models, optimizing
for both efficiency and performance. It focuses on simple, scalable structures
with uniform depth, width, and group convolution patterns, incorporating
features like bottleneck blocks. We use RegNetY-3.2GF for our experiments.

ResNeXt [18]: ResNeXt is a CNN architecture that extends the ResNet model by
introducing a cardinality dimension, using grouped convolutions to aggregate
multiple transformations, which improves performance and efficiency. We use
ResNeXt-50 $32\times 4$d for our experiments.

ShuffleNet [19]: ShuffleNet is a lightweight CNN architecture designed for
efficient computation on mobile devices. It uses channel shuffling and point-
wise group convolution to optimize speed and accuracy. We use ShuffleNetV2
$2.0\times$ for our experiments.

##  4 Datasets

Since we wanted to check the performance of various backbones on data
efficient fine-tuning, we decided to perform our experiments on datasets with
utmost 100,000 training images. We choose 20 publicly available datasets
belonging to 7 different domains such as natural, textures, remote sensing,
plants, astronomy, surface defect and medical images with number of training
images ranging from 1000 to 100,000 and number of classes ranging from 2 to
200. Details of datasets in each domain is given in table 3.

Datasets | Domain Description |  | # Training
---
Images
| # Testing
---
Images
# Classes
CIFAR-10 [20] | Natural Images | 50,000 | 10,000 | 10
CIFAR-100 [20] | Natural Images | 50,000 | 10,000 | 100
Tiny ImageNet [21] | Natural Images (ImageNet subset) | 100,000 | 10,000 | 200
Stanford dogs [22] | Natural Images (Dog breeds) | 12,000 | 8,580 | 120
Flowers-102 [23] | Natural Images (Flower species) | 2,040 | 6,149 | 102
CUB-200-2011 [24] | Natural Images (Bird species) | 5,994 | 5,794 | 200
Stanford Cars [25] | Natural Images (Car models) | 8,144 | 8,041 | 196
Food-101 [26] | Natural Images (Food categories) | 75,750 | 25,250 | 101
DTD [27] | Texture Images | 1,880 | 1,880 | 47
NEU Surface Defects [28] | Surface Defect Images | 1,440 | 360 | 6
UC Merced Land Use [29] | Remote Sensing Images | 1,680 | 420 | 21
EuroSAT [30] | Remote Sensing Images | 18,900 | 8,100 | 10
PlantVillage [31] | Plant Images | 44,343 | 11,105 | 39
PlantCLEF [32] | Plant Images | 10,455 | 1135 | 20
Galaxy10 DECals [33] | Astronomy Images (Galaxy Morphology) | 15,962 | 1,774 | 10
BreakHis $40\times$ [34] | Medical Images (Histopathology) | 1,398 | 606 | 2
BreakHis $100\times$ [34] | Medical Images (Histopathology) | 1,458 | 632 | 2
BreakHis $200\times$ [34] | Medical Images (Histopathology) | 1,411 | 611 | 2
BreakHis $400\times$ [34] | Medical Images (Histopathology) | 1,276 | 553 | 2
RSNA Pneumonia Detection [35] | Medical Images (Radiology) | 24,181 | 6046 | 2
Table 3: Details of the datasets used specifying the domain details, number of
images in the training and testing set and the number of classes for
classification

###  4.1 Natural

CIFAR-10 [20]: CIFAR-10 dataset is a widely-used benchmark dataset for image
classification, consisting of 60,000 $32\times 32$ color images in 10
different classes, with 6,000 images per class divided into 50,000 training
images and 10,000 test images.

CIFAR-100 [20]: CIFAR-100 dataset uses the same images as CIFAR-10, but images
are distributed across 100 different classes, with 600 images per class
divided into 50,000 training images and 10,000 test images. It provides a more
challenging classification task compared to CIFAR-10 due to the larger number
of classes.

Tiny ImageNet [21]: Tiny ImageNet dataset is a subset of the ImageNet dataset,
consisting of 200 image classes with 500 training images and 50 test images
per class, each resized to $64\times 64$ pixels. It is widely used for
benchmarking image classification algorithms, particularly in low-resource
scenarios.

Stanford Dogs [22]: Stanford Dogs dataset is a comprehensive dataset for fine-
grained image classification, containing 20,580 images of 120 different dog
breeds. It is widely used for benchmarking algorithms, particularly in
distinguishing between closely related categories.

Flowers-102 [23]: The Flowers 102 dataset is a dataset for fine-grained image
classification, consisting of 8,189 images of flowers categorized into 102
different species. Each class has between 40 to 258 images, and the dataset is
commonly used to benchmark algorithms in classification tasks due to its
diversity and challenging nature.

CUB-200-2011 [24]: Caltech-UCSD Birds-200-2011 is a comprehensive dataset for
fine-grained image classification, consisting of 200 bird species with 11,788
annotated images. It is widely used for benchmarking algorithms in fine-
grained visual recognition tasks due to its high level of granularity.

Stanford Cars [25]: The Stanford Cars dataset is a large-scale dataset for
fine-grained image classification, consisting of 16,185 images of 196 classes
of cars. It is widely used for evaluating and benchmarking computer vision
algorithms in tasks involving fine-grained visual recognition and object
detection.

Food-101 [26]: Food-101 dataset is a large-scale dataset for food
classification, containing 101,000 images of 101 different food categories,
with 750 training images and 250 test images per class. It is commonly used to
benchmark image recognition algorithms in the context of food and culinary
applications.

###  4.2 Texture

DTD [27]: Describable Textures Dataset (DTD) is a collection of 5,640 texture
images categorized into 47 classes based on human-describable attributes. It
is used to evaluate and benchmark algorithms in texture recognition and
classification tasks.

###  4.3 Surface Defect

NEU Surface Defect [28]: The Northeastern University (NEU) surface defect
database is a benchmark dataset for surface defect detection and
classification, featuring images of six types of surface defects. The dataset
includes 1,800 images with a resolution of $200\times 200$ pixels, divided
into 1,200 training images and 600 testing images. Each defect type has an
equal number of images, making it a well-balanced dataset for training and
evaluating machine learning models.

###  4.4 Remote Sensing

UC Merced Land Use [29]: UC Merced Land Use dataset is a high-resolution
dataset for land use classification, containing 2,100 aerial images
categorized into 21 land use classes with 100 images per class. Each image is
$256\times 256$ pixels, and the dataset is commonly used for evaluating and
benchmarking algorithms in remote sensing and geospatial analysis tasks.

EuroSAT [30]: The EuroSAT dataset is a benchmark dataset for land use and land
cover classification, consisting of 27,000 RGB and multi-spectral images
covering 10 classes, with images derived from Sentinel-2 satellite data. It is
widely used for evaluating the performance of machine learning algorithms in
remote sensing and geospatial analysis tasks.

###  4.5 Plant

PlantVillage [31]: PlantVillage dataset is a comprehensive dataset for plant
disease classification, containing over 54,000 images of healthy and diseased
leaves across 39 different plant categories. It is widely used for
benchmarking machine learning algorithms in agricultural and plant pathology
applications.

PlantCLEF [32]: The PlantCLEF dataset is a large-scale dataset for plant
identification, comprising millions of images covering thousands of plant
species, including trees, flowers, fruits, and leaves. It is used for
evaluating and benchmarking algorithms in botanical classification and plant
biodiversity studies. We use a subset of this dataset.

###  4.6 Astronomy

Galaxy 10 DECals [33]: Galaxy 10 DECals dataset is a dataset for galaxy
classification, consisting of 17,000 images of galaxies classified into 10
different morphological categories. It is used for evaluating and benchmarking
machine learning algorithms in astronomy and astrophysical research.

###  4.7 Medical

BreakHis [34]: Breast Cancer Histopathology Database is a dataset specifically
designed for the classification of breast cancer histopathological images. It
contains 7,909 microscopic images of breast tumor tissue, divided into benign
and malignant categories. It provides microscopic images of breast tumor
tissue at four different magnification levels: $40\times$, $100\times$,
$200\times$ and $400\times$. Each magnification level offers a different level
of detail, allowing for a comprehensive analysis of histopathological
features. The dataset is widely used for evaluating and bench-marking
algorithms in medical image analysis and computer-aided diagnosis.

RSNA Pneumonia Detection Challenge Dataset [35]: The RSNA Pneumonia Detection
Challenge dataset, provided by the Radiological Society of North America,
consists of over 30,000 chest X-ray images annotated for the presence of
pneumonia. It is designed to facilitate the development of machine learning
models for pneumonia detection, promoting advancements in medical imaging
analysis.

##  5 Experimental Details

Since the task is to measure fine-tuning performance of these backbones in
image classification via transfer learning, a standard training protocol was
used to compare the performance of all models. We do not freeze any layers of
the pre-trained backbones and fine-tune the full model on the given dataset or
a fraction of it, and measure the top-1 accuracy in the test set. Only the
final linear layer of the backbone was modified to match the number of classes
of each fine-tuning dataset.

All images were resized to $256\times 256$ for our experiments, except for
BreakHis fine-tuning where we resized images to $672\times 448$ since reducing
the resolution of histopathology images lead to poor results across models.
TrivialAugment [36] was used as data augmentation for all datasets except
BreakHis, Galaxy 10 DEcals, Augmentation was only used after checking whether
using this augmentation improved the model performance or not.

Due to limited computational resources, we use early stopping where training
is stopped if accuracy stops improving after 10 epochs. All experiments were
done on a single 80 GB Nvidia A100 GPU. Training was done using DualOpt [37]
where we used AdamW optimizer
($\alpha=0.001,\beta_{1}=0.9,\beta_{2}=0.999,\epsilon=10^{-8}$) with a weight
decay of 0.01 during the initial phase and then used SGD with a learning rate
of $0.001$ and momentum $=0.9$ during the final phase.

Cross-entropy loss was used for image classification. Batch-size was chosen so
that the entire GPU will be utilized while training. We used automatic mixed
precision in PyTorch during training. Top-1 accuracy on the test set of best
of three runs with random initialization is reported as a generalization
metric based on prevailing protocols [38].

##  6 Results

###  6.1 Fine-tuning Performance

The results of our fine-tuning experiments on datasets of all domains are
shown in Table 4, Table 5 and Table 6. We find that ConvNeXt-Tiny outperforms
all the other models in almost all natural image datasets (except CIFAR-10 and
CIFAR-100). EfficientNetV2-S also performs well on atleast 5 out of 7
datasets. RegNetY-3.2GF also performs well on 5 of the 7 datasets. WaveMix
performs the best in CIFAR-10 and CIFAR-100 but could not replicate the same
performance on other datasets.

ConvNeXt retains the good performance even amoung other domains such as
textures, plant and astronomy. RegNet also performs well on texture, remote
sensing and plant domains. We observe that SwinV2-Tiny performs the best in UC
Merced Land Use dataset, WaveMix and ResNeXt top the EuroSAT dataset, DenseNet
tops the PlantVillage dataset and WaveMix significantly outperforms all the
other models in Galaxy10 DECals dataset.

###  6.2 Performance with less training data

We observe from Table 7 that the models which are performing really well at
100% of the training data are also performing better than other models at 10%
and 1% of the training data. We can see from CIFAR-10 and Tiny Imaginary
Dataset results that ConvNeXt, WaveMix, EfficientNet, and even RegNet are
actually performing better than others even when the training data is
significantly reduced. This shows that the fine-tuning superiority of these
models is actually not dependent much on the amount of training data. Even
with less data, they can generalize better and create better representations
than other models.

Figure 1 and Figure 2 shows the performance of top 3 models with training
data. We see that the difference in accuracy amoung the three models is large
and significant when the training data is less (1%) and this difference
reduces as the training data increases.

We observe similar behaviour for other domains in Table 8. In plant datasets,
the top performing models at 100% of training data like ConvNeXt,
EfficientNet, DenseNet, and RegNet are again performing better than the other
models even at 10% of the training data. Similar behaviour is also observed in
radiology images, where ResNeXt and ResNeXt are performing better.

Natural Images
---
Backbones | Stanford Dogs | Flowers102 | CUB200 | Stanford Cars | Tiny ImageNet | CIFAR10 | CIFAR100
ResNet50 | 82.03 | 90.77 | 75.34 | 90.84 | 75.33 | 94.58 | 79.05
WaveMix | 83.12 | 88.43 | 67.32 | 87.86 | 76.30 | 97.26 | 83.88
ConvNeXt-Tiny | 89.47 | 94.41 | 81.92 | 92.26 | 83.42 | 96.48 | 82.60
Swin-Tiny | 83.81 | 90.07 | 78.16 | 90.04 | 76.85 | 95.38 | 81.25
SwinV2-Tiny | 84.44 | 90.06 | 66.89 | 91.38 | 73.93 | 94.52 | 75.37
EfficientNetV2-S | 86.59 | 93.65 | 79.10 | 91.59 | 81.35 | 96.42 | 83.20
DenseNet-161 | 80.02 | 89.96 | 76.44 | 91.06 | 75.03 | 96.03 | 81.01
MobileNetV3-large | 78.85 | 89.36 | 73.91 | 85.50 | 76.74 | 95.92 | 79.51
RegNetY-3.2GF | 85.94 | 92.22 | 81.24 | 91.33 | 80.14 | 96.82 | 82.89
ResNeXt-50 $32\times 4$d | 85.09 | 91.78 | 78.29 | 90.22 | 77.86 | 96.16 | 81.30
ShuffleNetV2 $2.0\times$ | 78.11 | 91.54 | 74.73 | 88.46 | 76.43 | 96.30 | 83.32

Table 4: Top-1 classification accuracy for fine-tuning pre-trained backbones on natural image datasets. The top 3 results for each dataset are highlighted with best in green and bold, second best in lighter green, and third best in lightest green.  Domain | Texture | Remote Sensing | Plant | Astronomy
---|---|---|---|---
Backbones | DTD |  | UC Merced
---
Land Use
EuroSat | PlantVillage | PlantCLEF | Galaxy10
ResNet50 | 68.21 | 96.90 | 98.75 | 99.83 | 80.60 | 84.80
WaveMix | 68.25 | 97.72 | 98.96 | 99.79 | 79.52 | 95.10
ConvNeXt-Tiny | 73.70 | 98.33 | 98.74 | 99.80 | 82.71 | 87.29
Swin-Tiny | 70.15 | 97.86 | 98.52 | 99.70 | 79.03 | 84.64
SwinV2-Tiny | 68.78 | 98.81 | 98.50 | 99.76 | 78.16 | 83.15
EfficientNetV2-S | 70.18 | 98.22 | 98.88 | 99.81 | 81.09 | 84.75
DenseNet-161 | 66.14 | 97.08 | 98.83 | 99.88 | 76.49 | 86.70
MobileNetV3-large | 68.69 | 97.14 | 98.72 | 99.80 | 75.52 | 82.47
RegNetY-3.2GF | 71.16 | 98.33 | 98.69 | 99.83 | 82.76 | 82.47
ResNeXt-50 $32\times 4$d | 72.73 | 98.33 | 98.96 | 99.75 | 80.32 | 85.82
ShuffleNetV2 $2.0\times$ | 68.64 | 97.86 | 98.65 | 99.70 | 81.76 | 83.69
Table 5: Top-1 classification accuracy for fine-tuning pre-trained backbones
on different domain datasets. The top 3 results are highlighted with best in
green and bold, light green, and third best in lightest green.  Medical Images
(Histopathology) BreakHis Dataset
---
Backbones | $40\times$ | $100\times$ | $200\times$ | $400\times$ | Average
ResNet50 | 97.91 | 99.53 | 99.22 | 98.44 | 98.78
WaveMix | 99.42 | 99.42 | 99.38 | 99.35 | 99.39
ConvNeXt-Tiny | 95.10 | 90.73 | 88.59 | 88.72 | 90.79
Swin-Tiny | 88.11 | 93.89 | 90.00 | 82.61 | 88.65
SwinV2-Tiny | 89.96 | 92.44 | 88.65 | 83.83 | 88.72
EfficientNetV2-S | 99.44 | 99.42 | 99.11 | 99.06 | 99.26
DenseNet-161 | 98.62 | 99.24 | 99.28 | 98.20 | 98.84
MobileNetV3-Large | 99.56 | 98.24 | 99.50 | 98.94 | 99.06
RegNetY-3.2GF | 99.84 | 99.22 | 99.48 | 98.02 | 99.14
ResNeXt-50 $32\times 4$d | 99.46 | 99.22 | 99.34 | 98.05 | 99.02
ShuffleNetV2 $2.0\times$ | 99.38 | 98.59 | 99.22 | 98.02 | 98.80
Table 6: Top-1 classification accuracy for fine-tuning pre-trained backbones on medical datasets. The top 3 results are highlighted with best in green and bold, light green, and third best in lightest green.  Figure 1: Variation of accuracy of top 3 backbones with increasing training data. The model performance at 3 orders of training data, 1% (500 images), 10% (5000 images) and 100% (50,000 images).  Figure 2: Variation of accuracy of top 3 backbones with increasing training data. The model performance at 3 orders of training data, 1% (1000 images), 10% (10,000 images) and 100% (100,000 images).  | CIFAR-10 | Tiny ImageNet
---|---|---
Number of training images | 50,000 | 5,000 | 500 | 100,000 | 10,000 | 1,000
Percentage of training data | 100% | 10% | 1% | 100% | 10% | 1%
ResNet | 94.58 | 86.5 | 65.42 | 76.30 | 66.56 | 28.11
WaveMix | 97.26 | 93.16 | 80.98 | 75.33 | 55.15 | 25.24
ConvNeXt | 96.48 | 91.06 | 72.22 | 83.42 | 67.47 | 37.38
Swin | 95.38 | 85.64 | 39.77 | 76.85 | 48.50 | 25.62
SwinV2 | 94.52 | 85.66 | 46.51 | 73.93 | 39.31 | 12.29
EfficientNet | 96.42 | 89.44 | 77.06 | 81.35 | 72.97 | 48.42
DenseNet | 96.03 | 86.92 | 64.83 | 75.03 | 51.89 | 18.00
MobileNet | 95.92 | 88.85 | 61.27 | 76.74 | 62.70 | 31.08
RegNet | 96.82 | 90.55 | 73.71 | 80.14 | 64.40 | 36.56
ResNeXt | 96.16 | 90.24 | 73.12 | 77.86 | 67.01 | 34.40
ShuffleNet | 96.30 | 90.62 | 77.45 | 76.43 | 61.65 | 29.89
Table 7: The variation of accuracy of backbones with reduction of training
data. Performance at 100%, 10% and 1% of training data for CIFAR-10 and Tiny
ImageNet datasets

| PlantVillage | PlantCLEF | RSNA | Food-101 | EuroSAT
---|---|---|---|---|---
Number of training images | 44,343 | 4,434 | 10,455 | 1,045 | 24,181 | 2,418 | 75,750 | 7,575 | 18.900 | 1,890
Percentage of training data | 100% | 10% | 100% | 10% | 100% | 10% | 100% | 10% | 100% | 10%
ResNet | 99.83 | 98.07 | 80.60 | 57.47 | 87.3 | 81.26 | 88.36 | 74.26 | 98.75 | 96.49
WaveMix | 99.79 | 97.39 | 79.52 | 57.64 | 86.94 | 80.93 | 85.88 | 68.86 | 98.96 | 96.48
ConvNeXt | 99.80 | 98.92 | 82.71 | 64.68 | 86.71 | 80.03 | 89.43 | 78.47 | 98.74 | 97.29
Swin | 99.70 | 97.75 | 79.03 | 48.57 | 70.36 | 68.22 | 88.81 | 76.21 | 98.52 | 95.95
SwinV2 | 99.76 | 97.72 | 78.16 | 21.47 | 86.38 | 67.63 | 89.04 | 74.17 | 98.50 | 96.48
EfficientNet | 99.81 | 98.42 | 81.09 | 64.4 | 87.06 | 79.96 | 89.84 | 76.68 | 98.88 | 97.23
DenseNet | 99.88 | 98.57 | 76.49 | 58.59 | 87.1 | 81.51 | 87.02 | 72.44 | 98.83 | 96.86
MobileNet | 99.80 | 98.29 | 75.52 | 54.93 | 87.05 | 79.55 | 85.73 | 71.12 | 98.72 | 96.35
RegNet | 99.83 | 98.25 | 82.76 | 64.53 | 87.05 | 79.56 | 90.4 | 77.89 | 98.69 | 96.49
ResNeXt | 99.75 | 98.01 | 80.32 | 63.61 | 87.89 | 81.42 | 88.89 | 74.69 | 98.96 | 96.53
ShuffleNet | 99.70 | 98.30 | 81.76 | 58.69 | 87.53 | 80.42 | 86.21 | 70.97 | 98.65 | 96.00

Table 8: The variation of accuracy of backbones with reduction of training
data. Performance at 100% and 10% training data.

##  7 Observations

$\bigtriangleup$ Higher pre-training accuracy in ImageNet does not mean higher
fine-tuning accuracy. From Table 2 we can see that amoung all the backbones we
used for our experiments, EfficientNetV2-S has the highest pre-training
ImageNet-1k accuracy (84.23%). Even though the fine-tuning performance of
EfficientNet was decent, it did not perform the best in any of the domains or
datasets we evaluated on. Even amoung datasets such as Tiny ImageNet and
Stanford Dogs whose images are sourced from ImageNet dataset, we find that
ConvNeXt outperforms EfficientNet. Therefore, we recommend the practitioners
to not use the pre-training accuracy as a metric to choose the backbone.

$\bigtriangleup$ Convolutional models strongly outperform transformers in
resource-efficient low-data fine-tuning. Even though Swin transformer has a
hierarchical structure capable of exploiting the spacial inductive bias [7],
they still perform poorly in almost all our tasks compared to modern CNN
architectures such as ConvNeXt (which was designed based on Swin transformer).
Hence, we recommend that for the low data fine-tune regime, it is better to
avoid using transformer architectures like Swin and use pure CNN backbones
such as ConvNeXt, EfficientNet or RegNet.

$\bigtriangleup$ ConvNeXt architecture consistently outperforms other models
when fine-tuning on natural image datasets. This superior performance can be
attributed to the innovative design of ConvNeXt, which integrates
architectural advancements that bridge the gap between traditional
convolutional networks, like ResNet, and modern Swin transformer models.
ConvNeXt retains the beneficial convolutional inductive bias, enabling it to
learn more effectively than other convolution-based architectures and even
some attention-based transformer models. Our data clearly indicates that for
natural images, ConvNeXt stands out as the best model, delivering exceptional
fine-tuning performance.

$\bigtriangleup$ RegNet and EfficientNet models are excellent choices for
fine-tuning across a wide range of image domains. While ConvNeXt excels
predominantly with natural images, EfficientNet closely follows in
performance, and RegNet also shows strong results in this domain. However, the
versatility of RegNet and EfficientNet extends beyond natural images. Our
experiments reveal that these models also perform exceptionally well on
diverse datasets, including remote sensing images, plant datasets, and medical
images, such as histopathology images. Therefore, we recommend practitioners
to consider RegNet and EfficientNet when working with datasets beyond natural
images, as their adaptability and robust performance across various domains
make them valuable tools for fine-tuning tasks.

$\bigtriangleup$ ShuffleNet is a better choice than MobileNet when very light-
weight models are needed. When we observe the performance of very lightweight
models, specifically those with a model size of less than 50 MB, which are
ideal for on-device applications, we find that ShuffleNetV2 generally
outperforms MobileNetV3 across multiple domains. Although MobileNetV3 has
shown better performance on medical domain, ShuffleNetV2 demonstrates more
consistent and slightly superior performance across a broader range of image
domains. Therefore, we recommend ShuffleNet v2 as a better choice for
practitioners dealing with on-device applications, where fine-tuning a model
on a domain-specific dataset is required.

$\bigtriangleup$ WaveMix performs well in datasets where multi-resolution
token-mixing aid in learning WaveMix outperforms all other models (9% increase
from the second best, ConvNeXt) in galaxy morphology classification. WaveMix
also performs well in medical domain performing better than all other
datasets, and also maintaining the performance across different magnification.
WaveMix, which uses 2D-DWT might possess inductive bias that can analyse the
domains of astronomy and medical images better than other convolutional models
due its multi-resolution token-mixing. We recommend using WaveMx in domains
where features across different resolutions are needed for better performance.
WaveMix also performs better than ConvNeXt in CIFAR-10 and CIFAR-100 datasets,
which were actually low resolution natural images ($32\times 32$) which were
resized to ($256\times 256$) for our training. Similarly, it gives best
performance in remote sensing dataset, EuroSAT, whose images ($64\times 64$)
were resized to ($256\times 256$) for our training. WaveMix is also state-of-
the-art in many low-resolution datasets such as EMNIST ($28\times 28$). The
only low resolution image dataset where WaveMix did not perform well is Tiny
ImageNet whose images ($64\times 64$) were also resized to ($256\times 256$).
We attribute this to the fact that Tiny ImageNet is an ImageNet-1k subset and
the other models which has better performance in ImageNet-1k naturally
performed better. So, we recommend using WaveMix for low-resolution image
datasets.

$\bigtriangleup$ Age of ResNet dominance is over. Our results also reveal that
ResNet is no longer competitive compared to these modern architectures in any
domain. Our experimental data shows that for natural images, ResNet does not
even rank among the top three performers. It significantly lags behind newer
models such as ConvNeXt. While ResNet does perform relatively well on one
medical dataset, even in these cases, other models achieve similar performance
levels. Therefore, it is clear that practitioners should transition from using
ResNet to these newer architectures to achieve better results in their fine-
tuning tasks.

$\bigtriangleup$ ConvNeXt and Swin transformers perform poorly in medical
domain. Our results show that both the architectures perform perform really
bad compared to all the other models when used in medical domain. Therefore,
we suggest practitioners working in medical domain to not use these models
despite their high performance in other domains.

$\bigtriangleup$ Most of the top models in every domain retain their higher
performance even with less training data. We find that even when we fine-tune
with a small percentage of training data (even 1% $\sim 1000$ images), the
models which performed well with full training set still retained their
superiority. This points to the presence of a domain specific inductive bias
present in these models since they can learn better representations with very
less data. The failure of these models to perform well on other domains
similarly with less training data also alludes to this inductive bias.

Backbone | Block |  | Kernels
---
in block
| #
---
Stages
#Blocks/stage | Non-linearity | CEF | Stem | DConv | SE | Norm
ResNet-50 | Bottleneck | 1$\times$1, 3$\times$3, 1$\times$1 | 4 | 3,4,6,3 | ReLU | 4 | 7$\times$7, /2 | No | No | BN
DenseNet-161 | Inverted BN | 1$\times$1, 3$\times$3 | 4 | 6,12,36,24 | ReLU | 4 | 7$\times$7, /2 | No | No | BN
ResNeXt-50-32$\times$4d | Bottleneck | 1$\times$1, 3$\times$3, 1$\times$1 | 4 | 3,4,6,3 | ReLU | 2 | 7$\times$7, /2 | No | No | BN
ShuffleNetV2-$\times$2.0 | Isotropic | 1$\times$1, 3$\times$3, 1$\times$1 | 3 | 4,8,4 | ReLU | 1 | 3$\times$3, /2 | Yes | No | BN
MobileNetV3-Large | Inverted BN | 1$\times$1, 3$\times$3/5$\times$5, 1$\times$1 | 4 | 2,3,6,3 | ReLU, SiLU | 3-6 | 3$\times$3, /2 | Yes | Yes | BN
RegNet-Y3-2GF | Isotropic | 1$\times$1, 3$\times$3, 1$\times$1 | 4 | 2, 5, 13, 1 | ReLU | 1 | 3$\times$3, /2 | No | Yes | BN
EfficientNet-V2-S | Inverted BN | 1$\times$1, 3$\times$3, 1$\times$1 | 6 | 2, 4, 4, 6, 9, 15 | SiLU | 4-6 | 3$\times$3, /2 | Yes | Yes | BN
WaveMix-192/16 | Tokenmixer | 1$\times$1, 1$\times$1, 1$\times$1 | 1 | 16 | GELU | 2 | 4$\times$4, /4 | No | No | BN
ConvNeXt-Tiny | Tokenmixer | 7$\times$7, 1$\times$1, 1$\times$1 | 4 | 3,3,9,3 | GELU | 4 | 4$\times$4, /4 | Yes | No | LN

Table 9: Architectural comparison of all the convolutional backbones showing
the block structure of the residual block (eg. BottleNeck (BN)), kernels used
in each block in sequence, number of stages in pyramidal models, number of
blocks in each stage in pyramidal models, the non-linearities used in networks
blocks, the channel expansion factor (CEF) in each block, the kernel and
stride used in the initial stem layer, whether depth-wise convolutions (DConv)
and squeeze-excitation (SE) operations were used in blocks, and the
normalization used in blocks (whether BatchNorm (BN) or LayerNorm (LN)). The
architectures are ordered in descending order of age.

##  8 Architectural Discussions

Table 9 shows the architectural details of all the convolutional architectures
used in our experiments from oldest to latest. From the Table, we can see
certain architectural trends that has improved the performance of the newer
models compared to older ones. All top performing models in all the domains
are the latest models such as RegNet, EfficientNet, WaveMix and ConvNeXt.
These observations can be used to develop better resource-efficient backbones
for fine-tuning tasks.

Most of the models use a $1\times 1$, $3\times 3$, $1\times 1$ kernel
structure in their blocks with a normalisation and activation following each
convolutional operation. MobileNet uses $5\times 5$ kernels towards the final
stages, while ConvNeXt uses $7\times 7$ kernel for spatial token-mixing in
each block. All architectures use $1\times 1$ convolution to change the number
of channels.

While increasing the channel dimensions inside bottlenecks/inverted
bottlenecks, almost all models use a channel expansion/contraction factor of
4. MobileNet and EfficientNet uses even higher values up to 6.

All models use adaptive average pooling in the classification head to down-
sample the output feature maps from the final stage to $1\times 1$ spatial
resolution.

$\bigtriangleup$ Inverted bottleneck design is better. The older models used
bottleneck blocks where channel dimension was reduced using $1\times 1$
convolutions before being passed to the main $3\times 3$ convolutions to
increase efficiency. Newer architectures such as EfficientNet and MobileNet
use an inverted bottleneck structure where the $1\times 1$ convolutions
increases the channel dimension before $3\times 3$ convolutions. The increase
in parameters is offset mainly by using the parameter efficient depth-wise
convolution which also increases performance. Models like ShuffleNet and
RegNet keeps the channel dimension invariant throughout the block resulting in
an isotropic design. WaveMix and ConvNeXt follow the token-mixing structure
where spatial and channel mixing is performed separately, with spacial mixing
done with token-mixing operations such as wavelet transform and large kernel
depth-wise convolution respectively. The channel dimension is expanded only
for channel mixing using $1\times 1$ convolutions in an inverted bottleneck
design.

$\bigtriangleup$ Depth-wise convolution replacing regular convolution. Depth-
wise convolutions have fewer parameters and requires less operations than
regular convolutions. Most of the newer models such as ShuffleNet, MobileNet
and EfficientNet replace the $3\times 3$ kernel regular convolution with
$3\times 3$ kernel depth-wise convolution. ConvNeXt uses $7\times 7$ kernel
depth-wise convolution for efficient spacial token-mixing. All $1\times 1$
convolutions in all these models are point-wise convolutions (regular
convolutions). SiLU activation has been observed to give better performance
when used with depth-wise convolutions [17].

$\bigtriangleup$ 4 stage models with largest number of blocks in penultimate
stage. From old to new models, 4 stage structure has been most popular and has
resulted in good performance. Other than EfficientNet with 6 stages,
ShuffleNet with 3 stages and WaveMix with a single stage, all other backbone
has 4 stages. The design principle of increasing the number of blocks with
each subsequent stage with the exception of final stage is also observed in
most models. Only Efficient has more blocks in the final stage than the
penultimate stage. Most of the 4 stage models use a channel expansion factor
greater than or equal to 2 during feature resolution down-sampling between
stages to increase the number of channels.

$\bigtriangleup$ Strided convolution for down-sampling. With the exception of
DenseNet which used average pooling for down-sampling the feature resolution
between different stages of the network, all other convolutional models used
strided convolutions with stride 2 for feature resolution down-sampling. So
strided convolutions are better for model performance than pooling operations.

$\bigtriangleup$ Small kernel stem layer. Stem layer is the initial layer used
to down-sample the full size input image whose output is then sent to the body
of architectures which has multiple stages. Compared to older models which
used large $7\times 7$ kernel with stride 2, newer models use $3\times 3$ and
$4\times 4$ kernels. The token-mixers use patchify stem ($4\times 4$ non-
overlapping convolutions).

$\bigtriangleup$ SiLU and GELU replacing ReLU. Older models used ReLU
activation after convolution operations. The newer models use a combination of
ReLU and SiLU activations while the token-mixers inspired from transformer
architecture uses GELU activation. Compared to other convolutional models
which uses 2 to 3 activations per block, token-mixers use much less
activations (one per block).

$\bigtriangleup$ Addition of squeeze-excitation (SE) module. [39] SE module
enhance CNNs by focusing on the most informative features through explicit
modeling of channel interdependencies and can be easily integrated into
various architectures without significant changes. The top performing
architectures such as RegNet and EfficientNet used SE modules to improve
performance.

##  9 Conclusions

Computer vision backbones are critical for fine-tuning on domain specific
datasets, particularly when using pre-trained backbones available in libraries
such as Torchvision. In our study, we have compared the performance of various
lightweight, resource-efficient backbones across different domains, including
medical images, natural images, astronomy images, plant images, and remote
sensing images.

Our analysis revealed that modern architectural models like ConvNeXt,
EfficientNet, and RegNet excel in handling multiple domains images.
Additionally, models with specific inductive biases, such as WaveMix, are
particularly useful for tasks requiring multi-resolution analysis. Among
lightweight, on-device models, ShuffleNet consistently outperformed MobileNet
in fine-tuning tasks. We also observed that transformer-based or attention-
based models, such as Swin Transformer, do not perform well when the fine-
tuning dataset is small.

Based on our findings, we offer practical recommendations for using pre-
trained computer vision backbones. Our comparative analysis on image
classification fine-tuning serves as a valuable guide for practitioners and
researchers alike, who aim to optimize their models for various image domains.
We hope that our work will contribute to the development of better model
architectures capable of performing well across diverse image datasets.

Limitations: We restricted our comparison to models available in Torchvision,
focusing specifically on lightweight and resource-efficient architectures.
Consequently, we did not analyze any larger models with more than 30 million
parameters, limiting our ability to test the scalability of these models with
larger fine-tuning datasets. Furthermore, our analysis was confined to fine-
tuning datasets containing fewer than 100,000 images, which may not fully
represent scenarios involving significantly larger datasets and scalability of
these backbones.

Another limitation of our work is that we exclusively focused on the
computational task of image classification. We did not extend our analysis to
other important tasks in computer vision, such as object detection or image
retrieval. The performance of various backbones on these other tasks remains
unexplored in our study. While we hope that there might be some correlation
with the performance we observed to these other computer vision tasks, this
remains speculative and requires further investigation to confirm.

Computation Cost and Carbon Footprint: The experiments in this paper took a
cumulative 1500 GPU hours on NVIDIA RTX A100 cards. Assuming the GPUs were
running with an average carbon efficiency of 0.37 kgCO2eq/kWh, the total
emissions are estimated to be 222 kgCO2eq.

## References

  * [1] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  * [2] TorchVision maintainers and contributors.  Torchvision: Pytorch’s computer vision library.  https://github.com/pytorch/vision, 2016.
  * [3] Ross Wightman.  Pytorch image models.  https://github.com/rwightman/pytorch-image-models, 2019.
  * [4] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby.  A large-scale study of representation learning with the visual task adaptation benchmark, 2020.
  * [5] Lorenzo Brigato, Björn Barz, Luca Iocchi, and Joachim Denzler.  Tune it or don’t use it: Benchmarking data-efficient image classification, 2021.
  * [6] Mohammad Reza Hosseinzadeh Taher, Fatemeh Haghighi, Ruibin Feng, Michael B. Gotway, and Jianming Liang.  A systematic benchmarking analysis of transfer learning for medical image analysis, 2021.
  * [7] Micah Goldblum, Hossein Souri, Renkun Ni, Manli Shu, Viraj Prabhu, Gowthami Somepalli, Prithvijit Chattopadhyay, Mark Ibrahim, Adrien Bardes, Judy Hoffman, Rama Chellappa, Andrew Gordon Wilson, and Tom Goldstein.  Battle of the backbones: A large-scale comparison of pretrained models across computer vision tasks, 2023.
  * [8] Pranav Jeevan, Kavitha Viswanathan, Anandu A S, and Amit Sethi.  Wavemix: A resource-efficient neural network for image analysis, 2024.
  * [9] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik.  Emnist: an extension of mnist to handwritten letters, 2017.
  * [10] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows, 2021.
  * [11] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo.  Swin transformer v2: Scaling up capacity and resolution, 2022.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * [13] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s, 2022.
  * [14] Mingxing Tan and Quoc V. Le.  Efficientnetv2: Smaller models and faster training, 2021.
  * [15] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger.  Densely connected convolutional networks, 2018.
  * [16] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam.  Searching for mobilenetv3, 2019.
  * [17] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár.  Designing network design spaces, 2020.
  * [18] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks, 2017.
  * [19] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.  Shufflenet v2: Practical guidelines for efficient cnn architecture design, 2018.
  * [20] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, 2009.
  * [21] Ya Le and X. Yang.  Tiny imagenet visual recognition challenge, 2015.
  * [22] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei.  Novel dataset for fine-grained image categorization.  In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
  * [23] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
  * [24] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona.  Caltech-UCSD Birds 200.  Technical Report CNS-TR-2010-001, California Institute of Technology, 2010.
  * [25] Afshin Dehghan, Syed Zain Masood, Guang Shu, and Enrique. G. Ortiz.  View independent vehicle make, model and color recognition using convolutional neural network, 2017.
  * [26] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101 – mining discriminative components with random forests.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 446–461, Cham, 2014. Springer International Publishing.
  * [27] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi.  Describing textures in the wild.  In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
  * [28] Kechen Song and Yunhui Yan.  A noise robust method based on completed local binary patterns for hot-rolled steel strip surface defects.  Applied Surface Science, 285:858–864, 2013.
  * [29] Yi Yang and Shawn Newsam.  Bag-of-visual-words and spatial extensions for land-use classification.  In ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM GIS), 2010.
  * [30] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification, 2019.
  * [31] David. P. Hughes and Marcel Salathe.  An open access repository of images on plant health to enable the development of mobile disease diagnostics, 2016.
  * [32] Hervé Goëau, Pierre Bonnet, and Alexis Joly.  Overview of plantclef 2021: cross-domain plant identification.  In CLEF (Working Notes), pages 1422–1436, 2021.
  * [33] Henry W Leung and Jo Bovy.  Deep learning of multi-element abundances from high-resolution spectroscopic data.  Monthly Notices of the Royal Astronomical Society, November 2018.
  * [34] Fabio A. Spanhol, Luiz S. Oliveira, Caroline Petitjean, and Laurent Heutte.  A dataset for breast cancer histopathological image classification.  IEEE Transactions on Biomedical Engineering, 63(7):1455–1462, 2016.
  * [35] Anouk Stein, MD, Carol Wu, Chris Carr, George Shih, Jamie Dulkowski, kalpathy, Leon Chen, Luciano Prevedello, Marc Kohli, MD, Mark McDonald, Peter, Phil Culliton, Safwan Halabi MD, and Tian Xia.  Rsna pneumonia detection challenge, 2018.
  * [36] Samuel G. Müller and Frank Hutter.  Trivialaugment: Tuning-free yet state-of-the-art data augmentation, 2021.
  * [37] Pranav Jeevan and Amit sethi.  Convolutional xformers for vision, 2022.
  * [38] Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi.  Escaping the big data paradigm with compact transformers, 2021.
  * [39] Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu.  Squeeze-and-excitation networks, 2019.


# Constraint on Lorentz invariance violation from Vela pulsar

Hao Li  Jie Zhu  Bo-Qiang Ma<EMAIL_ADDRESS>
###### Abstract

The High Energy Stereoscopic System (H.E.S.S) Collaboration reported the
discovery of a novel radiation component from the Vela pulsar by their
Cherenkov telescopes. It is of great importance that gamma rays with energies
of at least 20 TeV are recorded unexpectedly. The H.E.S.S Collaboration argued
that such results may challenge the state-of-the-art models for the high-
energy emission of pulsars. We point out in this work that these results also
provide a unique opportunity to constrain certain Lorentz invariance violation
parameters, leading to the realization of studying Lorentz invariance
violation by using gamma-ray pulsars. The Lorentz invariance violation scale
is constrained at the level of $E_{\mathrm{LV,}1}>1.66\times 10^{17}\rm GeV$
for the linear scenario, and $E_{\mathrm{LV,}2}>3.53\times 10^{10}\rm GeV$ for
the quadratic scenario. We anticipate that digging into the detailed features
of the data of the Vela pulsar and analyzing potentially more very-high-energy
photon data from pulsars in the future would improve the constraints on
Lorentz invariance violation.

###### keywords:

Lorentz invariance violation , the Vela pulsar , very-high-energy gamma rays

††journal: arXiv

[first]organization=School of Physics, Peking University, city=Beijing 100871,
country=China

[second]organization=Center for High Energy Physics, Peking
University,city=Beijing 100871, country=China
[third]organization=Collaborative Innovation Center of Quantum Matter,
city=Beijing, country=China

Utilizing high-energy photons from distant objects in the universe is one of
the most important approaches to searching Lorentz invariance violation (LIV)
[1, 2]. In general, these sources are required to be at a very large distance
from the Earth and the emissions of energetic photons are highly variable,
with a time-structure of milliseconds [1, 2]. Therefore, gamma-ray bursts
(GRBs) have long been the centre focus for the study of LIV following this
method (see Refs. [3, 4] and references therein). Just recently, results from
the High Energy Stereoscopic System (H.E.S.S.) open a new window to searching
Lorentz invariance violation from very-high-energy (VHE) cosmic photons. The
Cherenkov telescopes of H.E.S.S. detected multi-TeV gamma-rays from the Vela
pulsar, PSR B0833045, of which the radiation component extends up to energies
of at least 20 TeV [5]. These results are already analyzed by the H.E.S.S.
Collaboration, to show that the state-of-the-art models for the high-energy
emission of pulsars may be challenged [5]. However, besides using these
results to limit the models for pulsars, it is noteworthy that the photons
from the Vela pulsar are very energetic, and the phasograms of the Vela pulsar
have time structures of milliseconds. Although this object is quite close to
the earth, at a distance of 287 pc [6, 5], we suggest that the Vela pulsar can
be used to constrain parameters of LIV, with the same method as that utilized
in the case of GRBs, because of the photons of energies of at least 20 TeV and
the high temporal variability of this source, which has a spin period of 89 ms
[5]. In the following, we perform a quick and rough analysis without more
detailed data and constrain the leading order and next-to-leading order LIV
parameters for photons.

It is speculated from quantum gravity that the Lorentz invariance might be
broken at the Planck scale ($E_{\mathrm{Pl}}\simeq 1.22\times
10^{19}~{}\mathrm{GeV}$), and that the light speed may have a variation with
the energy of the photon. For energy $E\ll E_{\rm Pl}$, the modified
dispersion relation (MDR) of the photon can be expressed to the leading order
as

$E^{2}=p^{2}c^{2}\left[1-s_{n}\left(\frac{pc}{E_{\mathrm{LV,}n}}\right)^{n}\right].$
(1)

Assuming that the traditional relation $v=\partial E/\partial p$ holds, we
have the following speed relation

$v(E)=c\left[1-s_{n}\frac{n+1}{2}\left(\frac{pc}{E_{\mathrm{LV,}n}}\right)^{n}\right],$
(2)

where $n=1$ or $n=2$ as usually assumed, $s_{n}=\pm 1$ indicates whether high-
energy photons travel faster ($s_{n}=-1$) or slower ($s_{n}=+1$) than low-
energy photons, and $E_{\rm{LV},n}$ represents the nth-order Lorentz violation
scale. Usually, we should consider the expansion of the Universe [7, 8].
However, since the Vela pulsar is located nearby at the distance of
$L=287~{}{\rm pc}$, we can ignore the expansion of the Universe and assume
that the traveling time of cosmic particles can be written as $T=L/v(E)$. Thus
the arrival time difference between a high-energy photon with energy $E$ and
the low-energy light is

$\Delta
T=\frac{L}{v(E)}-\frac{L}{c}=s_{n}\frac{(n+1)L}{2c}\left(\frac{E}{E_{\mathrm{LV,}n}}\right)^{n},$
(3)

and the nth-order Lorentz violation scale can be written as

$E_{\mathrm{LV,}n}=\left(s_{n}\frac{(1+n)L}{2c\Delta
T}\right)^{\frac{1}{n}}E.$ (4)

From Fig. 1 of Ref. [5], the phasograms of Vela as measured with H.E.S.S.
CT1–4 for energies above 5 TeV and with H.E.S.S. CT5 in the 10 to 80 GeV
range, we can see a possible time difference of peak P2 between energy above 5
TeV and $10-80~{}\rm GeV$. We can see that the high-energy photons travel
slower than low-energy photons, and that suggests $s_{n}=+1$. The time
difference between the two peaks is one-half of the bin width of the phasogram
for energies above 5 TeV ($0.01\times$ spin period). Considering that the time
difference read from the phasograms can be also from bin division, which means
that the actual time difference can be smaller, we can obtain that $\Delta
T<0.01T_{\rm period}=0.89~{}\rm ms$. Since $5~{}{\rm TeV}\gg 80~{}{\rm GeV}$,
we can ignore the contribution of the time difference from low-energy photons,
and the energy of high-energy photons has a relation $E>5~{}\rm TeV$. From Eq.
(4), for $n=1$, we have

$E_{\mathrm{LV,}1}=\frac{L}{c\Delta T}E>1.66\times 10^{17}~{}\rm GeV.$ (5)

For $n=2$, we have

$E_{\mathrm{LV,}2}=\left(\frac{3L}{2c\Delta
T}\right)^{\frac{1}{2}}E>3.53\times 10^{10}~{}\rm GeV.$ (6)

It is worth noting that, the results we obtained in Eq. (5) and Eq. (6) are
somewhat weaker compared to some existing constraints in the literature [3, 4]
(see, also, a conservative bound $E_{\mathrm{LV},1}\geq 3.6\times
10^{17}~{}\mathrm{GeV}$ in Refs. [9, 10, 11, 12, 13, 14, 15, 16, 17, 18]).
However these results can be expected to be refined by analyzing the data of
the Vela pulsar more thoroughly and utilizing potentially more observations of
very-high-energy photons from pulsars in the future. We anticipate that thus
more stringent constraints on Lorentz invariance violation parameters can be
obtained. On the other hand, the constraints obtained in this work, albeit
weak, reveal the possibility of utilizing pulsar photons to constrain Lorentz
invariance violation parameters. Therefore in this work we show that the Vela
pulsar expands the range of sources that can be chosen to effectively study
Lorentz invariance violation.

## Acknowledgements

This work is supported by National Natural Science Foundation of China under
Grants No. 12075003 and No. 12335006.

## References

  * [1] G. Amelino-Camelia, J. Ellis, N. E. Mavromatos, D. V. Nanopoulos, Distance measurement and wave dispersion in a Liouville-String approach to quantum gravity, International Journal of Modern Physics A 12 (03) (1997) 607–623.  arXiv:9605211, doi:10.1142/S0217751X97000566.
URL https://www.worldscientific.com/doi/abs/10.1142/S0217751X97000566

  * [2] G. Amelino-Camelia, J. Ellis, N. E. Mavromatos, D. V. Nanopoulos, S. Sarkar, Tests of quantum gravity from observations of $\gamma$-ray bursts, Nature 393 (6687) (1998) 763–765.  doi:10.1038/31647.
URL http://www.nature.com/articles/31647

  * [3] J.-J. Wei, X.-F. Wu, Tests of lorentz invariance (2022).  arXiv:2111.02029, doi:10.1007/978-981-16-4544-0_132-1.
URL https://doi.org/10.1007/978-981-16-4544-0_132-1

  * [4] P. He, B.-Q. Ma, Lorentz Symmetry Violation of Cosmic Photons, Universe 8 (6) (2022) 323.  arXiv:2206.08180, doi:10.3390/universe8060323.
  * [5] The H.E.S.S. Collaboration et al. and R. Zanin, M. Kerr et al., Discovery of a radiation component from the Vela pulsar reaching 20 teraelectronvolts, Nat Astron (2023).  doi:10.1038/s41550-023-02052-3.
  * [6] R. Dodson, D. Legge, J. E. Reynolds, P. M. McCulloch, The Vela pulsar’s proper motion and parallax derived from VLBI observations, Astrophys. J. 596 (2003) 1137–1141.  arXiv:astro-ph/0302374, doi:10.1086/378089.
  * [7] U. Jacob, T. Piran, Lorentz-violation-induced arrival delays of cosmological particles, Journal of Cosmology and Astroparticle Physics 2008 (01) (2008) 031.  arXiv:0712.2170, doi:10.1088/1475-7516/2008/01/031.
URL https://iopscience.iop.org/article/10.1088/1475-7516/2008/01/031

  * [8] J. Zhu, B.-Q. Ma, Lorentz-violation-induced arrival time delay of astroparticles in Finsler spacetime, Phys. Rev. D 105 (12) (2022) 124069.  arXiv:2206.07616, doi:10.1103/PhysRevD.105.124069.
  * [9] L. Shao, Z. Xiao, B.-Q. Ma, Lorentz violation from cosmological objects with very high energy photon emissions, Astroparticle Physics 33 (5-6) (2010) 312–315.  arXiv:0911.2276, doi:10.1016/j.astropartphys.2010.03.003.
URL https://linkinghub.elsevier.com/retrieve/pii/S0927650510000654

  * [10] S. Zhang, B.-Q. Ma, Lorentz violation from gamma-ray bursts, Astroparticle Physics 61 (2015) 108–112.  arXiv:1406.4568, doi:10.1016/j.astropartphys.2014.04.008.
  * [11] H. Xu, B.-Q. Ma, Light speed variation from gamma-ray bursts, Astroparticle Physics 82 (2016) 72–76.  arXiv:1607.03203, doi:10.1016/j.astropartphys.2016.05.008.
URL http://linkinghub.elsevier.com/retrieve/pii/S0927650516300792

  * [12] H. Xu, B.-Q. Ma, Light speed variation from gamma ray burst GRB 160509A, Physics Letters B 760 (2016) 602–604.  arXiv:1607.08043, doi:10.1016/j.physletb.2016.07.044.
URL https://linkinghub.elsevier.com/retrieve/pii/S0370269316303859

  * [13] G. Amelino-Camelia, G. D’Amico, G. Rosati, N. Loret, In vacuo dispersion features for gamma-ray-burst neutrinos and photons, Nature Astronomy 1 (2017) 1–12.  arXiv:1612.02765v1, doi:10.1038/s41550-017-0139.
  * [14] H. Xu, B.-Q. Ma, Regularity of high energy photon events from gamma ray bursts, Journal of Cosmology and Astroparticle Physics 2018 (01) (2018) 050–050.  arXiv:1801.08084, doi:10.1088/1475-7516/2018/01/050.
URL https://iopscience.iop.org/article/10.1088/1475-7516/2018/01/050

  * [15] Y. Liu, B.-Q. Ma, Light speed variation from gamma ray bursts: criteria for low energy photons, The European Physical Journal C 78 (10) (2018) 825.  arXiv:1810.00636, doi:10.1140/epjc/s10052-018-6294-y.
URL http://link.springer.com/10.1140/epjc/s10052-018-6294-y

  * [16] H. Li, B.-Q. Ma, Light speed variation from active galactic nuclei, Science Bulletin 65 (4) (2020) 262–266.  doi:10.1016/j.scib.2019.11.024.
URL https://linkinghub.elsevier.com/retrieve/pii/S2095927319306693

  * [17] J. Zhu, B.-Q. Ma, Pre-burst events of gamma-ray bursts with light speed variation, Physics Letters B 820 (2021) 136518.  doi:10.1016/j.physletb.2021.136518.
  * [18] Y. Chen, B.-Q. Ma, Novel pre-burst stage of gamma-ray bursts from machine learning, Journal of High Energy Astrophysics 32 (2021) 78–86.  doi:https://doi.org/10.1016/j.jheap.2021.09.002.
URL https://www.sciencedirect.com/science/article/pii/S2214404821000380


# WaterNeRF: Neural Radiance Fields for Underwater Scenes

Advaith Venkatramanan Sethuraman1, Manikandasriram Srinivasan Ramanagopal1 and
Katherine A. Skinner1 *This work was supported by a University of Michigan
Robotics Institute Fellowship.1A. V. Sethuraman, M. Srinivasan Ramanagopal,
and K. A. Skinner are with the Department of Robotics, University of Michigan,
Ann Arbor, MI 48109 USA<EMAIL_ADDRESS>
###### Abstract

Underwater imaging is a critical task performed by marine robots for a wide
range of applications including aquaculture, marine infrastructure inspection,
and environmental monitoring. However, water column effects, such as
attenuation and backscattering, drastically change the color and quality of
imagery captured underwater. Due to varying water conditions and range-
dependency of these effects, restoring underwater imagery is a challenging
problem. This impacts downstream perception tasks including depth estimation
and 3D reconstruction. In this paper, we advance state-of-the-art in neural
radiance fields (NeRFs) to enable physics-informed dense depth estimation and
color correction. Our proposed method, WaterNeRF, estimates parameters of a
physics-based model for underwater image formation, leading to a hybrid data-
driven and model-based solution. After determining the scene structure and
radiance field, we can produce novel views of degraded as well as corrected
underwater images, along with dense depth of the scene. We evaluate the
proposed method qualitatively and quantitatively on a real underwater dataset.

##  I INTRODUCTION

Improving underwater perception will advance autonomous capabilities of
underwater vehicles, enabling increased complexity of their missions. This
will have impacts across marine science and industry [1, 2]. However, unlike
in air, underwater images are often severely degraded due to water column
effects on underwater light propagation. In underwater environments,
absorption contributes to wavelength-dependent attenuation, resulting in color
distortion in the image. Backscattering causes a haze effect across the scene,
similar to fog in air [3]. This can pose a problem for marine robotic systems
that rely on accurate and consistent color and dense structural information to
enable high level perceptual tasks. Ideally, underwater images could be
restored to appear as if taken in air. However, water column effects depend on
many factors including scene structure, water characteristics, and light
properties. Furthermore, there is often a lack of ground truth for color and
structure of underwater scenes. These factors make underwater image
restoration and dense scene reconstruction an ill-posed problem.

On land, there have been impressive advances in neural scene representation,
which can learn a volumetric scene function to enable rendering of novel
viewpoints. Mildenhall et al. developed Neural Radiance Fields (NeRF), which
can be trained on a set of sparse input images to learn a function that maps
position and viewing direction to view-dependent radiance and volume density
[4]. Neural reflectance fields have been developed to enable novel view
rendering with scene relighting [5, 6]. Zhang et al. extended neural
reflectance fields to further decompose reflectance to recover material and
illumination properties [7]. These advances in neural scene representation
demonstrate the ability to learn scene properties based on a set of input
images by accurately modeling the physics of image formation. We propose to
extend these advances to underwater environments to learn to model – and
correct for – effects of underwater light propagation, while simultaneously
learning a dense structural model of the scene.

Our proposed method, WaterNeRF, is the first method to extend neural volume
rendering to underwater environments. This enables novel view synthesis for
underwater scenes with full scene restoration as well as dense depth
estimation. The WaterNeRF framework, shown in Fig. 1, takes as input monocular
images of an underwater scene, constructs a volume represented as a NeRF, then
infers the wavelength-specific attenuation parameters required to restore the
scene to its in-air appearance. Notably, WaterNeRF exploits the strengths of
NeRFs to encode a scene and uses the resulting depth information to inform a
physics-based, data-driven restoration approach. We evaluate our method
against prior work on a dataset collected in a controlled lab environment with
ground truth color and 3D structure available.

Figure 1: Overview of the WaterNeRF framework. The input for training is a
collection of posed images captured in underwater environments. A multi-layer
perceptron learns to map this input to volume density, underwater RGB color,
and water column parameters. With the learned volume density, the depth is
computed using ray marching. The depth, RGB, and water column parameters are
then leveraged to output a corrected RGB image. Loss is computed against the
known raw underwater images and a set of reference images obtained by
histogram equalization. During inference, the framework can compute an
estimate of scene structure and water column parameters to produce corrected
color images that have consistent color from varying viewpoints.

##  II Related Work

###  II-A Image Processing for Underwater Imagery

Image processing approaches such as histogram equalization [8] and the gray
world assumption [9] can be used for underwater image enhancement. These
approaches act to stretch the effective contrast of underwater images, but are
not aware of the underlying physics-based process of underwater image
formation. As a result, they do not correct for various range-dependent
effects. This can lead to inconsistent image restoration across images of the
same scene taken from different viewpoints.

###  II-B Physics-based Models for Underwater Image Formation

The attenuation of light transmitted from an object underwater is dependent on
water conditions, scene structure, and material properties. Many physics-based
image restoration methods first use priors such as the Dark Channel Prior
[10], or haze line prior [11] to estimate the backscatter in an image. For
example, Berman et al. [12] first estimates backscatter from a single image
then produces a transmission map [13]. While this method produces appealing
results, it requires both a stereo camera setup and knowledge of specific
water types. Akkaynak et al. developed an updated model for underwater image
formation and restoration [3, 14, 15]. However, this method requires a known,
dense depth map of a scene.

###  II-C Deep Learning for Underwater Image Restoration

Recent work has focused on leveraging deep learning to learn a model for
underwater image formation to demonstrate underwater image rendering [16, 17,
18]. UGAN uses generated data to train a GAN model to remove the degradation
present in low quality underwater images [17]. WaterGAN is another adversarial
network that considers the inverse problem: generation of synthetic underwater
imagery from in-air images [18]. Our work will build on advances in learning-
based image restoration, but instead structure our solution as a NeRF
framework to enable learning a dense structural model of the scene in addition
to corrected color.

###  II-D Neural Radiance Fields

Most relevant to this work, neural reflectance fields have been developed to
enable novel view rendering with scene relighting [5, 6]. The effects of light
transport through different mediums have also been considered in the context
of NeRFs. Specifically, NeRFs have been used to estimate albedo and ambient
light from satellite imagery [19]. Recently, NeRFs have been used to model the
refraction of light at the interface between air and water [20]. Similarly, a
NeRF formulation for our problem is convenient because it allows explicit
reasoning and optimization of the various components of light received by the
imaging sensor.

##  III Technical Approach

Figure 1 shows an overview of the proposed pipeline for the WaterNeRF
framework. During training, the network takes as input 3D position and viewing
direction of a scene point and learns to output volume density, underwater RGB
color, and water column parameters, including attenuation and backscatter
parameters. The depth, underwater RGB, and water column parameters are inputs
to a physics-based model for underwater image restoration that outputs a
corrected RGB image. The training images require known poses, which can be
obtained by a Structure from Motion (SfM) pipeline such as COLMAP [21]. During
inference, we are able to render images from novel viewpoints with and without
water column effects.

###  III-A WaterNeRF Formulation

NeRFs take as input a query location $x,y,z$ and viewing directions
$\theta,\phi$ and output the color $r,g,b$ and density $\sigma$ [4]. We
specifically leverage mip-NeRF, which replaces rays with conical frustums to
avoid aliasing issues [22]. Mip-NeRF was chosen because both image quality and
depth map quality are crucial to image restoration underwater.

Mip-NeRF uses a multilayer perceptron (MLP) parameterized by weights $\Theta$.
The MLP is used to compute the color of a ray, say $r(t)=o+td$, where
$t\in[t_{n},t_{f}]$ lies between near plane $t_{n}$ and far plane $t_{f}$, $o$
is the ray origin, and $d$ is the ray direction. The query location is encoded
using an integrated positional encoding $\gamma(r(t))$ before being passed to
the MLP [22].

$\forall t_{k}\in
t,[\sigma_{k},c_{k}]=MLP(\gamma(r(t_{k})),\theta,\phi;\Theta)$ (1)

Each $\sigma_{k},c_{k}$ is used in a numerical quadrature sum to estimate the
color $C(r;\Theta,t)$ of a ray.

$C(r;\Theta,t)=\sum_{k}T_{k}(1-exp(-\sigma_{k}(t_{k+1}-t_{k})))c_{k}$ (2)

The depth at a given ray can be found similarly:

$D(r;\Theta,t)=\sum_{k}T_{k}(1-exp(-\sigma_{k}(t_{k+1}-t_{k})))t_{km}$ (3)

where $t_{km}=\frac{1}{2}(t_{k+1}+t_{k})$ and

$T_{k}=exp\left(-\sum_{k^{\prime}<k}\sigma_{k^{\prime}}(t_{k^{\prime}+1}-t_{k^{\prime}})\right)$
(4)

Finally, a color loss $\mathcal{L}_{c}$ is computed between the predicted
color $C(r;\Theta,t)$ and the ground truth pixel values $C^{*}(r)$ from the
observed images, which in our case are the original underwater images. This
loss is composed of coarse $C(r;\Theta,t^{c})$ and fine $C(r;\Theta,t^{f})$
queries to a single MLP network.

$\mathcal{L}_{c}=\lambda||C^{*}(r)-C(r;\Theta,t^{c})||^{2}_{2}+||C^{*}(r)-C(r;\Theta,t^{f})||^{2}_{2}$
(5)

###  III-B Color Correction Branch

In order to determine the color corrected appearance of the scene, we use the
model for light transport in water presented in [13]. $I_{c}(x)$ is the
acquired image at pixel location $x$ for channel $c\in\\{R,G,B\\}$, $t_{c}(x)$
is the transmission map, $J_{c}(x)$ is the object’s original radiance prior to
attenuation, and $A_{c}$ is the veiling light or backscatter. The transmission
map $t_{c}(x)$ is related to the wavelength specific attenuation coefficients
$\beta_{c}$ and the depth at the pixel $D(x)$.

$I_{c}(x)=t_{c}(x)J_{c}(x)+(1-t_{c}(x))A_{c}$ (6)
$t_{c}(x)=exp(-\beta_{c}D(x))$ (7)

Using this relationship, we can define the color corrected output of the
network $J(r;\Theta,t)$ as shown in Fig. 1.

$J(r;\Theta,t)=\frac{C(r;\Theta,t)-(1-t_{c}(x))A_{c}}{t_{c}(x)}$ (8)

Since we do not have ground truth color corrected images $I^{\ast}(r)$, we
cannot directly minimize a loss between $J(r;\Theta,t)$ such as in Equation
(5). Instead, we can rely on a supporting dataset of histogram equalized
underwater images $H(r)$ to provide an initialization. However, histogram
equalized imagery does not respect Equation (6) nor does it produce
consistently corrected images across the scene. To address these problems, we
reframe the color correction loss as an optimal transport problem and attempt
to match the distribution of pixels in histogram equalized images. We wish to
estimate the attenuation coefficients $\beta_{c}$ and veiling light $A_{c}$ to
correct an underwater image.

###  III-C Optimal Transport Primer

Discrete optimal transport problems involve distributions such as
$\mu,\nu\in\mathcal{P}(X)$ where $\mu=\sum_{i=1}^{n}a_{i}\delta_{x_{i}}$ and
$\nu=\sum_{j=1}^{m}b_{j}\delta_{y_{j}}$ and $\delta$ is a Dirac-delta centered
at points $\\{x_{i}|\forall i\in[1,n]\\}$ and $\\{y_{j}|\forall j\in[1,m]\\}$
[23]. The weights associated with this discrete distribution are such that
$a=(a_{1},\dots,a_{n})^{\top}\in\Delta_{n}$ and
$b=(b_{1},\dots,b_{m})^{\top}\in\Delta_{m}$, where

$\Delta_{n}=\\{p\in\mathbb{R}^{n}_{+}\ |\ p^{\top}\textbf{1}_{n}=1\\}$ (9)

$\textbf{1}_{n}$ is a vector containing ones of length $n$. The goal is to
compute a transport plan $T$ such that the Wasserstein distance of the
following form is minimized:

$W^{p}_{p}(\mu,\nu)=\min_{T\in\Pi(a,b)}\langle T,M\rangle$ (10)

$M$ is a cost matrix $\mathbb{R}^{n\times m}$ where $M_{ij}=d(x_{i},y_{j})$, a
valid distance between points $x_{i}$ and $y_{j}$. $\langle T,M\rangle$ is the
Frobenius product between matrices $Tr(T^{\top}M)$. The transport plan
$T\in\Pi(a,b)$ must be within the set

$\Pi(a,b)=\\{T\in\mathbb{R}^{n\times
m}_{+}:T\mathbf{1}_{m}=a,T^{\top}\mathbf{1}_{n}=b\\}$ (11)

This expensive optimization problem can be regularized and solved using a
differentiable process called the Sinkhorn algorithm $\tilde{S}_{\lambda}$.
The Sinkhorn algorithm is considered an efficient approximation of the
Wasserstein distance [24, 25].

$\tilde{S}_{\lambda}=\min_{T\in\Pi(a,b)}\langle
T,M\rangle-\frac{1}{\lambda}h(T)$ (12)
$h(T)=-\sum_{i,j=1}^{n,m}T_{i,j}(logT_{ij}-1)$ (13)

In this work, the Sinkhorn Loss is used between probability distributions
$\mu$, which denotes the distribution of corrected colors output by WaterNeRF
$J(r;\Theta,t)$ and $\nu$, which denotes the distribution of colors in the
histogram equalized images. Both $\mu$ and $\nu$ are distributions in the RGB-
space $\mathbb{R}^{3}$. Formally, the optimization problem becomes:

$\displaystyle\beta_{c}^{*},A^{*}_{c}$
$\displaystyle=\arg\min_{\beta_{c},A_{c}}\tilde{S}_{\lambda}(\mu,\nu)$ (14)
$\displaystyle\textrm{such that }\mu$
$\displaystyle=\frac{\nu-(1-t_{c}(x))A_{c}}{t_{c}(x)}$

The framework must optimize parameters $\beta_{c}$ in order to minimize the
Sinkhorn Loss between two distributions of colors. Since the optimization is
physically constrained, the optimal point $\beta_{c}^{*}$ will provide the
attenuation parameters required to map in-water images to histogram equalized
images while respecting the geometric (depth map) and physical (attenuation)
properties of the image formation model in Equation (6). This formulation
allows the model to infer parameters $\beta^{*}_{c},A^{*}_{c}$ that yield a
color corrected image. The Sinkhorn loss used in this work is the following:

$\mathcal{L}_{s}=\tilde{S}_{\lambda}(J(r;\Theta,t),H(r))$ (15)

The final objective is:

$\mathcal{L}=\mathcal{L}_{c}+\alpha\mathcal{L}_{s}$ (16)

TABLE I: Angular error (degrees) in color correction for novel views (N=3). Lower is better ($\downarrow$). Best is shown in bold Method | Blue $\bar{\psi}$ | Red $\bar{\psi}$ | Magenta $\bar{\psi}$ | Green $\bar{\psi}$ | Cyan $\bar{\psi}$ | Yellow $\bar{\psi}$ | Mean $\bar{\psi}$
---|---|---|---|---|---|---|---
Histogram Equalization | 6.67 | 14.07 | 8.06 | 9.55 | 3.78 | 3.11 | 7.54
Underwater Haze Lines | 4.39 | 16.09 | 5.17 | 13.91 | 8.80 | 4.68 | 8.84
FUnIE-GAN | 36.39 | 16.83 | 25.96 | 10.08 | 23.23 | 12.04 | 20.75
HENeRF | 5.12 | 14.89 | 7.72 | 11.13 | 4.08 | 5.59 | 7.78
WaterNerf (ours) | 7.88 | 19.51 | 9.71 | 11.32 | 2.80 | 5.59 | 9.47

##  IV Experiments & Results

###  IV-A Training Details

The network was trained in JAX, modified from the mip-NeRF repository
[22][26]. An initial learning rate of 5e-4 and a final learning rate of 5e-6
are used with learning rate decay. The Adam optimizer is used [27]. The
networks are trained for 600,000 iterations, which takes around 8 hours on an
Nvidia GeForce RTX 3090 GPU. We use a $\alpha=0.5$ for weighting the Sinkhorn
Loss.

###  IV-B Dataset

We evaluated our system on the open source UWBundle dataset, which consists of
36 images of an artificial rock platform submerged in a controlled in-lab
water tank environment [28]. Multiple views of the scene were captured with an
underwater camera in a lawnmower pattern, common in underwater surveying. The
rock platform contains a colorboard in the scene, making it ideal for
performing quantitative evaluation of color correction. An in-air ground truth
3D reconstruction was made using KinectFusion.

###  IV-C Comparison to Baselines

We compare the performance of WaterNeRF to prior work on image processing,
physics-based underwater image restoration, and deep learning-based methods.

FUnIE-GAN – This is a conditional GAN-based model that learns the mapping
between degraded underwater imagery and restored images [29].

Underwater Haze Lines Model – Using the haze lines prior, this model estimates
backscatter light and transmission map. Then, attenuation coefficients are
chosen from Jerlov’s classification of water types based on performance under
the Gray-World assumption [30, 12].

Histogram Equalization – We use histogram equalized imagery to provide an
initialization point for our optimization algorithm and we compare to these
images as a baseline. Histogram equalization is performed separately for each
color channel.

HENeRF – This is a mipNeRF trained directly on the histogram equalized images.
This has no color correction module and uses an MSE loss on the histogram
equalized pixel values.

WaterNeRF-No Sinkhorn – This is an ablation on the Sinkhorn loss. The
WaterNeRF is trained with an MSE loss between the corrected color prediction
$J(r;\Theta,t)$ and the histogram equalized color $H(r)$.

Figure 2: (top) Raw images, (second row) histogram equalized images, (third
row) UWHazeLines output, (fourth row) FuNIE-GAN output, (fifth row) HENeRF
Output, (sixth row) WaterNeRF Output, (seventh row) Depth Output from HENeRF,
(eight row) Depth Output from WaterNeRF

###  IV-D Qualitative Evaluation

Figure 2 shows qualitative performance against the baselines on the UWBundle
dataset. Histogram equalization and HENeRF result in visually appealing
images, but the corrected images are not consistent across multiple
viewpoints. The Haze Lines model suffers from vignetting artifacts. WaterNeRF
and FUnIE-GAN both produce consistent color corrected images across multiple
images. HENeRF and WaterNerF both produce dense maps of the scene. Since the
histogram equalized images are inconsistent, HENeRF produces poor depth
structure with holes. WaterNeRF produces dense depth maps without visible
discontinuities.

###  IV-E Color Correction Evaluation

Our primary evaluation metric for color correction is average angular error
$\bar{\psi}$ in the pixel space [12]. $I(x_{i})$ is the color at pixel
location $x_{i}$ and $\langle R_{g},G_{g},B_{g}\rangle$ is the color at the
ground truth pixel location. The colors are normalized then the angle in
degrees between the two unit vectors is computed by:

$\bar{\psi}=\textrm{cos}^{-1}\left(\frac{I(x_{i})\cdot\langle
R_{g},G_{g},B_{g}\rangle}{\|{I(x_{i})}\|\cdot\|\langle
R_{g},G_{g},B_{g}\rangle\|}\right)$ (17)

Table I shows the average angular error for patches of the colorboard within
corrected images from 3 held out test images containing the colorboard. The
error was calculated between patches of the colorboard imaged in air and
patches within corrected images underwater. For each color, a rectangular
region was taken from the patch and averaged. This average color was used to
calculate $\bar{\psi}$. Histogram equalization performs the best for color
accuracy. However, it is important to note that the histogram equalization
process will be biased for images that contain the color board. FUnIE-GAN
suffers on color accuracy relative to other methods. WaterNeRF is able to
correct images comparably to histogram equalization and the Haze Lines model.

###  IV-F Underwater Image Quality

Next, we evaluate the quality of restored images using the Underwater Image
Quality Metric (UIQM) [31]. UIQM considers the Underwater Image Colorfulness
Measure (UICM), the Underwater Image Contrast Measure (UIConM), and the
Underwater Image Sharpness Measure (UISM):

$UIQM=c_{1}\times UICM+c_{2}\times UISM+c_{3}\times UIConM$ (18)

We use the coefficients: $c_{1}=0.282,c_{2}=0.2953,c_{3}=3.5753$ from [31].
Table II reports the UIQM for corrected images produced by the baselines and
WaterNeRF on the UWBundle dataset. There are 5 held out test images, 3
containing the colorboard and 2 with no colorboard. WaterNeRF produces images
of higher underwater image quality than the Haze Lines model and FUnIE-GAN,
but lower quality than histogram equalization.

TABLE II: UIQM performance on novel views within the UWBundle dataset (N=5). Higher is better ($\uparrow$). Best is shown in bold.  Method | UIQM
---|---
Histogram Equalization | 2.96
Underwater Haze Lines | 2.60
FUnIE-GAN | 2.06
HENeRF | 2.40
WaterNeRF (ours) | 2.88

###  IV-G Scene Consistency Evaluation

In order to quantify the consistency of the color corrections, we calculate
the average standard deviation of the intensity-normalized RGB values of
pixels tracked through the scene. We call this the scene consistency metric
(SCM). First, we find a set of SURF features $\mathcal{P}$ between frames in
the underwater images, where $N=|\mathcal{P}|$ [32]. Then we track the pixels
$x_{i}\in\mathbb{R}^{3},i\in[1,N_{x}]$ corresponding to a feature
$x\in\mathcal{P}$ through $N_{x}$ corrected images. Finally, we calculate the
standard deviation of the RGB values of those pixels. This metric provides a
way to quantify the consistency of the correction method across different
views within the same scene.

$SCM=\frac{1}{N}\sum_{x\in\mathcal{P}}\sqrt{\frac{\sum_{x_{i}\in
x}(x_{i}-\mu_{x})^{2}}{N_{x}}}$ (19)

Table III reports the Scene Consistency Metric (SCM). Each color channel is
reported individually. FUnIE-GAN performs well on the SCM metric. WaterNeRF
also produces consistent images across scenes, with the lowest SCM in the red
channel and second lowest in the blue channel.

TABLE III: Scene Consistency Metric (SCM) across novel views in the UWBundle dataset (N=5). Lower is better ($\downarrow$). Best is shown in bold.  Method | $SCM_{R}$ | $SCM_{G}$ | $SCM_{B}$
---|---|---|---
Histogram Equalization | 0.1263 | 0.1287 | 0.1120
Underwater Haze Lines | 0.3840 | 0.3426 | 0.3068
FUnIE-GAN | 0.1064 | 0.0619 | 0.0908
HENeRF | 0.2051 | 0.1731 | 0.1800
WaterNeRF (ours) | 0.0423 | 0.1975 | 0.0919

###  IV-H Evaluation of Depth Maps

In addition to underwater and corrected color images, WaterNeRF also produces
a dense depth map for each novel view generated. To evaluate the quality of
this reconstruction, we project the 2D pixels into 3D points using the
camera’s intrinsic matrix. Finally, we use CloudCompare [33] to compute the
distance between the point cloud and a ground truth mesh for the UWBundle
dataset. Table IV reports the evaluation between a point cloud of the scene
generated by WaterNeRF and a ground truth mesh. WaterNeRF demonstrates higher
accuracy on depth reconstruction than HENeRF. Figure 3 shows sample point
clouds projected from the dense depth estimates and color corrected imagery
for WaterNerf.

TABLE IV: RMSE in meters between a ground truth 3D mesh for the UWBundle dataset and the pointclouds generated by WaterNeRF and HENeRF. Model | $Pred\rightarrow GT$ | G$T\rightarrow Pred$
---|---|---
WaterNeRF | 0.036 | 0.0014
HENeRF | 0.105 | 0.099
|
---|---
Figure 3: (left) 3D reconstruction from WaterNeRF. Color corrected RGB points
are projected onto the pointcloud provided from the dense depth map. (right)
Ground truth in-air 3D scan.

###  IV-I Ablation: WaterNeRF-No Sinkhorn Loss

We performed ablation studies with no Sinkhorn loss to demonstrate that a
direct MSE loss on the histogram equalized images can produce inconsistent
views (Fig. 4).

Figure 4: Two views generated by WaterNeRF with no Sinkhorn loss results in
inconsistent restorations.

##  V Conclusion & Future Work

This work is the first to develop neural volume rendering for underwater
scenes to enable novel view synthesis of underwater scenes with full scene
restoration and dense depth estimation. WaterNeRF performs comparably for
color correction, produces consistent scenes, and enables more accurate depth
estimation. Improvement of underwater image quality can accelerate adoption of
traditional robotic perception algorithms. Future work includes using 3D
object information and color from WaterNeRF to inform underwater manipulation
and exploration tasks for marine robotic platforms.

## References

  * [1] M. Johnson-Roberson, M. Bryson, A. Friedman, O. Pizarro, G. Troni, P. Ozog, and J. C. Henderson, “High-resolution underwater robotic vision-based mapping and three-dimensional reconstruction for archaeology,” Journal of Field Robotics, vol. 34, no. 4, pp. 625–643, 2017.
  * [2] M. Saberioon, A. Gholizadeh, P. Cisar, A. Pautsina, and J. Urban, “Application of machine vision systems in aquaculture with emphasis on fish: state-of-the-art and key issues,” Reviews in Aquaculture, vol. 9, no. 4, pp. 369–387, 2017.
  * [3] D. Akkaynak and T. Treibitz, “A revised underwater image formation model,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6723–6732, 2018.
  * [4] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” in European conference on computer vision, pp. 405–421, Springer, 2020.
  * [5] S. Bi, Z. Xu, P. Srinivasan, B. Mildenhall, K. Sunkavalli, M. Hašan, Y. Hold-Geoffroy, D. Kriegman, and R. Ramamoorthi, “Neural reflectance fields for appearance acquisition,” arXiv preprint arXiv:2008.03824, 2020\.
  * [6] P. P. Srinivasan, B. Deng, X. Zhang, M. Tancik, B. Mildenhall, and J. T. Barron, “Nerv: Neural reflectance and visibility fields for relighting and view synthesis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7495–7504, 2021.
  * [7] X. Zhang, P. P. Srinivasan, B. Deng, P. Debevec, W. T. Freeman, and J. T. Barron, “Nerfactor: Neural factorization of shape and reflectance under an unknown illumination,” arXiv preprint arXiv:2106.01970, 2021.
  * [8] S. M. Pizer, E. P. Amburn, J. D. Austin, R. Cromartie, A. Geselowitz, T. Greer, B. ter Haar Romeny, J. B. Zimmerman, and K. Zuiderveld, “Adaptive histogram equalization and its variations,” Computer vision, graphics, and image processing, vol. 39, no. 3, pp. 355–368, 1987.
  * [9] G. Buchsbaum, “A spatial processor model for object colour perception,” Journal of the Franklin institute, vol. 310, no. 1, pp. 1–26, 1980.
  * [10] N. Carlevaris-Bianco, A. Mohan, and R. M. Eustice, “Initial results in underwater single image dehazing,” in Oceans 2010 Mts/IEEE Seattle, pp. 1–8, IEEE, 2010.
  * [11] D. Berman, T. Treibitz, and S. Avidan, “Non-local image dehazing,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1674–1682, 2016.
  * [12] D. Berman, D. Levy, S. Avidan, and T. Treibitz, “Underwater single image color restoration using haze-lines and a new quantitative dataset,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 8, pp. 2822–2837, 2021.
  * [13] Y. Schechner and N. Karpel, “Recovery of underwater visibility and structure by polarization analysis,” IEEE Journal of Oceanic Engineering, vol. 30, no. 3, pp. 570–587, 2005.
  * [14] D. Akkaynak, T. Treibitz, T. Shlesinger, Y. Loya, R. Tamir, and D. Iluz, “What is the space of attenuation coefficients in underwater computer vision?,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 568–577, 2017.
  * [15] D. Akkaynak and T. Treibitz, “Sea-thru: A method for removing water from underwater images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1682–1691, 2019.
  * [16] T. Ye, S. Chen, Y. Liu, Y. Ye, E. Chen, and Y. Li, “Underwater light field retention: Neural rendering for underwater imaging,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 488–497, 2022.
  * [17] C. Fabbri, M. J. Islam, and J. Sattar, “Enhancing underwater imagery using generative adversarial networks,” 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 7159–7165, 2018.
  * [18] J. Li, K. A. Skinner, R. M. Eustice, and M. Johnson-Roberson, “Watergan: Unsupervised generative network to enable real-time color correction of monocular underwater images,” IEEE Robotics and Automation letters, vol. 3, no. 1, pp. 387–394, 2017.
  * [19] D. Derksen and D. Izzo, “Shadow neural radiance fields for multi-view satellite photogrammetry,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1152–1161, 2021.
  * [20] Z. Wang, W. Yang, J. Cao, L. Xu, J. ming Yu, and J. Yu, “Neref: Neural refractive field for fluid surface reconstruction and implicit representation,” ArXiv, vol. abs/2203.04130, 2022.
  * [21] A. Fisher, R. Cannizzaro, M. Cochrane, C. Nagahawatte, and J. L. Palmer, “Colmap: A memory-efficient occupancy grid mapping framework,” Robotics and Autonomous Systems, vol. 142, p. 103755, 2021.
  * [22] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan, “Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields,” ICCV, 2021.
  * [23] G. Luise, A. Rudi, M. Pontil, and C. Ciliberto, “Differential properties of sinkhorn approximation for learning with wasserstein distance,” in Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, (Red Hook, NY, USA), p. 5864–5874, Curran Associates Inc., 2018.
  * [24] R. Sinkhorn, “Diagonal equivalence to matrices with prescribed row and column sums,” The American Mathematical Monthly, vol. 74, no. 4, pp. 402–405, 1967.
  * [25] M. Cuturi, “Sinkhorn distances: Lightspeed computation of optimal transport,” Advances in neural information processing systems, vol. 26, 2013.
  * [26] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang, “JAX: composable transformations of Python+NumPy programs,” 2018.
  * [27] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (Y. Bengio and Y. LeCun, eds.), 2015.
  * [28] K. A. Skinner, E. I. Ruland, and M. Johnson-Roberson, “Automatic color correction for 3d reconstruction of underwater scenes,” in IEEE International Conference on Robotics and Automation, 2017.
  * [29] M. J. Islam, Y. Xia, and J. Sattar, “Fast underwater image enhancement for improved visual perception,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 3227–3234, 2020.
  * [30] N. G. Jerlov, Marine optics / by N.G. Jerlov.  Elsevier Scientific Pub. Co Amsterdam ; New York, 1976.
  * [31] K. Panetta, C. Gao, and S. Agaian, “Human-visual-system-inspired underwater image quality measures,” IEEE Journal of Oceanic Engineering, vol. 41, no. 3, pp. 541–551, 2016.
  * [32] H. Bay, T. Tuytelaars, and L. Van Gool, “Surf: Speeded up robust features,” in Computer Vision – ECCV 2006 (A. Leonardis, H. Bischof, and A. Pinz, eds.), (Berlin, Heidelberg), pp. 404–417, Springer Berlin Heidelberg, 2006.
  * [33] D. Girardeau-Montaut, “Cloud compare—3d point cloud and mesh processing software,” Open Source Project, 2015.