#  Toward Edge-Efficient Dense Predictions with Synergistic Multi-Task Neural
Architecture Search

Thanh Vu1 Yanqi Zhou2 Chunfeng Wen3† Yueqi Li3 Jan-Michael Frahm1

1UNC at Chapel Hill 2Google Research 3X, The Moonshot Factory
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Work
done during an internship at X. †Work done while at X.

###### Abstract

In this work, we propose a novel and scalable solution to address the
challenges of developing efficient dense predictions on edge platforms. Our
first key insight is that Multi-Task Learning (MTL) and hardware-aware Neural
Architecture Search (NAS) can work in synergy to greatly benefit on-device
Dense Predictions (DP). Empirical results reveal that the joint learning of
the two paradigms is surprisingly effective at improving DP accuracy,
achieving superior performance over both the transfer learning of single-task
NAS and prior state-of-the-art approaches in MTL, all with just 1/10th of the
computation. To the best of our knowledge, our framework, named EDNAS, is the
first to successfully leverage the synergistic relationship of NAS and MTL for
DP. Our second key insight is that the standard depth training for multi-task
DP can cause significant instability and noise to MTL evaluation. Instead, we
propose JAReD, an improved, easy-to-adopt Joint Absolute-Relative Depth loss,
that reduces up to 88% of the undesired noise while simultaneously boosting
accuracy. We conduct extensive evaluations on standard datasets, benchmark
against strong baselines and state-of-the-art approaches, as well as provide
an analysis of the discovered optimal architectures.

Figure 1:  An overview of our proposed methods. First, EDNAS framework
leverages the synergy and joint learning of multi-task dense prediction (MT-
DP) and hardware-aware NAS to both complement each component and boost on-
device performance. On the left is an illustration of the synergistic
relationship of these components. Second, JAReD loss reduces depth estimation
noise and further improves accuracy. On the right is the performance of our
proposed techniques on CityScapes compared to state-of-the-art MT-DP
approaches.

##  1 Introduction

Recent years have witnessed a strong integration of computer vision in many
downstream edge applications such as autonomous driving [2, 11, 38, 44, 52,
65, 68], mobile vision [16, 24, 25, 60, 61, 63], robotics [27, 35, 42], and
even computational agriculture [12, 28, 37], fueled by rapid innovations of
deep neural networks. In many of these applications, pixel-level dense
prediction tasks such as semantic segmentation or depth estimation can play a
critical role. For example, self-driving agents are using semantic and depth
information to detect lanes, avoid obstacles, and locate their own positions.
In precision agriculture, the output of these tasks can be used for crop
analysis, yield prediction, in-field robot navigation, etc. As more and more
neural models are being deployed into the real world, there has been a
continuously growing interest in developing edge-efficient architectures for
dense predictions over the years.

However, designing fast and efficient dense prediction models for edge devices
is challenging. First of all, pixel-level predictions such as semantic
segmentation and depth estimation are fundamentally slower than some other
popular vision tasks, including image classification or object detection. This
is because after encoding the input images into low-spatial resolution
features, these networks need to upsample them back to produce high-resolution
output masks. In fact, dense estimation can be several times or even an order
of magnitude slower than their counterparts, depending on the specific model,
hardware, and target resolution. Thus, real-time dense prediction models are
not only non-trivial to design, they can easily become a latency bottleneck in
systems that utilize their outputs. Such problems are intensified for edge
applications on platforms like the Coral TPU [13] due to the limited
computational resources, despite the need for low latency, e.g., to inform the
users or process subsequent tasks in real time.

Second, developing models for these edge environments is costly and hard to
scale in practice. On one hand, the architectural design process requires a
significant amount of time, human labor, and expertise, with the development
process ranging from a few months to a couple of years. On the other hand,
edge applications may require deployment on various platforms, including cell
phones, robots, drones, and more. Unfortunately, optimal designs discovered
for one hardware may not generalize to another. All of these together pose
challenges to the development of fast and efficient models for on-edge dense
predictions.

To tackle these problems, our first key insight is that Multi-Task Learning of
Dense Predictions (MTL-DP or MT-DP) and hardware-aware Neural Architecture
Search (h-NAS) can work in synergy to not only mutually benefit but also
significantly improve accuracy and computation. To the best of our knowledge,
our framework, named EDNAS111short for “Edge-Efficient Dense Predictions via
Multi-Task NAS”, is the first to successfully exploit such a synergistic
relationship of NAS and MTL for dense predictions. Indeed, on one hand, state-
of-the-art methods for multi-task dense predictions [4, 22, 36, 40, 53, 58,
66], in which related tasks are learned jointly together, mostly focus on
learning how to share a fixed set of model components effectively among tasks
but do not consider if such a set itself is optimal for MTL to begin with.
Moreover, these works typically study large models targeting powerful graphic
accelerators such as V100 GPU for inference and are not readily suitable for
edge applications. On the other hand, NAS methods aim to automatically learn
an optimal set of neural components and their connections. However, the
current literature often focuses on either simpler tasks such as
classification [7, 33, 62] or single-task training setup [19, 34]. In
contrast, we jointly learn MTL-DP and NAS and leverage their strengths to
tackle the aforementioned issues simultaneously, resulting in a novel and
improved approach to efficient dense predictions for edge.

Our second key insight is that the standard depth estimation training used in
MTL-DP can produce significant fluctuation in the evaluation accuracy. Indeed,
our analysis reveals a potential for undesirably large variance in both
absolute and relative depth. We hypothesize that this is caused by the
standard depth training practice that relies solely on $\mathcal{L}_{1}$ loss
function. This can significantly and negatively affect the accuracy of MT-DP
evaluation as arbitrary “improvement” (or “degradation”) can manifest purely
because of random fluctuation in the relative error. It is important that we
raise awareness of and appropriately address this issue as segmentation and
depth information are arguably two of the most commonly jointly learned and
used tasks in edge applications. To this end, we propose JAReD, an easy-to-
adopt augmented loss that jointly and directly optimizes for both relative and
absolute depth errors. The proposed loss is highly effective at simultaneously
reducing noisy fluctuations and boosting overall prediction accuracy.

We conduct extensive evaluations on CityScapes [14] and NYUv2 [50] to
demonstrate the effectiveness and robustness of EDNAS and JAReD loss.
Experimental results indicate that our methods can yield significant gains, up
to +8.5% and +10.9% DP accuracy respectively, considerably higher than the
previous state of the art, with only 1/10th of the parameter and FLOP counts
(Fig.  Toward Edge-Efficient Dense Predictions with Synergistic Multi-Task
Neural Architecture Search ).

##  2 Background and Related Works

(a) Hard parameter sharing [36, 66]

(b) Learning to branch [22, 4, 58]

(c) Learning to skip layers [53]

(d) Searching for layers (ours)

Figure 2:  Conceptual comparison with existing approaches. While current MT-DP
methods focus on how to better share a fixed set of layers, we instead learn
better sets of layers to share. Components in red are learnable while others
are fixed

In general, dense prediction models are often designed manually, in isolation,
or not necessarily constrained by limited edge computation [10, 27, 34, 35].
Specifically, works on multi-task learning for dense predictions (MTL-DP) [4,
5, 20, 22, 53, 58] often take a fixed base architecture such as DeepLab [9]
and focus on learning to effectively shared components, e.g. by cross-task
communication modules [5, 20], adaptive tree-like branching [4, 22, 58], layer
skipping [53], etc. (Fig. 2). On the other hand, neural architecture search
(NAS) studies up until recently have focused mostly on either image
classification problems[1, 7, 29, 33, 39, 62] or learning tasks in isolation
[19, 34, 54, 67]. Few have explored architecture search for joint training of
dense prediction tasks. However, as mentioned earlier, edge efficiency can
potentially benefit both MTL-DP and NAS. To the best of our knowledge, our
study is the first to report successful joint optimization of these two
learning paradigms for dense predictions. Next, we give an overview of the
most relevant efforts in the two domains of MTL and NAS. For more details,
please refer to these comprehensive surveys: MTL [8, 15], MTL for dense
predictions [59], NAS [46], and hardware-aware NAS [3], .

#### Neural Architecture Search (NAS)

In the past few years, neural architecture search (NAS) has emerged as a
solution to automate parts of the network design process. NAS methods have
shown remarkable progress and outperformed many handcrafted models [34, 54,
55, 56]. In our case, we are interested in hardware-aware NAS [6, 63, 67]
which can discover efficient architectures suitable for one or multiple
targeted edge platforms. This is typically done by casting hardware-aware NAS
as a multi-objective optimization problem [6, 54, 63] and adding hardware
cost, e.g. latency, memory, and energy, alongside prediction accuracy, to
guide the search. However, current studies often focus on image classification
[1, 7, 29, 33, 39, 62] or learning tasks in isolation [54, 67]. However,
performing multiple dense prediction tasks simultaneously can have significant
benefits for both inference speed and accuracy since tasks can leverage each
other’s training signals as inductive biases to improve their own learning and
the model’s generalization [8]. Thus, we are interested in combining hardware-
aware NAS with multi-task learning of dense prediction tasks to achieve both
better accuracy and better inference speed on edge devices. To this end, there
have been only a limited number of studies [4, 22, 53, 58] that started to
explore similar problems, which we will discuss next.

#### MTL for Dense Predictions

The goal of Multi-Task Learning (MTL) [8, 15] is to jointly learn multiple
tasks together to leverage cross-task information to improve per-task
prediction quality. In the context of edge applications, we are also
interested in the property of MTL that lets tasks share computation and output
multiple task predictions in one pass, thereby improving the overall inference
speed. This is particularly useful for dense predictions because they tend to
be more computationally expensive than their counterparts such as
classification [24, 26, 48, 55, 56] or detection [57, 64]. A popular
formulation of MTL that accomplishes this goal is called hard parameter
sharing (HPS) [36, 66]. Compared to soft parameter sharing (SPS) [20], whose
multi-task model size scales linearly with the number of tasks due to separate
per-task sub-networks, HPS models are more edge-friendly due to their compact
architectural structure. Specifically, HPS architectures are typically
composed of a shared trunk that extracts joint features for all tasks and
multiple per-task heads or branches that take the extracted features as input
and produce specific task prediction. The most standard setup is to have all
task heads branch off at the same point [36]. This is also our setup of choice
for the scope of this work. In addition, recent studies have begun to explore
strategies to learn adaptive sharing architectures from data [4, 22, 40, 53,
58]. Attention [40] and Layer-skipping[53] have been used to efficiently learn
a single shared model while modifying their behaviors to output the desired
task-specific prediction, given a task. Other studies [4, 22, 58] opt to
augment the HPS architectures by learning the branching of tasks. In other
words, the learned models may have multiple splitting points, where some tasks
can branch off earlier while some others share more layers. A common theme of
these approaches is that given a fixed starting architecture, the focus is on
learning which components of such network should be shared. Our work shifts
the focus to the base network and instead asks what components should be
included in such architecture to best benefit multi-task dense predictions.

##  3 Methodology

###  3.1 EDNAS: Joint MTL-DP and h-NAS

#### Synergistic Joint Learning

Our key idea is that we can leverage multi-task inference to significantly
reduce computation across several dense prediction tasks, while utilizing
hardware-aware NAS to simultaneously improve edge latency, design scalability,
and multi-task learning. Combining these two paradigms, MT-DP and NAS, is
beneficial not only to edge inference but also to each other. Fig.  Toward
Edge-Efficient Dense Predictions with Synergistic Multi-Task Neural
Architecture Search  illustrates these relationships. First, regarding edge
applications, multi-task models [59] that output several predictions at once
are attractive since they share computation across tasks to avoid multiple
inference runs and improve the overall latency linearly by design. However,
this multi-task setup also leads to performance degradation, known as negative
transfer. While most current works attribute this problem to improper sharing
of neural components, we hypothesize that components of popular base networks
such as DeepLab [9] -ResNet [23] may be well-tuned for their original
individual task, but not necessarily optimal for multi-task setting. It is
possible that certain layers, for example, may need more channels to capture
nuanced features required when the number of tasks increases. Moreover, these
models may need to be deployed on different edge platforms and thus, their
components need to be optimized accordingly. This motivates us to explore NAS
as a systematic and scalable method to discover components that could be more
suitable for multi-task learning and edge inference. Second, from the
perspective of NAS, directly searching for multi-task architectures can
potentially yield better results than transferring single-task searched
architectures to multi-task settings post NAS. In a way, we are removing a
proxy target and its assumption that architectures, which are good for an
individual task such as segmentation, are also optimal for multi-task
learning.

#### Hardware-Aware Multi-Task Objective

Given a fixed set of $N$ tasks $T=\\{T_{1},T_{2},...T_{N}\\}$, we formulate
the problem of multi-task NAS as a multi-objective search. Our goal is to
discover optimal models with both high accuracy for all tasks in $T$ and low
inference latency on specific edge devices. Let $a$ be an architecture with
weights $w_{a}$ sampled from the search space $A$ and $h$ be a target edge
hardware. Our optimization can then be expressed as follows:

$\max_{a\subset A}Rwd(a,T,h,w_{a}^{*})$ (1) $\text{s.t.}\quad
w_{a}^{*}=arg\min_{w_{a}}Loss(a,T,w_{a})$ (2) $\text{and}\quad Lat(a,h)\leq
l_{h}$ (3)

with $Rwd()$ being the objective or reward function and $l_{h}$ being the
target edge latency dependent on the hardware and application domain. Inspired
by [54], we use a weighted product for the reward function $Rwd()$ to jointly
optimize for models’ accuracy and latency constrained by hardware-dependent
requirements such as inference latency, chip area, energy usage, etc. This
allows for flexible customization and encourages Pareto optimal solutions of
multi-objective learning [17]. In this work, we focus on inference latency
$Lat(a,h)$ as the main hardware constraint.

$Rwd(a,T,h,w_{a})=Acc(a,T,w_{a})\left[\frac{Lat(a,h)}{l_{h}}\right]^{\beta}$
(4) $\text{s.t.}\quad\beta=\begin{cases}p\text{ if }Lat(a,h)\leq l_{h}\\\
q\text{ otherwise}\end{cases}$ (5)

We use an in-house cycle-accurate performance simulator to estimate the on-
device latency of sampled architectures during NAS. This offers a middle
ground between the accurate-but-expensive benchmarking methods that use real,
physical devices and the cheap-but-inaccurate one that use proxy metrics like
FLOPs, MACs, or number of parameters. Moreover, by configuring such a
simulator differently, we can inject hardware-specific information and bias
the search to adapt to different targeted edge platforms.

Unlike prior works [54, 67], we extend the notion of $Acc()$ to multi-task
setting using a simple-yet-effective nested weighted product of metrics and
tasks. Let $M_{i}=\\{m_{i,1},m_{i,2},...,m_{i,K}\\}$ be the set of metrics of
interest for tasks $T_{i}$, e.g. {mIoU, PixelAcc} for semantic segmentation.
Our multi-task $Acc()$ can be expressed as:

$Acc(a,T,w_{a})=\left[\prod_{i}m_{i}\right]^{1/N}\;$ (6)
$\text{s.t.}\;\;m_{i}=\left[\prod_{j}m_{i,j}^{w_{i,j}}\right]^{1/\sum_{j}w_{i,j}}$
(7)

This extended formulation is straightforward and scalable even when the number
of tasks or metrics increases. Since our goal is to discover multi-task
networks that can perform well across all tasks without bias to individual
tasks, we treat all task rewards equally in our formulation.

#### Edge-Friendly Base Architecture

Previously works [4, 22, 36, 53, 58] typically use bigger networks such as
ResNet [23] or VGG [51] backbone with ASPP [9] decoder. Such models, however,
are not suitable for edge platforms like the Coral TPU [13] due to their
limited computational resources. To this end, we propose the use of
EfficientNet [55, 56] backbone and BiFPN fusion modules [57], which have been
shown to have significantly better FLOPs and parameter efficiency (e.g. an
order of magnitude lower) compared to their counterparts [32, 55, 57, 67].
These advantages make them promising candidate modules to build edge-friendly
models. To generate multi-task outputs while saving computation, we share the
majority of the network, including both the EfficientNet backbone and BiFPN
modules, across all tasks and use only small per-task heads. This keeps our
model compact and avoids a significant increase in size as the number of tasks
goes up . We also replace Swish activation and attention-based fusion with
ReLU6 and Sum operations in [55] to further improve efficiency on edge. We
balance the compact EfficientNet backbone with 4 BiFPN fusion modules instead
of 3 like [57] to boost accuracy. The multi-scale fusion modules take features
$\\{P_{3},P_{4},P_{5},P_{6},P_{7}\\}$ from levels 3-7 of the backbone. These
components together make up our edge-friendly base architecture, which we will
use as both the seed for our NAS and the baseline model for evaluating MTL
performance.

#### Edge-Friendly Search Space

Modern NAS usually retains some aspect of the base architecture in order to
keep the search space tractable and to reduce the computational cost. Thus, it
is important to have a good initialization architecture to seed the search.
For this, we leverage the base architecture designed above and Pyglove [43], a
Python AutoML library that supports flexible layer-level mutation for NAS
components via symbolic programming. This allows us to transform the static
EfficientNet backbone into a tunable search space by replacing any standard
computational node with a PyGlove’s mutable object, e.g. converting
Conv2d(kernel=3) into Conv2d(kernel=oneof([3,5,7])). Furthermore, we expand
the search space to include Fused-IBN [56, 64, 67] modules alongside the
standard Inverted Bottleneck (IBN) [48]. Despite inciting more trainable
parameters, Fused-IBN can potentially offer better efficiency on edge devices
if strategically placed, e.g. via NAS. This is because industry accelerators
are better tuned for regular convolution than their depthwise counterparts,
e.g. resulting in 3$\times$ speedup for certain tensor shapes and kernel
dimensions [64]. Our final search space is defined by the following per-layer
decisions:

  * •

Layer type: {IBN, Fused-IBN}

  * •

Kernel size: {3, 5}

  * •

Output channel multiplier: {0.5, 0.75, 1.0, 1.5}

  * •

Expansion ratio: {3, 6}

The search is performed for all 16 IBN blocks of our base EfficientNet
backbone, together with the other search parameters, producing an expressive
search space of size $(2*2*4*2)^{16}=2^{80}\approx 1.2e24$.

###  3.2 Depth Estimation Noise and JAReD Loss

Depth Loss | mIoU | $\sigma\%$ | AbsE | $\sigma\%$ | RelE | $\sigma\%$
---|---|---|---|---|---|---
$\mathcal{L}_{1}$ | 38.6 | 2.5 | 0.01763 | 4.4 | 0.3541 | 4.1
JAReD | 38.9 | 1.6 | 0.01680 | 1.9 | 0.3237 | 0.5
% improved |  $\uparrow$0.8 |  $\downarrow$36.0 |  $\downarrow$4.7 |  $\downarrow$56.8 |  $\downarrow$8.6 |  $\downarrow$87.8

Table 1:  Depth estimation noise and JAReD loss. We train a model for
segmentation and depth prediction on CityScapes with the standard
$\mathcal{L}_{1}$ and proposed JAReD loss. The mean and standard deviation are
taken across 3 runs. Except for mIoU, lower is better.

#### Instability in Depth Estimation

During our study, we discover that depth prediction accuracy can vary greatly
across different training runs of the same setting. This is illustrated in
Tab. 1 by the results of standard depth training with $\mathcal{L}_{1}$ loss.
Note that the standard deviation of depth errors across identical runs are
fairly large at 4.4% and 4.1%, $\times 2$ higher than that of segmentation
mIoU. Such large variation is problematic for the multi-task evaluation as one
model could potentially arbitrarily and falsely “improve” or “degrade” purely
by chance. Moreover, this may even interfere with the joint learning MT-DP and
NAS through noisy task accuracy in the objective function in Eq 4. In other
words, it would be challenging for NAS to identify good architectures if
training accuracy itself is unstable and unreliable.

#### Joint Absolute-Relative Depth

We hypothesize that the noisy depth result is due to the fact that popular MT-
DP training [36, 53, 59] relies only $\mathcal{L}_{1}$ loss, which focuses on
optimizing for absolute depth and only implicitly learn relative depth. For
monocular setting, learning absolute depth directly is ill-posed and
challenging due to the scale ambiguity [18, 31]. Instead, we propose to
augment the standard loss using a weighted relative-error component, resulting
in a Joint Absolute-Relative Depth loss, or JAReD:

$\mathcal{L}_{JAReD}=\frac{1}{N}\Sigma|y-\hat{y}|+\lambda\frac{1}{N}\Sigma\left|\frac{y-\hat{y}}{y}\right|$
(8)

Tab. 1 shows that JAReD can help significantly reduce depth estimation
noise—the STDs of all tasks decrease, especially for relative error with 87.8%
lower fluctuation. Moreover, JAReD can simultaneously improve accuracy, with
both absolute and relative errors dropping by 4.7% and 8.6%.

##  4 Experiments

| Model | Seg | Depth |  $\Delta$Seg |  $\Delta$Depth | Avg
---|---|---|---|---|---|---
Method | #P | GFLOP | Speed | mIoU | PAcc | AbsE | RelE |  $\Delta$mIoU |  $\Delta$PAcc |  $\Delta$AbsE |  $\Delta$RelE | $\Delta T_{S}$ | $\Delta T_{D}$ | $\Delta T$
ST baseline [53] | 42.6 | 87.1 | — | 40.20 | 74.70 | .0170 | .330 | — | — | — | — | — | — | —
MT baseline [53] | 21.3 | 43.6 | — | 37.70 | 73.80 | .0180 | .340 | -6.2 | -1.2 | -5.9 | -3.0 | -3.7 | -4.5 | -4.1
Cross-Stitch [41] | 42.6 | 48.4 | — | 40.30 | 74.30 | .0150 | .300 | +0.2 | -0.5 | +11.8 | +9.1 | -0.1 | +10.4 | +5.1
Sluice [47] | 42.6 | 48.4 | — | 39.80 | 74.20 | .0160 | .310 | -1.0 | -0.7 | 5.9 | 6.1 | -0.8 | +6.0 | +2.6
NDDR-CNN [21] | 44.1 | 50.1 | — | 41.50 | 74.20 | .0170 | .310 | 3.2 | -0.7 | 0.0 | 6.1 | +1.3 | +3.0 | +2.2
MTAN [36] | 51.3 | 57.9 | — | 40.80 | 74.30 | .0150 | .320 | +1.5 | -0.5 | +11.8 | +3.0 | +0.5 | +7.4 | +3.9
DEN [1] | 23.9 | 51.2 | — | 38.00 | 74.20 | .0170 | .370 | -5.5 | -0.7 | 0.0 | -12.1 | -3.1 | -6.1 | -4.6
AdaShare [53] | 21.3 | 87.1 | — | 41.50 | 74.90 | .0160 | .330 | 3.2 | 0.3 | 5.9 | 0.0 | +1.8 | +2.9 | +2.3
ST edge baseline | 3.4 | 2.3 |  $\times$1.0 | 40.04 | 88.68 | .0157 | .340 | — | — | — | — | — | — | —
MT edge baseline | 3.4 | 1.2 |  $\times$1.2 | 38.64 | 88.49 | .0171 | .354 | -3.5 | -0.2 | -8.5 | -4.1 | -1.9 | -6.3 | -4.1
EDNAS | 4.3 | 4.1 |  $\times$1.3 | 46.52 | 90.61 | .0143 | .316 | +16.2 | +2.2 | +8.9 | +6.9 | +9.2 | +7.9 | +8.5
EDNAS+JAReD | 4.3 | 4.1 |  $\times$1.3 | 46.11 | 90.47 | .0143 | .281 | +15.1 | +2.0 | +9.1 | +17.4 | +8.6 | +13.3 | +10.9

Table 2:  Two-task CityScapes results. Best numbers are in bold, the second
best are underlined. ST stands for single-tasks. MT stands for multi-task. We
multiply the FLOPs by the number of tasks for methods that need multiple runs
to get different per-task predictions. FLOP counts are in gigas(G) and
parameter counts are in millions(M). Both of these, along with our model’s
edge latency, are measured for 256x256 resolution. We consult Table 8 and
Table 11 in [53] as well as its first author to acquire the full measurements
of prior works

###  4.1 Setup

#### Datasets and Tasks

We evaluate our proposed method using two popular datasets for multi-task
dense predictions: CityScapes [14] and NYU-v2 [50]. CityScapes contains 2975
training images and 500 validation images of driving scenes while NYU-v2 is
composed of 1449 densely labeled RGBD indoor images, with a stand training-to-
validation split of 795 to 654. We use the preprocessed versions provided by
AdaShare [53]. We jointly learn semantic segmentation (19 classes) and depth
prediction for CityScapes. For NYU-v2, we study 3-task learning of
segmentation, depth prediction, and surface normal estimation.

#### Baselines

We adopt the standard practice of evaluating our proposed techniques against
the Single-Task (ST) and vanilla Multi-Task (MT) versions, which are
EfficientNet-based in our case. We refer to these as edge baselines. For fair
comparisons, we consult the training hyperparameters used by AdaShare [53] to
match their baseline performance and only compare the relative improvements.

#### Implementation Details

For all experiments, we use EfficientNet-B0 [55] as our backbone. We use
Regularized Evolution [45] as our search controller as it can produce compact
and accurate models with less search time, thus shortening the experimentation
cycle. Nonetheless, we expect other controllers, e.g. PPO [49] as used by
prior works [54, 67], to also work. We use Adam [30] optimizer and cosine
learning rate scheduler for all our training, including both the proxy task
during NAS and the final training of the best candidates, to reduce
hyperparameter tuning effort. For full training, we train each model 3 times
and take the average results similar to Table 1 to reduce noise. All models
are trained from scratch without any pretrained weights. We acquire wall-clock
latency measurements by benchmarking models on a Coral EdgeTPU [13]. Further
details are included in the supplementary.

#### Evaluation Metrics

We use mean Intersection over Union (mIoU) and pixel accuracy (PAcc) for
semantic segmentation, and mean absolute error (AbsE) and mean relative error
(RelE) for depth prediction. For surface normal estimation on NYU-v2, we use
mean angle distance error (MeanE) across all pixels, as well as the percentage
of pixels with angle distances less a threshold
$\theta\in\\{11.25\degree,22.5\degree,30\degree\\}$, denoted as {$\theta$11,
$\theta$22, $\theta$30} respectively. Following other works [40, 53, 59], we
calculate a single evaluation score $\Delta T$ averaging over all relative
gains $\Delta T_{i}$ of all tasks $T_{i}$ relative to the Single-Task
baseline. A formal definition of these metrics are provided in our
supplementary materials.

| Seg | Depth | Surface Normal | Avg
---|---|---|---|---
Method | mIoU | PAcc | AbsE | RelE | MeanE |  $\theta$11 |  $\theta$22 |  $\theta$30 | $\Delta T$
ST edge | 23.1 | 58.3 | 0.50 | 0.20 | 13.8 | 50.8 | 81.2 | 90.8 | —
MT edge | 19.5 | 54.8 | 0.55 | 0.22 | 16.5 | 41.9 | 73.0 | 85.1 | -11.3
ST [53] | 27.5 | 58.9 | 0.62 | 0.25 | 17.5 | 34.9 | 73.3 | 85.7 | —
MT [53] | 24.1 | 57.2 | 0.58 | 0.23 | 16.6 | 42.5 | 73.2 | 84.6 | +2.0
MT edge | 19.5 | 54.8 | 0.55 | 0.22 | 16.5 | 41.9 | 73.0 | 85.1 | -0.1

Table 3:  NYUv2 baselines. ST and MT are prior large-scale models from [53]
while edge denotes our edge-friendly baselines

| Seg | Depth | Surface Normal | Avg
---|---|---|---|---
Method | mIoU | PAcc | AbsE | RelE | MeanE |  $\theta$11 |  $\theta$22 |  $\theta$30 | $\Delta T_{S}$ | $\Delta T_{D}$ | $\Delta T_{SN}$ | $\Delta T$
MT baseline [53] | 24.1 | 57.2 | 0.58 | 0.23 | 16.6 | 42.5 | 73.2 | 84.6 | — | — | — | —
Cross-Stitch [41] | 25.4 | 57.6 | 0.58 | 0.23 | 17.2 | 41.4 | 70.5 | 82.9 | +3.0 | +0.0 | -3.0 | +0.0
Sluice [47] | 23.8 | 56.9 | 0.58 | 0.24 | 17.2 | 38.9 | 71.8 | 83.9 | -0.9 | -2.2 | -3.7 | -2.3
NDDR-CNN [21] | 21.6 | 53.9 | 0.66 | 0.26 | 17.1 | 37.4 | 73.7 | 85.6 | -8.1 | -13.4 | -3.3 | -8.3
MTAN [36] | 26.0 | 57.2 | 0.57 | 0.25 | 16.6 | 43.7 | 73.3 | 84.4 | +3.9 | -3.5 | +0.7 | +0.4
DEN [1] | 23.9 | 54.9 | 0.97 | 0.31 | 17.1 | 36.0 | 73.4 | 85.9 | -2.4 | -51.0 | -4.1 | -19.2
AdaShare [53] | 30.2 | 62.4 | 0.55 | 0.20 | 16.6 | 45.0 | 71.7 | 83.0 | +17.2 | +9.1 | +0.5 | +8.9
MT edge baseline | 19.5 | 54.8 | 0.55 | 0.22 | 16.5 | 41.9 | 73.0 | 85.1 | — | — | — | —
EDNAS | 22.1 | 57.7 | 0.51 | 0.20 | 14.3 | 49.5 | 79.2 | 89.4 | +9.3 | +8.2 | +11.3 | +9.6
EDNAS+JAReD | 22.1 | 58.1 | 0.51 | 0.20 | 12.6 | 56.1 | 83.9 | 92.4 | +9.7 | +8.2 | +20.3 | +12.7

Table 4:  Three-task NYUv2 results . Our tasks of interest include 40-class
semantic segmentation, depth estimation, and surface normal estimation. Best
numbers are in bold, the second best are underlined. ST stands for single-task
and MT stands for multi-task. We multiply the FLOPs by the number of tasks for
methods that need multiple runs to get different per-task predictions. The
measurements of prior works are from Table 9 and Table 11 in [53]

| Model | Seg | Depth | Seg$\Delta$ | Depth$\Delta$ | Avg$\Delta$
---|---|---|---|---|---|---
Method | #P | GFLOP | Speed | mIoU | PAcc | AbsE | RelE |  $\Delta$mIoU |  $\Delta$PAcc |  $\Delta$AbsE |  $\Delta$RelE | $\Delta T_{S}$ | $\Delta T_{D}$ | $\Delta T$
ST edge baseline | 3.4 | 2.3 |  $\times$1.0 | 40.04 | 88.68 | .0157 | .340 | — | — | — | — | — | — | —
ST edge+maxLR | 3.4 | 2.3 |  $\times$1.0 | 55.02 | 92.29 | .0121 | .288 | +37.4 | +4.1 | +23.2 | +15.3 | +20.7 | +19.3 | +20.0
ST edge+maxLR+JAReD | 3.4 | 2.3 |  $\times$1.0 | 55.02 | 92.29 | .0116 | .168 | +37.4 | +4.1 | +26.7 | +50.5 | +20.7 | +38.6 | +29.7
ST edge+maxLR+JAReD | 3.4 | 2.3 |  $\times$1.0 | 55.02 | 92.29 | .0116 | .168 | — | — | — | — | — | — | —
MT edge+maxLR+JAReD | 3.4 | 1.2 |  $\times$1.2 | 53.80 | 91.94 | .0124 | .159 | -2.2 | -0.4 | -7.4 | +5.7 | -1.3 | -0.9 | -1.1
Transfer: NAS-Seg$\xrightarrow{}$MT | 4.1 | 2.5 |  $\times$1.3 | 58.17 | 92.78 | .0118 | .156 | +5.4 | +0.5 | -2.9 | +6.2 | +3.1 | +2.8 | +3.0
Transfer: NAS-Dep$\xrightarrow{}$MT | 3.6 | 2.5 |  $\times$1.3 | 57.97 | 92.73 | .0119 | .158 | +5.7 | +0.5 | -1.8 | +7.4 | +2.9 | +1.6 | +2.3
EDNAS+maxLR+JAReD | 4.3 | 4.1 |  $\times$1.3 | 58.54 | 92.78 | .0117 | .156 | +6.4 | +0.5 | -1.3 | +7.4 | +3.5 | +3.1 | +3.3

Table 5:  Stronger baselines on CityScapes. ST edge baseline and ST edge+maxLR
have identical training setting with the only exception of their learning rate
being 3e-4 and 1e-2 respectively

###  4.2 Results

#### EDNAS for 2-task CityScapes

Tab. 2 shows our experiments for the 2-task learning of 19-class semantic
segmentation and depth estimation on CityScapes dataset. In this experiment,
the same $\Delta T$ of -4.1 is shared by the MT edge baseline and its large-
scale counterpart, indicating that they both experience a similar level of
negative transfer and MTL difficulty. Following [53], we present MTL gains
relative to the ST baseline model. The proposed EDNAS exhibits a strong multi-
task performance with $\Delta T$=+8.5, outperforming all prior methods. Since
the full training of MT edge baseline and EDNAS-found architecture are
identical, it shows that joint MTL-DP and NAS can produce a superior relative
improvement of +8.5 - (-4.1) = +12.6 compared to the vanilla multi-task model.

#### JAReD Loss

From Tab. 2, we see that the proposed JAReD loss is able to greatly improve
depth estimation with a relative gain of $\Delta T_{D}$=13.3%. This in turn
further strengthens the overall multi-task performance by a significant margin
of +2.4 on top of the already-strong result ($\Delta T$=+8.5) of EDNAS.
Together, our two proposed techniques outperform all previous approaches on 3
out of 4 individual metrics, namely $\Delta$mIoU, $\Delta$PAcc, and
$\Delta$RelE, as well as on all the average metrics, which are $\Delta T_{S}$,
$\Delta T_{D}$, and $\Delta T$.

#### Edge-Efficient Inference

Regarding edge efficiency, EDNAS and EDNAS+JAReD use only 1/5th of the
parameters and 1/10th of the FLOPs compared to prior ResNet-based methods.
More importantly, the EDNAS-found model is able to practically maintain the
same on-device speed as the vanilla MT baseline, if not slightly faster,
despite the +12.6 improvement. This equates to a 30% improvement in latency
compared to separate single-task inferences, and further demonstrates the
benefits of our proposed joint learning for discovering and training better
multi-task architectures for dense predictions on edge platforms.

#### Generalization to 3-Task NYUv2

Unlike with Cityscapes where the MT baselines have similar accuracy drop, for
NYUv2, we notice a large difference between the amount of negative transfer in
MT edge baseline ($\Delta T$=-11.3) and in the large-scale MT model ($\Delta
T$=+2.0), as shown in Tab. 3. This indicates that multi-task training on NYUv2
data may be more challenging for edge models with limited computation. Because
of such discrepancy in the level of MTL difficulty, we directly use the MT
models (instead of ST models) as the baselines to benchmark the improvement
gained. Note that despite such a large gap compared to the ST edge setting,
our MT edge model is still comparable to the computationally heavy ST
baselines of prior studies, with a negligible $\Delta T$=-0.1. The NYUv2
results from Tab. 4 show that EDNAS and JAReD continue to achieve consistent
and significant improvements ($\Delta T$ of +9.6 and +12.7) over the baseline,
similar to what we observed for Cityscapes.

#### Robustness to Stronger Baselines

To further demonstrate the robustness of EDNAS as a solution for discovering
better multi-task architectures for dense predictions, we are interested in
examining its performance with stronger baselines (Tab. 5). Although prior
work [53] only uses learning rates in the order of 1e-4 to 1e-3, we also
experiment with other rates and observe a huge jump of $\Delta T$=+20.3 in
performance when simply increasing the learning rate while holding other
settings the same. We utilize this simple adjustment to obtain our stronger
edge baseline with the largest learning rate of maxLR=1e-2. Taking a step
further, we add JAReD loss to our ST edge baseline both to demonstrate the
effectiveness of JAReD loss even for single-task depth estimation and to
acquire our strongest baseline for evaluation. Our result of training the
EDNAS-found architecture with similar setup (+maxLR and +JAReD) illustrates
the strength of our proposed method with a relative multi-task gain of $\Delta
T$=+3.3. We emphasize that +3.3, despite being smaller than the improvements
we have seen so far, is still comparable to the majority of state-of-the-art
methods shown in Tab. 2, and that is on top of a +30% stronger ST baseline!

#### Joint Learning vs Transfer Learning

Tab. 5 also shows the performance of EDNAS when compared to the transferring
of NAS-found single-task models to the multi-task setting. Although
transferred architectures can bring a considerable amount of improvement
compared to our baseline ST and MT models, EDNAS’ joint learning of multi-task
dense predictions and hardware-aware NAS evidently offers the optimal
performance among these models, achieving either the best or second best
scores in all categories. Moreover, it is also important to note that there is
a significant difference in the performance gains of the transferred depth
estimation network compared to that of the transferred segmentation model.
Therefore, we may not know in advance which specific tasks transfer better
than the other, further illustrating the power and benefits of our EDNAS.

#### Analysis of EDNAS-Found Architectures

Index | Layer | Stride | Kernel | Filters | Expansion
---|---|---|---|---|---
0 | Conv2D | 2 | 3 | 32 | –
1 | FusedIBN | 1 | 3 | 16 | 1
2 | IBN | 2 | 5 | 36 | 6
3 | FusedIBN | 1 | 5 | 24 | 6
4 | FusedIBN | 2 | 3 | 60 | 6
5 | FusedIBN | 1 | 3 | 40 | 3
6 | FusedIBN | 2 | 5 | 120 | 3
7 | IBN | 1 | 3 | 120 | 3
8 | FusedIBN | 1 | 5 | 80 | 6
9 | FusedIBN | 1 | 5 | 168 | 6
10 | FusedIBN | 1 | 5 | 84 | 3
11 | FusedIBN | 1 | 5 | 84 | 6
12 | FusedIBN | 2 | 5 | 288 | 3
13 | FusedIBN | 1 | 3 | 96 | 3
14 | FusedIBN | 1 | 3 | 96 | 6
15 | FusedIBN | 1 | 3 | 96 | 3
16 | FusedIBN | 1 | 5 | 160 | 6

Table 6:  Backbone Architecture found by EDNAS – Backbone architecture found
EDNAS for multi-task segmentation and depth estimation on CityScapes, same
model as presented in Tab. 2.

Index | Layer | Stride | Kernel | Filters | Expansion
---|---|---|---|---|---
0 | Conv2D | 2 | 3 | 32 | –
1 | FusedIBN | 1 | 3 | 24 | 1
2 | IBN | 2 | 3 | 36 | 6
3 | IBN | 1 | 3 | 36 | 6
4 | FusedIBN | 2 | 5 | 40 | 6
5 | FusedIBN | 1 | 5 | 40 | 3
6 | IBN | 2 | 3 | 80 | 6
7 | FusedIBN | 1 | 3 | 120 | 3
8 | FusedIBN | 1 | 3 | 80 | 6
9 | FusedIBN | 1 | 3 | 168 | 3
10 | FusedIBN | 1 | 3 | 56 | 6
11 | FusedIBN | 1 | 3 | 112 | 3
12 | FusedIBN | 2 | 5 | 192 | 6
13 | FusedIBN | 1 | 3 | 192 | 6
14 | IBN | 1 | 5 | 192 | 3
15 | IBN | 1 | 5 | 192 | 3
16 | FusedIBN | 1 | 5 | 240 | 3

Table 7:  Backbone Architecture found by Single-task NAS – An example of the
backbone architecture found the single-task NAS targeting depth estimation on
CityScapes . We suspect that multi-task learning can benefit from more
expressive layers such as FusedIBN; thus, fewer of such layers compared Tab. 6
may correlate to the lower accuracy as seen in the previous experiments.

Tab. 6 gives a summary of the backbone architecture found by EDNAS for multi-
task segmentation and depth estimation on CityScapes. This is the same model
as presented in CityScapes experiment section. Except for the first Conv2D
layer, which is a fixed stem, the following 16 layers (1-16) are all tunable.
Our first observation is that FusedIBN is heavily favored by the search
algorithm over regular IBN, occupying 14 out of 16 tunable layers. This is
likely due to the fact that modern edge accelerators such as the Coral Edge
TPU [13] are more optimized for normal convolution than for depthwise
separable convolution. Therefore, they can leverage the dense computations to
improve both accuracy and inference latency. Second, we notice that 4 out of
our top 5 searched models have an IBN module at layer 2 and 7, including the
one in Tab. 6. The remaining architecture also has IBN for layer 7 but not for
layer 2. Hence, we believe that even though sparsely used, IBN layers can
still be beneficial if placed strategically, e.g. via EDNAS.

Tab. 7 provides an example of architectures found by our single-task NAS for
depth estimation. We observe that there are consistently and considerably
lower numbers of FusedIBN modules, namely 11 compared to 14 in Table Tab. 6,
which is produced by EDNAS, a multi-task NAS algorithm. Similar observation
also applies to the single-task NAS for segmentation, which has 12 FusedIBN
layers. We conjecture that multi-task learning might require more powerful and
expressive layers to capture cross-task nuances. As a result, single-task NAS,
which performs an indirect search using individual tasks, may fail to
recognize and meet these needs, leading to fewer FusedIBN blocks and poorer
accuracy as seen in the transferring experiments.

##  5 Conclusion

In this work, our two main contributions include EDNAS and JAReD loss. The
former is a novel and scalable solution that exploits the synergy of MTL and
h-NAS to improve both accuracy and speed for dense prediction task on edge
platforms. The latter is an easy-to-adopt augmented depth loss that
simultaneously mitigates noise and further boosts accuracy. Through extensive
experimentation, we show that the proposed techniques can outperform state-of-
the-art methods, minimize on-device computational cost, generalize to
different data and training settings, as well as discover meaningful and
effective architectures.

## References

  * [1] Chanho Ahn, Eunwoo Kim, and Songhwai Oh.  Deep Elastic Networks with Model Selection for Multi-Task Learning.  In ICCV, 2019.
  * [2] Tanya Amert, Ming Yang, Saujas Nandi, Thanh Vu, James H. Anderson, and F. Donelson Smith.  The price of schedulability in multi‑object tracking: The history‑ vs.‑accuracy trade‑off.  In Proceedings of the IEEE International Symposium on Real‑Time Distributed Computing (ISORC), 2020.
  * [3] Hadjer Benmeziane, Kaoutar El Maghraoui, Hamza Ouarnoughi, Smail Niar, Martin Wistuba, and Naigang Wang.  Hardware-Aware Neural Architecture Search: Survey and Taxonomy.  In IJCAI, 2021.
  * [4] David Bruggemann, Menelaos Kanakis, Stamatios Georgoulis, and Luc Van Gool.  Automated Search for Resource-Efficient Branched Multi-Task Networks.  In BMVC, 2020.
  * [5] David Bruggemann, Menelaos Kanakis, Anton Obukhov, Stamatios Georgoulis, and Luc Van Gool.  Exploring Relational Context for Multi-Task Dense Prediction.  In ICCV, 2021.
  * [6] Han Cai, Ligeng Zhu, and Song Han.  Proxylessnas: Direct neural architecture search on target task and hardware.  In ICLR, 2019.
  * [7] Ronghong Cai and Jianping Luo.  Multi-Task Learning for Multi-Objective Evolutionary Neural Architecture Search.  In CEC, 2021.
  * [8] Rich Caruana.  Multitask Learning.  In Machine learning, 1997.
  * [9] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, , and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets and atrous convolution and and fully connected CRFs.  In TPAMI, 2017.
  * [10] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin P. Murphy, and Alan Loddon Yuille.  Semantic image segmentation with deep convolutional nets and fully connected crfs.  CoRR, abs/1412.7062, 2015.
  * [11] Yuntao Chen, Chenxia Han, Yanghao Li, Zehao Huang, Yi Jiang, Naiyan Wang, and Zhaoxiang Zhang.  Simpledet: A simple and versatile distributed framework for object detection and instance recognition, 2019.
  * [12] Mang Tik Chiu, Xingqian Xu, Yunchao Wei, Zilong Huang, Alexander G. Schwing, Robert Brunner, Hrant Khachatrian, Hovnatan Karapetyan, Ivan Dozier, Greg Rose, David Wilson, Adrian Tudor, Naira Hovakimyan, Thomas S. Huang, and Honghui Shi.  Agriculture-vision: A large aerial image database for agricultural pattern analysis.  In CVPR, 2020.
  * [13] Coral Edge TPU.  https://coral.ai/.
  * [14] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, , and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In CVPR, 2016.  License: Cityscapes is freely available to academic and non-academic entities for non-commercial purposes.
  * [15] Michael Crawshaw.  Multi-Task Learning with Deep Neural Networks: A Survey.  arXiv preprint arXiv:2009.09796, 2020.
  * [16] Xiaoliang Dai, Peizhao Zhang, Bichen Wu, Hongxu Yin, Fei Sun, Yanghan Wang, Marat Dukhan, Yunqing Hu, Yiming Wu, Yangqing Jia, Peter Vajda, Matt Uyttendaele, and Niraj K. Jha.  Chamnet: Towards efficient network design through platform-aware model adaptation, 2018.
  * [17] K. Deb.  Multi-objective optimization.  In Search methodologies, pages 403–449, 2014.
  * [18] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  In NeurIPS, 2014.
  * [19] Jiemin Fang, Yuzhu Sun, Qian Zhang, Yuan Li, Wenyu Liu, and Xinggang Wang.  Densely connected search space for more flexible neural architecture search.  In CVPR, 2020.
  * [20] Yuan Gao, Haoping Bai, Zequn Jie, Jiayi Ma, Kui Jia, and Wei Liu.  MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning.  In CVPR, 2020.
  * [21] Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, , and Alan L Yuille.  Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction.  In CVPR, 2019.
  * [22] Pengsheng Guo, Chen-Yu Lee, and Daniel Ulbricht.  Learning to Branch for Multi-Task Learning.  In ICML, 2020.
  * [23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  arXiv preprint arXiv:1512.03385, 2015.
  * [24] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam.  Searching for mobilenetv3, 2019.
  * [25] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications, 2017.
  * [26] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [27] Ping Hu, Federico Perazzi, Fabian Heilbron, Oliver Wang, Zhe Lin, Kate Saenko, and Stan Sclaroff.  Real-time semantic segmentation with fast attention.  In IEEE Robotics and Automation Letters, volume 6, pages 1–1, 01 2020.
  * [28] Andreas Kamilaris and Francesc Xavier Prenafeta-Boldú.  Deep learning in agriculture: A survey.  Comput. Electron. Agric., 147:70–90, 2018.
  * [29] Eunwoo Kim, Chanho Ahn, and Songhwai Oh.  Auto-VirtualNet: Cost-adaptive dynamic architecture search for multi-task learnin.  In Neurocomputing, 2021.
  * [30] Diederik P. Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  In ICLR, 2015.
  * [31] Jae-Han Lee and Chang-Su Kim.  Monocular depth estimation using relative depth maps.  In CVPR, 2019.
  * [32] Yanwei Li, Lin Song, Yukang Chen, Zeming Li, X. Zhang, Xingang Wang, and Jian Sun.  Learning dynamic routing for semantic segmentation.  In CVPR, 2020.
  * [33] Jason Liang, Elliot Meyerson, and Risto Miikkulainen.  Evolutionary Architecture Search For Deep Multitask Networks.  In GECCO, 2018.
  * [34] Chenxi Liu, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan Yuille, and Li Fei-Fei.  Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation.  In CVPR, 2019.
  * [35] Haotian Liu, Rafael A. Rivera Soto, Fanyi Xiao, and Yong Jae Lee.  Yolactedge: Real-time instance segmentation on the edge.  In ICRA, 2021.
  * [36] Shikun Liu, Edward Johns, and Andrew J. Davison.  End-to-End Multi-Task Learning with Attention.  In CVPR, 2019.
  * [37] Nils Lüling, David Reiser, Alexander Stana, and Hans W. Griepentrog.  Using depth information and colour space variations for improving outdoor robustness for instance segmentation of cabbage.  2021 IEEE International Conference on Robotics and Automation (ICRA), 2021.
  * [38] Chenxu Luo, Xiaodong Yang, and Alan Yuille.  Self-supervised pillar motion learning for autonomous driving.  In CVPR, 2021.
  * [39] Jiaqi Ma, Zhe Zhao, Jilin Chen, Ang Li, L. Hong, and Ed H. Chi.  SNR: Sub-Network Routing for Flexible Parameter Sharing in Multi-task Learning.  In AAAI, 2019.
  * [40] Kevis-Kokitsi Maninis, Ilija Radosavovic, and Iasonas Kokkinos.  Attentive Single-Tasking of Multiple Tasks.  In CVPR, 2019.
  * [41] I. Misra, A. Shrivastava, A. Gupta, , and M. Hebert.  Cross-stitch networks for multi-task learning.  In CVPR, 2016.
  * [42] Kiru Park, Timothy Patten, Johann Prankl, and Markus Vincze.  Multi-task template matching for object detection, segmentation and pose estimation using depth images.  In 2019 International Conference on Robotics and Automation (ICRA), pages 7207–7213, 2019.
  * [43] Daiyi Peng, Xuanyi Dong, Esteban Real, Mingxing Tan, Yifeng Lu, Hanxiao Liu, Gabriel Bender, Adam Kraft, Chen Liang, and Quoc V. Le.  PyGlove: Symbolic Programming for Automated Machine Learning.  In NeurIPS, 2020.
  * [44] Aditya Prakash, Kashyap Chitta, and Andreas Geiger.  Multi-modal fusion transformer for end-to-end autonomous driving.  In CVPR, 2021.
  * [45] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le.  Regularized Evolution for Image Classifier Architecture Search.  In AAAI, 2019.
  * [46] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang.  A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions.  In ACM Computing Surveys, 2021.
  * [47] Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, , and Anders Søgaard.  Latent multi-task architecture learning.  In AAAI, 2019.
  * [48] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  MobileNetV2: Inverted Residuals and Linear Bottlenecks.  arXiv preprint arXiv:1801.04381, 2018.
  * [49] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [50] Pushmeet Kohli Nathan Silberman, Derek Hoiem, and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In CVPR, 2012.
  * [51] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  CoRR, abs/1409.1556, 2015.
  * [52] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Sheng Zhao, Shuyang Cheng, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov.  Scalability in perception for autonomous driving: Waymo open dataset, 2020\.
  * [53] Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko.  AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning.  In NeurIPS, 2020.
  * [54] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le.  MnasNet: Platform-Aware Neural Architecture Search for Mobile.  In CVPR, 2019.
  * [55] Mingxing Tan and Quoc V. Le.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks.  In ICML, 2019.
  * [56] Mingxing Tan and Quoc V. Le.  EfficientNetV2: Smaller Models and Faster Training.  In ICML, 2021.
  * [57] Mingxing Tan, Ruoming Pang, and Quoc V. Le.  EfficientDet: Scalable and Efficient Object Detection.  In CVPR, 2020.
  * [58] Simon Vandenhende, Stamatios Georgoulis, Bert De Brabandere, and Luc Van Gool.  Branched Multi-Task Networks: Deciding What Layers To Share.  In BMVC, 2020.
  * [59] Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool.  Multi-Task Learning for Dense Prediction Tasks: A Survey.  In TPAMI, 2021.
  * [60] Thanh Vu, Marc Eder, True Price, and Jan-Michael Frahm.  Any-width networks.  In CVPRW, 2020.
  * [61] Thanh Vu, Daniel Piros, and Amir Sadovnik.  How your phone recognizes your home: An investigation of mobile object recognition.  In National Conference on Undergraduate Research (NCUR), 2016.
  * [62] Qifei Wang, Junjie Ke, Joshua Greaves, Grace Chu, Gabriel Bender, Luciano Sbaiz, Alec Go, Andrew Howard, Feng Yang, Ming-Hsuan Yang, Jeff Gilbert, and Peyman Milanfar.  Multi-path Neural Networks for On-device Multi-domain Visual Classification.  In WACV, 2021.
  * [63] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer.  FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search.  In CVPR, 2019.
  * [64] Yunyang Xiong, Hanxiao Liu, Suyog Gupta, Berkin Akin, Gabriel Bender, Yongzhe Wang, Pieter-Jan Kindermans, Mingxing Tan, Vikas Singh, and Bo Chen.  MobileDets: Searching for Object Detection Architectures for Mobile Accelerators.  In CVPR, 2021.
  * [65] Ming Yang, Shige Wang, Joshua Bakita, Thanh Vu, F. Donelson Smith, James H. Anderson, and Jan-Michael Frahm.  Re-thinking cnn frameworks for time-sensitive autonomous-driving applications: Addressing an industrial challenge.  In Proceedings of the IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), 2019.
  * [66] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn.  Gradient Surgery for Multi-Task Learning.  In NeurIPS, 2020.
  * [67] Yanqi Zhou, Xuanyi Dong, Berkin Akin, Mingxing Tan, Daiyi Peng, Tianjian Meng, Amir Yazdanbakhsh, Da Huang, Ravi Narayanaswami, and James Laudon.  Rethinking co-design of neural architectures and hardware accelerators.  arXiv preprint arXiv:2102.08619, 2021.
  * [68] Ming Zhu, Chao Ma, Pan Ji, and Xiaokang Yang.  Cross-modality 3d object detection.  In WACV, 2021.

Toward Edge-Efficient Dense Predictions with Synergistic Multi-Task Neural
Architecture Search
Supplementary Material

Figure 3:  A system-level overview of our proposed methods. We leverage multi-
objective, hardware-aware neural architecture search to discover optimal
neural components suitable for multi-task dense predictions, while
simultaneously ensuring efficient edge inference.

##  Appendix A Experimental details

#### Hyperparameters of NAS

We use a Regularized Evolution controller with a population size of 50, random
initialization, uniform mutator, and a tournament sample size of 10. We let
the search run for about 2000 generations. These parameters were simply chosen
to fit our computational budget and were not tuned. During the search, we
train models for 5000 iterations as a proxy task to save computation. The
final models are trained for 20000 iterations following AdaShare. For the
$\beta$ in the objective function in Eq. 5, we use (p=0.0) to set up a hard
constraint function and (q=-0.07) to promote Pareto optimality, following
MnasNet. We use $w_{i,j}$=1.0 to equally weight all evaluation metrics
$M_{i,j}$ of any task $T_{i}$ in Eq. 6 and Eq. 7. These can be adjusted to
suit downstream applications. With 512 TPUv2 cores, our multi-trial search
takes about 1.5 days for Cityscapes and 3.5 days for NYUv2. Since EDNAS is not
constrained by the specific NAS algorithm, one can also use a one-shot search
with weight sharing [6, 63] instead for better computational efficiency.
Finally, Fig. 4 provides a visual comparison of IBN and Fused-IBN blocks.

|  | Seg | Depth | SN
---|---|---|---|---
Reference | Model | $\mathcal{L}_{CE}$ | $\mathcal{L}_{1}$ | $\mathcal{L}_{RE}$ | $\mathcal{L}_{ICS}$
Tab. 2: | MT edge baseline | 0.4 | 1.000 | — | —
Cityscapes | EDNAS | 0.4 | 1.000 | — | —
| EDNAS+JAReD | 0.5 | 0.950 | 0.050 | —
Tab. 3: | MT edge baseline | 1.0 | 1.000 | — | 40.0
NYUv2 | EDNAS | 1.0 | 1.000 | — | 50.0
| EDNAS+JAReD | 1.0 | 0.999 | 0.001 | 60.0

Table 8:  Final loss weights. This table specifies the per-task loss weights
for models trained on 2-task Cityscapes and 3-task NYUv2. “SN” stands for
surface normal estimation.

| Loss weight | Seg | Depth | Avg
---|---|---|---|---
Method | $\mathcal{L}_{CE}$ | $\mathcal{L}_{1}$ | $\mathcal{L}_{RE}$ | mIoU | PAcc | AbsE | RelE | $\Delta T_{S}$ | $\Delta T_{D}$ | $\Delta T$
Single-task seg | 1.000 | 0.000 | 0.000 | 40.04 | 88.68 | — | — | — | — | —
Single-task depth | 0.000 | 1.000 | 0.000 | — | — | 0.0157 | 0.340 | — | — | —
Multi-task seg-depth | 0.500 | 1.000 | 0.000 | 38.64 | 88.49 | 0.0171 | 0.354 | -1.9 | -6.3 | -4.1
Multi-task seg-depth | 0.500 | 0.999 | 0.001 | 46.78 | 90.62 | 0.0149 | 0.323 | +9.5 | +5.1 | +7.3
Multi-task seg-depth | 0.500 | 0.990 | 0.010 | 46.83 | 90.56 | 0.0144 | 0.304 | +9.5 | +9.5 | +9.5
Multi-task seg-depth | 0.500 | 0.950 | 0.050 | 46.11 | 90.47 | 0.0143 | 0.281 | +8.6 | +13.3 | +10.9
Multi-task seg-depth | 0.500 | 0.900 | 0.010 | 46.41 | 90.56 | 0.0146 | 0.300 | +9.0 | +9.5 | +9.3

Table 9:  Impact of loss weighting

#### Task Loss and Weighting

Following AdaShare [53], we use Cross-Entropy loss $\mathcal{L}_{CE}$ to train
semantic segmentation, $\mathcal{L}_{1}$ loss for the base training of
monocular depth estimation, and the inverse of cosine similarity loss
$\mathcal{L}_{ICS}$ for surface normal prediction. Our JAReD loss also
includes a weighted mean relative error component $\mathcal{L}_{RE}$ as
specified in Eq. 8 We manually tune the loss weights to avoid ineffective
weighting interfering with the evaluation of NAS-found architectures, using
two guidelines: (1) We set task weights so that our MT edge baseline best
matches AdaShare’s (Sec. 4.1), then use similar weights for EDNAS. (2) For
EDNAS+JAReD, we keep the $\lambda$ in Eq. 8 small to avoid overwhelming the
$\mathcal{L}_{1}$ and other tasks such as segmentation. Tab. 8 details the
final weights of our main models, as presented in Tab. 2 and Tab. 3. In
addition, Tab. 9 illustrates the impact of different loss weighting strategies
on the multi-task performance of segmentation and depth prediction.

####  $\Delta$ Metrics for MTL Evaluation

Following the standard metrics for evaluating multi-task learning [40, 53,
59], we calculate the scores of multi-task learning relative to the single-
task performance. Specifically, given a multi-task model $a$ for evaluation,
let $T_{i}\in T$ be a task of interest (e.g. semantic segmentation) and
$m_{ij}\in M_{i}$ be an evaluation metric for task $T_{i}$ (e.g. mIoU). Let
$\hat{m}_{ij}$ be the baseline score of a corresponding singe-task model (e.g.
single-task segmentation mIoU). We define the per-metric relative score
$\Delta m_{ij}$ (e.g. $\Delta$mIoU) of the multi-task model $a$ with regard to
its baseline $\hat{m}_{ij}$ as followed:

$\Delta m_{ij}=(-1)^{l_{j}}\frac{m_{ij}-\hat{m}_{ij}}{\hat{m}_{ij}}*100\%$ (9)
$\text{with }l_{j}=\begin{cases}1\text{ if lower is better for metric
$M_{j}$}\\\ 0\text{ otherwise}\end{cases}$ (10)

We then define the per-task relative score $\Delta{T_{i}}$ (e.g. $\Delta$Seg)
of any task $T_{i}$ and the overall multi-task score $\Delta{T}$ of model $a$
respectively as:

$\Delta{T_{i}}=\frac{1}{|M_{i}|}\sum_{j=1}^{|M_{i}|}\Delta m_{ij}$ (11)
$\Delta T=\frac{1}{|T|}\sum_{i=1}^{|T|}\Delta T_{i}$ (12)

with $|M_{i}|$ and $|T|$ being the cardinality of the corresponding metric set
and task set respectively.

##  Appendix B Qualitative Results

Figure 5 presents some qualitative results of semantic segmentation and depth
estimation on CityScapes dataset

(a) Inverted Bottleneck (IBN)

(b) Fused-IBN

Figure 4:  A visual comparison of the Inverted Bottleneck (IBN) [48] and
Fused-IBN [64, 67, 56] blocks.

Input Image Ground Truth Single-Task Multi-Task EDNAS

Figure 5: Qualitative results for semantic segmentation (top) and depth
estimation (bottom) on CityScapes dataset


# Using homemade spectrometers to perform accurate measurements of discrete
and continuous spectra

Ana R. Romero Castellanos  Universidad ECCI, Carrera 19 No. 49-20, 111311,
Bogotá D.C, Colombia  H. E. Castellanos  Laboratorio de Física, Universidad
Manuela Beltrán, Avenida Circunvalar No. 60-00, 110231, Bogotá D.C, Colombia
C.E. Alvarez-Salazar  Laboratorio de Física, Universidad Manuela Beltrán,
Avenida Circunvalar No. 60-00, 110231, Bogotá D.C, Colombia

###### Abstract

Homemade spectrometers are commonly used tools to analyze light sources and
determine its physical characteristics. We perform an assessment of homemade
spectrometers in terms of spectral resolution and accuracy in the
determination of intensity, through the comparison of results with a
spectroscope commonly used in the physics and chemistry labs. We found that
the homemade spectrometer used is sufficiently accurate in wavelength, and can
be used by undergraduate students to perform precise measurements as, for
example, the spectrum of the Sun, leading to the determination of its
temperature.

##  1 Introduction

Spectrometry, originally thought as the analysis of the interaction of
radiation with matter with changing wavelength, but understood nowadays as the
determination of the characteristics of any quantity as a function of
wavelength or frequency, is one of the most relevant techniques to
characterize a system.

Spectroscopy has found application in many fields, due to the fact that the
properties of light emitted by a source can give quantitative information
about a system. In this way, spectroscopy has been used for the study of
atomic structure through the determination of emission and absorption spectra,
the analysis of chemical structures, the investigation of chemical equilibrium
and the kinetics of chemical reactions [1, 2]. Spectroscopy has found
applications in environmental analysis to determine the acidity of water or
the presence of metallic compounds in a sample [3, 4], in medicine has been
used for diagnostic and therapeutic applications as well as for pharmaceutical
analysis [5, 6], and in astronomy has been used, both long ago and nowadays,
to determine the chemical composition of stars or planets [7, 8, 9].

In this way, spectroscopy is a very important tool for different professionals
and scientists, and this makes teaching the basics of spectroscopy a must in
physics courses. Furthermore, spectroscopes are broadly used in the modern
physics lab to understand the structure of the atom and introducing students
to the concepts of quantum mechanics and the quantization of light [10, 11].

Nevertheless, a precision spectrometer is an expensive device which many
institutions can not afford, and this situation has raised different
experimental designs to supply this need. On the other hand, it could be very
enlightening for students, as mentioned in [12, 13], to build a spectrometer
to understand the measurement process and the physics behind the device used
to characterize a given chemical solution, as performed by the authors of the
same references.

Taking these considerations into account, homemade spectrometers have been
designed and built to perform different experiments. A measurement of
wavelengths of Balmer series for hydrogen has been obtained (with very low
relative errors) using spectrometers with transmission/reflection diffraction
gratings [14]. A determination of the concentration of ferrous
($\textrm{Fe}^{2+}$) and sodium ($\textrm{Na}^{+}$) ions in saline solutions
and natural water samples have been made in [15], using a low cost
spectroscope connected to a smartphone. On the same vain, Hosker [16] has
built a shoebox spectrophotometer using the light sensor on a smartphone to
perform measurements of absorbance as a function of concentration for
different solutions.

Certainly, with the COVID-19 pandemic, virtual labs using simulators, homemade
experiments and the use of both smartphone and online apps have become common
tools in education to perform all kind of experiments in physics [17, 18, 19,
20, 21, 22, 23], chemistry [24, 25], microbiology [26, 27], acoustics [28],
and all fields of knowledge. All these strategies have been assessed in terms
of their effectiveness and usefulness for scientific education [29, 30].
Nevertheless, it would be good to ask how different are the results obtained
with a homemade measurement device to the ones measured using specialized
equipment commonly used in the undergraduate physics and chemistry labs, and
if the results obtained can be used by students to perform accurate
measurements.

In this work, we present an assessment of the results obtained with a homemade
spectrometer and make a comparison with the results taken using a common
physics lab device. We have captured the spectrum of three different light
sources (hydrogen, helium and neon) using the Public Lab foldable mini-
spectrometer [31] available at publiclab.org/wiki/foldable-spec and using the
do-it-yourself (DIY) material analysis tool Spectral Workbench by Public Lab,
available at spectralworkbench.org/, and compare the results with the ones
obtained with the Pasco wireless spectrometer, using the free Pasco
Spectrometry Software available at pasco.com/products/software/spectrometry.
After comparison, we conclude that the results obtained with the homemade
spectrometer are sufficiently accurate and can be used by undergraduate
students to perform more complex measurements for both discrete and continuous
light sources.

In this sense, we propose to use the homemade spectrometer to determine the
spectrum of sunlight and determine the value of the temperature of the Sun, a
measurement which could be of interest both for physics and astronomy
undergraduate students. We suggest a comparison of the results with the
Standard Solar Spectra, freely available at
pveducation.org/pvcdrom/appendices/standard-solar-spectra, in order to compare
the fits obtained by a homemade experiment and a more sophisticated one, what
reinforces the skills of data analysis and evaluation.

This experiment and its related data analysis can help students to understand
the phenomena of light diffraction and its use for the measurement of light
properties, the characteristics and differences of discrete and continuous
spectra, the atomic structure and the characteristics of black-body radiation.

This paper is organized as follows. In Sec. 2 we present the experimental
setup used, describe the homemade spectrometer and the measurement process
used, and make a slight description of the capture of spectra using the
wireless Pasco spectrometer. In Sec. 3 we present the results obtained using
both the Pasco Wireless Spectrometer and the homemade spectrometer for three
different light sources, and compare the results in order to assess the
viability of using the homemade spectrometer to perform more complex studies.
In section 4 we show the results for the measurement of the spectrum of
sunlight, from which a blackbody spectrum has fitted in order to determine the
surface temperature of the Sun. Finally, we present our conclusions in Sec. 5.

##  2 Experimental setup and calibration

The first step to develop this home experiment to analyze light sources is to
build the measuring device using the foldable mini-spectrometer designed by
Public Lab and available at
https://publiclab.org/sites/default/files/8.5x11mini-spec3.8.pdf. In order to
build the spectrometer, students need black cardboard, a DVD-R, scissors and
glue or adhesive tape, which makes the instrument inexpensive, easy to build
and suitable for use by any student of any institution.

For our homemade experiment we have chosen to use a pre-designed spectrometer
already available and not to design a different configuration, as done, for
example, in [12, 13, 32, 33], in order to perform an assessment of the results
obtained with the homemade apparatus, and compare these measurements with the
ones obtained with a commonly used equipment in physics teaching lab, the
PASCO wireless spectrometer.

Once the spectrometer has been built, the online application
spectralworkbench.org should be used to get the spectrum of a light source.
Using a laptop webcam or a mobile phone camera, it is possible to get the
spectrum (a curve of light intensity as a function of wavelength) for
different light sources. The calibration process is fundamental, as the image
pixels are converted in nanometers, giving results of acceptable precision, as
analyzed later in this paper. Students can perform the calibration at home by
direct comparison of the light spectrum emitted by a compact fluorescent lamp
(CFL) with the mercury reference spectrum used in the app, matching the main
blue (436 nm) and green (546 nm) lines emitted by the mercury gas contained in
the CFL with the reference spectrum given in the web-based application.

In our experiment, we have used a mercury spectral tube to perform the
calibration, finding the results in figure 1, where a plot of relative
intensity as a function of wavelength is shown, and we can compare the
wavelengths of the peaks with the spectrum presented in Basic atomic
spectroscopy data. In this graph, we can see the emission spectrum as a
continuous black line showing four different peaks, matching very closely the
reference emission lines at 405, 436, 546 and 577 nm, approximately, indicated
by vertical colored lines in the graph. Additionally, we have inserted the
line spectrum, as obtained with spectralworkbench.org using a webcam, where we
can see the strong green and blue lines used to perform the calibration.

Figure 1: Mercury spectrum obtained using the homemade spectrometer and the
web application spectralworkbench.org. The vertical colored lines show the
main emission lines for this element in the visible range, where the
corresponding wavelengths are indicated. The spectrum obtained in the
application is shown in the insert.

From figure 1 we can see that the mercury spectrum, as obtained through the
homemade spectrometer and the web-based application spectralworkbench.org, is
composed by a set of four intensity peaks at 403, 438, 545 and 575 nm with a
specific width, characterizing the resolution of the spectrometer, a property
which we discuss later in this paper. We can see that these results have a
maximum percentage error of 1.2%, showing a very good approximation to the
position of peaks reported elsewhere [34], what makes the homemade
spectrometer a good option to characterize other light sources.

##  3 Characterization of discrete sources using the homemade spectrometer

In this section, we present the results obtained with the homemade
spectrometer for the analysis of light emitted by three different spectral
tubes: hydrogen, helium, and neon. We also assess the characteristics of the
homemade spectrometer and compare the results with the ones obtained using the
PASCO wireless spectrometer, commonly used in physics and chemistry teaching
labs.

In figure 2 we can see the hydrogen spectrum obtained using both the homemade
and PASCO spectrometers, in black and gray, respectively. In this graph, we
can see the spectral lines reported by [34] as vertical colored lines, where
the corresponding wavelengths are indicated alongside each line. The line
spectrum for this source is shown in the insert. From this graph, we can see
that the spectrum obtained with the homemade spectrometer consists of four
peaks with wavelengths 403, 428, 486 and 667 nm, which correspond to radiation
emitted in the transition of electrons from a higher energy level to a lower
energy level111The violet (403 nm) line corresponds to a transition from the
6th to the 2nd energy level, the blue line (428 nm) to a transition from the
5th to the 2nd, the green (486 nm) is produced in the 4th to the 2nd
transition, and the red (667 nm) in the 3rd to 2nd energy level transition.
This analysis has already done, for example, in [11, 14].. The lines obtained
deviate from the most intense emission lines of hydrogen in, at most, 2%.
Again, this deviation is very small and reinforces the observation that the
homemade spectrometer gives results of sufficient accuracy to perform the
analysis of different light sources, which can reinforce the knowledge of
modern physics in undergraduate students.

Figure 2: The hydrogen spectrum obtained with the homemade spectrometer
(black) and with the PASCO spectrometer (gray). The vertical colored lines
show the main emission wavelengths for this element, written alongside each
line. The insert shows the capture in the web-base application
spectralworkbench.org.

In this sense, we have found the resolving power of the homemade spectrometer,
defined as

$R=\frac{\lambda}{\Delta\lambda},$ (1)

where $\Delta\lambda$ is the smallest resolvable wavelength difference between
two monochromatic lines observable in the spectrum [35]. To determine
$\Delta\lambda$ for the homemade spectrometer, we found the full width at half
maximum (FWHM) by fitting a Gaussian distribution to each peak and finding the
position of the maximums, as can be seen in table 1, where the best fit for
each data set around the corresponding wavelength is shown as a continuous
line, and the FWHM is indicated in each graph. From these results we have
found a resolving power of 18 for a wavelength of 403 nm, 25 for 428 nm, 27
for 486 nm and 46 for the 667 nm wavelength. These results show that the
greatest resolving power of the spectrometer corresponds to the red color
which, nevertheless, has the lowest intensity in the spectrum.

It is important to note that the results of the homemade spectroscope are
accurate in terms of wavelength, but not in relative intensity, as can be seen
in figure 2, which shows its maximum intensity at a wavelength of 486 nm, the
green line of the spectrum. On the other hand, the PASCO spectroscope shows
the most intense line in the red color, at a wavelength of 648 nm, very close
to the value reported in [34]. From this result, and the absence of the orange
line in the mercury spectrum in figure 1, we conclude that there is a
suppression of intensity in the measurement of higher wavelengths, a point
which will be discussed later in this paper.

$\lambda_{\textrm{exp}}=403$ nm |  $\lambda_{\textrm{exp}}=428$ nm
---|---
|
$\lambda_{\textrm{exp}}=486$ nm |  $\lambda_{\textrm{exp}}=667$ nm
|
Table 1: Best fits for each of hydrogen peaks observed from figure 2, where
the resonance wavelength observed with the homemade spectrometer is shown in
the upper part, and the FWHM for each peak is shown.

In figure 3 we show the spectrum obtained with the homemade spectrometer
(black continuous line) for the helium spectral tube, and the corresponding
comparison with the PASCO spectrometer (gray continuous line) with the
emission lines reported in [34] indicated by vertical colored lines with the
corresponding wavelengths written alongside. The line spectrum for this source
is shown in the insert. From this graph we can see that the homemade
spectrometer gives six peaks at wavelengths 400, 446, 470, 490, 499 and 580
nm, which deviate at most 1.4% with respect to the lines reported in [34].

Figure 3: Helium spectrum obtained with the homemade spectrometer (black), and
its comparison with the PASCO spectrometer (gray) and reference emission lines
[34]. The spectrum obtained is shown in the insert.

From the helium spectrum shown in figure 3 we can notice several
characteristics. We can see interference effects in the green lines of the
spectrum, separated approximately 10 nm (less than the FWHM of the
spectrometer for the green lines, approximately 18 nm, as can be seen in table
1). On the other hand, it is important to remember that the spectrum obtained
with the homemade spectrometer is accurate in terms of wavelengths but not in
intensities. This remark is revealed in the helium spectrum too, as the blue
line with measured wavelength 446 nm should have, according to the
measurements reported in [34], an intensity greater than the intensities of
the two green lines at 492 nm and 502 nm, which are shown with higher
intensity in the spectrum obtained with the homemade spectrometer.

This effect, and also the absence of the red lines at 667, 687 and 707 nm
persistent in the helium spectrum shows the effects of the filter in the
webcam or mobile phone camera. This is due to the fact that digital cameras
used in laptops and smartphones have a decrease in their relative spectral
sensitivity, describing their relative efficiency of light detection for
different wavelengths, for wavelengths in the interval between 600 and 700
nanometers, corresponding to the red part of the visible spectrum. For this
color, the relative sensitivity is at most 50%, while the relative
sensitivities for green and blue reach, approximately, 100% and 85%,
respectively [36, 37]. This also explains why the web-based application
spectralworkbench.org, when used with a laptop or smartphone camera, shows the
green peaks with greater intensity than the blue or the red peaks, even though
their intensity should not be that high, as can be seen in figure 2 with the
red line measured at 667 nm (which should be the one with maximum intensity).

To close this section, we present in figure 4 the spectrum obtained for the
neon spectral tube, drawn as a continuous black line. In the same graph, the
reference spectrum is indicated by the vertical colored lines with the
corresponding wavelengths indicated alongside, and the spectrum observed in
the web-based application spectralworkbench.org is shown in the insert. We
have chosen to avoid the inclusion of the PASCO spectrum in this case in order
to emphasize the characteristics of the neon spectrum, and perform an analysis
with the same tools which a student performing the experiment would have. We
can see that this spectrum, as measured by the homemade spectrometer, consists
of seven intensity peaks with wavelengths 541, 581, 587, 600, 621, 629 and 642
nm, which deviate, at most, 1% with the lines reported in reference [34].

Figure 4: Neon spectrum obtained with the homemade spectrometer (black), and
its comparison with the reference emission lines at [34]. The insert shows the
spectrum obtained.

The characteristics of the homemade spectrometer, obtained from the analysis
of light emitted by the hydrogen and helium spectral tubes, are shown
completely in the neon spectrum of figure 4.

The spectrometer shows an interference between sets of spectral lines which
are very close, as the yellow lines shown in the spectrum. The first observed
peak seen in the yellow part of the visible spectrum corresponds to the
interference of three spectral lines at wavelengths 576, 580 and 582 nm, and
the second observed peak in the same color corresponds to the interference of
radiation emitted at wavelengths of 585 and 588 nm, as reported in [34]. It is
important to notice that, even though we can distinguish two peaks, they are
not fully disentangled, as the five wavelengths in the yellow part of the
visible spectrum are very close, and the resolving power of the homemade
spectrometer is not enough to identify each line independently. This behavior
is also seen in the orange line measured at 600 nm with the homemade
spectrometer, which corresponds to the interference of emission lines at 597
and 598 nm, very close to the observed wavelength, explaining the wide peak
observed around 600 nm.

Another characteristic previously discussed is the effect of the filter in the
webcam or mobile phone camera, which makes the spectrum sufficiently accurate
in wavelength, but not in terms of intensity. This aspect is revealed in the
small intensities shown in the red part of the spectrum (emission lines at 627
and 640 nm), which are shown very low in comparison with the yellow and orange
peaks, even though the red line near 627 nm should have the same intensity as
the two orange lines, as reported in Basic atomic spectroscopy data.

An aspect which calls for attention in the results obtained with the homemade
spectrometer is the attenuation in intensity observed for the green peak near
540 nm and the red peak near 640 nm. According to experimental data reported
in [34], these two lines, together with the yellow line near 585 nm, should be
the more intense lines and have equal intensities. The spectrum in figure 4
shows a strong attenuation of these two lines, but the reason for each one is
very different. For the green line near 540 nm, the attenuation is due to the
position and geometry of the homemade spectrometer, which bother the
observation of this line. For the case of the red line near 640 nm, we observe
a strong attenuation due to the filter effects of the webcam or mobile phone
camera.

It should be desirable for students to perform a correction of the intensities
measured, according to the methods explained, for example, in [36, 37], in
order to improve their experimental data.

##  4 Sunlight spectrum and Blackbody radiation

The spectra analyzed so far correspond to the intensity as a function of the
wavelength of light emitted by atomic sources, and are composed of different
discrete lines emitted when an electron makes a transition from a higher
energy level to a lower energy level. In this section, we use the homemade
spectrometer to analyze a light source with a continuous spectrum, the Sun,
and determine the characteristics of its radiation.

In figure 5 we show the sunlight spectrum obtained with the homemade
spectrometer (continuous black line) and the blackbody radiation spectrum
obtained for this data set, as a continuous red line. In the same graph, we
can see the standard solar spectrum [38] as a continuous gray line and its
corresponding fit using the blackbody radiation spectrum, as a continuous blue
line.

In this figure, we can see that the measurement taken with the homemade
spectrometer for the spectrum of sunlight, shown as a continuous black line in
the graph, gives an appropriate characterization of this light source for
wavelengths between 400 and 540 nm, where the attenuation of relative
intensity is not very significant and the observations with the homemade
spectrometer are very close to the standard solar spectrum. As was previously
mentioned in the analysis of sources with discrete spectrum, the attenuation
observed for the red part of the visible spectrum is due to the effect of
camera filters. This strong attenuation causes a drop in the red part of the
spectrum which affects the fit of the data set, giving a smaller maximum than
the observed from the raw data set.

From the fit of the blackbody radiation spectrum performed on the data set
obtained with the homemade spectrometer (red line) and its comparison with the
corresponding fit to the standard solar spectrum (blue line), we can see,
besides the strong attenuation for higher wavelengths, a displacement of the
maximum of the graph, and this will give a lower value of the temperature of
the Sun. More specifically, from the fit to the blackbody radiation spectrum
to the data obtained with the homemade spectrometer, we obtained a Sun
temperature of $(5591\pm 70)$ K, with a percentage error of approximately 3%
with respect to the reported temperature of the Sun [39, 40].

Figure 5: Sunlight spectrum obtained with the homemade spectrometer (black),
and its comparison with the standard solar spectrum (gray). The red and blue
lines show the blackbody radiation spectrum for the homemade and solar
spectra, respectively.

##  5 Conclusions

In this work we have shown that homemade spectrometers can be used to perform
an accurate characterization of both discrete and continuous light sources. We
have seen that the homemade spectrometer is accurate in terms of wavelength,
but presents an attenuation in intensity, specially for the red part of the
visible spectrum, due to the effect of laptop or mobile phone cameras used by
students performing the experiment

We have seen that the spectral resolution is enough to disentangle discrete
lines with separation greater than 10 nm, but interference effects for lines
with separation very close to this threshold often appear. This characteristic
allows students to understand the physical processes behind the spectroscope,
and the phenomenon of interference between light waves.

Additionally, we have seen that filters in laptop and mobile phone cameras
cause major effects in the properties of spectra, especially in the red part
of the spectrum, where attenuation can decrease the intensity of light up to
50%.

Finally, we have seen that students can perform the characterization of
continuous light sources as the Sun, and use the measured spectra to
understand the properties of blackbody spectrum and determine the surface
temperature of the Sun.

## Acknowledgments

A.R.R.C is grateful for the academic support of the Basic Sciences Department
at Universidad ECCI. H.E.C and C.E.A.S are grateful for the support of
Laboratory coordination at Universidad Manuela Beltrán.

## References

  * [1] Robert Eisberg.  Fundamentals of modern physics.  Krieger Publishing Company, 1961.
  * [2] Heinz-Helmut Perkampus.  UV-VIS Spectroscopy and its Applications.  Springer Science & Business Media, 2013.
  * [3] Markku T Mesilaakso.  Application of nmr spectroscopy to environmental analysis: Detection of trace amounts of chemical warfare agents and related compounds in organic extract, water, and sand.  Environmental science & technology, 31(2):518–522, 1997.
  * [4] Zhen Li, Jinxing Wang, and Daoliang Li.  Applications of raman spectroscopy in detection of water quality.  Applied Spectroscopy Reviews, 51(4):333–357, 2016.
  * [5] Pavel Matousek and Nicholas Stone.  Recent advances in the development of raman spectroscopy for deep non-invasive medical diagnosis.  Journal of biophotonics, 6(1):7–19, 2013.
  * [6] Jang Ah Kim, Dominic J Wales, and Guang-Zhong Yang.  Optical spectroscopy for in vivo medical diagnosis—a review of the state of the art and future perspectives.  Progress in Biomedical Engineering, 2(4):042001, 2020.
  * [7] Guillermo Gonzalez.  The chemical compositions of stars with planets: A review.  Publications of the Astronomical Society of the Pacific, 118(849):1494, 2006.
  * [8] Yann Alibert and Willy Benz.  Formation and composition of planets around very low mass stars.  Astronomy & Astrophysics, 598:L5, 2017.
  * [9] T Mishenina, N Basak, V Adibekyan, C Soubiran, and V Kovtyukh.  Chemical composition of stars with massive planets.  Monthly Notices of the Royal Astronomical Society, 504(3):4252–4273, 2021.
  * [10] Alberto Galindo and Pedro Pascual.  Quantum mechanics I.  Springer Science & Business Media, 2012.
  * [11] Ruediger Scholz, Susanne Wessnigk, and Kim-Alessandro Weber.  A classical to quantum transition via key experiments.  European Journal of Physics, 41(5):055304, 2020.
  * [12] Sara Taha, Gelan Rafat, Fatma Aboshosha, and Fotouh R Mansour.  A simple homemade spectrophotometer.  Journal of analytical chemistry, 72(2):239–242, 2017.
  * [13] Carlos Balado Sanchez, Rebeca P Diaz Redondo, Ana Fernandez Vilas, and Angel M Sanchez Bermudez.  Spectrophotometers for labs: A cost-efficient solution based on smartphones.  Computer Applications in Engineering Education, 27(2):371–379, 2019\.
  * [14] Pasquale Onorato, Massimiliano Malgieri, and A De Ambrosis.  Measuring the hydrogen balmer series and rydberg’s constant with a homemade spectrophotometer.  European Journal of Physics, 36(5):058001, 2015.
  * [15] Helton Jader Souza de Oliveira, Pedro Lemos de Almeida Jr, Bárbara Araújo Sampaio, Julys Pablo Atayde Fernandes, Osmundo Dantas Pessoa-Neto, Eduardo Antônio de Lima, and Luciano Farias de Almeida.  A handheld smartphone-controlled spectrophotometer based on hue to wavelength conversion for molecular absorption and emission measurements.  Sensors and Actuators B: Chemical, 238:1084–1091, 2017.
  * [16] Bill S Hosker.  Demonstrating principles of spectrophotometry by constructing a simple, low-cost, functional spectrophotometer utilizing the light sensor on a smartphone.  Journal of Chemical Education, 95(1):178–181, 2018.
  * [17] Tommaso Rosi and Pasquale Onorato.  Video analysis-based experiments regarding malus’ law.  Physics Education, 55(4):045011, 2020.
  * [18] Majid Iqbal, Fayzan Ahmed, Arshad Iqbal, and Zaheer Uddin.  Teaching physics online through spreadsheets in a pandemic situation.  Physics Education, 55(6):063006, 2020.
  * [19] Da Yang Tan and Jer-Ming Chen.  Bringing physical physics classroom online–challenges of online teaching in the new normal.  The Physics Teacher, 59(6):410–413, 2021.
  * [20] Pascal Klein, Lana Ivanjek, Merten Nikolay Dahlkemper, Katarina Jeličić, M-A Geyer, Stefan Küchemann, and Ana Susac.  Studying physics during the covid-19 pandemic: Student assessments of learning achievement, perceived effectiveness of online recitations, and online laboratories.  Physical Review Physics Education Research, 17(1):010117, 2021.
  * [21] Ari Syahidul Shidiq, Anna Permanasari, and Hernani Hernani.  Simple, portable, and inexpensive spectrophotometers for high schools lab activity.  Advances in Social Science, Education and Humanities Research (ASSEHR), 438:150–154, 2020.
  * [22] Razieh Koohkan, Massoud Kaykhaii, Mojtaba Sasani, and Brett Paull.  Fabrication of a smartphone-based spectrophotometer and its application in monitoring concentrations of organic dyes.  ACS omega, 5(48):31450–31455, 2020.
  * [23] Vilma Nohemí Yanchapanta Bastidas, Rosa Maricela Ormaza Hugo, Miguel Ángel Sáez Paguay, and Richard Willians Pachacama Choca.  Construcción de un espectrómetro casero para el análisis de espectros de luz, mediante el software spectral workbench.  Polo del Conocimiento: Revista científico-profesional, 5(7):209–222, 2020.
  * [24] Jessica L Andrews, Juan Pablo de Los Rios, Mythreyi Rayaluru, Seungwon Lee, Lilly Mai, Anna Schusser, and Chi H Mak.  Experimenting with at-home general chemistry laboratories during the covid-19 pandemic.  Journal of Chemical Education, 97(7):1887–1894, 2020.
  * [25] Rajesh Sunasee.  Challenges of teaching organic chemistry during covid-19 pandemic at a primarily undergraduate institution.  Journal of Chemical Education, 97(9):3176–3181, 2020.
  * [26] ALS López, DE Navarro-López, and J Castañeda-Sedano.  Home lab as a strategy for meaningful learning of microbiology during covid-19 pandemic.  In Proceedings of the International Conference on Education, volume 7, pages 247–256, 2021.
  * [27] Joanna Koort and Silja Åvall-Jääskeläinen.  Redesigning and teaching veterinary microbiology laboratory exercises with combined on-site and online participation during the covid-19 pandemic.  FEMS Microbiology Letters, 368(16):fnab108, 2021.
  * [28] Murray S Korman and Leah E Burge.  Teaching underwater acoustics and sonar during the covid pandemic.  The Journal of the Acoustical Society of America, 149(4):A45–A45, 2021.
  * [29] Xiaoshan Z Gordy, Wesley Sparkmon, Hyllore Imeri, Andrew Notebaert, Marie Barnard, Caroline Compretta, Erin Dehon, Juanyce Taylor, Stephen Stray, Donna Sullivan, et al.  Science teaching excites medical interest: A qualitative inquiry of science education during the 2020 covid-19 pandemic.  Education Sciences, 11(4):148, 2021.
  * [30] Nagihan Tanik-Önal and Nezih Önal.  Teaching science through distance education during the covid-19 pandemic.  International Online Journal of Education and Teaching, 7(4):1898–1911, 2020.
  * [31] Papercraft spectrometer intro kit.  Available at https://publiclab.org/wiki/foldable-spec.  Accessed on November 16th, 2021.
  * [32] Young-Gu Ju.  Fabrication of a low-cost and high-resolution papercraft smartphone spectrometer.  Physics Education, 55(3):035005, 2020.
  * [33] Tommaso Rosi, Massimiliano Malgieri, Pasquale Onorato, and Stefano Oss.  What are we looking at when we say magenta? quantitative measurements of rgb and cmyk colours with a homemade spectrophotometer.  European Journal of Physics, 37(6):065301, 2016.
  * [34] David R Lide.  CRC handbook of chemistry and physics, volume 85.  CRC press, 2004.
  * [35] J. Gordon Robertson.  Quantifying resolving power in astronomical spectra.  Publications of the Astronomical Society of Australia, 30:e048, 2013\.
  * [36] Jun Jiang, Dengyu Liu, Jinwei Gu, and Sabine Süsstrunk.  What is the space of spectral sensitivity functions for digital color cameras?  In 2013 IEEE Workshop on Applications of Computer Vision (WACV), pages 168–179. IEEE, 2013.
  * [37] D Andrew Rowlands.  Color conversion matrices in digital cameras: a tutorial.  Optical Engineering, 59(11):110801, 2020.
  * [38] C. Honsberg and S. Bowden.  Standard Solar Spectra.  https://www.pveducation.org/pvcdrom/appendices/standard-solar-spectra.  Accessed: 2021-11-28.
  * [39] JR Kuhn, KG Libbrecht, and RH Dicke.  The surface temperature of the sun and changes in the solar constant.  Science, pages 908–911, 1988.
  * [40] Johan Holmberg, Chris Flynn, and Laura Portinari.  The colours of the sun.  Monthly Notices of the Royal Astronomical Society, 367(2):449–453, 2006.