# Rethinking Co-design of Neural Architectures and Hardware Accelerators

Yanqi Zhou  Xuanyi Dong  Berkin Akin  Mingxing Tan  Daiyi Peng  Tianjian Meng
Amir Yazdanbakhsh  Da Huang  Ravi Narayanaswami  James Laudon

###### Abstract

Neural architectures and hardware accelerators have been two driving forces
for the progress in deep learning. Previous works typically attempt to
optimize hardware given a fixed model architecture or model architecture given
fixed hardware. And the dominant hardware architecture explored in this prior
work is FPGAs. In our work, we target the optimization of hardware and
software configurations on an industry-standard edge accelerator. We
systematically study the importance and strategies of co-designing neural
architectures and hardware accelerators. We make three observations: 1) the
software search space has to be customized to fully leverage the targeted
hardware architecture, 2) the search for the model architecture and hardware
architecture should be done jointly to achieve the best of both worlds, and 3)
different use cases lead to very different search outcomes. Our experiments
show that the joint search method consistently outperforms previous platform-
aware neural architecture search, manually crafted models, and the state-of-
the-art EfficientNet on all latency targets by around 1% on ImageNet top-1
accuracy. Our method can reduce energy consumption of an edge accelerator by
up to 2x under the same accuracy constraint, when co-adapting the model
architecture and hardware accelerator configurations.

Machine Learning, ICML

##  1 Introduction

Figure 1: Chip Energy ($Power\times Latency$) vs. ImageNet top-1 accuracy.
Joint search for hardware and model architectures (NAHAS) significantly
outperforms other platform-aware NAS, or manually crafted efficient ConvNets.
Manual-EdgeTPU are manually crafted models on an evolved EdgeTPU search space
defined in Section 3.2.2.

The traditional approach to hardware design is to build a general purpose
processor that is optimized for a particular set of workloads. For example,
CPUs are optimized for sequential workloads such as desktop applications, and
GPUs are designed for massive parallel workloads such as gaming, graphics
rendering, scientific computing, etc. The optimization of these designs is
typically done by evaluating the performance of the designs on a set of
benchmarks (e.g., SPEC (SPE, 2006)), which act as a proxy for the full set of
workloads. In addition to the difficulty for the benchmarks to capture all the
salient characteristics of the workload set, the benchmarks tend to stay fixed
for a substantial period of time, which can result in the hardware design
lagging behind the algorithmic changes, particularly in a rapidly changing
field such as machine learning.

As a result of hitting the end of Moore’s Law in the recent decade, the focus
for continued processing improvements has moved to hardware specialization to
provide additional speedups and efficiency for a narrowed-down application or
domain. Google’s TPU (TPU, 2016), Intel’s Nervana NNP (Yang, 2019), and
Apple’s Neural Engine in the M1 SoC (app, 2020) are representative
accelerators specialized for deep learning primitives. MLPerf (MLP, 2019) has
become prevalent for benchmarking the state-of-the-art design of machine
learning (ML) accelerators. However, rapid progress in deep learning has given
birth to numerous more expressive and efficient models in a short time, which
results in both benchmarking and accelerator development lagging behind. For
example, squeeze-and-excite with global pooling and SiLU/Swish non-linearity
(Ramachandran et al., 2017; Elfwing et al., 2018) are found to be useful in
EfficientNet (Tan & Le, 2019); however, they are often not supported or
inefficient in many specialized accelerators like GPUs, EdgeTPUs and DSPs.

Platform-aware neural architecture search (NAS) (Tan et al., 2019; Wu et al.,
2019; Cai et al., 2019) optimizes the neural architectures for a target
inference device. The target device has a fixed hardware configuration that
can significantly limit NAS performance and efficiency. For example, the
target device may have a sub-optimal compute and memory ratio for the target
application combined with an inference latency target, which can shift the
optimal NAS model distributions and result in underperformance.

There is also a shift towards optimizing hardware architecture given a fixed
model architecture. For example, design space exploration in computer systems
has become more crucial due to the surge of specialized hardware (Parsa et
al., 2020; Iqbal et al., 2020; Sun et al., 2020; Nardi et al., 2019; Cong et
al., 2018; Koeplinger et al., 2018; Balaprakash et al., 2016; Ansel et al.,
2014). Combining this success with platform-aware NAS, some recent attempts
have been made in jointly co-optimizing hardware and model architectures
(Jiang et al., 2020a; Choi et al., 2020; Jiang et al., 2020b; Achararit et
al., 2020; Yang et al., 2020; Kwon et al., 2018; Yang et al., 2019). However,
most of the work target FPGAs or academic accelerators (Chen et al., 2016),
which have some limited capabilities such as not being able to run arbitrary
networks on large datasets like ImageNet. They typically study the co-
optimization of a fixed neural network without changing the NAS search space.

Building on this prior work, we propose to study the joint search of hardware
and model architectures. We generalize the above ideas into a framework called
NAHAS – Neural Hardware and Hardware Accelerator Search. In NAHAS, we
parameterize neural architecture search and hardware accelerator search in a
unified joint search space. We use a highly parameterized industry-standard
edge accelerator as our target device, which has a tunable set of important
hardware configurations. These knobs fundamentally determine hardware
characteristics such as the number of computing units, compute to memory
ratio, bandwidth, etc., which we find very critical to model performance. We
formulate the optimization problem as a bi-level optimization with hardware
resource constraints on the chip area and model latency or energy (which
encapsulates both power and latency).

Unlike the conventional search approach, NAHAS is task driven, where the task
is a problem (e.g., image classification, object detection) or a domain of
problems (e.g., vision, NLP), not a set of fixed programs or graphs (e.g.,
ResNet, Transformers). This effectively creates generalization across the
vertical stack, making the hardware evolve with the applications. NAHAS can be
practically used to design customized accelerators for autonomous driving and
mobile SoC (system-on-chip), where a set of highly optimized accelerators are
combined into a system.

Figure 2: Different accelerator configurations have different Pareto frontiers
consisting of different NAS models. Joint search effectively extends the
Pareto frontier by joining multiple frontiers.

Figure 2 shows that while conventional platform-aware NAS selects models along
the Pareto frontier with different latency and accuracy tradeoffs for one
target device, NAHAS further expands the Pareto frontier by enabling different
hardware accelerator configurations a high-level workflow of NAHAS. To
summarize our contributions:

  * •

We develop a fully automated framework that can jointly optimize neural
architectures and hardware accelerators. For the first time, we demonstrate
the effectiveness of co-optimizing neural architecture search with the
parameterization of a highly optimized industry-standard edge accelerator.

  * •

We propose a latency-driven search method, which is hardware-agnostic and
achieves state-of-the-art results across multiple search spaces. NAHAS
outperforms platform-aware search including MnasNet (Tan et al., 2019) and
EfficientNet (Tan & Le, 2019), and manually crafted models including
MobileNet-v2-1.4 and Manual-EdgeTPU (Xiong et al., 2020; Gupta & Akin, 2020),
on all latency targets by around 1% in ImageNet top-1 accuracy or around 20%
in inference latency.

  * •

We observe that different NAS search spaces consisting of different
fundamental operations, coupled with accuracy and inference latency
objectives, require completely different accelerator configurations. Co-
adapting NAS search spaces and accelerator configurations for different model
sizes and latency targets becomes essential for domains such as autonomous
driving, edge computing, etc.

  * •

We compare different optimization strategies and find that joint search
consistently outperforms optimizing NAS and HAS in an alternating fashion or
in a nested loop. Oneshot search can significantly reduce search cost (total
number of samples and search time); however, it is less suitable for large
models when constructing the super network is too costly.

##  2 Related Work

ML-driven Hardware Accelerator Search: Hierarchical-PABO (Parsa et al., 2020)
and FlexiBO (Iqbal et al., 2020) leveraged multi-objective Bayesian
optimization for neural network accelerator design. In order to reduce
computational costs, (Sun et al., 2020) applied a genetic algorithm to design
CNN models without modifying the underlying architecture. HyperMapper (Nardi
et al., 2019) used random forests in automatic tuning of hardware accelerator
configuration in a multi-objective setting.

Platform-Aware & Efficient NAS: There has been growing interest in leveraging
NAS to automate the design process of edge models (Howard et al., 2019; Tan et
al., 2019; Dai et al., 2019). ProxylessNAS directly searches for ImageNet by
proposing a gradient-based approach to train binarized parameters (Cai et al.,
2019). FBNet proposed a differentiable platform-aware NAS using Gumbel Softmax
(Wu et al., 2019). NetAdapt (Yang et al., 2018) and AMC (He et al., 2018)
utilize a direct metric like latency to finetune the number of channels of a
pre-trained model. Accelerator-aware NAS (Gupta & Akin, 2020) targets
industry-standard accelerators and identified state-of-the-art (SoTA) models
for an edge TPU. However, none of these works optimize the underlying hardware
accelerators together with NAS. There are many recent efforts to reduce the
search cost of NAS (Brock et al., 2018; Bender et al., 2018; Pham et al.,
2018; Liu et al., 2019; Cai et al., 2020; Dong & Yang, 2019; Cai et al., 2019;
Wu et al., 2019). Existing works are heavily focusing on highly regularized
search spaces consisting of mainly Inverted Bottleneck (IBN) layers, for FLOPs
on mobile CPUs. Less attention has been paid to the adaptation of search space
or a latency/power driven search for modern accelerators such as DSPs or
EdgeTPUs.

Hardware Metrics: Platform-aware NAS typically searches for efficient models
with lower parameter counts and FLOPs (number of multiply-add operations).
However, optimizing _indirect metrics_ like parameter counts or FLOPs won’t
necessarily improve the _direct metrics_ of latency (Bender et al., 2020; Wu
et al., 2019).. For example, a multi-branch network like NASNet (Zoph et al.,
2018) has lower parameter counts and FLOPs compared to the layer-wise network
such as ResNet and MobileNet (Sandler et al., 2018), but its fragmented cell-
level structure is not hardware friendly and is often slow on real-world
devices and accelerators. The effects of _indirect metrics_ like parameter
counts and FLOPs do not have a simple correlation with _direct metrics_ such
as latency and power. For example, the model can run out of memory if the
number of parameters is too large.

Co-design: There is a growing body of work exploring neural architecture
search and hardware design (Jiang et al., 2020a; Choi et al., 2020; Jiang et
al., 2020b; Achararit et al., 2020; Yang et al., 2020; Kwon et al., 2018; Yang
et al., 2019). However, most of the work targets FPGAs or academic
accelerators (Chen et al., 2016), where the hardware is less optimized on real
workloads (e.g. ImageNet) and the performance model is less accurate. NAHAS,
however, targets a highly optimized industry-standard ML accelerator and the
performance simulator is a validated, cycle-accurate model. Moreover, instead
of using a fixed NAS search space, NAHAS demonstrates effectiveness on a real
ImageNet workload, on multiple search spaces including an adapted search space
for EdgeTPU. NAHAS is the first to adapt the NAS search space for the co-
design of NAS and accelerators.

##  3 Methodology

In this section, we will generalize the ideas of joint optimization of
hardware and model architectures with NAHAS and present two different
approaches.

###  3.1 Formulation

The objective for NAHAS is to find a neural architecture configuration
$\alpha$ and hardware accelerator configuration $h$ such that the validation
accuracy can be maximized while meeting a chip area and latency target.

$\displaystyle\min\limits_{\alpha,h}{\mathcal{L}}(\alpha,h,\omega_{\alpha}^{*},{\mathbb{D}}_{val})$
(1) $\displaystyle\mathrm{s.t.}~{}$
$\displaystyle~{}\omega_{\alpha}^{*}=\arg\min_{\omega_{\alpha}}{\mathcal{L}}(\alpha,h,\omega_{\alpha},{\mathbb{D}}_{train}),$
(2) $\displaystyle\mathrm{and}~{}$ $\displaystyle~{}Latency(\alpha,h)\leq
T_{latency},~{}Area(h)\leq T_{area},$ (3)

where ${\mathcal{L}}$ indicates the objective function of the tasks (e.g.,
cross-entropy for classification) and $\omega_{\alpha}$ denotes the weights of
the architecture $\alpha$. NAHAS introduces a broader search space than NAS
(neural architecture search) or HAS (hardware accelerator search) alone, with
the flexibility to fix either $\alpha$ or $h$ therefore, the optimization
problem is reduced to NAS or HAS. We empirically compare different
optimization strategies in Section 4.4.

###  3.2 NAS Search Space

####  3.2.1 IBN-based Search Space

MobileNetV2: We build the architecture search space ${\mathcal{S}}_{1}$ based
on the standard MobileNetV2. The search space is tailored for mobile edge
processors, and therefore consists mostly of efficient operations such as
mobile IBN. Specifically, we search for the kernel size from {3, 5, 7} for
each IBN layer, and we also search for the expansion ratio from {3, 6} for
each block except for the first one, which has the default expansion ratio of
1. In MobileNetV2, there are 17 inverted residual blocks, and thus the
cardinality of ${\mathcal{S}}_{1}$ is about 8.4e12.

EfficientNet: In order to create larger NAS models and to better leverage
modern edge accelerators which have larger number of compute units and memory
capacities, we build the architecture search space ${\mathcal{S}}_{2}$ based
on the standard EfficientNet-B0. Similar to ${\mathcal{S}}_{1}$, we also
search for the kernel size from {3, 5, 7} and the expansion ratio from {3, 6}.
Since there are 16 inverted residual blocks in EfficientNet-B0, the
cardinality of ${\mathcal{S}}_{2}$ is about 1.4e12.

####  3.2.2 Evolved NAS Search Space

Figure 3: Tunable Layers for Edge Accelerators. In a Fused-IBN layer, we use a
$one\\_of$ operation in PyGlove to select whether to use a regular depthwise
convolution or a fused convolution.

IBNs remain the predominant building blocks in many SoTA mobile models. The
previously described search spaces, built around MobileneV2 and EfficientNet
in Section 3.2.1, are both IBN based. Although IBN layers are good at reducing
the parameter count and FLOPs, depthwise-separable convolutions in IBN can
significantly increase inference latency or reduce accelerator utilization.
For example, many industry standard edge accelerators, such as Google’s
EdgeTPU and Qualcomm DSPs, are designed for regular convolutions with
sufficient compute intensity (more computations per data transfer) (Zhou et
al., 2018). For certain tensor shapes and kernel dimensions, a regular
convolution can utilize the hardware up to 3x more efficiently than the depth-
wise variation on an EdgeTPU despite the much larger computation cost (7x more
FLOPs) (Xiong et al., 2020). This leads to our decision to re-engineer the
search space for edge accelerators.

Fused IBN Layers: We adopted the fused IBN layer from MobileDets (Xiong et
al., 2020) that modifies an IBN layer by fusing together the first $1$x$1$
convolution (which usually comes with an expansion ratio) and its subsequent
$K$x$K$ depthwise convolution with a single $K$x$K$ full convolution. We keep
the expansion factor as conventional MobileNet models to enable channel size
expansion. We name this Fused-IBN. Fused-IBN provides more trainable
parameters over baseline IBN but runs faster on edge accelerators for some
tensor shapes.

Figure 4: Evolved Search Space: NAHAS respects Efficientnet’s compound scaling
ratios, while introducing edge efficient Fused-IBN with tunable parameters. We
use a symbolic programming library named PyGlove (Peng et al., 2021) to tune
filter size, kernel size, expansion factor, and groups.

Tunable Layers via Symbolic Programming: Intuitively, Fused-IBN becomes less
efficient when the network gets deeper and the channel dimension gets larger,
due to the cost of full convolution over large input channels. We introduce a
more flexible, tunable search space via PyGlove (Peng et al., 2021) symbolic
library, to select ops type at each layer as well as selecting layer
parameters such as kernel size, expansion factor, filter scaling multiplier,
and number of groups. This flexibility provides a middle ground between IBN
and Fused-IBN layers.

Figure 3 demonstrates idea of a switched Fused-IBN layer. For some tensor
shapes, Fused-IBN can be more effective when running on edge accelerators,
such as early layers with smaller channel sizes in a convolutional network.
However, as the network goes deeper and channel size becomes larger, we may
still adopt the conventional IBN to reduce latency and cost. Figure 4
demonstrates our evolved search space, which consists of a mixture of Fused-
IBN, and conventional IBN layers. In our manually crafted model (Manual-
EdgeTPU presented in our evaluation results) based on the evolved search
space, we use a fixed number of fused-IBN in the early layers. However, in
NAHAS, we use PyGlove (Peng et al., 2021) to symbolize each layer as a tunable
search space. For example, we can replace any static node in a computational
graph, with a tunable node with a predefined search space (e.g., filter
scaling, number of groups, kernel size, and expansion factor, etc.).
Therefore, we can essentially transform any static neural network into a
tunable search space. In the following sections, we evaluate NAHAS on multiple
search spaces, leveraging the powerful symbolic programming.

###  3.3 Accelerator Search Space

The target device is an industry-standard, highly parameterized edge
accelerator which allows us to create various configurations in a large design
space with tradeoffs between performance, power, area, and cost. The
accelerator in Figure 5 features a set of parallel processing elements (PE)
organized in a 2D tile. The number of PEs in each dimension determines the
aspect ratio of the chip. In each PE there are multiple compute lanes that
shares a local memory and each lane has a register file and a series of
single-instruction multiple-data (SIMD) style multiply-accumulate (MAC)
compute units. Each of these architecture components provide a degree of
parallelism and a corresponding area cost. With a fixed chip area budget, the
hardware accelerator search (HAS) optimizes the on-chip resource allocation,
balancing the compute and memory and searching for the best chip
parameterization for a given application.

Figure 5: Edge Accelerator Configuration Details. parameters | type | search space
---|---|---
PEs_in_x_dimension | int | 1, 2, 4, 6, 8
PEs_in_y_dimension | int | 1, 2, 4, 6, 8
SIMD_units | int | 16, 32, 64, 128
compute_lanes | int | 1, 2, 4, 8
local_memory_MB | int | 0.5, 1, 2, 3, 4
register_file_KB | int | 8, 16, 32, 64, 128
io_bandwidth_gbps | float | 5, 10, 15, 20, 25
Table 1: Edge Accelerator Search Space.

As our baseline we take the default accelerator configuration which is
optimized while considering a series of production workloads from multiple
domains. The baseline configuration features 4x4 PEs where each PE has 2 MB
local memory and 4 compute lanes. Each compute lane has a 32 KB register file
and 64 4-way SIMD units. Since the accelerator is targeted for edge use cases,
SIMD units can sustain the peak throughput for 8-bit quantized operations.
This baseline configuration can deliver a peak throughput of 26 TOPS/s at 0.8
GHz.

Unlike the NAS search space, the HAS search space contains many invalid
points, which makes training a cost model or joint search with the in-house
simulator more challenging. Invalid configurations can be caused by many
reasons. For example, the created accelerator configuration in combination
with the NAS model may not be supported by the compiler or a NAS model may be
created that is too large for the generated HAS configuration, etc.

###  3.4 Search Objective

Chip power and area are two important efficiency metrics for accelerator
design. In this work, we focus on maximizing model accuracy while meeting an
inference latency constraint on a target device. The latency constraint can be
easily swapped with an energy (average power times latency) constraint that
considers both power and latency. We also impose a chip area constraint, which
is set equal to the baseline accelerator design. This paper does not focus on
minimizing chip area or latency beyond these set constraints. In a more
general setting, the chip resource constraints can be set to area, power,
energy, or a combination of many, however, experimentation on the mixture of
constraints is not the focus of this paper.

Similar to MnasNet (Tan et al., 2019), we use a customized weighted product to
encourage Pareto optimal solutions. More specifically, we have a optimization
metric of model accuracy and two hardware constrain metrics of model latency
and chip area. The optimization goal is to maximize model accuracy while
meeting the latency and area constraints.

$\displaystyle\max\limits_{\alpha,h}Accuracy(\alpha,h)(\frac{Latency(\alpha,h)}{T_{latency}})^{w_{0}}(\frac{Area(h)}{T_{area}})^{w_{1}},$
(4)

where $w_{0},w_{1}$ are the weight factors:

$\displaystyle w_{0}$
$\displaystyle=\begin{cases}p,&\textrm{if}~{}Latency(\alpha,h)\leq
T_{latency}\\\ q,&\textrm{otherwise}\end{cases},$ (5) $\displaystyle w_{1}$
$\displaystyle=\begin{cases}p,&\textrm{if}~{}Area(h)\leq T_{area}\\\
q,&\textrm{otherwise}\end{cases}$ (6)

When $p=0,q=-1$, the reward function imposes a hard latency constraint and we
simply use accuracy as the objective if the measured latency is meeting the
latency target $T$ and only sharply penalize the objective value if the sample
violates the latency constraint. When $p=q=-0.07$111(Tan et al., 2019) found
-0.07 to empirically ensure Pareto-optimal solutions have similar reward under
different accuracy-latency trade-offs., the reward becomes a soft constraint
function. In the NAHAS evaluations, we use both rewards for different
experiments.

###  3.5 Search Strategy

####  3.5.1 Joint search w/o weight sharing

Similarly as NASNet (Zoph et al., 2018) and MnasNet (Tan et al., 2019), we use
PPO (Schulman et al., 2017) as the controller algorithm to optimize the joint
search space from NAS and HAS. The controller samples the search space using a
recurrent network, each sample is trained by a child program. For the
MobileNetV2 search space, we use a proxy task that trains each sample on
ImageNet for only 5 epochs and it takes 5000 samples for the controller to
converge. For larger models using EfficientNet search space, training the
proxy task for 15 epochs while reducing the total number of samples to 2000
improves the results.

####  3.5.2 Joint search with weight sharing

To further reduce the search cost, we employ an efficient search method with
weight sharing. Similarly as ProxylessNAS (Cai et al., 2019) and TuNAS (Bender
et al., 2020), we use the controller decisions from the NAS space to construct
a super-network for optimizing the architecture, meanwhile using the decisions
from the HAS space to create a sub-graph for computing the cost. Decision
points from both spaces are optimized by a RL algorithm within the same graph.
For each training step, we train the model weights and the controller decision
points in an interleaved way. To achieve better results, we apply the absolute
reward function and RL warm-up procedure introduced in TuNAS.

To estimate the latency of the model on a given accelerator configuration, we
train a cost model with random generated samples using an in-house accelerator
simulator. We need a cost model because as NAS becomes much faster with
oneshot search, the query to the accelerator performance simulator for chip
area and inference latency becomes the new bottleneck for NAHAS oneshot
search. Given a sampled neural architecture configuration and an accelerator
configuration, the cost model predicts the accelerator area $f_{a}(h)$ and
model accuracy $f_{l}(\alpha,h)$. We use an MLP network with ReLU to
encapsulate the non-linearity in the latency prediction. The area predictor
and latency predictor largely share parameters with only separate
parameterization in the prediction heads.

$\displaystyle Loss=MSE(L_{a},f_{a}(h))+\lambda MSE(L_{l},f_{l}(\alpha,h)),$
(7)

The cost model was trained with 500k labeled data randomly generated by
permuting the neural architecture configurations and accelerator
configurations. Because labeled data for accelerator performance is much
cheaper than labeled data for NAS accuracy, and the collection of data can
utilize the vast amount of CPU resources, we do not consider the cost of
training a cost model. We use a 3-layer MLP of hidden size 256 and apply a
dropout of 0.1 to mitigate overfitting at each layer.

##  4 Evaluation

###  4.1 Experiment Setup

Accelerator Performance Simulator: Evaluating the candidate neural and
hardware architectures accurately is a key requirement for the NAHAS
framework. Simply using the number of MACs or parameters of the neural model
as a proxy for performance can be highly inaccurate as its performance highly
depends on how the neural network is mapped on the hardware architecture and
its unique compute characteristics. To this end, we have utilized an in-house
cycle-accurate performance simulator and an analytical area model based on
hardware synthesis. We deployed both of these estimators as a service where
multiple NAHAS clients can send parallel requests. This provides a flexible
way to scale-up the performance and area evaluations.

Search Hyperparameters: For the end2end search, we choose PPO as it is tested
by time. We use the average performance of 10 trials as a reward. We use Adam
optimizer with the learning rate of 0.0005 to update the controller, where the
policy gradients are clipped by 1.0. For each trial, we train the sampled
candidate by five epochs using RMSProp. For these five epochs, we will first
warm up the model by two epochs using the learning rate from 0 to 0.66, and
then cosine decay it from 0.66 to 0 for the rest three epochs. For the oneshot
search, we utilize REINFORCE to optimize the controller following TuNAS. We
use Adam with a learning rate of 0.0048 to optimize it and use the momentum as
0.95 for baseline. In addition, we use RMSProp to optimize the shared weights
following the same learning rate schedule as TuNAS. The latency predictor is
pre-trained.

Cost Model Hyperparameters: The cost model was trained with hyperparameters
defined in Table 2. In the oneshot search, we replace the accelerator
performance simulator with the trained cost model and search for the best
model for five different latency targets (0.3ms, 0.5ms, 0.8ms, 1.1ms, and
1.3ms). The average error between the latency target and the estimated latency
of the best model using the accelerator performance simulator is only 0.4%.

Figure 6: Cost Model Accuracy.

Optimizer | Adam
---|---
Loss Re-weight $\lambda$ | 10
Batch size | 128
Hidden dimension | 256
Learning rate | 0.001
Training steps | 600k
Input feature size | 394

Table 2:  Cost Model Hyperparameters.

###  4.2 Sample Distributions

To study the sample distribution during search, we first compare our NAHAS
search with previous platform-aware NAS, as shown in Figure 7. Search is
performed on EfficientNet-B0 with Squeeze-and-Excite layers and Swish non-
linearity. In platform-aware NAS, the target device is fixed to the baseline
accelerator design, as described in Section 3.3; whereas in NAHAS search, the
target chip area is set to the same as the target device of platform-aware
search. Latency targets are set of 1ms. We observe that without the
flexibility to change the hardware configuration, platform-aware NAS always
converges to sub-optimal solutions of either higher latency or lower accuracy.
For NAHAS, not all the samples traversed meet the chip area constraint.
However, traversing through samples violating the resource constraints (red
points in the figure) can help converge to more pareto-optimal samples
eventually with both higher accuracy and lower inference latency.

Figure 7:  Comparison of the sample distributions during search.

###  4.3 Latency/Energy-Driven NAHAS

Figure 8: Inference Latency vs. ImageNet Accuracy. NAHAS significantly
outperform other platform-aware, efficient ConvNets.

Inference latency can be a critical metric for applications with stringent
service-level-agreement (SLA) requirements (e.g. mobile face recognition) or
real-time requirements (e.g. autonomous driving). In this section, we compare
a latency-driven NAHAS based on both the IBN-only search space in Section
3.2.1 and adapted search space discussed in Section 3.2.2 with related
platform-aware NAS and manually crafted models for EdgeTPUs. Manutal-EdgeTPU-S
and Manual-EdgeTPU-M are manually designed models targeting EdgeTPU on the
evolved search space described in Section 3.2.2. However, manually crafted
models can be inferior in identifying efficient models with lower inference
time, as model latency is a complex function of accelerator resources,
compute-to-memory ratio, and neural model characteristics (operation types and
tensor shapes). Without an active interaction with an accurate accelerator
performance simulator, it is extremely hard to predict model latency given an
arbitrary neural model architecture and a target accelerator configuration.

Demonstrated in Figure 8, NAHAS consistently outperforms related work by
around 1% ImageNet top-1 accuracy at all latency targets. With the same
accuracy, NAHAS on average reduces inference latency by around 20%. We
empirically found that a IBN-only search space is good for identifying small,
low-latency models (e.g. NAHAS-XS, NAHAS-S) while the proposed evolved search
space is good for identifying larger, more accurate models (e.g. NAHAS-M,
NAHAS-L, NAHAS-XL) with more relaxed latency targets. Manually crafted models
can be very efficient at a particular latency target (e.g. Manual-EdgeTPU-S)
however, they can be less efficient when comparing the entire latency spectrum
(e.g. Manual-EdgeTPU-M), which manifests the importance of automation.

Apart from inference latency, accelerator energy consumption is another
important factor when co-optimizing neural architectures and accelerators.
Mobile processors have a stringent power budget due to limited battery life,
therefore, maximizing model accuracy while meeting a power budget is another
useful way to co-design neural architectures and accelerators. We evaluated a
energy-driven NAHAS, where energy is a product of power and latency, comparing
to related mobile-efficient NAS and Manual-EdgeTPU. We found that our method
can improve energy efficiency by up to 2x comparing to related work at the
same ImageNet top-1 accuracy, as shown in Figure 1.

Model | Top-1 Acc. | Latency in ms (Ratio-to-best) | Energy in mJ (Ratio-to-best)
---|---|---|---
EfficientNet-B0 (Tan & Le, 2019) wo SE/Swish | 74.7% | 0.35 (1.17x) | 1.00 (1.64x)
MobileNetV2 (Sandler et al., 2018) | 74.4% | 0.30 (1.00x) | 0.70 (1.15x)
MnasNet-B1 (Tan et al., 2019) | 74.5% | 0.41 (1.37x) | 0.88 (1.44x)
ProxylessNAS (Cai et al., 2019) | 74.8% | 0.42 (1.40x) | 0.98 (1.61x)
Manual-EdgeTPU-small | 76.2% | 0.42 (1.40x) | 1.78 (2.91x)
IBN-only fixed accelerator | 74.6% | 0.38 (1.27x) | 0.82 (1.34x)
IBN-only NAHAS multi-trial | 74.9% | 0.30 | 0.75 (1.23x)
IBN-only NAHAS oneshot | 76.5% | 0.35 (1.17x) | 0.61
EfficientNet-B1 (Tan & Le, 2019) wo SE/Swish | 76.9% | 0.51 (1.04x) | 1.50 (1.53x)
MnasNet-D1 (Tan et al., 2019) | 75.1% | 0.55 (1.12x) | 1.75 (1.78x)
Fixed accelerator multi-trial w fused-IBN | 75.3% | 0.52 (1.06x) | 1.32 (1.35x)
IBN-only NAHAS multi-trial | 77.4% | 0.52 (1.06x) | 1.50 (1.53x)
NAHAS multi-trial w fused-IBN | 77.9% | 0.51 (1.04x) | 1.12 (1.14x)
IBN-only NANAS oneshot | 76.8% | 0.49 | 0.98
EfficientNet-B3 (Tan & Le, 2019) wo SE/Swish | 78.8% | 0.72 (1.12x) | 2.28 (1.56x)
Manual-EdgeTPU-medium | 77.2% | 0.62 | 2.72 (1.86x)
MobilenetV3 w SE | 76.8% | 1.44 (2.32x) | 4.00 (2.74x)
Fixed accelerator multi-trial w fused-IBN | 78.2% | 0.74 (1.19x) | 1.75 (1.20x)
NAHAS multi-trial w fused-IBN | 79.5% | 0.72 (1.12x) | 1.46

Table 3:  Comparison on accuracy v.s. latency with previous approaches. Models
are grouped into different regimes and sorted by accuracy. On the fast latency
regime, “NAHAS oneshot” has the best performance, while on the high accuracy
regime, “NAHAS multi-trial” achieves the best performance.

Model | mIOU Acc. | Latency in ms (Ratio-to-best) | Energy in mj (Ratio-to-best)
---|---|---|---
EfficientNet-B0 wo SE/Swish | 73.8% | 3.29 (1.08x) | 6.71 (1.08x)
EfficientNet-B1 wo SE/Swish | 72.8% | 3.66 (1.20x) | 8.04 (1.29x)
EfficientNet-B2 wo SE/Swish | 72.6% | 3.71 (1.21x) | 8.48 (1.36x)
Manual-EdgeTPU-S | 71.2% | 4.15 (1.36x) | 13.31 (2.14x)
Manual-EdgeTPU-M | 74.4% | 4.75 (1.55x) | 17.68 (2.84x)
IBN-only NAHAS multi-trial | 73.6% | 3.06 | 6.54 (1.05x)
NAHAS multi-trial w fused-IBN |  74.8% | 4.17 (1.36x) | 6.23

Table 4:  Comparison on Cityscapes Segmentation Dataset (Cordts et al., 2016).

###  4.4 Detailed Comparison with SoTA

We compare NAHAS with mobile-efficient models (including NAS-based and
manually crafted) and NAHAS variants on fixed accelerator NAHAS (NAS only with
default accelerator configuration), multi-trial NAHAS (without parameter
sharing) and oneshot NAHAS (with parameter sharing). We categorize the models
into three: small, medium, and large. We use a latency target of 0.3ms, 0.5ms,
and 0.7ms and use a energy target of 0.7mJ, 1.0mJ, and 1.5mJ for these three
categories. According to Table 3, NAHAS significantly outperforms related work
on all three perspectives (accuracy, latency, and energy), except for latency
when comparing to medium sized manually crafted Manual-EdgeTPU. However,
Manual-EdgeTPU is much worse (1.82x) in energy than NAHAS. We conclude that:

  * •

Adapting hardware accelerator configurations helps identify more pareto
efficient models in terms of model accuracy, inference latency, and energy
consumption, which cannot be identified by a fixed hardware NAS.

  * •

The NAS search space also needs to be adapted to create more edge efficient
neural architectures. For example, removing SE and Swish non linearity
significantly improves inference latency and energy while not impairing
accuracy significantly. Some expensive ops such as fused-IBN, can be more
effective when running on edge accelerators, for certain tensor shapes.

  * •

Different neural architectures with different performance targets lead to
drastically different accelerator configurations. For example, larger models
require a higher memory-to-compute ratio in the accelerator design. NAHAS
identifies edge accelerator configurations with larger number of processing
elements (PE) and smaller memory capacity, compared to the baseline edge
accelerator, for small models with very tight latency/energy target; It
identifies accelerator configurations with larger local memory and register
file size for large models and models containing fused-IBN.

  * •

Model inference time and energy consumption on the target accelerator is a
complex function of accelerator resources, chip area, neural architectures,
tensor shapes, that is intricate to capture with manual engineering. The
manually designed Manual-EdgeTPU can be competitive on one hardware
performance metric, but can be significantly worse in the other.

  * •

Oneshot search is more effective than a multi-trial search without parameter
sharing for smaller models with lower than 0.5ms latency. However, it has
limited performance when the model becomes larger with a more relaxed
latency/energy target. Constructing a super-network in a oneshot search can be
impractically too expensive when the search space is larger or the model size
becomes larger.

###  4.5 Ablation Study

Search Method: Figure 9 compares joint search with phase search. More
particularly, in a phase-based NAHAS, we start with a hardware accelerator
search on a fixed initial neural architecture in the search space
(MobileNetV2, EfficientNet-B1, and EfficientNet-B2) with a soft constraint
function described in Section 3.4, aiming to find an accelerator configuration
which is pareto-optimal in terms of latency and chip area. Then we apply a NAS
with a hard constraint function described in Section 3.4 on the selected best
accelerator configuration, aiming to identify good neural architectures that
strictly meet the hardware latency constraint. We compare Phase-NAHAS using 1x
and 2x total searched samples, compared to the NAHAS multi-trial baseline.

Figure 9: Joint Search vs. Phase-based Search.

NAHAS phase search with the same number of searched samples performs much
worse than the NAHAS multi-trial joint search. Doubling the total searched
samples thus the search time, improves the quality of results. However, we
find that the initial neural architecture creates a large variance in search
quality.

Segmentation Results: We also conduct extensive experiments on segmentation
task to verify the generalization ability of our proposed NAHAS. Specifically,
we use the popular CityScapes dataset (Cordts et al., 2016) to benchmark model
performance. We train all models from scratch with a SGD optimizer and a
cosine learning rate of 0.08 and warmup proportion of 0.01. We use a training
batch size of 64 and evaluation batch size of 32. During the NAHAS search, we
use a proxy training of 20 epochs. In the final evaluation, we train the
models for 1000 epochs.

##  5 Conclusion

We systematically studied the importance and strategies of jointly optimizing
neural architectures and hardware accelerators. Our identified NAHAS models
outperform related work by 1% in ImageNet top-1 accuracy and reduce
accelerator energy consumption by up to 2x.

## References

  * SPE (2006) Spec cpu 2006, 2006.  URL https://www.spec.org/cpu2006/.
  * TPU (2016) Tpu, 2016.  URL https://cloud.google.com/tpu/.
  * MLP (2019) Mlperf, 2019.  URL https://mlperf.org.
  * app (2020) Apple m1, 2020.  URL https://www.apple.com/mac/m1/.
  * Achararit et al. (2020) Achararit, P., Hanif, M. A., Putra, R. V. W., Shafique, M., and Hara-Azumi, Y.  Apnas: Accuracy-and-performance-aware neural architecture search for neural hardware accelerators.  _IEEE Access_ , 8:165319–165334, 2020.
  * Ansel et al. (2014) Ansel, J., Kamil, S., Veeramachaneni, K., Ragan-Kelley, J., Bosboom, J., O’Reilly, U.-M., and Amarasinghe, S.  OpenTuner: An extensible framework for program autotuning.  In _PACT_ , 2014.
  * Balaprakash et al. (2016) Balaprakash, P., Tiwari, A., Wild, S. M., Carrington, L., and Hovland, P. D.  AutoMOMML: Automatic multi-objective modeling with machine learning.  In _International Conference on High Performance Computing_ , pp. 219–239. Springer, 2016.
  * Bender et al. (2020) Bender, G., Liu, H., Chen, B., Chu, G., Cheng, S., Kindermans, P.-J., and Le, Q. V.  Can weight sharing outperform random architecture search? an investigation with tunas.  In _CVPR_ , 2020.
  * Bender et al. (2018) Bender, G. M., jan Kindermans, P., Zoph, B., Vasudevan, V., and Le, Q.  Understanding and simplifying one-shot architecture search.  In _ICML_ , 2018.
  * Brock et al. (2018) Brock, A., Lim, T., Ritchie, J. M., and Weston, N.  Smash: One-shot model architecture search through hypernetworks.  In _ICLR_ , 2018.
  * Cai et al. (2019) Cai, H., Zhu, L., and Han, S.  Proxylessnas: Direct neural architecture search on target task and hardware.  In _ICLR_ , 2019.
  * Cai et al. (2020) Cai, H., Gan, C., Wang, T., Zhang, Z., and Han, S.  Once-for-all: Train one network and specialize it for efficient deployment, 2020.
  * Chen et al. (2016) Chen, Y., Emer, J. S., and Sze, V.  Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks.  In _ISCA_ , 2016.
  * Choi et al. (2020) Choi, K., Hong, D., Yoon, H., Yu, J., Kim, Y., and Lee, J.  Dance: Differentiable accelerator/network co-exploration.  _arXiv preprint arXiv:2009.06237_ , 2020.
  * Cong et al. (2018) Cong, J., Wei, P., Yu, C. H., and Zhang, P.  Automated accelerator generation and optimization with composable, parallel and pipeline architecture.  In _DAC_ , 2018.
  * Cordts et al. (2016) Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B.  The cityscapes dataset for semantic urban scene understanding.  In _CVPR_ , 2016.
  * Dai et al. (2019) Dai, X., Zhang, P., Wu, B., Yin, H., Sun, F., Wang, Y., Dukhan, M., Hu, Y., Wu, Y., Jia, Y., Vajda, P., Uyttendaele, M., and Jha, N. K.  Chamnet: Towards efficient network design through platform-aware model adaptation.  In _CVPR_ , pp. 11398–11407, 2019.
  * Dong & Yang (2019) Dong, X. and Yang, Y.  Searching for a robust neural architecture in four gpu hours.  In _CVPR_ , 2019.
  * Elfwing et al. (2018) Elfwing, S., Uchibe, E., and Doya, K.  Sigmoid-weighted linear units for neural network function approximation in reinforcement learning.  _Neural Networks_ , 107:3–11, 2018.
  * Gupta & Akin (2020) Gupta, S. and Akin, B.  Accelerator-aware neural network design using automl.  In _MLSys On-device Intelligence Workshop_ , 2020.
  * He et al. (2018) He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., and Han, S.  Amc: Automl for model compression and acceleration on mobile devices.  In _ECCV_ , 2018.
  * Howard et al. (2019) Howard, A., Sandler, M., Chu, G., Chen, L.-C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., et al.  Searching for mobilenetv3.  In _ICCV_ , 2019.
  * Iqbal et al. (2020) Iqbal, M. S., Su, J., Kotthoff, L., and Jamshidi, P.  Flexibo: Cost-aware multi-objective optimization of deep neural networks.  _arXiv preprint arXiv:2001.06588_ , 2020.
  * Jiang et al. (2020a) Jiang, W., Yang, L., Dasgupta, S., Hu, J., and Shi, Y.  Standing on the shoulders of giants: Hardware and neural architecture co-search with hot start.  _arXiv preprint arXiv:2007.09087_ , 2020a.
  * Jiang et al. (2020b) Jiang, W., Yang, L., Sha, E. H.-M., Zhuge, Q., Gu, S., Dasgupta, S., Shi, Y., and Hu, J.  Hardware/software co-exploration of neural architectures.  _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , 2020b.
  * Koeplinger et al. (2018) Koeplinger, D., Feldman, M., Prabhakar, R., Zhang, Y., Hadjis, S., Fiszel, R., Zhao, T., Nardi, L., Pedram, A., Kozyrakis, C., et al.  Spatial: A language and compiler for application accelerators.  In _PLDI_ , 2018.
  * Kwon et al. (2018) Kwon, K., Amid, A., Gholami, A., Wu, B., Asanovic, K., and Keutzer, K.  Co-design of deep neural nets and neural net accelerators for embedded vision applications.  In _DAC_ , 2018.
  * Liu et al. (2019) Liu, H., Simonyan, K., and Yang, Y.  Darts: Differentiable architecture search, 2019.
  * Nardi et al. (2019) Nardi, L., Koeplinger, D., and Olukotun, K.  Practical design space exploration.  In _MASCOTS_ , 2019.
  * Parsa et al. (2020) Parsa, M., Mitchell, J. P., Schuman, C. D., Patton, R. M., Potok, T. E., and Roy, K.  Bayesian multi-objective hyperparameter optimization for accurate, fast, and efficient neural network accelerator design.  _Frontiers in Neuroscience_ , 14:667, 2020.
  * Peng et al. (2021) Peng, D., Dong, X., Real, E., Tan, M., Lu, Y., Liu, H., Bender, G., Kraft, A., Liang, C., and Le, Q. V.  Pyglove: Symbolic programming for automated machine learning, 2021.
  * Pham et al. (2018) Pham, H., Guan, M. Y., Zoph, B., Le, Q. V., and Dean, J.  Efficient neural architecture search via parameter sharing, 2018.
  * Ramachandran et al. (2017) Ramachandran, P., Zoph, B., and Le, Q. V.  Searching for activation functions.  _arXiv preprint arXiv:1710.05941_ , 2017.
  * Sandler et al. (2018) Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In _CVPR_ , 2018.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Sun et al. (2020) Sun, Y., Xue, B., Zhang, M., Yen, G. G., and Lv, J.  Automatically designing cnn architectures using the genetic algorithm for image classification.  _IEEE Transactions on Cybernetics_ , 2020.
  * Tan & Le (2019) Tan, M. and Le, Q. V.  Efficientnet: Rethinking model scaling for convolutional neural networks.  _ICML_ , 2019.
  * Tan et al. (2019) Tan, M., Chen, B., Pang, R., Vasudevan, V., and Le, Q. V.  Mnasnet: Platform-aware neural architecture search for mobile.  _CVPR_ , 2019.
  * Wu et al. (2019) Wu, B., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., Jia, Y., and Keutzer, K.  Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search.  In _CVPR_ , 2019.
  * Xiong et al. (2020) Xiong, Y., Liu, H., Gupta, S., Akin, B., Bender, G., Kindermans, P.-J., Tan, M., Singh, V., and Chen, B.  Mobiledets: Searching for object detection architectures for mobile accelerators.  _arXiv preprint arXiv:2004.14525_ , 2020.
  * Yang (2019) Yang, A.  Deep learning training at scale spring crest deep learning accelerator (intel® nervana™ nnp-t).  In _Proceedings of the Hot Chips_ , 2019.
  * Yang et al. (2020) Yang, L., Yan, Z., Li, M., Kwon, H., Lai, L., Krishna, T., Chandra, V., Jiang, W., and Shi, Y.  Co-exploration of neural architectures and heterogeneous asic accelerator designs targeting multiple tasks, 2020.
  * Yang et al. (2018) Yang, T.-J., Howard, A., Chen, B., Zhang, X., Go, A., Sandler, M., Sze, V., and Adam, H.  Netadapt: Platform-aware neural network adaptation for mobile applications.  In _ECCV_ , 2018.
  * Yang et al. (2019) Yang, Y., Huang, Q., Wu, B., Zhang, T., Ma, L., Gambardella, G., Blott, M., Lavagno, L., Vissers, K., Wawrzynek, J., et al.  Synetgy: Algorithm-hardware co-design for convnet accelerators on embedded fpgas.  In _FPGA_ , 2019.
  * Zhou et al. (2018) Zhou, Y., Ebrahimi, S., Arık, S. Ö., Yu, H., Liu, H., and Diamos, G.  Resource-efficient neural architect, 2018.
  * Zoph et al. (2018) Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V.  Learning transferable architectures for scalable image recognition.  In _CVPR_ , 2018.