Enabling Hard Constraints in Differentiable Neural Network and
  Accelerator Co-Exploration
Co-exploration of an optimal neural architecture and its hardware accelerator
is an approach of rising interest which addresses the computational cost
problem, especially in low-profile systems. The large co-exploration space is
often handled by adopting the idea of differentiable neural architecture
search. However, despite the superior search efficiency of the differentiable
co-exploration, it faces a critical challenge of not being able to
systematically satisfy hard constraints such as frame rate. To handle the hard
constraint problem of differentiable co-exploration, we propose HDX, which
searches for hard-constrained solutions without compromising the global design
objectives. By manipulating the gradients in the interest of the given hard
constraint, high-quality solutions satisfying the constraint can be obtained.
Deokki Hong, Kanghyun Choi, Hye Yoon Lee, Joonsang Yu, Noseong Park, Youngsok Kim, Jinho Lee
# Enabling Hard Constraints in Differentiable Neural Network and Accelerator Co-Exploration

###### Abstract.

Co-exploration of an optimal neural architecture and its hardware accelerator is an approach of rising interest which addresses the computational cost problem, especially in low-profile systems. The large co-exploration space is often handled by adopting the idea of differentiable neural architecture search. However, despite the superior search efficiency of the differentiable co-exploration, it faces a critical challenge of not being able to systematically satisfy hard constraints such as frame rate. To handle the hard constraint problem of differentiable co-exploration, we propose HDX, which searches for hard-constrained solutions without compromising the global design objectives. By manipulating the gradients in the interest of the given hard constraint, high-quality solutions satisfying the constraint can be obtained.

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote†: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none
+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote †: copyrighted: none

+
Footnote: copyrighted: none

+
FootnoteFootnote: copyrighted: none

+
FootnoteFootnote: copyrighted: none
+
FootnoteFootnote: copyrighted: none

+
Footnote: copyrighted: none

+
FootnoteFootnote: copyrighted: none

+
FootnoteFootnote: copyrighted: none

+
Footnote: copyrighted: none
+
FootnoteFootnote: copyrighted: none

+
FootnoteFootnote: copyrighted: none

+
FootnoteFootnote: copyrighted: none

+
Footnote: copyrighted none: none

+
FootnoteFootnote: copyrighted: none

+
FootnoteFootnote: copyrighted: none

+
FootnoteFootnote: copyrighted: none

+
FootnoteFootnote: copyrighted: none

+
FootnoteFootnote: copyrighted: none

+
FootnoteFootnote: copyrighted: none

+
FootnoteFootnote: copyrighted: none

+
FootnoteFootnote: copyrighted: none none

+
FootnoteFootnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
FootnoteFootnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
FootnoteFootnote: copyrighted: none

+
Footnote: copyrighted: none
+
FootnoteFootnote: copyrighted: none

+
FootnoteFootnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
FootnoteFootnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted none:

+
FootnoteFootnote: copyrighted: none

+
FootnoteFootnote: copyrighted: none

+
Footnote: copyrighted: none

+
Footnote: copyrighted: none

+
FootnoteFootnote:
### DNN-Accelerator Co-exploration

The early work on the co-exploration utilize variants of reinforcement learning, or evolutionary algorithm to leverage its simplicity (Bahdan et al., 2015; Chen et al., 2015; Chen et al., 2016; Li et al., 2017; Li et al., 2018; Li et al., 2019). Each candidate network is trained for evaluation, while the accelerator design is analyzed for hardware efficiency. These values create rewards used by the agent to create the next candidate solution. However, they all inherit the same problem from RL-based NAS methods in which they require expensive training to evaluate each candidate solution. To worsen the matter, co-exploration requires even larger network/hardware search space than searching only for networks.

In such regard, differentiable approaches were adopted to co-exploration (Li et al., 2017). Auto-NBA (Li et al., 2018) used a differentiable accelerator search engine to build a joint-search pipeline, and DANCE (Li et al., 2019) trained auxiliary neural networks for hardware search and cost evaluation. However, none of the above properly addresses the hard constraint problem. In this work, we propose a holistic method of handling hard constraints on differentiable co-exploration.

## 3. Motivational Experiment

The most straightforward and naive way to handle hard constraints within differentiable co-exploration would be to tune the relative weight to the hardware cost. For example, below is the loss function used in differentiable co-exploration (Li et al., 2018; Li et al., 2018).

\[\mathcal{Loss}=\mathcal{Loss}_{CE}+\lambda_{Cost}Cost_{HW}, \tag{1}\]

which is designed co-optimize accuracy and hardware cost simultaneously, and \(\lambda_{Cost}\) balances the two terms1. By increasing \(\lambda_{Cost}\), one can indirectly instruct the search process to consider hardware metrics more. However, giving a larger penalty does not directly lead to reduction in the value of a constrained metric. Figure 1 plots how changing \(\lambda_{Cost}\) in Eq. (1) from 0.001 to 0.010 affects the latency/energy and the classification error for CIFAR-10 dataset. Searches were done three times for each setting and plotted with the same colors with large dots for their averages. Even though some trend is observed that depends on \(\lambda_{Cost}\), inconsistency in both direction and variance of the trajectory is more dominant.

Footnote 1: It is different from hyperparameters of typical machine learning formulation where the two terms serve toward a single objective.

Consider a scenario where a designer wants to design a neural network-accelerator architecture pair with latency under some constraint (e.g., 33.3 ms), using the conventional co-exploration methods. The designer would try searching with some initial \(\lambda_{Cost}\) and try adjusting the value over the course of multiple searches. However, such inconsistency between \(\lambda_{Cost}\) and the latency makes it extremely difficult to obtain the adequate solution, not to mention the huge time cost of performing the search numerous times. Despite the difficulties that lie in tackling a hard-constrained co-exploration problem, designing an effective strategy is necessary.

## 4. Hard-Constrained Co-Exploration

### Problem Definition

The mathematical formulation of hard-constrained differentiable co-exploration is as below:

\[\operatorname*{arg\,min}_{\alpha,\beta}(\mathcal{Loss}_{NAS}(w^{* },net(\alpha))+\lambda_{Cost}Cost_{HW}(eval(\alpha,\beta))),\] \[\text{s.t.}\;\;w^{*}=\operatorname*{arg\,min}_{w}(\mathcal{Loss} _{NAS}(w,net(\alpha))),\;\;\;t\leq T, \tag{2}\]

where \(t\) denotes the current value of constrained metric such as latency or energy, and \(T\) is the target value (e.g., 33.3 ms for latency). \(\alpha\) and \(\beta\) denote network architecture parameters and hardware accelerator configuration, respectively. \(w\) is the weights of the NAS supernet and \(net(\alpha)\) is the current dominant network architecture selected. \(eval(\alpha,\beta)\) indicates the hardware metrics evaluated for \(\alpha\) and \(\beta\). The objective of co-exploration is expressed using two distinct evaluation metrics, which are neural architecture loss (\(\mathcal{Loss}_{NAS}\)) and hardware cost (\(Cost_{HW}\)) defined from the user.

### Differentiable Co-exploration Framework

Although our main contribution is that we enable hard constraints, we explain our framework for the co-exploration since they are closely related. Figure 2 illustrates the overall architecture of the proposed method, being similar to existing methods (Li et al., 2018; Li et al., 2018). Figure 2 (a) is the network search module. This module searches for network architecture by choosing a path from the supernet. The network structure is then fed to the evaluator module.

The evaluator network \(eval()\) is the key to the differentiable co-exploration that enables the gradient to flow into the supernet, considering the relation between the hardware accelerator. It is a composition of two subnetworks: a hardware generator \(gen()\) and an estimator \(est()\). The hardware generator takes the neural architecture parameters as inputs and uses them to output the optimal hardware implementation (\(\beta\) from Eq. 2). It is jointly trained during the co-exploration so that the generator does not depend on certain cost function, and can adapt to the constraint. The estimator network outputs the hardware-related metrics by taking output of the generator and the network. It is pre-trained according to the network and the accelerator search space. For pre-training the estimator, traditional (non-differentiable) cost estimation frameworks such as MAESTRO (Li et al., 2018), Timeloop (Tlemelo et al., 2019), and Accelergy (Li et al., 2019) are used as ground truth. After pre-training, the estimator is frozen during the exploration and is only used to infer the hardware cost given a

Figure 1. A motivational experiment. In each plot, we swept the value \(\lambda_{Cost}\) 0.001 to 0.010. It is clear that the trajectory is not strictly linear to \(\lambda_{cost}\) with high variations.
network architecture. With these, we convert Eq. 2 as below:

\[\operatorname*{s.t.} w^{*}=\operatorname*{arg\,min}_{w}(\mathcal{Loss}_{NAS}(w,net( \alpha))),\] \[v^{*}=\operatorname*{arg\,min}_{w}(Cost_{HW}(est(\alpha,gen(v, \alpha)))), \tag{3}\]

where \(v\) is the weights for the hardware generator.

### Enabling Hard-Constraints with Gradient Manipulation

In addition to the differentiable co-exploration methodology, we suggest the novel idea of gradient manipulation as an effective solution to the hard constraint problem. Direct manipulation of gradients is a strategy often used in achieving multiple goals, such as in continual learning (Srivastava et al., 2015) or differential equations (Kang et al., 2016). In this paper, we present a solution to apply gradient manipulation to the co-exploration problem in the interest of satisfying hard constraints.

The diagrams on Figure 2 (b) and (c) show a high-level abstraction of our gradient manipulation method. The main idea is to artificially generate a force that can push the gradient in the direction that _agrees_ with the constraint. The conditions under which the method is applied to compute the new gradient \(g\) are defined as below:

\[g=\begin{cases}g_{\mathcal{Loss}}&,\text{if }t\leq T\\ &\text{or }t>T\wedge g_{\mathcal{Loss}}\cdot g_{Const}\geq 0,\\ m_{\alpha}+g_{\mathcal{Loss}}&,\text{otherwise}\end{cases} \tag{4}\]

\[g_{Const}=\frac{\partial\max(t-T,0)}{\partial\alpha}. \tag{5}\]

In the above equation, \(g_{\mathcal{Loss}}\) is the original gradient from the global loss function defined as

\[\mathcal{Loss}=\mathcal{Loss}_{NAS}+\lambda_{Cost}\cdot Cost_{HW}, \tag{6}\]

as in Eq. 3, and \(g_{Const}\) is the gradient of constraint loss that we define as: \(Const=max(t-T,0)\). Note that \(t\) is a function of \(\alpha\), and thus can be backpropagated to find the gradient with respect to \(\alpha\). In an ideal case where the \(t\leq T\), the constraint is already met so we do nothing. In the unfortunate case when the constraint is still not met, we calculate for the dot product of the two gradients to determine the agreement in their directions. If \(g_{\mathcal{Loss}}\cdot g_{Const}\geq 0\) (i.e., the angle between two gradients is less than 90), it means gradient descent update will contribute towards satisfying the constraint. Thus it is interpreted as an agreement in direction and the same \(g_{\mathcal{Loss}}\) is used unmodified. Figure 2 (b) depicts this scenario. However, if they disagree as illustrated in Figure 2 (c) (i.e., \(g_{\mathcal{Loss}}\cdot g_{Const}<0\)), we force the gradient to shift its direction by \(m_{\alpha}\), which is obtained from \((m_{\alpha}+g_{\mathcal{Loss}})\cdot g_{Const}\geq 0\) to guarantee decrease in target cost after gradient descent. It can be reformulated as \(m_{\alpha}\cdot g_{Const}+g_{\mathcal{Loss}}\cdot g_{Const}=\delta\) where \(\delta\geq 0\) is a small value for ensuring gradual movement towards satisfying the constraint.

For updating \(\alpha\) and \(w\), we solve for optimal \(m_{\alpha}\) with respect to \(\alpha\), which are the parameters for the network architecture. To minimize the effect of \(m_{\alpha}\) on \(g_{\mathcal{Loss}}\), we use a pseudoinverse-based solution that is known to minimize the size of \(||m_{\alpha}||_{2}^{2}\) as below:

\[m_{\alpha}^{*}=\frac{-(g_{\mathcal{Loss}}\cdot g_{Const})+\delta}{||g_{Const }||_{2}^{2}}g_{Const}. \tag{7}\]

In order to control the magnitude of the pull, we use a small multiplying factor \(p>0\) on \(\delta\). The policy for updating \(\delta\) using \(p\) is as follows: Some initial value \(\delta_{0}\) exists for \(\delta\). If the target metric fails to meet the constraint, \(\delta\) is multiplied by \(1+p\) to strengthen the pull (\(\delta^{\prime}=(1+p)\delta\)). In the other case when the constraint is satisfied, \(\delta\) is reset to its initial value (\(\delta^{\prime}=\delta_{0}\)).

Note that we also train \(v\), weights for the hardware generator using gradient descent. Thus we compute for \(m_{v}^{*}\) in the same manner, but use \(g_{Cost_{HW}}\) in place of \(g_{\mathcal{Loss}}\) for updating the generator.

Although a single constraint is already a challenging target, our method can be further generalized to accommodate multiple constraints. Now the gradient is modified only in the direction of individual constraints that do not comply. We provide a more generalized formulation:

\[g=\begin{cases}g_{\mathcal{Loss}}&,\text{if }\bigwedge_{i=1}^{n}(t_{i}\leq T _{i})\\ &\text{or }\bigvee_{i=1}^{n}(t_{i}>T_{i})\wedge g_{\mathcal{Loss}}\cdot g_{Const }\geq 0,\\ m_{\alpha}+g_{\mathcal{Loss}}&,\text{otherwise}\end{cases} \tag{8}\]

\[g_{Const}=\frac{\partial\sum_{i=1}^{n}\max(t_{i}-T_{i},0)}{\partial\alpha}. \tag{9}\]

### Implementation Details

**Hardware cost function**. In this work, we choose the inference latency, energy, and the chip area as the widely used hardware metrics. Considering all of them, a commonly used cost function is multiplying them (i.e., EDP, EDAP) as in (Gang et al., 2016; Wang et al., 2016). However, we found that the energy is usually easier to optimize for, and using such cost function unfairly favors energy-oriented designs. Therefore, we use a balanced weighted sum for the cost function as below.

\[Cost_{HW}=C_{E}Energy+C_{L}Latency+C_{A}Area. \tag{10}\]

**Estimator and Generator Network**. Following (Gang et al., 2016), we model both the estimator and generator with five-layer Multi-Layer Perceptron (MLP) with residual connections. To train the estimator, we first build a dataset by randomly sampling 10.8M network-accelerator pairs (2.95e\(-9\) % of the total search space) from our search space which are evaluated on hardware metrics using Timeloop (Timeteloop, 2016) and Accelergy (Miller et al., 2017). Using this dataset, the estimator

Figure 2. Overall structure of HDX.
is trained for 200 epochs with the batch size of 256. The weight update is done using Adam optimizer with the learning rate of 1e-4. The accuracy of the estimator was over 99% for all metrics, being powerful enough as an engine for co-exploration. The generator is randomly initialized and jointly trained with the NAS supernet. As the manipulated gradient from the hard-constraint is back-propagated, the generator learns to create accelerators that comply with the constraint on given neural network architecture.

**Search Space.** We use ProxylessNAS (He et al., 2017) as a NAS backbone with path sampling to train \(\alpha\). It consists of multiple settings of MBConv operation with kernel size (He et al., 2017; He et al., 2017; He et al., 2018) and expand ratio (He et al., 2017; He et al., 2018). The total number of layers is 18 and 21 for CIFAR-10 (He et al., 2018) and ImageNet (He et al., 2018) dataset, respectively. However, our method is orthogonal to the NAS implementation and has the flexibility to choose from any differentiable NAS algorithms, such as DARTS (Krizhevsky et al., 2014) or OFA (He et al., 2017).

We use Eyeriss (Eyeriss, 2015) as the accelerator's backbone architecture. It is composed of a two-dimensional Processing Element (PE) array where each PEs has a Multiply-Accumulate (MAC) unit attached to a register file. Therefore, hardware accelerator design space comprises PE array size from 12x8 to 20x24, register file size per PE from 16B to 256B. In addition, the search space includes dataflow of Weight-Stationary (WS) similar to (He et al., 2017), Output-Stationary (OS) similar to (He et al., 2017) and Row-Stationary (RS) similar to (Eyeriss, 2015).

## 5. Experiments

### Experimental Environment

We have conducted experiments on HDX using CIFAR-10 (He et al., 2018) and ImageNet (He et al., 2018) dataset. For all the hardware metrics (latency, energy, and chip area) reported, we have used the direct evaluation on the designed hardware from Timeloop (Tel
Figure 3 (left) and (mid) show the relation between error and latency. The colored horizontal bars represent the two latency targets we applied. It can be easily seen that all solutions found by HDX satisfy the given hard constraints regardless of the value of \(\lambda_{Cost}\). Furthermore, all solutions have the latency right below the constraint, showing that the solutions did not over-optimize for the constrained metric (latency). DANCE (DANCE, 2017) and Auto-NBA (Beng et al., 2017) were able to exploit the trade-off between hardware metrics and accuracy, but has no control over meeting the constraint. Even with soft-constraint terms, they mostly failed to obtain in-constraint solutions. Auto-NBA at a glance seems to be slightly better at meeting the constraints, but it is because its baseline method favors hardware-efficient solutions over high-accuracy ones, not because of its ability to meet the constraint, exemplified by the fact that there is no solution with high accuracy, or latency under \(16.6\,\mathrm{ms}\).

### Solution Quality Found by HDX

In this subsection, we demonstrate that HDX can 1) handle constraints from all three metrics (latency, energy, and chip area), 2) handle multiple constraints, and 3) obtain solutions of good overall quality. Figure 3 (right) plots \(Cost_{HW}\) and error together, which allows evaluating quality of the solutions in terms of Pareto-optimality. Because Figure 3 (left) and (mid) overlook the other metrics, comparing the \(Cost_{HW}\) together is required to be fair.

From the plot, it is clear that quality of solutions from HDX is better than the NAS\(\rightarrow\)HW method, and has no degradation from the existing co-exploration methods. In fact, the tightly constrained (\(16.6\mathrm{ms}\)) solutions even find better solution than those of the existing solutions in terms of Pareto-optimality.

To further study the quality of the solutions found by HDX, we have conducted another set of experiments. We selected a few solutions found from DANCE method as 'Anchor' solutions and listed them in Table 2. From those, we chose either one or all three of the hardware metrics to be fixed as the hard constraint, and performed co-explorations using HDX. Because it is guaranteed that such solution exists, a good method should be able to find a solution meeting the constraint, of at least a similar quality. As in the Section 5.3, all of the 8 cases we have examined succeeded in finding a valid solution. Furthermore, all the solutions show similar global loss values from the anchor solutions as shown in the rightmost column.

### Results from ImageNet Dataset

Table 3 shows the co-exploration results from ImageNet dataset (Dosov et al., 2017), under \(125\) ms constraint. As displayed in the table, HDX always succeeded in finding a solution within constraint where the others often failed to satisfy. Furthermore, the Top-1 error and the global loss shows that the quality of the solution found by HDX is not compromised at all, compared to DANCE or its variant.

### Sensitivity Study on Pulling Magnitude

In HDX, the only hyperparameter is \(p\) that controls the pulling magnitude. Figure 4 illustrates how the global loss and latency changes over latency-constrained (\(33.3\) ms) explorations, with varying \(p\) of \(1\)e-\(2\), \(7\)e-\(3\), and \(4\)e-\(3\).

Regardless of the value of \(p\), the curve for the constrained value shows a similar trend. At the beginning, the global loss becomes mainly optimized, while the latency stays steady. During this phase the pulling magnitude \(\delta\) (See Eq. 7) is still growing, and is not strong enough to make meaningful changes. At certain point, \(\delta\) becomes strong enough to pull the solution towards lowering latency. When the latency satisfies the constraint, global loss starts to decrease while maintaining the latency. There is no significant discrepancy between the final solution in the global loss and the latency, which shows that HDX is insensitive to the hyperparameter \(p\).

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline Index & Constrained & Lat. (ms) & E (mJ) & Area (mm\({}^{2}\)) & Error (\%) & \(Cost_{HW}\) & Loss \\ \hline \multirow{4}{*}{A} & Anchor & 69.23 & 37.00 & 2.53 & \(4.10\pm 0.16\) & 21.84 & 0.632 \\  & Latency & **43.99** & 21.79 & 2.10 & \(4.20\pm 0.07\) & 13.87 & 0.624 \\  & Energy & 51.98 & **29.18** & 2.53 & \(4.38\pm 0.17\) & 17.44 & 0.630 \\  & Chip Area & 64.00 & 34.82 & 2.53 & \(4.05\pm 0.06\) & 20.56 & 0.629 \\  & All & **63.72** & **12.09** & 1.86 & \(4.12\pm 0.18\) & 13.29 & 0.623 \\ \hline \multirow{4}{*}{B} & Anchor & 49.65 & 27.53 & 2.53 & \(4.22\pm 0.06\) & 16.67 & 0.638 \\  & Latency & **48.02** & 27.33 & 2.53 & \(4.27\pm 0.09\) & 16.41 & 0.644 \\ \cline{1-1}  & Energy & 95.02 & **24.45** & 1.89 & \(4.05\pm 0.10\) & 20.76 & 0.648 \\ \cline{1-1}  & Chip Area & 54.74 & 29.81 & 2.53 & \(4.11\pm 0.13\) & 17.96 & 0.645 \\ \cline{1-1}  & All & **41.32** & **8.59** & 1.86 & \(4.35\pm 0.05\) & 9.50 & 0.629 \\ \hline \hline \end{tabular}

* "Bold colored numbers indicate that they are under constraint of the same colored non-bold numbers.

\end{table}
Table 2. Results Showing the Quality of Solutions

Figure 4. Sensitivity to \(p\) on HDX. The red lines represents latency constraint at 33.3 ms.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline Method & in-const\({}^{2}\) & Lat. (ms) & Error (\%) & CostHW & Loss \\ \hline \multirow{2}{*}{NAS\(\rightarrow\)HW} & ✗ & 242.92 & 24.84 & 46.29 & 1.99 \\  & ✗ & 135.39 & 28.83 & 24.26 & 2.17 \\ \hline \multirow{2}{*}{DANCE} & ✗ & 165.98 & 25.46 & 28.37 & 2.04 \\  & ✓ & 125.18 & 25.28 & 25.32 & 2.09 \\ \hline \multirow{2}{*}{DANCE;Soft Coast.} & ✗ & 188.69 & 25.69 & 33.14 & 1.99 \\  & ✓ & 105.65 & 26.37 & 25.58 & 2.08 \\ \hline \multirow{2}{*}{**HDX (Proposed)**} & ✓ & 92.06 & 25.01 & 24.48 & 1.98 \\ \cline{1-1}  & ✓ & 112.11 & 25.20 & 22.63 & 2.00 \\ \hline \hline \end{tabular}
\end{table}
Table 3. Experimental Results for ImageNet

Figure 3. Co-exploration results. (left) and (mid) represent the latency and (right) represent the hardware cost. Colored marks are methods with constraints of the same color.
### Analysis on the Searched Solutions

Fig. 5 visualizes the network and accelerator searched for 60 fps (a) and 30 fps (b) constraints. For the found design pair (a), the design contains relatively smaller kernels, more layers, and a powerful accelerator. To meet a tight constraint while maintaining accuracy, the network has small kernels, mainly of 3\(\times\)3. Using smaller kernels quadratically reduces the number of multiplications. Therefore, decreasing the kernel size and increasing number of layers is a good choice for reducing inference latency. Looking at the accelerator design, it has relatively large PE array (16\(\times\)16) to achieve low latency. It takes weight stationary (WS) dataflow, which is known to have low latency. In addition, there are some kernels with high channel expand ratio in the network. WS exploits channel parallelism for fast execution, and thus has advantage over the found network.

On the other hand, in the design for 30 fps (b), the design settles at a solution that can optimize the energy consumption while satisfying the constraint. The design uses larger kernels in the network and row stationary (RS) dataflow in the accelerator. RS is known to have good energy efficiency (Cheng et al., 2019), and exploits parallelism from spatial dimensions of kernel and the activation. Thus, having larger kernels have advantages on RS dataflow. To reduce the energy consumption, the design has fewer PEs (12\(\times\)8), larger RFs to save off-chip access energy, and fewer layers in the network.

## 6. Conclusion

In this paper, we proposed HDX, a hard-constrained differentiable co-exploration method. By conditionally applying gradient manipulation that moves the solution towards meeting the constraints, hard constraints can be reliably satisfied with high-quality solutions. We believe this proposal would ease the development of DNN based systems by a significant amount.

###### Acknowledgements.

 This work has been supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (2022R1C1C1008131, 2022R1C1C1011307), and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (2020-0-01361, Artificial Intelligence Graduate School Program (Yonsei University)).
