# HAO: Hardware-aware Neural Architecture Optimization for Efficient Inference

Zhen Dong∗, Yizhao Gao∗, Qijing Huang, John Wawrzynek, Hayden K.H. So, Kurt
Keutzer
University of California, Berkeley
The University of Hong Kong
<EMAIL_ADDRESS><EMAIL_ADDRESS>∗
Equal contribution.

###### Abstract

Automatic algorithm-hardware co-design for DNN has shown great success in
improving the performance of DNNs on FPGAs. However, this process remains
challenging due to the intractable search space of neural network
architectures and hardware accelerator implementation. Differing from existing
hardware-aware neural architecture search (NAS) algorithms that rely solely on
the expensive learning-based approaches, our work incorporates integer
programming into the search algorithm to prune the design space. Given a set
of hardware resource constraints, our integer programming formulation directly
outputs the optimal accelerator configuration for mapping a DNN subgraph that
minimizes latency. We use an accuracy predictor for different DNN subgraphs
with different quantization schemes and generate accuracy-latency pareto
frontiers. With low computational cost, our algorithm can generate quantized
networks that achieve state-of-the-art accuracy and hardware performance on
Xilinx Zynq (ZU3EG) FPGA for image classification on ImageNet dataset. The
solution searched by our algorithm achieves 72.5% top-1 accuracy on ImageNet
at framerate 50, which is 60% faster than MnasNet [37] and 135% faster than
FBNet [43] with comparable accuracy.

##  I Introduction

Modern complex deep neural networks (DNNs) are able to achieve unparalleled
accuracy in a wide range of applications at the expense of their much
increased computing requirements. To successfully deploy these computationally
demanding DNNs in resource-constrained edge systems such as an embedded FPGA,
while maintaining real-time performance, system designers must therefore
engage in difficult tradeoffs between model accuracy and implementation
efficiency. There are three common approaches to improve the efficiency of DNN
and the corresponding hardware design for edge deployment: 1) quantize the
model to achieve efficient representations of DNNs, 2) select less compute-
intensive operations and design efficient DNN architectures, and 3) design
specialized hardware. The three design techniques altogether form a large
design space for developing efficient DNN accelerator solutions at the edge.

Quantization [9, 19, 50, 53] is a general and effective technique that uses
low bitwidth (such as 4-bit or 8-bit) to represent the floating-point
weights/activations in neural networks. To achieve a better trade-off between
accuracy and efficiency, mixed-precision quantization was introduced to allow
different layers to have different bitwidths. Mixed-precision quantization
leads to an exponentially large search space to find the optimal bitwidths.
Prior work [44, 40, 11] adopts differentiable search, reinforcement learning,
or sensitivity analysis to tackle this problem. However, the computational
cost of these approaches is non-trivial. Besides, these works solely focus on
quantization without co-considering the neural architecture design or
efficient hardware implementation.

The second approach to achieve efficient inference is designing compact neural
network architecture. Compared to manually designing networks, neural
architecture search (NAS) algorithms [55, 26, 27, 37, 6] can automatically
find network architectures that are more accurate and efficient. However, the
NAS algorithm typically requires training sampled networks/sub-networks to get
feedback on different neural architectures, which makes NAS algorithms
computationally expensive to gain enough feedback and achieve good
performance. In practice, NAS algorithms either heuristically prune the
architectural search space or use proxy tasks to reduce the computational
cost, leading to sub-optimal DNN architectures.

The hardware design, in common practice, is performed separately with
software. Such practice can lead to sub-optimal performance because
quantization and NAS target hardware-agnostic metrics such as model size or
FLOPs. These performance proxies do not guarantee high inference speed on
different hardware designs. As an example, the quantization algorithm may
select a mixture of every bitwidth from 1 bit to 8 bit, and the NAS algorithm
may choose to jointly use convolution with different kernel and group sizes.
Though this solution can be small in model size or FLOP counts, on embedded
FPGA devices with limited resources (such as Zynq ZU3EG), supporting all these
operations at the same time is inefficient or even infeasible.

Consequently, a joint search among quantization, neural architectures, and
hardware implementation is necessary to expose the optimal configurations of
DNN and the corresponding accelerator design. Previous work [40, 44, 49]
searched quantization schemes with different hardware configurations, but left
their DNN architecture untouched. [37, 43, 6] searched for efficient neural
architectures on specific hardware platforms, but did not consider the impact
of quantization and hardware design. [21, 15, 14, 1] covered hardware design
and neural architectures in their search space but did not include
quantization. Though [20, 28, 25, 17] considered all three aspects, their
search space is limited.

In this work, we explore the joint search space of neural architecture,
quantization, and hardware design. Instead of pruning the space by heuristics,
or applying reinforcement learning or derivative-based search algorithms, we
formulate the search as an integer programming problem, so that efficient
optimization algorithms can be used to reduce computational cost. Based on our
hardware latency model and network accuracy predictor, we propose a hardware-
aware neural architecture optimization (HAO) method to generate pareto-optimal
DNN designs to run on embedded FPGAs. Our contributions are as follows:

  1. 1.

We formulate the design of neural architecture, quantization, and hardware
design jointly as an integer programming problem.

  2. 2.

We use a subgraph-based latency model for FPGAs, and we use a network accuracy
predictor to reduce the computational cost of the automatic design flow.

  3. 3.

HAO achieves state-of-the-art performance on ImageNet with Zynq ZU3EG FPGA.
Our model can achieve 72.5% Top-1 accuracy running with 50 FPS, which is 60%
faster than MnasNet and 135% faster than FBNet with comparable accuracy.

##  II Related Work

###  II-A Efficient Deep Learning

Quantization [9, 19, 50, 53, 40, 11, 7] is a practical method to achieve
efficient inference, which uses low bitwidth to represent weights and
activations in a given neural network model. Since uniformly applying ultra-
low precision quantization can cause accuracy degradation, mixed-precision
quantization [54, 40, 10] is used to recover the accuracy. Mixed-precision
quantization allows different layers in a neural network to have different
bitwidth, leading to an exponentially large search space for the optimal
bitwidth setting. [40] applies reinforcement learning to explore the space,
and [44] uses differentiable search to decrease the required search time. [10]
introduces Hessian-based sensitivity analysis to determine bitwidth, while
obtaining the Hessian information has a high computational cost.

Instead of compressing a large pre-trained model, previous work [18, 33, 29,
38] focus on directly designing compact neural network architectures that can
achieve decent accuracy with small model size or FLOPs. To avoid manual
efforts, neural architecture search (NAS) algorithms have been proposed to
automatically design pareto-optimal network architectures. Previous NAS
methods [55, 26] use a reinforcement learning agent to explore the design
space of neural architectures, which typically requires a large number of
computational resources (48,000 GPU hours). [32] applies evolutionary
algorithm to search for efficient neural architectures, which is feasible but
also costly (75,600 GPU hours). Differential search based NAS methods [27, 43,
37, 6] significantly reduce the search cost by 1) using a supernet with weight
sharing [30] and 2) applying continuous relaxation on the discrete search
space so that gradients can be used to assist searching. However,
differentiable NAS algorithms often lead to a small search space due to the
limitation of supernets, which makes it dependent on existing candidates of
good operations. They are sub-optimal if the design space is not already well-
explored.

###  II-B Hardware-aware Search

Since inference speed is dependent on characteristics of specific hardware
platforms, simply applying quantization or NAS algorithms based on proxy
metrics (model size or FLOPs) can be sub-optimal. To solve this problem, many
hardware-aware search algorithms have been introduced to seek efficient
deployment of DNNs on targeted hardware platforms. These methods [6, 40, 5,
37, 43, 34, 22, 48, 4] usually retrieve latency or energy feedback from a
given hardware platform, and search for optimal DNNs that can meet certain
application constraints. Note that the hardware design is fixed in these
methods, and thus is not in the search space.

To further improve the efficiency, in recent years, a few works have extended
the NAS framework by integrating hardware design into the search space [51,
28, 15, 14, 25, 20, 21, 46, 2, 1]. Generally, these software/hardware co-
search algorithms adopt pre-defined hardware design templates and incorporate
several high-level design hyperparameters in the search framework. In addition
to neural architectures, some works also incorporated quantization in their
search space. [28] captures the relationship between quantization bitwidth and
LUTs consumption on FPGA, and developed a NAS algorithm under the constraint
of LUTs. In [20], the authors integrate several model compression techniques
in the search framework and use quantization to reduce the latency of weight
loading. [25] proposes a uniformed differentiable search algorithm using
gumbel-softmax to sample discrete implementation hyperparameters including
quantization bitwidth.

Although previous methods consider hardware design choices, the size of
searchable space is still limited by the search algorithm efficiency and the
total computation budget. Consequently, enlarging hardware search space may
result in the shrinkage of software search space. In this work, we propose a
subgraph-based hardware latency model, together with an accuracy predictor for
neural architectures and quantization. Based on these, we are able to
formulate the software/hardware co-search as an integer programming problem,
which can be effectively optimized with a very small computational cost.

##  III Methodology

In HAO, we expose a large design space in both hardware and algorithm
configurations to accelerate DNNs. To efficiently navigate the search space,
we first apply integer programming to prune the hardware configuration space
by minimizing the latency subject to a set of hardware resource constraints.
We then narrow the DNN architecture space by adopting Monte Carlo tree search
(MCTS) [24] to minimize the quantization accuracy perturbation while
satisfying a given latency constraint. In addition, we develop an accuracy
predictor to estimate the accuracy of the DNN to further reduce the overall
feedback time for each sample. Our flow produces a pareto-optimal curve
between latency and accuracy.

###  III-A Hardware Design

We target FPGA in this work to demonstrate how co-designed hardware and DNN
fully exploit the optimization opportunities in hardware with limited
resources while achieving on-par accuracy. In this section, we model the
resource consumption and the computation latency for different types of
convolution kernels. On top of that, we formulate the overall resource
constraints and latency objectives as an integer programming problem for the
subgraph-based design, which will serve as the latency simulator in the
following DNN architecture optimization.

####  III-A1 Hardware Subgraph Template

Figure 1:  Hardware design space. The dataflow accelerator template consists
of $M$ convolution kernels that are selected from the kernel pool and
spatially mapped to hardware. The tunable design parameters include the number
of compute kernels $M$, the kernel type, filter size $K$, input and output
channel parallelization factor $PI$ and $PO$.

As shown in Fig. 1, in HAO, we adopt a subgraph-based hardware design. A
subgraph consists of several convolution kernels that are spatially mapped on
hardware, which also corresponds to the major building block of neural
architecture. For a given hardware subgraph, the possible building blocks for
neural architecture also include all the sub-layers of the subgraph since each
kernel is implemented with a skip signal to bypass its compute in hardware.
Each invocation to the accelerator computes one subgraph in the DNN
architecture. The intra-subgraph results are buffered and streamed on FPGA and
the inter-subgraph activations are communicated through DRAM.

We implement a parameterizable accelerator template in high-level synthesis
(HLS). The generated dataflow accelerator can contain M convolution kernels
chained through FIFOs to exploit pipeline-level parallelism. Each convolution
kernel can be chosen from one of the three convolutions from the kernel pool:
Conv $k\times k$, Depthwise Conv $k\times k$ [8], and Conv $1\times 1$. The
hardware implementation of each kernel typically comprises a weight buffer, a
line buffer, a MAC engine, and a quantization unit to rescale outputs.All the
computational units are implemented using integer-only arithmetics as in [19].

####  III-A2 Hardware Resource Modeling

This section describes the modeling details of different FPGA resources. We
adopt a bottom-up design flow to model the utilization of LUTs and DSPs for
low-bit multiply-accumulate (MAC) operations on FPGA. In addition, our model
derives the BRAM utilization based on data size and precisions as well as the
parallelization factors of the compute kernels. Table I lists the notations
used in this paper.

##### LUTs

Both DSPs and LUTs can be used for computation on FPGA. It is more efficient
to perform ultra low-bit computation on LUTs compared with DSPs. We use pragma
to direct the mapping of low-precision MAC operations to LUTs in HLS. To build
a precise model, we perform full logic synthesis to obtain the LUTs
consumption on low bitwidth multipliers and adders. Fig. 2 shows the LUTs
consumption on different activation and weight bitwidths ranging from 2 to 8.
We denote the LUT resource lookup function of multipliers as
$L_{M}(Q_{w},Q_{a})$ where $Q_{w}$ and $Q_{a}$ represent the bitwidth of
weights and input activations respectively. Derived from the logic synthesis
results, the LUT consumption of the adders $L_{A}(Q_{p})$ for carrying out
$Q_{p}$ bit partial sum accumulation can be expressed as
$L_{A}(Q_{p})=Q_{p}+7$.

Figure 2:  LUT usage of multipliers with different input precisions.  Figure
3:  Example mapping of two low-precision MACs $a\times w_{1}$ and $a\times
w_{2}$ onto a DSP with $27\times 18$ multiplier support. The multiplexer in
DSP can choose between self-accumulating or chaining mode.

##### DSP

The embedded DSP slice on FPGA supports the MAC operation in the following
format:

$\small P\mathrel{+}=A\times(B+C)$ (1)

In naive HLS mapping, one DSP slice is configured to support one MAC. To
improve DSP throughput for low-bit operations, we use the shift-and-pack
method in [12] to efficiently map two MACs on one DSP by leveraging the
additional pre-adder. Given the input activation $a$ and the weights $w_{1}$
and $w_{2}$ for two different output channels, as shown in Fig. 3, the packing
algorithm first sign-extends $w_{1}$ to 27 bits and left shifts $w_{2}$ by
$18$ bits. The output $P$ can be further accumulated with the partial sum or
separated into two products $P_{1}$ and $P_{2}$. This shift-and-pack method
can be applied to the situation when $w_{1}$ and $w_{2}$ are no larger than 8
bits.

##### BRAM

We assume a buffering scheme in which we fully exploit reuse opportunities.
The 18-Kb BRAMs usage $B_{w}$ for the weight buffer can be calculated as:

$\small B_{w}=\lceil N_{w}\times Q_{w}/PF/18{\rm{Kb}}\rceil\times PF$ (2)

where $N_{w}$ is the maximum number of weights to store on-chip, $Q_{w}$ is
the bitwidth of weights, and $PF$ is the BRAM partition factor of the weights
buffer. For convolution kernel with size $k>1$, we implement a line-buffer to
maximize input reuse. The number of BRAMs $B_{l}$ needed for line buffer is:

$\small B_{l}=\lceil(W\times C)_{\max}\times Q_{a}/18{\rm{Kb}}\rceil\times k$
(3)

where $(W\times C)_{max}$ is the maximum product between the size of image
width $W$ and channel $C$ over the entire network. Our line buffer
implementation merges the input width and channel dimension of the feature map
into one dimension, and $k$ rows of line buffers are allocated for $k\times k$
convolution kernel.

####  III-A3 Hardware Resource Allocation

With the resources modeling, we can further estimate the optimal resource
allocation for a hardware subgraph under the resource constraints of the
target FPGA. For full k $\times$ k Conv, given the input channel
parallelization factor $PI$ and output channel parallelization factor $PO$,
the compute engine loads $k^{2}\times PI$ inputs in parallel and computes $PO$
output partial sums. The total BRAM usage $N_{wbuf}$ for on-chip buffers is:

$\small\begin{gathered}\begin{aligned}
N_{\text{wbuf}}&=\left\\{\begin{aligned} B_{w}&+B_{l}&k>1\\\ &B_{w}&k=1\\\
\end{aligned}\right.\\\ \end{aligned}\\\ \end{gathered}$ (4)

The engine is composed of $k^{2}\times PI\times PO$ MAC units that can be
mapped to either DSPs or LUTs, incurring usage in LUTs $N_{luts}$ or DSPs
$N_{dsp}$:

$\small\begin{gathered}\begin{aligned} N_{\text{dsp}}&=k^{2}\times PI\times
PO/2\\\ \end{aligned}\\\ N_{\text{luts}}=k^{2}\times PI\times
PO\times(L_{M}(Q_{w},Q_{a})+L_{A}(Q_{p}))\\\ \end{gathered}$ (5)

For k $\times$ k Depthwise Conv where each output channel result is
corresponding to the inputs from the same channel, we use only $PO$ to denote
the channel dimension parallel factor. The $k\times k$ computation engine
takes $k^{2}\times PO$ input and computes $PO$ partial sums concurrently.
Similarly, the BRAM usage for the compute kernel is:

$\small\begin{gathered}N_{\text{wbuf}}=B_{w}+B_{l}\\\ \end{gathered}$ (6)

The LUT or DSP usage to support depthwise convolution grows linearly with the
$PO$ parallelism factor:

$\small\begin{gathered}N_{\text{dsp}}=k^{2}\times PO\\\
N_{\text{luts}}=k^{2}\times PO\times(L_{M}(Q_{w},Q_{a})+L_{A}(Q_{p}))\\\
\end{gathered}$ (7)

Regarding the Quantization unit that converts partial sum in high-precision to
quantized input for the next layer, we implement it with DSP with a
parallelization factor of $PO$. Its overall resource usage is:

$\small N_{\text{dsp}}=PO,N_{\text{sbuf}}=B_{s}\\\ $ (8)

Since we perform channel-wise quantization on weights, each output channel has
its own quantization scale. We thus set the number of buffered scales $N_{s}$
to $OC$. The calculation of $B_{s}$ is similar to $B_{w}$ in Eqn. 2. The
bitwidth of scale $Q_{s}$ ranges from 16-24 depending on the actual value
range after obtaining the integer scale using the inference scheme in [19].
The total BRAM usage $N_{\text{bram}}$ is a sum of weight buffer usage
$N_{\text{wbuf}}$ and scale buffer usage $N_{\text{sbuf}}$:

$\small N_{\text{bram}}=N_{\text{wbuf}}+N_{\text{sbuf}}\\\ $ (9)

####  III-A4 Hardware Latency Objective

TABLE I: Notations for hardware design Notation | Description |  Notation | Description
---|---|---|---
$H$ | feature map height |  $PI$ | parallelism on input channel
$W$ | feature map width |  $PO$ | parallelism on output channel
$Q$ | quantization setting |  $PF$ | array partition factor
$Q_{a}$ | activation bitwidth |  $L_{M}$ | LUTs usage of a Multiplier
$Q_{w}$ | weights bitwidth |  $L_{A}$ | LUTs usage of an Adder
$Q_{p}$ | partial sum bitwidth |  $B_{l}$ | line buffer BRAM usage
$k$ | kernel size |  $B_{w}$ | weights BRAM usage
$Lat_{\text{comp}}$ | computation latency |  $N_{w}$ | number of weights buffered
$Lat_{\text{on/off}}$ | latency of activation |  $N_{\text{dsp}}$ | total DSP usage of a kernel
| communication |  $N_{\text{bram}}$ | total BRAM usage of a kernel
$Lat_{w}$ | latency of loading weights |  $N_{\text{luts}}$ | total LUTs usage of a kernel
$S$ | hardware subgraph |  $N_{\text{wbuf}}$ | BRAM usage for weights buffer
$A$ | neural architecture |  $N_{\text{sbuf}}$ | BRAM usage for scale buffer
$M$ |  number of kernels in $S$ |  $N$ |  number of layers in $A$

Given a layer with input channel size $IC$, output channel size $OC$, input
height $H$ and width $W$, the compute latency is:

$\centering\small Lat_{\text{comp}}=\left\\{\begin{aligned} &H\times
W\times\lceil IC/PI\rceil\times\lceil OC/PO\rceil&~{}\text{if {full}}\\\
&H\times W\times\lceil IC/PO\rceil&~{}\text{if {depthwise}}\\\
\end{aligned}\right.\@add@centering$ (10)

depending on if the kernel type is full or depthwise convolution. The
communication latency for loading the activation on-chip and off-chip can be
roughly calculated as:

$\small\begin{gathered}Lat_{\text{on}}=H\times W\times IC\times Q_{a}/bw\\\
Lat_{\text{off}}=H\times W\times OC\times Q_{a}/bw\\\ \end{gathered}$ (11)

where $bw$ is the practical bandwidth of off-chip memory. Similarly, the
latency of loading weights can be estimated as:

$\centering\small Lat_{w}=\left\\{\begin{aligned} &k^{2}\times IC\times
OC\times Q_{w}/bw&~{}\text{if {full}}\\\ &k^{2}\times IC\times
Q_{w}/bw&~{}\text{if {depthwise}}\\\ \end{aligned}\right.\@add@centering$ (12)

Based on the latency model for a single layer, we can further derive the
latency of computing a subgraph. A hardware subgraph design with $M$
convolution kernels can be represented as $S=\\{K_{1},K_{2},...K_{M}\\}$ with
specific quantization bitwidths
$Q=\\{(Q_{a}^{1},Q_{w}^{1}),...,(Q_{a}^{M},Q_{w}^{M})\\}$. For a given network
architecture $A=\\{a_{1},a_{2},...a_{N}\\}$, the subgraph mapping
$\\{g_{1},...,g_{L}\\}$ can be generated using a grouping function $f_{m}$:

$\small\\{g_{1},g_{2},...,g_{L}\\}=f_{m}(\\{a_{1},a_{2},...,a_{N}\\})\\\ $
(13)

To model the overlapping of the dataflow architecture, the latency of
computing each $g_{i}$ can be approximated using the maximum latency over all
the subgraph layers. Besides, to execute each layer on hardware, the
accelerator will preload the weights to the on-chip buffer before the kernel
starts, and apply double-buffering to hide the communication overhead of the
input activations. The overall latency for computing a subgraph can be written
as:

$\displaystyle Lat(g_{i})=$
$\displaystyle\max(Lat_{\text{on}}^{i1},Lat(a_{i1}),...,Lat(a_{iM}),Lat_{\text{off}}^{iM})$
(14) $\displaystyle+\sum\nolimits_{j=1}^{M}Lat_{w}^{ij}$

Figure 4:  Illustration of HAO pipeline.

With the hardware analytical model above, we can then formulate the automatic
hardware design problem as an integer programming that minimizes the overall
latency:

min $\displaystyle\sum\nolimits_{i=1}^{L}Lat(g_{i})$ (15) s.t.
$\displaystyle\sum_{k\in S}N_{\text{dsp}}^{k}\leq T_{\text{dsp}}$
$\displaystyle\sum_{k\in S}N_{\text{luts}}^{k}\leq T_{\text{luts}}\times\beta$
$\displaystyle\sum_{k\in S}N_{\text{bram}}^{k}\leq T_{\text{bram}}$

where $T_{\text{dsp}}$, $T_{\text{luts}}$, $T_{\text{bram}}$ are the total
resources available on the target FPGA device. Note that $\beta$ is an
empirical parameter describing the percentage of total LUTs allocated for MAC
computation, which is set to 50% in our experiments. We treat this formulation
as a sub-program to the DNN design optimization which will be covered in the
next section. Given the explicitly expressed constraints and objective, we are
able to directly generate the corresponding hardware implementation that
minimizes the latency for different DNN design choices with different
quantization schemes and kernel types.

###  III-B DNN Design

Co-search of hardware-friendly neural network architectures and mixed
quantization precisions is computationally intensive and time-consuming. In
HAO, we formulate the search to an integer programming problem. In Sec. III-B1
we present our search space of neural architectures. Given a latency
constraint, we can first search feasible neural architectures and
corresponding mixed-precision bitwidth settings by applying the aforementioned
hardware latency model as well as a model quantifying the effect of
quantization perturbation. We then use an accuracy predictor to compare across
different networks and find the pareto-optimal architectures and quantization
settings among all candidates.

####  III-B1 Search Space of Neural Architectures

In HAO, we construct the neural network architectures from subgraphs with
feasible hardware mappings on FPGAs. Our subgraphs are combinations of
operations such as convolution or depthwise convolution with kernel size of
$1\times 1$ or $k\times k$ as mentioned in the previous section. Although only
one subgraph can be chosen on hardware, the possible building blocks for
neural architecture search include the sub-layers of the subgraph. This is
because each layer in the subgraph can be decided whether to bypass or not
using a skip signal in hardware.

We set no limit on the total number of subgraphs and choose the channel size
for different layers from $\\{16,32,64,128,256,512,1024\\}$. We also consider
input resolution in HAO with potential configuration from
$\\{96,128,160,192,224,256\\}$. Consequently, our search space is
significantly larger compared to the prior work [6, 27, 43, 45, 37]. For
example, in [37], the same cell configuration is repeated within every block.
A standard search setting is to use 5 blocks with 3 identical cells in each
block, and each cell, typically with 3 layers, has a sub-search space of 432,
resulting in a search space of size $432^{5}\approx 10^{13}$. In comparison,
even with a simple subgraph {1x1 convolution, 3x3 depthwise convolution},
assume the number of layers is $45$ (same as [37]), the size of search space
in HAO is $(2\times 7)^{45}\approx 10^{51}$. The large search space of HAO
makes it more likely to encompass designs with good efficiency and high
accuracy for broader deployment scenarios with various hardware and latency
constraints.

####  III-B2 Integer Programming

Given a latency constraint $Lat_{0}$, we use integer programming to obtain
feasible neural architectures and corresponding quantization settings.
Specifically, based on the aforementioned hardware simulator, inference
latency ($Lat$) is a function (denoted as $\mathbb{L}$) of neural architecture
($A$) and the quantization setting ($Q$) for subgraph. In Eqn. 16, i and j are
layer index, $N$ represents the total number of layers, and $M$ represents the
number of layers in a subgraph.

$\small\begin{gathered}Lat=\mathbb{L}(A,Q),\\\
A=\\{k_{i},H_{i},W_{i},IC_{i},OC_{i},S_{i},i\in[1,N]\\},\\\
Q=\\{Q_{a}^{j},Q_{w}^{j},j\in[1,M]\\}\\\ \end{gathered}$ (16)

In HAO, perturbation, denoted as $Pert$, is used to estimate the accuracy
degradation caused by quantization. For a given neural architecture, the
accuracy of the full-precision pretrained model is irrelevant to quantization
setting $Q$. The perturbation models the relative accuracy change to the
pertrained network among different $Q$. As shown in Eqn. 17, the perturbation
should be multiplied with a constant $\lambda$ to have the same scale as
accuracy, but this will not change relative accuracy ranking since
$PretrainedAcc$ in Eqn. 17 is a constant. As in [10], the total perturbation
$Pert$ can be estimated by summing the perturbation contributed from each
layer $Pert_{i}$. Using the norm of $\Delta W_{i}$ (the distance between the
quantized tensor and the original tensor $W_{i}$) and the trace of Hessian
matrix $H_{i}$, the $Pert_{i}$ can be calculated as follows (i is the layer
index).

$\small\begin{gathered}Acc=PretrainedAcc-\lambda Pert,\\\
Pert=\mathbb{P}(A,Q)=\sum_{i=1}^{N}{Pert_{i}},\\\
Pert_{i}=\overline{Tr}(H_{i})\cdot\|\Delta W_{i}\|_{2}^{2},\end{gathered}$
(17)

With a latency constraint $Lat_{0}$, we need to find feasible neural
architecture $A$ and then determine corresponding quantization setting $Q$ to
minimize perturbation. Note that $A$ contains integer architectural parameters
(kernel size, feature resolution, channel number, stride, etc), and $Q$
contains the bitwidths of layers in the subgraph, which are integer values
chosen from {2, 3, 4, 5, 6, 7, 8}. Therefore, the task to find $A$ and $Q$
satisfying latency constraint $Lat_{0}$ can be formulated as an integer
programming problem as shown in Eqn. 18.

$\small\begin{gathered}\underset{Q}{\text{min}}~{}\mathbb{P}(A,Q),\\\
s.t.~{}~{}\mathbb{L}(A,Q)\leq Lat_{0}\end{gathered}$ (18)

The latency constraint in Eqn. 18 can be modified to Eqn. 19 to reduce the
number of neural architecture candidates. This modification is based on the
assumption that neural architectures with higher latency tend to have more
complex structures and higher expression capability, and therefore higher
accuracy. $\alpha$ here is a hyperparameter ranging from 0 to 1. A larger
$\alpha$ can lead to a lower search cost.

$\small\begin{gathered}\alpha Lat_{0}\leq\mathbb{L}(A,Q)\leq
Lat_{0}\end{gathered}$ (19)

We apply Monte Carlo tree search (MCTS) [24] for better sample efficiency on
finding feasible neural architectures and quantization bitwidths that satisfy
Eqn. 18 and Eqn. 19. Benefiting from its online model, MCTS can dynamically
trade off exploration and exploitation, which makes MCTS hard to be trapped in
local optimum compared to other methods such as Bayesian optimization or
greedy algorithms. With the heuristic that
$\mathbb{L}(A,2bit)\leq\mathbb{L}(A,Q)\leq\mathbb{L}(A,8bit)$, we first find
$A$ that satisfies Eqn. 20 and then solve for appropriate quantization setting
$Q$. We follow the standard to set $A$ (then $Q$ in the next step) as state,
and our actions are selected from {increase/decrease channel,
increase/decrease resolution, skip/unskip a layer, add/delete a subgraph,
termination}. More details about MCTS can be found in [24, 3, 41].

$\small\begin{gathered}\alpha Lat_{0}\leq\mathbb{L}(A,8bit)\\\
\mathbb{L}(A,2bit)\leq Lat_{0}\end{gathered}$ (20)

####  III-B3 Accuracy Predictor

As discussed in Sec. III-B2, given a latency constraint $Lat_{0}$, neural
architecture candidates and corresponding quantization settings can be
obtained with different perturbation. To compare among different neural
architectures, a predictor is used to estimate the accuracy of pre-trained
models with given architectures. In HAO, we directly stack architectural
parameters of each layer together as the input vector, and then we apply a
support vector regression (SVR) model to predict the accuracy. It should be
noted that we choose SVR predictor for simplicity and better sample
efficiency, since SVR models generally require fewer data to train compared to
neural networks used in [42, 39]. To quickly train the predictor, we collect
{architecture, accuracy} data by training 10 large neural networks from
scratch and then reusing the weights while fine-tuning them to 200 different
architectures. In our experiments, all neural networks are built by linearly
stacking subgraphs, meaning that they are generally similar to each other. To
support more complicated architectures such as DenseNet [16] or LSTMs [36], as
suggested in [42, 39], using a better strategy (such as autoencoder) for
neural architecture representation, using semi-supervised learning with
unlabelled data, and using graph convolutional networks (GCN) as the predictor
can further improve performance, with the cost of more computation resources
and time.

We use the accuracy predictor to sort candidates that satisfy the latency
constraint $Lat_{0}$. Since the accuracy predictor can be shared with
different subgraphs, we repeat the aforementioned process for all subgraphs
and select the top neural architectures and corresponding quantization
settings111In our experiments we train top 5 architectures with corresponding
quantization settings and choose the best one for a given latency constraint..
We finally train them from scratch on ImageNet and then quantize the models as
the final results of HAO.

##  IV Results

###  IV-A Simulator Performance

Figure 5:  (Top) The correlation between latency predicted by the hardware
simulator (after calibration) and the latency directly measured on FPGA.
(Bottom) The correlation between predicted accuracy and the accuracy tested on
ImageNet validation set.

In Sec. III-A, we present an analytical latency simulator that can quickly
estimate the inference latency given a DNN architecture. The optimization
algorithm in Sec. III-B2 uses the simulator to obtain quick latency feedback.

To test the effectiveness of our latency simulator, we synthesize several
accelerators for different MobileNetV2 and HAO designs. The hardware
parameters of different implementations are automatically generated by
hardware optimization in Eqn. 15. To calibrate our latency model for the
target FPGA, we first perform linear regression to fit the cycle prediction to
the hardware execution latency. We obtain a calibrated latency model
$1.27\times Lat+3.8$ and use it for our latency prediction. Then for different
accelerator implementations, we obtain the latency pairs from our simulator
and the real hardware execution and plot them in Fig. 5. We observe a strong
linear relationship ($r=0.998$) between the real inference latency and the
estimated latency.

In addition to the hardware latency simulator, HAO also uses an accuracy
predictor to reduce the computational cost. We show the performance of the
predictor in Fig. 5. As can be seen, for different CNN models in our search
space, the results of our accuracy predictor align well with the actual test
accuracies on ImageNet validation dataset.

###  IV-B Experimental Results

In this section, we present the accuracy and latency results of HAO on the
Ultra 96 board with a Xilinx Zynq ZU3EG FPGA. We show that HAO outperforms
manually designed solutions, as well as solutions with automatically searched
DNN architectures and quantization settings.

Fig. 6 shows the pareto frontier of HAO with respect to accuracy and latency.
MobileNetV2 [33] is a popular neural architecture manually designed for
efficient inference. The original MobileNetV2 is in floating-point format. To
achieve a fair comparison, we quantize MobileNetV2 to 8-bit weights and 8-bit
activations, and then run it on FPGA with a {1x1 convolution, 3x3 depthwise
convolution, 1x1 convolution} subgraph. We follow [33] to change the channel
width multiplier (selected from $\\{1.0,0.75,0.5,0.3\\}$) and input resolution
(selected from $\\{224,192,160,128,96\\}$) of MobileNetV2, in order to trade-
off latency and accuracy. In comparison, the neural architecture (including
input resolution) and quantization bitwidth setting are automatically selected
in HAO. As can be seen, HAO outperforms MobileNetV2 on a wide range of latency
values. HAO can achieve 72.5% top-1 accuracy with 20ms latency (50 fps), which
is more than 1% higher accuracy than MobileNetV2 while running 15% faster. In
the cases with a more strict latency constraint (for example autonomous
vehicles), HAO can still preserve 66% accuracy with only 8ms latency (125
fps). This is significantly higher than the 63% of MobileNetV2 while being
faster. Furthermore, we compare with results from MnasNet [37], which is a
hardware-aware neural architecture search method. As in Fig. 6, HAO also
outperforms MnasNet by a large margin222Part of the MnasNet pareto curve is
out of the latency range in Fig. 6. We present these extra results in Table
II..

Figure 6:  Pareto frontier for accuracy and latency. We generate pareto frontier of MobileNetV2 and MnasNet by varying width multipliers as well as the input resolution, as suggested in the references [33, 37]. As can be seen, HAO results outperform MobileNetV2 and MnasNet by a large margin on Zynq ZU3EG.  TABLE II: Performance comparison on ImageNet with prior works. | Platform | Input Resolution | Framerate(fps) | Quantization Bitwidth | Top-1 Accuracy(%)
---|---|---|---|---|---
EDD-Net-2 [25] | Zynq ZU9EG | 224 $\times$ 224 | 125.6 | W16A16 | 74.6
HotNas-Mnasnet [20] | Zynq ZU9EG | 224 $\times$ 224 | 200.4 | NA | 73.24
HotNas-ProxylessNAS [20] | Zynq ZU9EG | 224 $\times$ 224 | 205.7 | NA | 73.39
EDD-Net-3 [25] | Zynq XC7Z045 | 224 $\times$ 224 | 40.2 | W16A16 | 74.4
VGG16 [52] | Zynq XC7Z045 | 224 $\times$ 224 | 27.7 | W16A16 | 69.3
VGG-SVD [31] | Zynq XC7Z045 | 224 $\times$ 224 | 4.5 | W16A16 | 64.64
VGG16 [35] | Stratix-V | 224 $\times$ 224 | 3.8 | W8A16 | 66.58
VGG16 [13] | Zynq 7Z020 | 224 $\times$ 224 | 5.7 | W8A8 | 67.72
Dorefa [23] | Zynq 7Z020 | 224 $\times$ 224 | 106.0 | W2A2 | 46.10
Synetgy [47] | Zynq ZU3EG | 224 $\times$ 224 | 66.3 | W4A4 | 68.30
FINN-R [4] | Zynq ZU3EG | 224 $\times$ 224 | 200.0 | W1A2 | 50.30
MobileNetV2 [33] | Zynq ZU3EG | 224 $\times$ 224 | 43.5 | W8A8 | 71.40
MnasNet-A1 [37] | Zynq ZU3EG | 224 $\times$ 224 | 22.3 | W8A8 | 74.60
MnasNet-A1 [37] | Zynq ZU3EG | 192 $\times$ 192 | 27.8 | W8A8 | 73.33
MnasNet-A1-0.75 [37] | Zynq ZU3EG | 224 $\times$ 224 | 31.0 | W8A8 | 72.70
MnasNet-A1 [37] | Zynq ZU3EG | 160 $\times$ 160 | 35.8 | W8A8 | 71.35
FBNet-B [43] | Zynq ZU3EG | 224 $\times$ 224 | 24.6 | W8A8 | 73.20
FBNet-iPhoneX [43] | Zynq ZU3EG | 224 $\times$ 224 | 21.3 | W8A8 | 72.62
HAO | Zynq ZU3EG | 256 $\times$ 256 | 44.9 | W-mixed A8 | 72.68
HAO | Zynq ZU3EG | 256 $\times$ 256 | 50.0 | W-mixed A8 | 72.45
HAO | Zynq ZU3EG | 224 $\times$ 224 | 58.9 | W6A8 | 71.76
HAO | Zynq ZU3EG | 224 $\times$ 224 | 77.0 | W-mixed A8 | 70.06
HAO | Zynq ZU3EG | 192 $\times$ 192 | 93.5 | W-mixed A8 | 68.80
Figure 7:  Illustration of neural architecture and quantization setting
searched by HAO. W and A stand for weight and activation quantization
bitwidth, S is the stride of a specific convolutional layer. DW Conv stands
for depth-wise convolution.

In addition to comparing pareto-frontier performance with our own hardware
implementation, we also compare HAO with various previous work in Table II.
[31, 23, 13, 35] are manually designed solutions. [25, 20] are search-based
methods. Note that these prior works target larger FPGA boards with more
resources, and some use more complex neural architectures, 16-bit fixed-point
or floating-point precision. For a fair comparison, we further compare HAO
with [47, 4, 37, 43, 33], which have the same hardware platform (Zynq ZU3EG)
as ours333Note that [37, 43] are well-known hardware-aware search algorithms,
and we implement their searched results on Zynq ZU3EG for comparison.. For
HAO, we apply layer-wise quantization for activations and channel-wise
quantization for weights, with standard linear quantizer and static
quantization for the simplicity of deployment. As can be seen in Table II, HAO
achieves state-of-the-art performance on embedded FPGA with limited resources.
With higher top-1 accuracy (68.8% vs 68.3%), HAO solution is significantly
faster than Synetgy [47] (94fps vs 66fps), albeit Synetgy is assisted by extra
operations such as shift. Moreover, when the framerate is 50fps, HAO can
achieve 72.5% top-1 accuracy on ImageNet, which is more than 1% higher than
MnasNet-A1 (71.4%) while being 14% faster. Comparing with FBNet-iPhoneX, HAO
obtains slightly better accuracy (72.7% vs 72.6%), while having a much higher
framerate (45 vs 21). It should be noted that for different hardware platforms
or different latency constraints, previous methods need to repeat the whole
search pipeline to find appropriate solutions, while the predictor in HAO can
be shared so that no additional search cost will be required.

TABLE III: Hardware resources utilization and power LUTs | FF | DSP | BRAM | Power
---|---|---|---|---
61362(87.0%) | 55136(39.0%) | 360(100%) | 431(99.8%) | 5.5W

Table III shows the hardware resource utilization and power usage for HAO on
Zynq ZU3EG FPGA. We observe 4.3W power consumption with no workload running on
the programming logic side and 5.5W power when running the network. Besides,
we are able to utilize 100% of DSP and 87% of LUTs on the FPGA, showing the
effectiveness of our hardware resource modeling. In the optimization program
in Eqn. 15, we allocate $\beta$ percent of LUTs as computation resource to
search for optimal design parameters, which makes the LUTs utilization more
controllable. In this way, the simulator can automatically decide whether to
implement a kernel on DSP or LUTs based on the quantization setting $Q$. As a
result, we can achieve high resource utilization by leveraging the benefits of
mix-precision operations on FPGA.

In Fig. 7, we show one of the searched results by HAO. A subgraph {1x1
convolution, 3x3 depthwise convolution, 1x1 convolution} is used in this
solution. As can be seen, HAO finds that a 6-bit/7-bit mixed-precision
quantization setting is better than 8-bit uniform quantization for weights. In
general, lower bit-width means more computation units under the same resource
constraints, but it can lead to larger quantization perturbation. HAO can
balance the efficiency and perturbation, and we observe that the 8-bit
counterpart of HAO 6/7-bit result runs 5% slower with negligible accuracy
gain. Moreover, the results of HAO show that, for our implementation on Zynq
ZU3EG, solutions with solely $3\times 3$ depthwise convolution perform better
than those with a mixture of $3\times 3$ and $5\times 5$ depthwise
convolution. This is due to the fact that when using a mixture of $3\times 3$
and $5\times 5$ depthwise convolution, either $3\times 3$ or $5\times 5$
kernel will be idle when invoking the accelerator, which is a waste on
platforms with limited hardware resources.

##  V Conclusions

In this work, we propose HAO to jointly optimize the neural architecture,
quantization, and hardware design. To reduce the computation required for
evaluating different designs, we develop a subgraph-based hardware latency
model as well as an accuracy predictor for neural architectures. We formulate
the algorithm and hardware co-search as an integer programming problem, which
significantly prunes the total search space. On an embedded FPGA device, we
show that our HAO method finds the pareto-optimal designs which outperform
previous solutions on both latency and accuracy.

## Acknowledgments

This work was supported by Facebook Reality Labs, Google Cloud, Alibaba,
Samsung SAIT, by the Berkeley ADEPT Lab, Berkeley Deep Drive, the Berkeley
Wireless Research Center, by the Croucher Innovation Award, and by CONIX
Research Center.

## References

  * [1] Mohamed S Abdelfattah, Łukasz Dudziak, Thomas Chau, Royson Lee, Hyeji Kim, and Nicholas D Lane.  Best of both worlds: Automl codesign of a cnn and its hardware accelerator.  arXiv preprint arXiv:2002.05022, 2020.
  * [2] Mohamed S Abdelfattah, Lukasz Dudziak, Thomas Chau, Royson Lee, Hyeji Kim, and Nicholas D Lane.  Codesign-nas: Automatic fpga/cnn codesign using neural architecture search.  In The 2020 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pages 315–315, 2020.
  * [3] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer.  Finite-time analysis of the multiarmed bandit problem.  Machine learning, 47(2-3):235–256, 2002.
  * [4] Michaela Blott, Thomas B Preußer, Nicholas J Fraser, Giulio Gambardella, Kenneth O’brien, Yaman Umuroglu, Miriam Leeser, and Kees Vissers.  Finn-r: An end-to-end deep-learning framework for fast exploration of quantized neural networks.  ACM Transactions on Reconfigurable Technology and Systems (TRETS), 11(3):1–23, 2018.
  * [5] Han Cai, Tianzhe Wang, Zhanghao Wu, Kuan Wang, Ji Lin, and Song Han.  On-device image classification with proxyless neural architecture search and quantization-aware fine-tuning.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [6] Han Cai, Ligeng Zhu, and Song Han.  Proxylessnas: Direct neural architecture search on target task and hardware.  arXiv preprint arXiv:1812.00332, 2018.
  * [7] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer.  Zeroq: A novel zero shot quantization framework.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13169–13178, 2020.
  * [8] François Chollet.  Xception: Deep learning with depthwise separable convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251–1258, 2017.
  * [9] Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio.  Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1.  arXiv preprint arXiv:1602.02830, 2016.
  * [10] Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer.  HAWQ-V2: Hessian aware trace-weighted quantization of neural networks.  Advances in neural information processing systems, 2020.
  * [11] Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer.  Hawq: Hessian aware quantization of neural networks with mixed-precision.  In Proceedings of the IEEE International Conference on Computer Vision, pages 293–302, 2019.
  * [12] Yao Fu, Ephrem Wu, Ashish Sirasao, Sedny Attia, Kamran Khan, and Ralph Wittig.  Deep learning with int8 optimization on xilinx devices.  White Paper, 2016.
  * [13] Kaiyuan Guo, Song Han, Song Yao, Yu Wang, Yuan Xie, and Huazhong Yang.  Software-hardware codesign for efficient neural network acceleration.  IEEE Micro, 37(2):18–25, 2017.
  * [14] Cong Hao, Yao Chen, Xinheng Liu, Atif Sarwari, Daryl Sew, Ashutosh Dhar, Bryan Wu, Dongdong Fu, Jinjun Xiong, Wen-mei Hwu, et al.  Nais: Neural architecture and implementation search and its applications in autonomous driving.  arXiv preprint arXiv:1911.07446, 2019.
  * [15] Cong Hao, Xiaofan Zhang, Yuhong Li, Sitao Huang, Jinjun Xiong, Kyle Rupnow, Wen-mei Hwu, and Deming Chen.  Fpga/dnn co-design: An efficient design methodology for 1ot intelligence on the edge.  In 2019 56th ACM/IEEE Design Automation Conference (DAC), pages 1–6. IEEE, 2019.
  * [16] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  * [17] Qijing Huang, Dequan Wang, Zhen Dong, Yizhao Gao, Yaohui Cai, Tian Li, Bichen Wu, Kurt Keutzer, and John Wawrzynek.  Codenet: Efficient deployment of input-adaptive object detection on embedded fpgas.  In The 2021 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pages 206–216, 2021.
  * [18] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer.  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size.  arXiv preprint arXiv:1602.07360, 2016.
  * [19] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko.  Quantization and training of neural networks for efficient integer-arithmetic-only inference.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2704–2713, 2018.
  * [20] Weiwen Jiang, Lei Yang, Sakyasingha Dasgupta, Jingtong Hu, and Yiyu Shi.  Standing on the shoulders of giants: Hardware and neural architecture co-search with hot start.  IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 39(11):4154–4165, 2020.
  * [21] Weiwen Jiang, Lei Yang, Edwin H-M Sha, Qingfeng Zhuge, Shouzhen Gu, Sakyasingha Dasgupta, Yiyu Shi, and Jingtong Hu.  Hardware/software co-exploration of neural architectures.  IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2020.
  * [22] Weiwen Jiang, Xinyi Zhang, Edwin H-M Sha, Lei Yang, Qingfeng Zhuge, Yiyu Shi, and Jingtong Hu.  Accuracy vs. efficiency: Achieving both through fpga-implementation aware neural architecture search.  In Proceedings of the 56th Annual Design Automation Conference 2019, pages 1–6, 2019.
  * [23] Li Jiao, Cheng Luo, Wei Cao, Xuegong Zhou, and Lingli Wang.  Accelerating low bit-width convolutional neural networks with embedded fpga.  In 2017 27th International Conference on Field Programmable Logic and Applications (FPL), pages 1–4. IEEE, 2017.
  * [24] Levente Kocsis and Csaba Szepesvári.  Bandit based monte-carlo planning.  In European conference on machine learning, pages 282–293. Springer, 2006.
  * [25] Yuhong Li, Cong Hao, Xiaofan Zhang, Xinheng Liu, Yao Chen, Jinjun Xiong, Wen-mei Hwu, and Deming Chen.  Edd: Efficient differentiable dnn architecture and implementation co-search for embedded ai solutions.  arXiv preprint arXiv:2005.02563, 2020.
  * [26] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy.  Progressive neural architecture search.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 19–34, 2018.
  * [27] Hanxiao Liu, Karen Simonyan, and Yiming Yang.  Darts: Differentiable architecture search.  arXiv preprint arXiv:1806.09055, 2018.
  * [28] Qing Lu, Weiwen Jiang, Xiaowei Xu, Yiyu Shi, and Jingtong Hu.  On neural architecture search for resource-constrained hardware platforms.  arXiv preprint arXiv:1911.00105, 2019.
  * [29] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.  Shufflenet v2: Practical guidelines for efficient cnn architecture design.  In Proceedings of the European conference on computer vision (ECCV), pages 116–131, 2018.
  * [30] Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean.  Efficient neural architecture search via parameter sharing.  arXiv preprint arXiv:1802.03268, 2018.
  * [31] Jiantao Qiu, Jie Wang, Song Yao, Kaiyuan Guo, Boxun Li, Erjin Zhou, Jincheng Yu, Tianqi Tang, Ningyi Xu, Sen Song, et al.  Going deeper with embedded fpga platform for convolutional neural network.  In Proceedings of the 2016 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pages 26–35, 2016.
  * [32] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le.  Regularized evolution for image classifier architecture search.  In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4780–4789, 2019.
  * [33] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [34] Florian Scheidegger, Luca Benini, Costas Bekas, and A Cristiano I Malossi.  Constrained deep neural network architecture search for iot devices accounting for hardware calibration.  In Advances in Neural Information Processing Systems, pages 6056–6066, 2019.
  * [35] Naveen Suda, Vikas Chandra, Ganesh Dasika, Abinash Mohanty, Yufei Ma, Sarma Vrudhula, Jae-sun Seo, and Yu Cao.  Throughput-optimized opencl-based fpga accelerator for large-scale convolutional neural networks.  In Proceedings of the 2016 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pages 16–25, 2016.
  * [36] Martin Sundermeyer, Ralf Schlüter, and Hermann Ney.  Lstm neural networks for language modeling.  In Thirteenth annual conference of the international speech communication association, 2012.
  * [37] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le.  Mnasnet: Platform-aware neural architecture search for mobile.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2820–2828, 2019.
  * [38] Mingxing Tan and Quoc V Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  arXiv preprint arXiv:1905.11946, 2019.
  * [39] Yehui Tang, Yunhe Wang, Yixing Xu, Hanting Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu.  A semi-supervised assessor of neural architectures.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1810–1819, 2020.
  * [40] Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han.  Haq: Hardware-aware automated quantization with mixed precision.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8612–8620, 2019.
  * [41] Linnan Wang, Saining Xie, Teng Li, Rodrigo Fonseca, and Yuandong Tian.  Sample-efficient neural architecture search by learning action space.  arXiv preprint arXiv:1906.06832, 2019.
  * [42] Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Pieter-Jan Kindermans.  Neural predictor for neural architecture search.  In European Conference on Computer Vision, pages 660–676. Springer, 2020.
  * [43] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer.  Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10734–10742, 2019.
  * [44] Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer.  Mixed precision quantization of convnets via differentiable neural architecture search.  arXiv preprint arXiv:1812.00090, 2018.
  * [45] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong.  Pc-darts: Partial channel connections for memory-efficient differentiable architecture search.  arXiv preprint arXiv:1907.05737, 2019.
  * [46] Lei Yang, Weiwen Jiang, Weichen Liu, HM Edwin, Yiyu Shi, and Jingtong Hu.  Co-exploring neural architecture and network-on-chip design for real-time artificial intelligence.  In 2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC), pages 85–90. IEEE, 2020.
  * [47] Yifan Yang, Qijing Huang, Bichen Wu, Tianjun Zhang, Liang Ma, Giulio Gambardella, Michaela Blott, Luciano Lavagno, Kees Vissers, John Wawrzynek, et al.  Synetgy: Algorithm-hardware co-design for convnet accelerators on embedded fpgas.  In Proceedings of the 2019 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pages 23–32, 2019.
  * [48] Yang Yang, Chao Wang, Lei Gong, and Xuehai Zhou.  Fpnet: Customized convolutional neural network for fpga platforms.  In 2019 International Conference on Field-Programmable Technology (ICFPT), pages 399–402. IEEE, 2019.
  * [49] Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan, Leyuan Wang, Qijing Huang, Yida Wang, Michael W Mahoney, et al.  Hawqv3: Dyadic neural network quantization.  arXiv preprint arXiv:2011.10680, 2020.
  * [50] Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua.  Lq-nets: Learned quantization for highly accurate and compact deep neural networks.  In Proceedings of the European conference on computer vision (ECCV), pages 365–382, 2018.
  * [51] Xinyi Zhang, Weiwen Jiang, Yiyu Shi, and Jingtong Hu.  When neural architecture search meets hardware implementation: from hardware awareness to co-design.  In 2019 IEEE Computer Society Annual Symposium on VLSI (ISVLSI), pages 25–30. IEEE, 2019.
  * [52] Xiaofan Zhang, Junsong Wang, Chao Zhu, Yonghua Lin, Jinjun Xiong, Wen-mei Hwu, and Deming Chen.  Dnnbuilder: an automated tool for building high-performance dnn hardware accelerators for fpgas.  In 2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pages 1–8. IEEE, 2018.
  * [53] Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen.  Incremental network quantization: Towards lossless CNNs with low-precision weights.  International Conference on Learning Representations, 2017.
  * [54] Yiren Zhou, Seyed-Mohsen Moosavi-Dezfooli, Ngai-Man Cheung, and Pascal Frossard.  Adaptive quantization for deep neural network.  In Thirty-Second AAAI Conference on Artificial Intelligence, 2018\.
  * [55] Barret Zoph and Quoc V Le.  Neural architecture search with reinforcement learning.  arXiv preprint arXiv:1611.01578, 2016.


11institutetext: SAILAB, University of Siena
http://sailab.diism.unisi.it,
22institutetext: MAASAI, Universitè Côte d’Azur
https://team.inria.fr/maasai/team-members

# An Overview on the Web of Clinical Data

Marco Gori 1122

###### Abstract

In the last few years there has been an impressive growth of connections
between medicine and artificial intelligence (AI) that have been characterized
by the specific focus on single problems along with corresponding clinical
data. This paper proposes a new perspective in which the focus is on the
progressive accumulation of a universal repository of clinical hyperlinked
data in the spirit that gave rise to the birth of the Web. The underlining
idea is that this repository, that is referred to as the Web of Clinical Data
(WCD), will dramatically change the AI approach to medicine and its
effectivness. It is claimed that research and AI-based applications will
undergo an evolution process that will likely reinforce systematically the
solutions implemented in medical apps made available in the WCD. The
distinctive architectural feature of the WCD is that this universal repository
will be under control of clinical units and hospitals, which is claimed to be
the natural context for dealing with the critical issues of clinical data.

## Introduction

Eric J. Topol, in his popular book [7], claims that medicine will inevitably
be Schumpetered in the coming years. No matter if this excitement is
motivated, we are still missing a crucial catalyzer for strongly accelerating
the Schumpetering. At the moment, most studies are characterized by the
relentless search for unified standards to share data. What if we get rid of
any standard? It is worth mentioning that while from one side the diffusion of
clinical data is carefully controlled, there are initiatives that come from
patients motivated by the willingness of sharing their clinical data and
experiences111See e.g. https://www.patientslikeme.com/. The explosion of
social networks have in fact strongly favored the birth of communities where
people like to exchange ideas, make helpful connections, and feel supported
when they need it most. On the other hand, there are also huge efforts in the
building of health systems that can strongly benefit also from recent the
development of AI, where the emphasis has been shifted to machine learning and
processing on unstructured data. Amongst a number of attempts, the Canadian
efforts222https://www.cifar.ca/docs/default-source/ai-reports/ai4health-
report-eng-f.pdf is definitely worth mentioning.

This paper promotes the idea of building a universal repository of clinical
data to collect the health records of people, that closely reminds the spirit
of the Web. This universal collection would not represent only a truly
paradigm shift on the access to clinical data by students and experts in
medicine, but it would open the doors to the new grand challenge of building
decision support systems that operate on a universal repository on the basis
of the content of the health records, as well as with recommendations based on
the discovering of similarities. This evolution of a such a project seems to
be intimately connected with the very critical nature of clinical data. When
considering the risks, the most natural answer is to refrain from such a crazy
adventure, but we claim that the time has come to an in-depth analysis of
risks and opportunities. The emerging novel solutions that could dramatically
improve long-term developments in medicine, which can also go beyond the
borders of single countries. Concerning the accessibility we propose that the
major difference with respect to the Web, is that only the Clinical Units (CU)
can access, while patients are expected to benefit from new services.
Depending on specific initiatives from CU, the act of patient data donation
could be rewarded in different ways (e.g. information services related to the
patient clinical condition). The original spirit of the Web leads to conceive
the traditional world-wide search engine service that can itself be
immediately very important for the clinical activities. However, most
important challenges are connected with the current developments of AI, which
is expected to facilitate the development of intelligence apps. One can also
expect that those apps will conquer the degree of autonomy that make them
nearly indistinguishable from a medical assistant operating in team under the
control of human experts.

An overview is given on the WCD by focussing mostly on the technological side
of the project. Preliminary discussions on legal issues and the business
model, along with the social implications are also given.

##  1 Web of Clinical Data: The universal repository

In the last few years the impressive growth of computer-based medical services
have been mostly based on appropriate organization of clinical data. The
structured organization of huge amount of such data has been playing a
fundamental role also in many artificial intelligence based challenges to
medicine. Based on an early intuition [5], this paper gives a view on a truly
different way of collecting and organizing clinical data, which is claimed to
open the doors to an explosive evolution of the field. We assume that medical
information is stored in any multimedia document connected by hyperlinks, just
like in the Web. It’s up to AI agents to interpret the information hidden in
the repository, which is referred to as the Web of Clinical Data (WCD). Links
in the WCD are expected to establish different levels of relations. The nodes
of the graph are medical documents that can properly be linked and included in
the WCD. Some links are used for connecting medical documents of a single
patient corresponding to a certain specific clinical event, others are used
for establishing relations between documents of different patients. No matter
how the repository is created, we assume that documents do not contain any
identifiable reference to the identity of patients, which are only
characterized by the Personal Code (PC). The repository can be supplied by
anonymized data coming from clinical centers as well as from patients, who
upload their data after having signed appropriate authorizations. As a result,
the repository exhibits the very simple structure depicted in Fig. 1, where
any type of multimedia medical document of a given patient is linked to
his/her Personal Code. These documents are collected as a list for any
individual thus exhibiting a forest structure. Ideally, one would like medical
information supplied according to the structure of Fig. 1, where the anamnesis
of any patient is represented by a list of clinical events, each of them
properly organized into clinical data, diagnosis and therapeutic treatments.
The distinctive underlying assumption of the WCD is that there is no need to
provide such a structure, since it is up to the WCD Intelligent System that
continuously process the clinical data to reconstruct the structure of Fig. 1.

Figure 1: Patients can upload their own data in a truly unstructured form.
Clinical data, diagnosis, and therapeutic documents are stored as a forest
with separate data for each patient, that is characterized by an anonymous
Personal Identification code. The separation and appropriate structured
organization of this material can be created by opportune intelligent agents
that contribute to the creation of the WCD.

The identification of different clinical events might arise from the detection
of the date on the documents. For example, it could be the case that the
therapy adopted by a patient is related to a set of clinical data and to a
specific diagnosis, but there is no hyperlink amongst these information. Just
like humans, it’s up to an intelligent agent to realize that something is
missing and reconstruct the links. Clearly, the presence of the date is a
fundamental cue for linking documents of the same clinical event. However,
documents with no identifiable date can also be supplied to the WCD.
Basically, the underlying principle is that they are also precious for medical
inference, though one must consider their reduced significance due to the lack
of a temporal collocation in the anamnesis of a patient.

The role of the WCD Intelligent System goes well beyond that of reconstructing
the structure of data of simple patients. As we can seen in Fig. 2, one can
connect patients of the WCD so as to enrich the forest structure into a graph-
based structure where links arise because of regularities discovered in the
repository. For example, an intelligent agent can discover similarities
between the clinical data of two patients. This can arise from data in
different formats, ranging from text and different formats of signals.
Similarities can also arise in the diagnosis and the therapeutic treatments,
so as the original forest evolves towards a truly WCD with precious
inferential information.

Figure 2: Automatic construction of the WCD: The forest of patients grows up
by appropriate links amongst similar data. Similarities arise from both text
and image data and are discovered by intelligent agents.

##  2 Data anonymization, storage, and legal issues

The actual evolution of the WCD does require to clean a number of crucial
legal issues connected with the distribution of clinical data. Their actual
distribution is supposed to under the control of the Governing Board of the
WCD, which will be composed of doctors, scientists in related disciplines, and
layers. First of all, we need to clarify who is supposed to gain the
permission for accessing the data along with the connected purpose. The WCD is
conceived for boosting research and world-wide medical treatment in clinical
centers. As a consequence, the permission is granted to scientists and doctors
upon approval from the Governing Board of the WCD. As it will be shown in the
next section, this restriction on the accessibility poses fundamental
constraints on the computer architecture, since the underlying assumption is
that the repository can only be made available to Clinical Units (CU), which
could be hospitals, research centers whose access comes with the duty of not
to distribute data elsewhere. Basically, the intention is that of using the
same legal framework which regulates the interaction of medical staff in
sensible clinical data. Following related technical developments in the field
of bank accounting, we only make the access available independently of the
geographical position, whereas we do inherit and keep all the restrictions
concerning the already established rules for the access. Clearly, this imposes
neither restrictions on the place where data are stored nor on who provides
the storing service, which does not necessarily corresponds with granting
accessibility.

The issue of anonymization has been the subject of an in-depth investigation
in the last few years, but only a few of them make significant efforts towards
a rigorous treatment (see e.g. [3] for a very good example). Many circulating
claims on the issue of anonymization are often quite generic; namely the
inferential context is not well-defined, which facilitates restrictive claims
on the supposed risks of making clinical data widely available. Formally, the
problem of disclosing the identity from medical data seems to be generic and
ill-posed. Clearly, textual documents on the anamnesis of a patient may
contribute to discover the identity, whereas the inference from signal
biomedical signals cannot rely on cues that can somehow reveal the identity.
One can at most believe of some geographical connections that are expected to
give rise to a certain disease. Notice that giving a collection of medical
data under the conditions of the WCD, the identity disclosure is only possible
provided that the databases of clinical centers are violated, which is in fact
an event that can happen regardless of the WCD initiative.

Basically, the WCD initiative faces the ill-position of the issue of sensible
data by adopting a strategy where anonymization and authorization is granted
by design. The act of autonomously data uploading does exhibit the stronger
patient’s intention to make data available. It is worth mentioning that the
interaction with clinical centers follows the other way around, namely data
are downloaded by the patients. Hence, we can think of keeping the same
communication channel where this time patients upload their own data, which
can come from different sources. This information flow with clinical centers
very much resembles the one they establish with their own banks. The level of
security and the type of information exchange shares in fact the same needs.
As patients decide to upload their data to their own preferred clinical center
(more are possible), we start creating a repository which is promoted to the
WCD, only after a further check, whose details are defined by the WCD
Governing Board. The bottom line is that the WCD does subscribe the current
scenario for the access to clinical data, the only difference being that of
making the data widely available to world-wide authorized clinical and
research units. In this framework, we expect that most claims on sensible
information might vanish thanks to the governing philosophy of the WCD.

##  3 Overall architecture

The access requirements pointed out in the previous sections for the WCD open
an interesting computer architecture problem that is mostly connected with the
storing of a growing repository that must be distributed worldwide. At the
same time, each CU is also expected to access the whole WCD repository. In
order to satisfy these requirement the architectural solution proposed for the
WCD is the one depicted in Fig. 3.

Figure 3: Overall architecture of the WCD, where Clinical Units act as macro-
nodes. Each such unit is used for storing patients’ data in a truly
unstructured form (Patient Local WCD - PLWCD), while additional meta-data are
created in the (Semantic-Local WCD - SLWCD). The WCD is also stored in an
encrypted form onto a cloud system.

_WCD uniform resource locator_
The repository can be given a formal description at local level thanks to the
Uniform Resource Locator of any document, that can be constructed by following
the same principles used in the Web. For example, the reference to the ECG
diagnosis in a certain CU could be wcd.ao-siena.1492.ecg-careggi-
june-2020.dia, whose structure is based on separate fields which provide
information on the patient, the specific document and where it was archived.
This is sketched below

${\tt wcd}.\underbrace{\tt ao-siena}_{\tt ClinicalUnit}.\underbrace{1492}_{\tt
patient\ no}.\underbrace{\tt ecg-careggi-june-2020}_{\tt
filename}.\underbrace{\tt dia}_{\tt diagnosis}$

where we easily induce the general structure of the URL. Basically, the ECG
document named ecg-careggi-june-2020.dia is a diagnostic document (extension
dia) corresponding to patient 1492 of Clinical Unit ao-siena of the wcd. This
document is supposed to be directly accessible within the CU where it is
stored for any type of processing. However, since this document is also
supposed to be accessed outside the CU where it has been generated, one needs
to make it available world-wide.
There are a number of studies on architectural solutions for related problems
that inspire the solution to this general framework (see e.g. [4, 1, 9, 6]).
Interestingly, related studies [8] have also been carried out in the context
of medical applications. The solution prospected in Fig. 3 is naturally
following classic search engine technology where there is a global repository.
Interestingly, this analogy comes with the fundamental difference that data
are encrypted and, consequently, not accessible within the global Web
repository.

_Search engine primitive_
The basic idea is that of caching of the documents on a Cloud System (CS) in
an encrypted format. Basically, any document created in a CU is encrypted and
backed up to the CS. This allows us to carry out a classic information
retrieval search [4] where all documents produced in the CU are uploaded to
the CS along with their encrypted keywords. This allows the CS to construct
the inverted indexes to be used for searching. Classic encryption solutions
can be used for handling the security of the interaction between the CUs and
the CS. Classic searching primitives based on propositional calculus are
supposed to be used, whereas the major assumption is not to assign the CS
higher-level services. The underlying assumption is in fact that of moving to
the CUs any solution based on AI agents. It is worth mentioning the this
horizontal keyword-based search service is in fact the first one which is
offered on top of the WCD. Interestingly, the search primitives are also of
crucial importance as a building layer for any service app.

_Service Apps_
Any service in the WCD is supposed to be given by apps running in CU computer
servers. Any computation does require to select the documents that are
pertinent to the service, so as they are temporarily downloaded in the CU
server. Basically, these apps are expected to operate on a proper selection of
information from the WCD that is expected to be useful for their objective.
The structure of any service is sketched as

1. create the primary app cache 2. q $\leftarrow$ QueryFromService(input) 3. RetrievedDocCollection $\leftarrow$ WCD-retrieve(q) 4. run BodyApp(RetrievedDocCollection,input) 5. free(RetrievedDocCollection)

First, the app may need to create its own cache (primary cache) from the WCD
to optimize the performance. This step may also be omitted, so as all the
processing is based only on the input being process. The second step consists
of formulating the searching query on the basis of the input to the app. Then,
as the query is fed to the QueryFromService, pertinent documents are retrieved
and stored in the secondary cache RetrievedDocCollection. These locally
retrieved collection, along with the primary cache is used for carrying out
the task assigned to the app. Basically, the actual processing is carried out
by BodyApp on the second argument ${\tt input}$ by exploiting the local cache.

######  Example 3.1.

Suppose we want to discover patients whose clinical data are similar to those
of a given patient. The reference for inspecting the similarity is defined by
document input=ecg-careggi-june-2020.dia, which is in fact a document under
diagnosis. This document comes with a number of attached metadata, like for
instance, the specific type of diagnosis (ECG signal). The document, along
with its attached metadata represent the input of the service, which is
expected to compose the query q automatically by using QueryFromService. As
the query is obtained, it is used to collect RetrievedDocCollection and,
finally, the body of specific part of the app BodyApp is used to discover
which documents are similar.

##  4 WCD enrichment apps

Patients and staff from the CUs are generally expected to upload clinical data
in the general unstructured form of a multi-media document. Clearly, it could
be the case that single patients directly provide information in a truly
structured way according to Fig. 1. Whenever this does not happen, the purpose
of WCD enrichment apps is that of creating the missing structured
representation.

Figure 4: Service and Enrichment apps in the WCD. For a service app to operate
on the WCD the connection with the cached info in the EWCD is required.

The first important enrichment task is that of segmenting patients’ clinical
events. The most informative cue comes from the eventual presence of the date
in the document. As it is available, the task of its detection is a well-posed
problem either for textual or image-based documents. In both cases the
extraction of the data could be quite a complex problem. However, in both
cases we are in front of classic problems of pattern recognition that have
been the subject of massive investigation (see e.g. [2], for early studies).
It is worth mentioning that whenever the date is either missing or badly
recognized, the segmentation of patients’ event can also rely on different
cues. For example, there are cases in which a certain drug is prescribed upon
a corresponding diagnosis, so as two documents can be placed in the same
medical event. Clearly, this task can become very sophisticated and very well
represents the type of challenges that are open with the WCD.

Other important enrichment tasks involve the separation between clinic,
diagnostic, and therapeutic data. This is essentially a document
classification task. While there is a huge literature for attacking this
problem the specific context of the WCD provide a number of additional cues to
solve it successfully. Hence enrichment apps contribute to create the
structured view of the forest indicated in Fig. 1. Moreover, the successive
discovery of links between different patients of the same CU depicted in Fig.
2 is another tasks of enrichment apps. This problem is basically one of
discovering the similarities between multimedia documents, which has also been
the subject of a massive investigation In addition to links between documents
of the same type, we can find links between text and documents, a task which
clearly needs the additional step of extracting textual description from
images. Overall, enrichment apps create the WCD locally to each CU. However,
their task goes beyond the discovery of local links. In order to generate
global links upon discovery of relationships between documents of different
CUs, enrichment apps rely on the software architecture described in Section 3.
In particular, the discussion of Example 3.1 concerning the discovery of
global similarities for a given document provides an insight on how to attack
the problem. Clearly, once the similarities have been found, we need to update
the index of the WCD by its enrichment with the discovered links. While this
is carried out in the CUs, the inverted indexes in the CS must be updated
according. Notice that the process of constructing the WCD by this similarity
inspection needs to be frequently carried out because of the continuous update
of documents on the CUs.

##  5 WCD service apps and medical assistants

The creation of the WCD along with the associated high-level meta-data
associated with the documents open the doors to the development of medical
service apps ranging from the field of diagnosis to that of therapeutic
treatment. The underlying philosophy in the development of service apps is
that they do not simply operate on a specific CU, but on the WCD. Depending on
the dimension of the CU, the apps might have different configurations, but the
are conceived for working at global level.

Figure 5: Service apps onion skin view. The lower level only relies on the
processing of data from the SLWCD. In a middle layers, apps operate by relying
on the computation of other apps. Finally, in the last layer, medical
assistant operate autonomously on the basis of all the info available in the
lower levels.

Because of the need to make them available on the WCD, service apps are
expected to run on popular operating systems. As we can see in Fig. 5, service
apps can operate at different level of abstraction. As the process conquer a
certain degree of autonomy, service apps become a sort of medical assistant
which can support decisions under the control of human experts. The long term
view of the WCD is that the progressive construction of service apps is
expected to contribute to medical treatment on specific patients, but also to
the permanent creation of metadata that are expected to provide an important
support for future medical medical decisions.

##  6 Conclusions

The described framework of the WCD suggests the development of a stable
marriage between medicine and artificial Intelligence. While there is a
benefit in the systematic developments of service apps thanks to the word-wide
competition for medicine, a much better benchmarking context is created for
research in artificial intelligence. Basically, it looks like the WCD can act
as a fundamental catalyzer for both the disciplines. The current focus on
specific benchmarks is expected to be translated into a sort of permanent AI
challenge in medicine, that will constantly refer to the WCD.

The philosophy behind the WCD is that of increasing the level of medical
treatment by strongly promoting the CUs, which turn out to be the nodes of the
WCD. As such, they are expected to control the world-wide repository of
clinical data and will likely realize that there is a crucial benefit of
assuming this role. The WCD will make available best medical practices
worldwide, a service that has an enormous value and that might also be paid
off. Pharma companies might be interested in advertisement on the WCD, which
could the source of huge resources especially for best CUs. Simple metrics
could in fact measure the number of access to documents uploaded to different
CUs, so as to distribute money from advertisement. This will definitely bless
the principle that in the WCD framework doctors carry out two different
specific roles: First, their primary role is that of providing the appropriate
medical assistance to single patients. Second, their successful treatments
will be useful for future worldwide medical decisions. Finally, the conception
of the WCD comes with the wish that its development will be primarily useful
for the medical support to poor countries.

## Acknowledgments

We thank Alessandro Rossi, Stefano Melacci, Sandro Bartolini (University of
Siena), Arturo Chiti (Humanitas, Milan), and Paolo Traverso (FBK, Trento) for
insightful discussions.

## References

  * [1] Pedro Geraldo M. R. Alves and Diego F. Aranha.  A framework for searching encrypted databases.  J. Internet Serv. Appl., 9(1):1:1–1:18, 2018.
  * [2] Heidi Bjering Stratti.  Optimal operators in digital image processing.  PhD thesis, 2008.
  * [3] Khaled El Emam and Luk Arbuckle.  Anonymizing Health Data: Case Studies and Methods to Get You Started.  O?Reilly Media, Inc., 1st edition, 2013.
  * [4] Z. Fu, K. Ren, J. Shu, X. Sun, and F. Huang.  Enabling personalized search over encrypted outsourced data with efficiency improvement.  IEEE Transactions on Parallel and Distributed Systems, 27(9):2546–2559, 2016.
  * [5] M. Gori, Campiani G, Rossi A, and Setacci C.  The web of clinical data.  Journal of Cardiovascular Surgery, 23:717–718, 2014.
  * [6] Yusheng Jiang, Tamotsu Noguchi, Nobuyuki Kanno, Yoshiko Yasumura, Takuya Suzuki, Yu Ishimaki, and Hayato Yamana.  A privacy-preserving query system using fully homomorphic encryption with real-world implementation for medicine-side effect search.  In Proceedings of the 21st International Conference on Information Integration and Web-Based Applications and Services, iiWAS2019, page 63?72, New York, NY, USA, 2019. Association for Computing Machinery.
  * [7] Eric Topol.  The Creative Destruction of Medicine: How the Digital Revolution Will Create Better Health Care.  Basic Books, 2011.
  * [8] Lei Xu, Chungen Xu, Joseph K. Liu, Cong Zuo, and Peng Zhang.  Building a dynamic searchable encrypted medical database for multi-client.  Inf. Sci., 527:394–405, 2020.
  * [9] Steven Zittrower and Cliff Changchun Zou.  Encrypted phrase searching in the cloud.  In 2012 IEEE Global Communications Conference, GLOBECOM 2012, Anaheim, CA, USA, December 3-7, 2012, pages 764–770. IEEE, 2012.