AIBench Scenario: Scenario-distilling AI Benchmarking

This paper has been accepted by The 30th International Conference on Parallel
Architectures and Compilation Techniques (PACT 2021)

BenchCouncil: International Open Benchmark Council
Chinese Academy of Sciences
Beijing, China
https://www.benchcouncil.org/aibench/scenario/

Technical Report No. BenchCouncil-AIBench-Scenario-2021-1

August, 2021

###### Abstract

Modern real-world application scenarios like Internet services consist of a
diversity of AI and non-AI modules with huge code sizes and long and
complicated execution paths, which raises serious benchmarking or evaluating
challenges. Using AI components or micro benchmarks alone can lead to error-
prone conclusions. This paper presents a methodology to attack the above
challenge. We formalize a real-world application scenario as a Directed
Acyclic Graph-based model and propose the rules to distill it into a
permutation of essential AI and non-AI tasks, which we call a scenario
benchmark. Together with seventeen industry partners, we extract nine typical
scenario benchmarks. We design and implement an extensible, configurable, and
flexible benchmark framework. We implement two Internet service AI scenario
benchmarks based on the framework as proxies to two real-world application
scenarios.

We consider scenario, component, and micro benchmarks as three indispensable
parts for evaluating. Our evaluation shows the advantage of our methodology
against using component or micro AI benchmarks alone. The specifications,
source code 111Zenodo: https://doi.org/10.5281/zenodo.5158715
GitHub: https://github.com/BenchCouncil/aibench_scenario, testbed, and results
are publicly available from https://www.benchcouncil.org/aibench/scenario/.

##  1 Introduction

Modern real-world application scenarios like Internet services are complex,
which raises serious benchmarking and evaluating challenges. First, they often
adopt a microservice-based architecture consisting of a diversity of AI and
non-AI modules with long and complicated execution paths across different
datacenters. Using AI micro or component benchmarks alone can lead to error-
prone conclusions.

For Internet services, the overall system tail latency deteriorates even
hundreds of times than that of a single AI component. For example, for text
classification–an AI component, the gap reaches up to 180x (Section 5.2.1). If
we evaluate a new AI model or accelerator with text classification, it reduces
the component tail latency. However, it contributes little to the overall
system tail latency. Note that tail latency represents the tail performance of
a small fraction of requests, e.g., 99th percentile latency, which is
significant since these small fractions may exceed the median latency by
orders of magnitude [1].

Second, they dwarf the traditional workloads in terms of code sizes and
deployment scales. For example, the traditional desktop workloads like data
compression [2], image manipulation [2], are about one hundred thousand lines
of code, running on a single node. The Web server workloads [3] are hundreds
of thousands of lines of code, running on a small-scale cluster, i.e., dozens
of nodes. However, the modern AI stacks (e.g., Spark [4], TensorFlow [5])
alone are more than millions of lines of code, and the real-world workloads
often run on a large-scale cluster, i.e., tens of thousands of nodes [6]. It
is almost impossible to evaluate a real-world application scenario directly.
Even it is possible, due to their huge code sizes, it will raise the fairness
challenge in assuring the benchmark implementations’ equivalence across
different systems and the repeatability challenge in measurement errors.

Third, it is prohibitively costly for porting a real-world application
scenario to an innovative system or architecture [7, 8]. Especially, the real-
world data sets, workloads, and even AI models are treated as first-class
confidential issues by their owners [9, 10].

While profiling has been applied to various aspects of benchmarking [11, 12],
we are not aware of evidence demonstrating it can holistically evaluate a
large and complex system. For example, for Internet services, a few publicly
available performance models or insights observed through profiling [9, 10,
13, 14] do not work across different systems or architectures. As there are no
publicly available benchmarks, the state-of-the-art and state-of-the-practice
are advanced only by the research staff within various service providers,
which poses a considerable obstacle for our communities towards developing an
open and mature research field.

This paper proposes a gray-box methodology that accesses no code but needs the
industry’s design input (Fig. 1), to attack the above challenge. We formalize
a real-world application scenario as a Directed Acyclic Graph-based model (in
short, DAG model) and propose six rules to distill it into a permutation of
essential AI and non-AI tasks as a scenario benchmark. We call it a scenario-
distilling methodology. Our methodology is to identify the critical path and
primary modules of a real-world scenario since they consume the most system
resources and are the core focuses for system design and optimization. We will
extend to white-box (apply to industry users who can access the code) and
black-box (without input) methodologies in the future. In cooperation with
seventeen industry partners, we extract nine vital scenario benchmarks. We
design and implement a reusing benchmark framework, including the AI and non-
AI component libraries, data input, online inference, offline training, and
deployment tool modules. Based on the framework, we implement two scenario
benchmarks–E-commerce Search Intelligence and Online Translation Intelligence.
With respect to its counterpart real-world application scenario, each scenario
benchmark reduces the complexity by one or two orders of magnitude. It is
easier to achieve the latter’s efficient implementation to avoid misleading
evaluations using a suboptimal implementation. For a real-world application
scenario, the complex software evolution also hinders high-performance
implementation.

We consider scenario, component, and micro benchmarks as three indispensable
parts other than using component or micro benchmark alone. A scenario
benchmark lets the software and hardware designers obtain the overall system
performance and determine the critical path’s primary components. Furthermore,
the system and architecture researchers use the AI component benchmarks that
occupy the critical path to evaluate the AI model performance and quality
targets. Finally, the code developers can use microbenchmarks obtained through
profiling scenario benchmarks and component benchmarks to optimize the hotspot
functions. They bridge a considerable gap from real-world application scenario
deployment to simulator-based architecture research. The community can tune
the system, architecture, and algorithm innovations on a scenario benchmark
without disclosing real-world application scenarios’ confidentiality. The
developer can test the designs on a scenario benchmark or obtain the trace of
a scenario benchmark through sampling, a component benchmark, or a
microbenchmark and feed them into simulators for earlier validation. AIBench
Scenario can use SimPoint [15] or Pin [16].

Figure 1: Scenario-distilling Benchmarking.

Our evaluations demonstrate how to drill down from a scenario benchmark into
the component benchmarks and zoom in on the microbenchmarks’ hotspot
functions. The evaluations on two CPU clusters and one GPU cluster show the
advantage of our benchmarks against MLPerf and TailBench. We have several
observations as follows: (1) The performance characteristic of only one AI
component varies from that of a scenario benchmark with a permutation of the
previous AI component and other AI and non-AI components. The benchmark user
must understand the implication of benchmarking using only one AI component
like MLPerf in the context of a scenario benchmark. In serving the same
requests for the online services, different AI components incur different
latency. Some AI components contribute little to the overall system
performance. (2) The overall system tail latency deteriorates dozens of times
or even hundreds of times over a single AI component, which we cannot predict
using two state-of-the-art statistical models [17, 18]. (3) We argue that on
AI benchmarking, systems can benefit from analyzing both online services and
offline AI training scenarios. Internet service architects generally attempt
to balance the tradeoffs among service quality, input data dimensions, AI
model complexity, accuracy, and model update intervals.

We organize the rest of this paper as follows. Section 2 explains the
motivation. Section 3 summarizes the related work. Section 4 presents how to
distill and implement scenario benchmarks. Section 5 performs the evaluation.
Section 6 concludes.

##  2 Motivation

###  2.1 Using component benchmarks alone may lead to error-prone conclusions.

Modern Internet services process millions of user queries daily; thus, the
tail latency is important in terms of user experience [17]. However, a
microservice-based architecture can contain various AI and non-AI modules, and
consequently forms long and complicated execution paths. Previous AI
benchmarking work provides a few micro or component benchmarks and fails to
model an industry-scale application scenario’s overall critical paths.

The overall system latency indicates that of the entire execution path,
including AI and non-AI components. Our experiments in Section 5.2.1 show the
overall system tail latency deteriorates dozens or even hundreds of times than
a single component. The gap ranges from 2.2x – 180x between text
classification and recommendation, respectively. When we evaluate a new AI
model or accelerator using text classification, suppose it reduces the
component tail latency. However, it still contributes little to the overall
system tail latency.

Another case study is as follows. Model pruning has been shown to be effective
in improving inference speed [19]. Evaluating with image-to-text a component
of the Translation Intelligence scenario benchmark, a model-pruning
implementation [19] decreases the model size from 31MB to 21MB. It reduces the
99th percentile latency by 51% from 4214 to 2081 milliseconds compared to the
original model. The accuracy loss is 1%, from 88% decreasing to 87%, which may
be considered acceptable considering the vast inference speedup. However,
under a scenario benchmark, the overall 99th percentile latency nearly remains
the same (about 0.2% reduction), with the same accuracy loss, which may be
considered unacceptable as it reduces the service accuracy while providing no
non-negligible computational benefit. Thus, scenario benchmarking is
essential.

###  2.2 Can a Statistical Model Predict the Overall System Tail Latency?

The tail latency of many components has been studied by using various
profiling-based statistical models. Recent work [20] proposes an algorithm to
predict the tail latency of database query systems. However, for complex
application scenarios like Internet services, the overall system tail latency
prediction is not fully addressed.

In Section 5.2.3, we use two state-of-the-art queueing-based statistical
models to estimate the overall system latency and tail latency: a simple
queueing model [17] and a sophisticated queueing network model [18]. We find
that the gaps between the actual average latency/tail latency and the
theoretical ones are large through the experiments using two scenario
benchmarks. For example, for E-commerce Intelligence, the average latency gap
is 8.6x and 4.9x using the simple queueing model and the queueing network
model, respectively. The 99th percentile latency gap is 3.3x using the simple
queueing model. Due to the non-superposition property of the overall system
tail latency–being not equal to the sum of each component–the 99th percentile
latency is hard to be predicted accurately using a queueing network model
[18], so we do not report the 99th percentile latency gap using the queueing
network model.

###  2.3 Why Offline AI Training is also Essential in Scenario Benchmarking

As our many industry partners have witnessed, they have to update the AI
models in a real-time manner using an AI model for an online service. For
example, one E-commence giant demands that the AI models are updated every one
hour, and the updated model will bring in the award of about 3% click-through
rate and millions of profits. In Section 5.3, our evaluation shows offline
training should be included in scenario benchmarking, as it is essential to
balance the tradeoffs among model update intervals, training overheads, and
accuracy improvements.

##  3 Related Work

Several AI benchmark suites only provide component benchmarks, which may lead
to error-prone conclusions. Fathom [21] provides eight deep learning component
benchmarks implemented with TensorFlow, targeting six AI tasks. DAWNBench [22]
is a component benchmark suite, which firstly pays attention to end-to-end
performance–the training time to achieve state-of-the-art accuracy. MLPerf
[23, 24, 25] is an ML component benchmark suite targeting seven AI tasks and
eight AI models for training, and six AI models for inference. Although MLPerf
provides the inference for server mode, the server mode only contains one
component and fails to cover a real-world industry-scale scenario’s critical
path. Our experiments illustrated in Section 2.1 also find that for complex
scenarios, containing the execution path with only one component without the
entire execution path is insufficient and may even lead to error-prone
conclusions. MLPerf Tiny [26] is a deep learning benchmark for embedded and
ultra-low-power devices. TBD Suite [27] is a component benchmark suite for DNN
training, with eight neural network models for six AI tasks.

Several AI benchmark suites only provide microbenchmarks, ignoring the quality
target in benchmarking. DeepBench [28] consists of four operations involved in
training deep neural networks. DNNMark [29] is a GPU benchmark suite that
consists of a collection of deep neural network primitives.

Additionally, Sirius [30] is an end-to-end IPA web-service application that
receives voice and image queries and responds with natural language. TailBench
[31] is a benchmark suite that consists of eight latency-critical workloads.
CloudSuite [32] provides eight workloads for benchmarking cloud services.
MLModelScope [33] proposes a distributed platform for scalable model
evaluation. DeathStarBench [34] is a benchmark suite for microservice,
including five workloads. They find a single poorly-configured microservice or
slow server can degrade end-to-end latency by several orders of magnitude [34,
35].

AIBench distills and abstracts real-world application scenarios into AI
Scenario, Training, Inference, Micro and Synthetic Benchmarks across
Datacenter, HPC, IoT, and Edge [36]. This paper focuses on scenario benchmark,
others include AIBench Training [37], HPC AI500 [38], Edge AIBench [39, 40],
and AIoTBench [41].

##  4 The Design and Implementation

Collaborating with the seventeen industry partners whose domains include
search engine, e-commerce, social network, a news feed, video, etc., we
distill their products or services into nine representative scenarios, as
summarized in Table 1. After identifying their primary components, we design
and implement a reusing benchmark framework. Finally, we implement two
scenario benchmarks on the reusing framework: E-commerce Search Intelligence
(in short, E-commerce Intelligence) and Online Translation Intelligence
(Translation Intelligence).

Subsection 4.1 demonstrates how to distill a scenario into a high-level
scenario benchmark specification. Subsection 4.2 explains the general steps to
implement a scenario benchmark on the reusing benchmark framework and the
implementation details of these two scenario benchmarks.

Table 1: Nine Representative Scenarios Extracted from the Seventeen Industry Partners. Application Scenario |  Involved AI Task |  Involved Non-AI Task |  Data |  Metrics |  Model Update Frequency
---|---|---|---|---|---
E-commerce Search Intelligence |  Text/Image Classification; Learning to Rank; Recommendation |  Query Parsing, Database Operation, Indexing |  User Data, Product Data, Query Data |  Precision, Recall, Latency |  High
Online Translation Intelligence |  Text-to-Text Translation; Speech Recognition; Image-to-Text |  Query Parsing, Audio/Image Preprocessing |  Text, Audio, Image |  Accuracy, Latency |  Low
Content-based Image Retrieval |  Object Detection; Classification; Spatial Transformer; Image-to-Text |  Query Parsing, Indexing |  Image |  Precision, Recall, Latency |  High
Web Searching |  Text Summarization; Learning to Rank; Recommendation |  Query Parsing, Indexing, Crawler |  Product Data, Query Data |  Precision, Recall, Latency |  High
Facial Authentication and Payment |  Face Embedding; 3D Face Recognition; |  Encryption |  Face Image |  Accuracy, Latency |  Low
News Feed |  Recommendation |  Database Operation, Basic Statistics, Filter |  Text |  Precision, Recall |  High
Live Streaming |  Image Generation; Image-to-Image |  Video Codec, Video Capture |  Image |  Latency |  Low
Video Services |  Image Compression; Video Prediction |  Video Codec |  Video |  Accuracy, Latency |  Low
Online Gaming |  3D Object Reconstruction; Image Generation; Image-to-Image |  Rendering |  Image |  Latency |  Low

###  4.1 The Distilling Methodology

We formalize a real-world application scenario as a DAG model, consisting of a
series of tasks with dependencies. Fig. 2(a) shows the DAG model of the real-
world E-commerce Intelligence derived from one Internet service giant. The
formalization requires the domain knowledge, needing input from the primary
industry partners. Our partners accept it as it does not expose confidential
real-world workloads, data sets, and AI models. After the formalization, the
high-level scenario specification is stated in a paper-and-pencil approach and
reasonably divorced from individual implementations [8, 42].

We distill the scenario’s DAG model into a permutation of essential AI and
non-AI tasks and consider it a high-level scenario benchmark specification.
Fig. 2 shows the DAG models before and after distilling of E-commerce
Intelligence. After distilling, we can reduce the complexity of the scenarios
by one or two orders of magnitude. Fig. 3 shows the DAG model of Translation
Intelligence after distilling. Due to the page limitations, we only describe
how we distill this scenario using text. To capture the essential parts of the
original DAG, we propose six distilling rules.

Figure 2: The DAG Models of E-commerce Intelligence Before and After
Distilling.

  * •

R1. We will keep only one essential branch among the DAG branches that have
similar processing logic.

_E-commerce Intelligence._ In the Recommender module, personalized
recommendation, product recommendation, and advertising (AD) have similar
processing logic though they consider different attributes and having other
purposes. For example, personalized and product recommendations think user
attributes and product attributes, respectively; recommendation cares about
the user experience, while AD mainly concerns the click-through rate of
advertised products. Here we only keep personalized recommendation as one
essential branch.

_Translation Intelligence._ In the Text Translator module, the translation
microservices for different languages use a similar AI model while different
training data. We only preserve the translation from English to Deutsch.

  * •

R2. We prune the DAG branch that occupies the smallest fraction that less than
1% in the whole scenario.

_E-commerce Intelligence._ Within Recommender, there are three branches
according to the query type—image, audio, and text. The audio query occupies a
minuscule fraction and less than 1%, so we prune it.

_Translation Intelligence._ In the translation scenario, the fractions of
text, image, and audio branches are more than 1% according to our industry
partner, so we reserve all three branches.

  * •

R3. We remove the components that implement the auxiliary end-user functions.

_E-commerce Intelligence._ E-commerce Intelligence focuses on the product
searching scenario, so we remove the OLAP analyzer that analyzes the product
ordering and the supporting systems such as monitoring and logging.

_Translation Intelligence._ We also remove the supporting systems for tracing,
monitoring, logging, etc.

  * •

R4. For each AI component, we provide only one state-of-the-art or state-of-
the-practice model for simplicity.

_E-commerce Intelligence._ A scenario usually provides multiple AI models,
which different businesses share. For simplicity, we only reserve one state-
of-the-art or state-of-the-practice model for each AI component like Ranker.

_Translation Intelligence._ We provide one state-of-the-art or state-of-the-
practice model for its three AI components.

  * •

R5. We combine the successive steps involved in similar components.

_E-commerce Intelligence._ We combine the entity identification and category
classifier within Recommender, as they both include image classifier and text
classifier.

_Translation Intelligence._ We combine the Text Translator, since all the
three branches, i.e., the image, text, and audio, involve in text-to-text
translation.

  * •

R6. After performing R1 to R5, we will remove the components related to the
pruned ones.

_E-commerce Intelligence._ After pruning the AD branch in the recommender, we
remove the AD cluster group in Searcher and AD ranking in Ranker accordingly.
Also, we remove the deduplication component in Search Planner since the AD
cluster group is removed, and Searcher will return no repeated items. Besides,
Indexer only builds the product indexes since Searcher mainly uses the product
attributes to search the product item.

_Translation Intelligence._ After removing the translation microservices for
the other languages, the related components are removed.

Figure 3: The Distilled DAG Model of Translation Intelligence.

###  4.2 How to Implement a Scenario Benchmark?

####  4.2.1 The Summary of Representative AI Tasks

To cover a broad spectrum of AI Tasks, we thoroughly analyze the real-world
application scenarios shown in Table 1. We summarize the essential AI tasks
and define the AI component specifications. Each AI component specifies an AI
task description, a reference AI model, datasets, evaluation metrics, and
state-of-the-art quality target [25]. In total, we identify sixteen
representative AI tasks illustrated as follows. For each AI task, we implement
the AI component benchmarks on both TensorFlow [5] and PyTorch [43].

_Classification_ is to extract different thematic classes within the input
data like an image [44] or text file [45]. _Image Generation_ is an
unsupervised learning problem to mimic the distribution of data and generate
images [46]. _Text-to-Text Translation_ needs to translate a text from one
language to another [47]. _Image-to-Text_ is to extract the text information
from an image, which is a process of optical character recognition (OCR) [48]
or generate the description of an image automatically [49]. _Image-to-Image_
is to convert an image from one representation to another one [50], e.g.,
season change. _Speech Recognition_ is to recognize and translate a spoken
language into text [51]. _Face Embedding_ is to transform a facial image into
a vector in an embedding space [52]. _3D Face Recognition_ is to recognize the
3D facial information from multiple images from different angles [53]. _Object
Detection_ is to detect the objects within an image [54]. _Recommendation_ is
to provide recommendations [55]. _Video Prediction_ is to predict the future
video frames through predicting previous frames transformation [56]. _Image
Compression_ is to compress the images and reduce the redundancy [57]. _3D
Object Reconstruction_ is to predict and reconstruct 3D objects [58]. _Text
Summarization_ is to generate a text summary [59]. _Spatial Transformer_ is to
perform spatial transformations [60] like stretching. _Learning to Rank_ is to
learn the attributes of a searched content and rank the scores for the results
[61].

Moreover, we profile all implemented component benchmarks and drill down into
frequently appearing and time-consuming units of computation. Each of them
constitutes a microbenchmark, which is easily portable to a new architecture
and system and hence beneficial for fine-grained profiling and tuning.

####  4.2.2 The Reusing Framework

Figure 4: The Reusing Framework.

As shown in Fig. 4, the reusing framework provides loosely coupled modules
that are easily configured. It includes the data input, offline training,
online inference, non-AI libraries, and deployment tool modules. Based on the
reusing framework, we can quickly implement a scenario benchmark. Since there
are many other scenarios that we do not investigate, e.g., machine programming
and event prediction, the framework may not suit all the scenarios. We will
explore more scenarios and involved AI tasks to achieve broader coverage.

The data input module is responsible for feeding data into the other modules.
It collects representative real-world data sets from the authoritative public
websites and our industry partners after anonymization. We design the data
schema to maintain the real-world data characteristics, alleviating the
confidential issues. Our framework covers a broad spectrum of data types and
sources, integrates various open-source data storage systems, and supports
large-scale data generation and deployment for partial data [62].

We provide offline training and online inference modules to implement a
scenario benchmark. First, the offline training module chooses one or more
component benchmarks by specifying the input data and execution parameters
like the batch size. Then the offline training module trains a model and
provides the trained model to the online inference module. The online
inference module loads the trained model onto the serving system, i.e.,
TensorFlow Serving [63]. The non-AI library module provides non-AI computation
and database access, including query parsing, database operations, image and
audio preprocessing, indexing, crawler, encryption, basic statistics, filter,
video codec, video capture, rendering. For a complex application scenario, the
online inference, non-AI libraries, and offline training modules constitute an
overall critical path together.

The framework provides the deployment tools that contain two automated
deployment templates using Ansible [64] and Kubernetes [65] to support large-
scale cluster deployments. The Ansible templates support scalable deployment
on physical or virtual machines, while the Kubernetes templates are used to
deploy on a container cluster. A configuration file needs to be specified for
installation and deployment, including the module parameters like input data
and the cluster parameters like nodes, memory, and network information. A user
does not need to know how to install and run each module through the
deployment tools.

####  4.2.3 Design and Implementation of Scenario Benchmarks

Based on the reusing framework, we implement two scenario
benchmarks—E-commerce Intelligence and Translation Intelligence. They model
the complete use cases of a realistic E-commerce search and a real-world
online translation scenario augmented with AI.

The implementation steps are as follows.

(a) First, we choose the essential AI and non-AI component benchmarks and
corresponding input data from the reusing framework according to Table 1.

(b) Next, the permutation of component benchmarks is specified by the scenario
benchmark specification. The permutations of E-commerce Intelligence and
Translation Intelligence are shown in Fig. 2(b) and Fig. 3, respectively. They
consist of four subsystems: Online Server, Offline Analyzer, Query Generator,
and Data Storage. We will provide the permutation specifications of the other
seven scenarios listed in Table 1.

(c) Then, we choose the deployment tools, i.e., Ansible and Kubernetes, and
modify the reusing framework’s deployment template. The deployment template
includes the module-related configurations, i.e., input data, execution
parameters, non-AI libraries, and cluster-related configurations, i.e., node,
memory, and network information. Note that we have provided the default
templates on our open-source website for both Ansible and Kubernetes. Users
can change the configurations according to their environments.

(d) Finally, we train the AI models of the selected AI component benchmarks
using the offline training module and transfer the trained models to the
online inference module. Note that we also provide pre-trained models for
direct use.

Implementation of E-commerce Intelligence. Query generator simulates
concurrent users and sends queries to Online Server based on a specific
configuration. A query item provides either a text or an image to reflect the
different search habits. The configuration designates the parameters like
concurrency, query arriving rate, distribution, user thinking time, and ratios
of text and image items. The configurations simulate different query
characteristics and satisfy multiple generation strategies. Query Generator is
based on JMeter [66].

Online Server receives query requests and performs searching and
recommendation, integrating AI inference, consisting of four modules.

_Search Planner_ receives and forwards the queries, using the Spring Boot
framework [67].

_Recommender_ is to analyze query items and provide a personalized
recommendation, according to the user information obtained from User Database.
It first conducts query preprocessing and then predicts a query item belongs
to which category based on two classification models—FastText [45] and
ResNet50 [44]. FastText is used to classify a text query, while ResNet50 [44]
is used to classify an image query. The Recommender module uses a deep neural
network, proposed for e-commerce tasks [68], to provide personalized
recommendations. We use TensorFlow Serving [63] to provide text
classification, image classification, and online recommendation services.

We deploy _Searcher_ on three different clusters on default, which follows an
industry-scale deployment. The clusters hold the inverted indexes of product
information in the memory to guarantee high concurrency and low latency.
According to the click-through rate and purchase rate, we classify the
products into three categories according to popularity: high, medium, and low,
and the proportion of data volume is 15%, 50%, and 50%, respectively. Note
that the high-popularity category is a subset of the medium-popularity one. We
store the indexes of products with different popularity in the other clusters.

The cluster containing high-popularity product data has more nodes and more
backups to guarantee searching efficiency. In comparison, the cluster
containing low-popularity has the least number of nodes and backups. We use
Elasticsearch [69] to set up and manage Searcher deploying on the three
clusters.

_Ranker_ uses the weight returned by _Recommender_ as initial weight and ranks
the scores of products through a personalized L2R neural network [68]. Ranker
uses TensorFlow Serving [63] to implement product ranking.

Offline Analyzer performs job scheduling, AI model training, and updating. Job
Scheduler includes batch and streaming-like processing, which updates the
models every few hours or seconds accordingly. AI Offline Trainer trains
learning models of chosen components or performs real-time model updates for
online inference.

Implementation of Translation Intelligence. Query generator simulates
concurrent users and sends either a text, an image, or audio to Online Server.
The configuration information is the same as that of E-commerce Intelligence.

Online Server contains four modules. _Search Planner_ is the entrance of the
Online Server, which also uses the Spring Boot framework [67].

_Image Converter_ receives an image query and extracts the text information
within the image. It first performs image preprocessing using a BASE64 image
encoder since a RESTful API requires encoding binary inputs as Base64 [70].
Then it converts the image into text using the Image-to-Text component—optical
character recognition (OCR) [48].

_Audio Converter_ receives an audio query and recognizes the text information
within the audio. It first converts input audio into a WAV format with 16KHz
and reuses Speech Recognition [51] component.

_Text Translator_ performs translations and reuses Text-to-Text Translation
component [47]. It receives text queries directly from Search Planner or the
converted text data from Image Converter and Audio Converter.

Offline Analyzer performs job scheduling, which is similar to that of
E-commerce Intelligence, except that they use different AI component
benchmarks and update intervals.

####  4.2.4 Validation of a Scenario Benchmark

Our methodology provides a two-level mechanism to verify whether a scenario
benchmark can represent the real-world application scenario.

Specification-level Validation. On the specification level, it is easier to
validate whether the scenario benchmark specification captures the essential
parts of a scenario, as we can easily reach a consensus about the efficiency
of the distilling rules with the industry partners’ feedback. Our methodology
and distilling rules only prune the auxiliary modules like the logging system
and the non-critical path that occupies little weight; thus, from the
specification level, the distilled DAG is close to the original DAG.

Implementation-level Validation. On the implementation level, we compare the
scenario benchmark with the real-world scenario from the perspectives of both
overall system performance and single component performance. From the overall
system performance perspective, we compare the overall system latency of the
whole execution path under the same deployment. We define their deviation as
the ratio of absolute latency difference between a scenario benchmark and a
real-world one to the value of the real-world one—the lower the deviation, the
closer the benchmark to reality. From the single component performance
perspective, we replay the user queries and run the component on the same
processor. Finally, we compare the latency of the component, system (e.g., CPU
utilization), and micro-architectural (e.g., instructions per cycle) behaviors
to check whether the scenario benchmark has similar behaviors with the real-
world one.

##  5 Evaluation

This section summarizes our evaluation using the scenario benchmarks. Through
the evaluations in Subsection 5.2 and Subsection 5.3, we demonstrate the
advantages of scenario benchmarking in online services and offline training.
We gain several insights that cannot be found using MLPerf [23] and TailBench
[31]. In Subsection 5.2 and Subsection 5.4, we demonstrate how to drill down
from a scenario benchmark into component benchmarks and zoom in on the hotspot
functions of the microbenchmarks. The evaluations emphasize the necessity of
scenario benchmarking and explain what benefits it can bring for the system
and architecture communities.

###  5.1 Experiment Setup

####  5.1.1 Node Configurations

Since CPU is widely used for inference in industry [9], the online server of
E-commerce Intelligence and Translation Intelligence is deployed on a 15-node
CPU cluster and a 5-node CPU cluster, respectively. Their offline trainers are
deployed on GPUs. Our another paper [37] has evaluated the AI offline training
on other platforms like TPU.

For the 15-node CPU cluster, each node is equipped with two Xeon E5645
processors and 32 GB memory. For the 5-node CPU cluster, each node is equipped
with two Xeon E5-2620 V3 (Haswell) processors and 64 GB memory. Each processor
of the above two CPU clusters contains six physical out-of-order cores and
disables Hyper-threading. They use the same OS version: Linux Ubuntu 18.04
with the Linux kernel version 4.15.0-91-generic, and the identical software
versions: Python 3.6.8 and GCC 5.4. All the nodes are connected with a 1 Gb
Ethernet network.

Each GPU node is equipped with Nvidia Titan XP GPU. Every Titan XP owns 3840
Nvidia Cuda cores and 12 GB memory. The CUDA and Nvidia driver versions are
10.0 and 410.78, respectively.

####  5.1.2 Benchmark Deployment

Online Server Settings of E-commerce Intelligence. We deploy E-commerce
Intelligence on the 15-node CPU cluster, containing one Query Generator node
(Jmeter 5.1.1), one Search Planner node (SpringBoot 2.1.3), four Recommender
nodes (TensorFlow Serving 1.14.0 for Text Classifier, Image Classifier,
Recommendation, and Python 3.6.8 for Preprocessor), six searcher nodes
(Elasticsearch 6.5.2 [69]), one Ranker node (TensorFlow Serving 1.14.0), and
two nodes for Data Storage (Neo4j 3.5.8 for User Database, Elasticsearch 6.5.2
for Product Database).

Online Server Settings of Translation Intelligence. We deploy Translation
Intelligence on the 5-node CPU cluster, containing one Query Generator node
(Jmeter 5.1.1), one Search Planner node (SpringBoot 2.1.3), one Image
Converter node (TensorFlow Serving 1.14.0 for Image-to-Text and Python 3.6.8
for Image Preprocessing), one Audio Converter node (TensorFlow Serving 1.14.0
for Speech Recognition and Python 3.6.8 for Audio Preprocessing), and one Text
Translator node (TensorFlow Serving 1.14.0).

Offline AI Trainer Settings. We deploy offline AI Trainers on 4-node GPUs for
E-commerce Intelligence and on 3-node GPUs for Translation Intelligence to
train AI models or update AI models in a real-time manner.

####  5.1.3 Performance Data Collection

We use the network time protocol (NTP) [71] for synchronizing cluster-wide
clock. We use Perf [72] to collect the CPU micro-architectural data through
the hardware performance monitoring counters (PMCs). For GPU profiling, we use
the Nvidia profiling toolkit—nvprof [73] to track GPU performance. We run each
benchmark three times and report the average numbers.

###  5.2 Benchmarking Online Services

This subsection demonstrates how to drill down from a scenario benchmark into
individual modules or even primary components for latency breakdown (Section
5.2.1), explains why a statistical model cannot predict the overall system
tail latency (Section 5.2.3), explores the factors impacting service quality
(Section 5.2.4), and characterizes the micro-architectural behaviors (Section
5.2.5).

(1) E-commerce Intelligence. (2) Translation Intelligence.

Figure 5: Overall System, Modules, and Components Latency Breakdown of Two
Scenario Benchmarks.

####  5.2.1 Latency Breakdown of Different Levels

The latency is an essential metric to evaluate the service quality. This
subsection demonstrates how to drill down into different levels for a detailed
latency breakdown using two scenario benchmarks on the two CPU clusters. The
scenario benchmark configurations are as follows.

For E-commerce Intelligence, Product Database contains a hundred thousand
products with 32-attribute fields. Query Generator simulates 2000 users with a
30-second warm-up time. A user sends query requests continuously every think
time interval, following a Poisson distribution. According to history logs, we
set the proportion of text and image queries as 99% and 1%, respectively. For
Translation Intelligence, Query Generator simulates ten users with a 30-second
warm-up time. The think time interval also follows a Poisson distribution. The
proportion of text, image, and audio queries is 90%, 5%, and 5%, respectively.
We collect the performance numbers until 20,000 query requests have finished
for both two scenario benchmarks. Also, we train each AI task to achieve the
quality target of the referenced paper.

Fig. 5 shows the overall-system and individual-module latency of two scenario
benchmarks, respectively. From Fig. 5-1(a) and Fig. 5-2(a), we find the
average, 90th percentile, and 99th percentile latency of the overall system of
E-commerce Intelligence is 178, 238, and 316 milliseconds, respectively.
Simultaneously, for Translation Intelligence, the number is 778.7, 934.4, and
5919.7 milliseconds, respectively. The two scenario benchmarks reflect
different latency characteristics because of various permutations of AI and
non-AI tasks.

We further give a latency breakdown of each module to identify the critical
paths. Fig. 5-1(b) shows the latency of the Recommender, Searcher, Search
Planner, and Ranker modules within E-commerce Intelligence. The latency of
Search Planner is negligible, so we do not report it. We find that the
Recommender occupies the most significant latency proportion: 117.06, 145.73,
and 197.63 milliseconds for the average, 90th percentile, 99th percentile
latency, respectively. The average, 90th percentile, 99th percentile latency
is 50.12, 102.3, 170.21 milliseconds for Searcher, 8.3, 8.9, 11.5 milliseconds
for Ranker, respectively. Fig. 5-2(b) shows the latency of Image Converter,
Audio Converter, and Text Translator modules within Translation Intelligence.
Among them, Audio Converter incurs the most considerable latency: 3897.4,
5431.4, and 6613.3 milliseconds for the average, 90th percentile, 99th
percentile latency, respectively. Text Translator also has a large proportion
of latency, 565.8, 815.2, and 1212.1 milliseconds for the average, 90th
percentile, and 99th percentile latency. Audio Converter’s high latency is due
to its high model complexity, spending thousands of milliseconds processing
each audio query. Text Translator needs to process all the queries eventually
and results in query queueing, thus becoming the bottleneck of the overall
system. We find that different AI components incur different latency for both
scenario benchmarks, which their model complexity and dominance in the
execution paths determine.

Furthermore, Fig. 5-1(c) and Fig. 5-2(c) shows the latency breakdown of the
most time-consuming modules: Recommender for E-commerce Intelligence, Audio
Converter, and Image Converter for Translation Intelligence. Recommender
includes Query Parsing, Preprocessor, User DB Access, Image Classifier, Text
Classifier, and Recommendation, as shown in Fig. 5-1(c). We find that
Recommendation, Image Classifier (two AI components), and User DB Access (non-
AI component) are the top three key components that impact the recommender’s
latency. The average latency of Recommendation (component) takes up 80% of the
average latency of Recommender (module) and occupies 54% of the total latency
of Online Server (overall system). The 99th percentile latency of
Recommendation is 149.9 milliseconds, while the number of Recommender and
Online Server is 197.63 and 316 milliseconds, respectively. For Text
Classifier, its average latency and tail latency is less than two
milliseconds, one-hundredth of the subsystem’s latency. The overall system
tail latency deteriorates dozens of times or even hundreds of times concerning
a single component because of the following reasons. First, a single component
may not be on the critical path. Second, even an AI component like
Recommendation is critical; there are cascading interaction effects with the
other AI and non-AI components.

From Fig. 5-2(c), we find that Speech Recognition has a significant impact on
the latency, more than thousands of milliseconds both for the average and tail
latency, due to its high model complexity.

We also analyze the execution time ratio of the AI components vs. non-AI
components of the overall system latency. If we exclude the communication
latency, the average time spent on the AI and the non-AI components is 137.12
and 58.16 milliseconds for E-commerce Intelligence. While for Translation
Intelligence, nearly 99% execution time is spent on the AI components. This
observation indicates that the AI components’ contributions to the overall
system performance vary from different scenarios.

Reproducibility. Reproducibility is significant for a benchmark. We repeat two
scenario benchmarks five times to verify their reproducibility. We use the
coefficient of variation (CV) as the metric, which is the standard deviation
ratio to the mean value. The CV for the average, 90th percentile, and 99th
percentile latency is 0.005, 0.007, and 0.014 for E-commerce Intelligence, and
0.006, 0.018, and 0.024 for Translation Intelligence, respectively. The
experiments show that our scenario benchmarks have extremely low variance.

####  5.2.2 Validation of Scenario Benchmarks

To validate how closely our scenario benchmark to real-world application
scenarios, we take E-commerce Intelligence as an example to verify the
effectiveness.

We compare the E-commerce Intelligence scenario benchmark with the real-world
E-commerce scenario from our industry partner. According to Section 4.2.4, we
conduct the comparison from the overall system performance and single
component performance. We deploy the real-world E-commerce scenario on our
15-node cluster and use the same configurations for the query generator. Fig.
6 shows the overall system latency of the whole execution path. We find that
the average, 90th percentile and 99th percentile latency of the real-world
scenario are 180.47, 225, and 299 milliseconds, respectively. The values are
178, 238, and 316 milliseconds respectively, for E-commerce Intelligence. The
deviations for the average, 90th percentile, and 99th percentile latency are
1.4%, 5.8%, and 5.7%, respectively, within the normal range. Hence, they share
the consistent latency behaviors, demonstrating that the scenario benchmark
captures the real-world E-commerce scenario’s overall system performance.

Figure 6: Comparison of Overall System Latency between E-commerce Intelligence
Benchmark and Real-world Scenario.

We run the Recommendation component both within E-commerce Intelligence and
real-world scenarios to compare a single component’s performance. We choose
Recommendation as it is the most time-consuming component and impacts the
overall system latency. The average latency of Recommendation for E-commerce
Intelligence and real-world scenario are 97.03 and 103.68 milliseconds,
respectively, and the deviation is 6%. For the CPU utilization and
instructions per cycle (IPC), the values are 51% vs. 50%, and 0.59 vs. 0.6,
respectively. The deviations are 2% and 1.7% for CPU utilization and IPC.
Thus, from a single component level, our benchmark still reflects its
importance on the critical path and reflects similar workload behaviors within
the real-world scenario.

####  5.2.3 Can a Statistical Model Predict the Overall System Tail latency?

As a scenario benchmark is more complicated than a microbenchmark or a
component benchmark, we conduct experiments to check whether we can use a
statistical model to predict the overall system tail latency.

The state-of-the-art work [17] uses the M/M/1 and M/M/K queueing models to
estimate the p’th percentile latency. Since our experiments in Section 5.2.1
only contain one Online Server instance, we adopt the M/M/1 model. The p’th
percentile latency ($T{p}$) and the average latency ($T{m}$) are calculated
using the formulas:
$T{p}=-\frac{\ln\left(1-\frac{P}{100}\right)}{\mu-\lambda}$ ,
$T{m}=\frac{1}{\mu-\lambda}$. $\mu$ is the service rate, following the
exponential distribution. $\lambda$ is the arrival rate, following the Poisson
distribution.

For E-commerce Intelligence, we obtain $\mu$ as 90 requests per second through
experiments. Then we set $\lambda$ as 3, 33, and 66 requests per second,
respectively, to estimate 100, 1000, and 2000 concurrent users. The average
latency theoretical number is 11, 17, and 41 milliseconds for different
settings, while the actual number is 141, 148, and 178 milliseconds,
respectively. The average gap is 8.6x. The theoretical number of the 99th
percentile latency is 52, 80, and 191 milliseconds, while the actual number is
261, 267, and 316 milliseconds, respectively. The average gap is 3.3x. For
Translation Intelligence, we get $\mu$ as 5.5 requests per second through
experiments. Then we set $\lambda$ as 0.3, 0.8, and 1.5 requests per second,
respectively, to estimate 5, 10, and 20 concurrent users. For different
settings, the average latency’s theoretical number is 192, 212, and 250
milliseconds, while the actual number is 788, 799, and 821 milliseconds,
respectively. The average gap is 3.7x. The theoretical number of the 99th
percentile latency is 885, 979, and 1151 milliseconds, while the actual number
is 4283, 5354, and 5362 milliseconds, respectively. The average gap is 5.0x.

The main reason for this vast gap is as follows. It is complex and uncertain
about executing a scenario benchmark, and the service rate does not follow the
exponential distribution. So, the M/M/1 model is far away from the realistic
situation.

We further build a queueing network model [18] for E-commerce Intelligence.
The model consists of nine M/M/1 components, each of which corresponds with
one component in E-commerce Intelligence, e.g., Text classifier in Fig. 2. We
set $\lambda$ as 3, 33, and 66 requests per second, too. The average gap of
the average latency between the theoretical and actual number is 4.9x. We do
not report the tail latency gap because the queueing network is hard to
accurately predict the overall tail latency due to its non-superposition
property.

####  5.2.4 Factors Impacting Service Quality

We explore the impacts of the data dimension and model complexity on the
service quality.

Impact of Data Dimension. The data input has a significant impact on workload
behaviors [74, 75]. We quantify its effect on the service quality through
resizing the dimensions of input image data for Image Classifier in E-commerce
Intelligence, including 8*8, 16*16, 32*32, 64*64, 128*128, and 256*256
dimensions. Fig. 7 shows the average latency curve of the Image Classifier. We
find that with the data complexity increases, the average latency
deteriorates, but its slope gradually decreases. For example, the data
dimension changes from 128*128 to 256*256, while the average latency
deteriorates three times. The reason is that with the enlargement of the data
dimension, the increase of continuous data accesses brings in better data
locality. From the micro-architectural perspective, we notice that with the
rise of data dimension, the IPC increases sharply from 0.34 to 2.37. The cache
misses of all levels and pipeline stalls decrease, going down about ten or
even dozens of times. Hence, resizing the input data dimension (data quality)
to an appropriately larger one is beneficial for improving resource
utilization. At the same time, there is a tradeoff between good service
quality and enlarged data dimensions.

Figure 7: Impact of Data Dimension on Service Quality.

Impact of Model Complexity. The online inference module needs to load the
trained model and conducts a forward computation to obtain the result.
Usually, increasing the depth of a neural network model may improve the model
accuracy, but the side effect is that a larger model size results in longer
inference time. For comparison, we replace ResNet50 with ResNet152 in the
Image Classifier. The model accuracy improvement is 1.5%, while the overall
system 99th percentile latency deteriorates by 10x.

Hence, Internet service architects should attempt to balance tradeoffs among
data quality, service quality, model complexity, and model accuracy.

####  5.2.5 Micro-architectural Characterization

Figure 8: The Cache and Pipeline Behavior Disparity between AI and non-AI
Components. The y-axis indicates the average misses or stall cycles per kilo
instructions of all AI and non-AI components; The labeled numbers reveal the
AI components’ ratios to Non-AI components.

We characterize the micro-architectural behaviors of both AI and non-AI
components of two scenario benchmarks using Perf. To compare AI and non-AI
behaviors as a whole, we first report their average numbers of all AI and non-
AI components. In our experiments, the AI components have lower IPC
(instructions per cycle) (0.35) than that of the non-AI components (0.99) on
average since they suffer from more cache misses, TLB misses, and execution
stalls, even up to a dozen times. Fig. 8 shows the cache and pipeline behavior
differences between AI and non-AI components. The reason is that the AI
components serving the services have more random memory accesses and worse
data locality than non-AI components. Thus, they exhibit a larger working set
and higher memory bandwidth.

For different AI components, we find in E-commerce Intelligence, the two
components with higher latency, i.e., Image Classifier and Recommendation,
suffer from higher backend bound while lower frontend bound than Text
Classifier and Ranker. The higher backend bound is mainly due to higher L1
data cache misses (more than 1.7x) for Image Classifier and more DRAM accesses
(more than 1.4x) for Recommendation. The higher frontend bound for Text
Classifier and Ranker is mainly due to the higher frontend latency bound
caused by large L1 instruction cache misses (more than 2.5x), and instruction
TLB misses (more than 3.9x). For Translation Intelligence, Image Converter and
Audio Converter suffer from higher backend bound while lower frontend bound
than those of Text Translator. The higher backend bound for Image Converter
and Audio Converter is mainly due to higher L1 data cache misses (more than
2x) and more DRAM accesses (more than 1.5x). The lower frontend bound,
incurred by inefficient utilization of the Decoded Stream Buffer (DSB),
attributes to the lower frontend bandwidth bound (about two times lower).

###  5.3 Benchmarking Offline Training

Updating AI models in a real-time manner is a significant industrial concern
in many scenarios shown in Table 1. We evaluate how to balance the tradeoffs
among model update intervals, training overheads, and accuracy improvements
using AI Offline Trainer of E-commerce Intelligence on the Titan XP GPUs.

We resize the input data volume to investigate how to balance tradeoffs among
model update intervals, training overheads, and accuracy improvements. For
Image Classifier, using 60% of training images, the training time is 2096.8
seconds to achieve the highest accuracy of 68.7%. When using 80% and 100% of
training images, the time spent is 2347.8 and 3170.7 seconds to achieve the
highest accuracy of 71.8% and 73.7%, respectively. Hence, resizing the volume
from 60% to 100%, 51% additional training time brings in a 4.96% accuracy
improvement; From 80% to 100%, 35% additional training time brings in a 1.9%
accuracy improvement. For Ranker, using 60%, 80%, and 100% of training data,
the training time is 839.3, 1222.8, and 1430,7 seconds, respectively. The
highest accuracy is 12.4%, 13.6%, and 13.72%, respectively, which is close to
[61]. So, for this AI component, resizing from 60% to 100%, 70% additional
training time brings a 1.36% accuracy improvement. Resizing from 80% to 100%,
17% extra training time brings in a 0.12% accuracy improvement.

We conclude that Internet service architects need to balance the tradeoffs
among model update intervals, training overheads, and accuracy improvements.
Moreover, for different AI components, the tradeoffs have subtle differences,
and the evaluation should be performed using a scenario benchmarking
methodology.

###  5.4 Zooming in on Hotspot Functions

The evaluations in Section 5.2.1 demonstrate that drilling down from a
scenario benchmark into the primary components lets us focus on the component
benchmarks without losing the overall critical path. This subsection will
further demonstrate how to zoom in on the hotspot functions of these primary
component benchmarks.

Section 5.2.1 shows the primary five AI components are on the critical path of
the two scenario benchmarks and have significant impacts on the overall system
latency and tail latency. They are Recommendation, Ranker, Image Classifier
for E-commerce Intelligence, and Speech Recognition and Text Translator for
Translation Intelligence. To zoom in on these component benchmarks’ hotspot
functions, we run both training and inference stages and use nvprof to trace
the running time breakdown. To profile accuracy-ensured performance, we first
adjust the parameters, e.g., batch size, achieve the state-of-the-art quality
target of that model on a given dataset, and then sample 1,000 epochs of the
training using the same parameter settings.

Table 2 shows the hotspot functions of the five primary components of the two
scenario benchmarks. Note that we merge the time percentages of the same
function and report the total sum. For example, the time percentage of
Eigen::internal::EigenMetaKernel merges all the functions that call
Eigen::internal::EigenMetaKernel with different parameters, like scalar_sum_op
and scalar_max_op.

Table 2: Hotspot Functions of Each AI Component. Component |  Function Name |  Time(%)
---|---|---
AI Components of E-commerce Benchmark
Recommendation |  Eigen::internal::EigenMetaKernel |  39.6
CUDA Memcpy |  23.4
tensorflow::GatherOpKernel |  8.1
tensorflow::scatter_op_gpu::ScatterOpCustomKernel |  4
Ranker |  Eigen::internal::EigenMetaKernel |  56.6
sgemm_32x32x32_NT_vec |  9.1
tensorflow::functor::ColumnReduceSimpleKernel |  8.7
sgemm_32x32x32_TN_vec |  5
Image Classifier |  Eigen::internal::EigenMetaKernel |  18.2
maxwell_scudnn_128x128_stridedB_splitK_interior_nn |  15.6
maxwell_scudnn_128x128_relu_interior_nn |  11.5
maxwell_scudnn_128x128_stridedB_interior_nn |  7.9
cudnn::detail::bn_bw_1C11_kernel_new |  7
AI Components of Translation Benchmark
Text Translator |  Eigen::internal::EigenMetaKernel |  26.3
sgemm_128x128x8_NN_vec |  17.6
sgemm_128x128x8_NT_vec |  15.8
sgemm_128x128x8_TN_vec |  8.3
Speech Recognition |  Eigen::internal::EigenMetaKernel |  23.9
cudnn::detail::dgrad2d_alg1_1 |  17.4
cudnn::detail::dgrad_engine |  13.4
sgemm_32x32x32_NT_vec |  9.4

We find that Eigen::internal::EigenMetaKernel is the most time-consuming
function among the five primary AI components. Eigen is a C++ template library
for linear algebra [76] used by TensorFlow. Through statistics in Fig. 9, we
further find that within Eigen::internal::EigenMetaKernel, the most commonly
used kernels are matrix multiplication, sqrt, compare, sum, quotient, argmax,
max, and data_format computations. Here data_format means variable assignment,
data slice, data resize, etc. We reveal that these five components and the
corresponding hotspot functions are the optimization points for software stack
and CUDA library optimizations and micro-architectural optimizations.

Figure 9: Eigen::internal::EigenMetaKernel Runtime Breakdown—the most time-
consuming function among the five AI components.

Focusing on Eigen::internal::EigenMetaKernel, we further analyze their GPU
stalls. We find that the constant memory dependency stall, which immediate
constant cache miss incurs, is the top one stall, occupying more than 50% of
total stalls on average. The memory dependency stall—a memory operation that
cannot be performed due to the required resources not being available or fully
utilized—occupies 20% on average. The instruction fetch stall and execution
dependency stall, which is due to an unready input required by the
instruction, occupies about 14% and 13%, respectively. Possible optimization
strategies include storing constants in global memory, improving data
placement, and recomputing constant.

##  6 Conclusion

This paper proposes a scenario-distilling methodology to tackle the challenges
of benchmarking modern real-world application scenarios. We formalize a real-
world application scenario as a DAG model and present the rules to distill it
into the permutation of essential AI and non-AI tasks as a high-level scenario
benchmark specification. Concerning the original scenarios, the scenario
benchmarks reduce the complexity by one or two orders of magnitude. Together
with seventeen industry partners, we extract nine scenarios and identify the
primary components. We design and implement an extensible, configurable, and
flexible benchmark framework. We implement two Internet service AI scenario
benchmarks as proxies to two real-world application scenarios based on the
framework. We consider scenario, component, and micro benchmarks as three
indispensable parts, and the evaluation shows the advantage of our methodology
against using component or micro AI benchmarks alone.

## Acknowledgment

We are grateful to the anonymous reviewers and our shepherd, Professor Justin
Gottschlich. This work is supported by the Standardization Research Project of
Chinese Academy of Sciences No.BZ201800001. Jianfeng Zhan is the corresponding
author.

##  7 Artifact Appendix

###  7.1 Abstract

The artifact contains AIBench Scenario benchmarks, running and profiling
scripts. It can support the characterization results in Chapter V.

###  7.2 Artifact check-list (meta-information)

  * •

Program: AIBench Scenario benchmarks

  * •

Compilation: Python 3.6.8, Java 1.8.0

  * •

Data set: Anonymous data from industry or open-sourced data sets

  * •

Run-time environment: Ubuntu 18.04

  * •

Hardware: CPUs and GPUs

  * •

Run-time state: Disable Hyper-Threading

  * •

Execution: root user or users that can execute sudo without password

  * •

Experiment: Deploy the benchmarks and corresponding software stacks; run
benchmarks; output the results

  * •

Publicly available?: Yes

  * •

Workflow frameworks used?: No

  * •

Archived (provide DOI)?: https://doi.org/10.5281/zenodo.5158715

###  7.3 Description

####  7.3.1 How delivered

The artifacts is available on Zenodo: https://doi.org/10.5281/zenodo.5158715,
including all the source code scripts, and instructions. The Zenodo DOI is
10.5281/zenodo.5158715. Also, the AIBench scenario benchmarks and framework
are open sourced on GitHub:

https://github.com/BenchCouncil/aibench_scenario.

Note that the running or profiling scripts we provide suit for our cluster
environment, like the node ip/hostname and port number, if you download and
use it in your cluster environment, you need to modify the scripts.

####  7.3.2 Hardware dependencies

The AIBench scenario benchmarks can be run on all processors that can deploy
Docker, Jmeter, SpringBoot, TensorFlow Serving, Elasticsearch and Neo4j
stacks. However, for cache and pipeline behavior analysis, user need to find
the performance counters corresponding to specific processor. We have provided
profiling scripts for Xeon E5645 processor.

####  7.3.3 Software dependencies

Python 3.6.8, Java 1.8.0, Jmeter 5.1.1, SpringBoot 2.1.3, TensorFlow Serving
1.14.0, Elasticsearch 6.5.2, Neo4j 3.5.8, Docker 19.03 and Docker compose
1.29.

####  7.3.4 Data sets

We provide both anonymous data from industry and open-sourced data sets. Users
can find where to download and how to use these data in the README file.

###  7.4 Installation

User need to install Python, Java, and Docker. For Jmeter, SpringBoot,
TensorFlow Serving, Elasticsearch, and Neo4j, user can install them manually
or use the deployment templates we provided. The install details can be found
in the README file.

###  7.5 Experiment workflow

Before profiling system and micro-architecture metrics of a scenario
benchmark, users should make sure there is no other workload running. Please
refer to the README file within the artifacts package for detailed
information.

####  7.5.1 Deploy the benchmarks.

We provide deployment scripts for two scenario benchmarks.

Deploy E-commerce Intelligence:

i. Under the following directory

#cd aibench_scenario/benchmarks/ecommerce_search

ii. Deploy benchmark: #docker-compose up

Deploy Translation Intelligence:

i. Under the following directory

#cd aibench_scenario/benchmarks/translation

ii. Deploy benchmark: #docker-compose up

####  7.5.2 Run the benchmarks.

We provide running scripts for two scenario benchmarks.

Run E-commerce Intelligence:

i. Load the data

#cd aibench_scenario/framework/online/non_AI_module/

benchmark-1.0-SNAPSHOT

#./create_index.sh

#./load_data.sh

Note that you need to modify the IP address of the above two shell scripts
according to your cluster configurations.

ii. Run the benchmark

#apache-jmeter-5.1.1/bin/jmeter -n -t aibench_scenario/

framework/online/non_AI_module/load_generator/

search_loop_2W.jmx -l result.txt -e -o report

Note that you need to modify the search_loop_2W.jmx file. First, use docker
command to check the IP address of Search Planner;

#docker ps

#docker inspect $ID

The ID is the output of the command ”docker ps” named Search Planner, the
output will include the IP address of Search Planner.

Then change all the IP address of ”172.18.11.91” in the search_loop_2W.jmx
file to the actual IP address of Search Planner.

iii. Check the results

After the Step ii, it will generate a directory named ”report”. Within the
directory, there is a file named ”index.html”, which can show the average
latency, 90th percentile latency, and 99th percentile latency. We also provide
”run_search.sh” to run the e-commerce intelligence benchmark. The user need to
modify the scripts according to their own cluster environment.

Run Translation Intelligence:

i. Run the benchmark

#cd aibench_scenario

#apache-jmeter-5.2.1/bin/jmeter -n -t apache-jmeter-5.2.1/translation.jmx -l
result.txt -e -o report

Note that you need to modify the translation.jmx file. First, use docker
command to check the IP address of Translation Planner;

#docker ps

#docker inspect $ID

The ID is the output of the command ”docker ps” named Translation Planner, the
output will include the IP address of Translation Planner.

Then change all the IP address of ”172.18.13.26” in the translation.jmx file
to the actual IP address of Translation Planner. The result.txt will log the
timestamp of all the queries.

ii. Check the results

After the Step i, it will generate a directory named ”report”. Within the
directory, there is a file named ”index.html”, which can show the average
latency, 90th percentile latency, and 99th percentile latency.

####  7.5.3 Process the metric data and plot the figures

For E-commerce Intelligence, the processing script is log_process.sh in the
“aibench_scenario/ecommerce_search” directory. The usage of log_process.sh is
./log_process.sh <log_dir >, for example, ./log_process.sh ../log-0310-1402,
the result.csv will be outputted in current directory, and run the command
“python e_commerce_search_figure.py” will produce the Fig.5(1).

For Translation Intelligence, in the “aibench_scenario/translation” directory,
run the “python translation_intelligence_fig.py” will produce the Fig.5(2).

For the experiment of Fig.7, users have to change the data dimension,
recompile the benchmark, and re-run the E-commerce Intelligence benchmark for
several times.

i. Change the data dimension

Change the Line #204 of the file

“aibench_scenario/framework/online/non_AI_module/recommender/cpu_app.py”:

_img = image.img_to_array(image.load_img(img_path, target_size=(8, 8)))_

Change (8, 8) to the other dimension you need to test, like (16, 16), (32,
32), (64, 64), (128, 128), or (256, 256).

ii. Recompile the benchmark

#cd aibench_scenario/benchmarks/ecommerce_search

#docker-compose up -d

After changing the data dimension and recompiling the benchmark, re-run the
E-commerce Intelligence benchmark according to the steps of ”Run E-commerce
Intelligence” and obtain the results.

iii. Use the image_dimension_figure.sh to produce the Fig.7.

#./image_dimension_figure.sh <8_logdir><16_logdir><32_logdir><64_logdir
><128_logdir><256_logdir>

Note that the <x_logdir>are the log directories generated in Step iv when the
data dimension is x*x.

For the experiment of Fig.8, lsdata-pact21-fig8.py can produce the
corresponding figure, the usage is python lsdata-pact21-fig8.py
<input_dir><output_name>, the input_dir refer the hw_log, for example python
lsdata-pact21-fig8.py log-0310-1402/hw_log output. This command will produce
an excel file named <output_name>.xls and a figure. Note that the
use_perf_new1.py we provide only suit for sampling micro-architecture metrics
on Intel Xeon E5645 processors. If you need to profile on another processor
type, you need to modify the corresponding hardware performance counters.

###  7.6 Evaluation and expected result

To evaluate the scenario benchmarks, users need to run and profile them. The
results should follow the similar trends as reported in the paper. Note that
if you download the artifacts and deploy them in your cluster, the results may
different from the values in the paper, due to the differences of cluster
scale, node configurations, network configurations, etc.

###  7.7 Experiment customization

Users can run these scenario benchmarks for different benchmarking purpose,
e.g. datacenter deployment strategies. The scenario benchmarks can be deployed
on different processors and cluster scales. Also, the AI related TensorFlow
serving components can be deployed on different AI chips like GPU and TPU.

###  7.8 Notes

For the artifact evaluation, it may cost a lot of time to run some benchmarks,
we also provide the scripts and the original running logs and profiling data
used in our paper, which are suit for our experimental environments and
configurations. Since the platform configurations, network configurations, and
hardware types (e.g. memory capacity, processing capacity) may be different,
so the performance data may be different on another platform.

## References

  * [1] J. Li, N. K. Sharma, D. R. Ports, and S. D. Gribble, “Tales of the tail: Hardware, os, and application-level sources of tail latency,” in Proceedings of the ACM Symposium on Cloud Computing, pp. 1–14, 2014.
  * [2] “Speccpu 2017.” https://www.spec.org/cpu2017/.
  * [3] “Nginx website.” http://nginx.org/.
  * [4] “Spark.” https://spark.apache.org/.
  * [5] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mane, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viegas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, “Tensorflow: Large-scale machine learning on heterogeneous distributed systems,” arXiv preprint arXiv:1603.04467, 2016.
  * [6] L. A. Barroso and U. Hölzle, “The datacenter as a computer: An introduction to the design of warehouse-scale machines,” Synthesis Lectures on Computer Architecture, vol. 4, no. 1, pp. 1–108, 2009.
  * [7] W. Gao, J. Zhan, L. Wang, C. Luo, D. Zheng, X. Wen, R. Ren, C. Zheng, X. He, H. Ye, H. Tang, Z. Cao, S. Zhang, and J. Dai, “Bigdatabench: A scalable and unified big data and ai benchmark suite,” arXiv preprint arXiv:1802.08254, 2018.
  * [8] D. Bailey, E. Barszcz, J. Barton, D. Browning, R. Carter, L. Dagum, R. Fatoohi, P. Frederickson, T. Lasinski, R. Schreiber, H. Simon, V. Venkatakrishnan, and S. Weeratunga, “The nas parallel benchmarks,” The International Journal of Supercomputing Applications, vol. 5, no. 3, pp. 63–73, 1991.
  * [9] K. Hazelwood, S. Bird, D. Brooks, S. Chintala, U. Diril, D. Dzhulgakov, M. Fawzy, B. Jia, Y. Jia, A. Kalro, J. Law, K. Lee, J. Lu, P. Noordhuis, M. Smelyanskiy, L. Xiong, and X. Wang, “Applied machine learning at facebook: A datacenter infrastructure perspective,” in 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 620–629, IEEE, 2018.
  * [10] G. Ayers, J. H. Ahn, C. Kozyrakis, and P. Ranganathan, “Memory hierarchy for web search,” in 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 643–656, IEEE, 2018.
  * [11] T. M. Conte, K. N. Menezes, and M. A. Hirsch, “Accurate and practical profile-driven compilation using the profile buffer,” in Proceedings of the 29th Annual IEEE/ACM International Symposium on Microarchitecture. MICRO 29, pp. 36–45, IEEE, 1996.
  * [12] M. Alam, J. Gottschlich, N. Tatbul, J. S. Turek, T. Mattson, and A. Muzahid, “A zero-positive learning approach for diagnosing software performance regressions,” Advances in Neural Information Processing Systems, vol. 32, pp. 11627–11639, 2019.
  * [13] A. Sriraman and A. Dhanotia, “Accelerometer: Understanding acceleration opportunities for data center overheads at hyperscale,” in Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 733–750, 2020.
  * [14] S. Karandikar, A. Ou, A. Amid, H. Mao, R. Katz, B. Nikolić, and K. Asanović, “Fireperf: Fpga-accelerated full-system hardware/software performance profiling and co-design,” in Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 715–731, 2020.
  * [15] G. Hamerly, E. Perelman, J. Lau, and B. Calder, “Simpoint 3.0: Faster and more flexible program phase analysis,” Journal of Instruction Level Parallelism, vol. 7, no. 4, pp. 1–28, 2005.
  * [16] C.-K. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V. J. Reddi, and K. Hazelwood, “Pin: building customized program analysis tools with dynamic instrumentation,” Acm sigplan notices, vol. 40, no. 6, pp. 190–200, 2005.
  * [17] C. Delimitrou and C. Kozyrakis, “Amdahl’s law for tail latency,” Communications of the ACM, vol. 61, no. 8, pp. 65–72, 2018.
  * [18] B. Urgaonkar, G. Pacifici, P. Shenoy, M. Spreitzer, and A. Tantawi, “An analytical model for multi-tier internet services and its applications,” ACM SIGMETRICS Performance Evaluation Review, vol. 33, no. 1, pp. 291–302, 2005.
  * [19] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” arXiv preprint arXiv:1608.08710, 2016.
  * [20] R. Marcus, P. Negi, H. Mao, N. Tatbul, M. Alizadeh, and T. Kraska, “Bao: Making learned query optimization practical,” in Proceedings of the 2021 International Conference on Management of Data, pp. 1275–1288, 2021.
  * [21] R. Adolf, S. Rama, B. Reagen, G.-Y. Wei, and D. Brooks, “Fathom: reference workloads for modern deep learning methods,” in Workload Characterization (IISWC), pp. 1–10, IEEE, 2016.
  * [22] C. Coleman, D. Narayanan, D. Kang, T. Zhao, J. Zhang, L. Nardi, P. Bailis, K. Olukotun, C. Ré, and M. Zaharia, “Dawnbench: An end-to-end deep learning benchmark and competition,” Training, vol. 100, no. 101, p. 102, 2017.
  * [23] “Mlperf website.” https://www.mlperf.org.
  * [24] V. J. Reddi, C. Cheng, D. Kanter, P. Mattson, G. Schmuelling, C.-J. Wu, B. Anderson, M. Breughe, M. Charlebois, W. Chou, R. Chukka, C. Coleman, S. Davis, P. Deng, G. Diamos, J. Duke, D. Fick, J. S. Gardner, I. Hubara, S. Idgunji, T. B. Jablin, J. Jiao, T. St. John, P. Kanwar, D. Lee, J. Liao, A. Lokhmotov, F. Massa, P. Meng, P. Micikevicius, C. Osborne, G. Pekhimenko, A. Tejusve Raghunath Rajan, D. Sequeira, A. Sirasao, F. Sun, H. Tang, M. Thomson, F. Wei, E. Wu, L. Xu, K. Yamada, B. Yu, G. Yuan, A. Zhong, P. Zhang, and Y. Zhou, “Mlperf inference benchmark,” arXiv preprint arXiv:1911.02549, 2019.
  * [25] P. Mattson, C. Cheng, C. Coleman, G. Diamos, P. Micikevicius, D. Patterson, H. Tang, G.-Y. Wei, P. Bailis, V. Bittorf, D. Brooks, D. Chen, D. Dutta, U. Gupta, K. Hazelwood, A. Hock, X. Huang, B. Jia, D. Kang, D. Kanter, N. Kumar, J. Liao, G. Ma, D. Narayanan, T. Oguntebi, G. Pekhimenko, L. Pentecost, V. J. Reddi, T. Robie, T. St. John, C.-J. Wu, L. Xu, C. Young, and M. Zaharia, “Mlperf training benchmark,” arXiv preprint arXiv:1910.01500, 2019.
  * [26] C. Banbury, V. J. Reddi, P. Torelli, J. Holleman, N. Jeffries, C. Kiraly, P. Montino, D. Kanter, S. Ahmed, D. Pau, et al., “Mlperf tiny benchmark,” arXiv preprint arXiv:2106.07597, 2021.
  * [27] H. Zhu, M. Akrout, B. Zheng, A. Pelegris, A. Phanishayee, B. Schroeder, and G. Pekhimenko, “Tbd: Benchmarking and analyzing deep neural network training,” arXiv preprint arXiv:1803.06905, 2018.
  * [28] “Deepbench.” https://svail.github.io/DeepBench/.
  * [29] S. Dong and D. Kaeli, “Dnnmark: A deep neural network benchmark suite for gpus,” in Proceedings of the General Purpose GPUs, pp. 63–72, ACM, 2017\.
  * [30] J. Hauswald, M. A. Laurenzano, Y. Zhang, C. Li, A. Rovinski, A. Khurana, R. G. Dreslinski, T. Mudge, V. Petrucci, L. Tang, and J. Mars, “Sirius: An open end-to-end voice and vision personal assistant and its implications for future warehouse scale computers,” in Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 223–238, 2015.
  * [31] H. Kasture and D. Sanchez, “Tailbench: a benchmark suite and evaluation methodology for latency-critical applications,” in 2016 IEEE International Symposium on Workload Characterization (IISWC), pp. 1–10, IEEE, 2016.
  * [32] M. Ferdman, A. Adileh, O. Kocberber, S. Volos, M. Alisafaee, D. Jevdjic, C. Kaynak, A. D. Popescu, A. Ailamaki, and B. Falsafi, “Clearing the clouds: A study of emerging workloads on modern hardware,” ASPLOS, 2012.
  * [33] A. Dakkak, C. Li, J. Xiong, and W.-m. Hwu, “Mlmodelscope: A distributed platform for model evaluation and benchmarking at scale,” arXiv preprint arXiv:2002.08295, 2020.
  * [34] Y. Gan, Y. Zhang, D. Cheng, A. Shetty, P. Rathi, N. Katarki, A. Bruno, J. Hu, B. Ritchken, B. Jackson, K. Hu, M. Pancholi, Y. He, B. Clancy, C. Colen, F. Wen, C. Leung, S. Wang, L. Zaruvinsky, M. Espinosa, R. Lin, Z. Liu, and J. Padilla, “An open-source benchmark suite for microservices and their hardware-software implications for cloud & edge systems,” in Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 3–18, ACM, 2019\.
  * [35] Y. Gan, Y. Zhang, K. Hu, D. Cheng, Y. He, M. Pancholi, and C. Delimitrou, “Seer: Leveraging big data to navigate the complexity of performance debugging in cloud microservices,” in Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 19–33, 2019.
  * [36] “Aibench tutorial on isca 2021.” https://www.benchcouncil.org/aibench-tutorial/isca21/file/01_AIBench_Overview.pdf.
  * [37] F. Tang, W. Gao, J. Zhan, C. Lan, X. Wen, L. Wang, C. Luo, Z. Cao, X. Xiong, Z. Jiang, et al., “Aibench training: balanced industry-standard ai training benchmarking,” in 2021 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 24–35, IEEE, 2021\.
  * [38] Z. Jiang, W. Gao, F. Tang, L. Wang, X. Xiong, C. Luo, C. Lan, H. Li, and J. Zhan, “Hpc ai500 v2.0: The methodology, tools, and metrics for benchmarking hpc ai systems,” 2021 IEEE International Conference on Cluster Computing (CLUSTER), 2021.
  * [39] T. Hao, Y. Huang, X. Wen, W. Gao, F. Zhang, C. Zheng, L. Wang, H. Ye, K. Hwang, Z. Ren, and J. Zhan, “Edge aibench: Towards comprehensive end-to-end edge computing benchmarking,” 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18), 2018.
  * [40] T. Hao, K. Hwang, J. Zhan, M. Chen, Y. Li, Y. Miao, and Y. Cao, “Distributed cloud/edge computing with weighted ai benchmarks on heterogeneous servers,” Technical Report, 2021.
  * [41] C. Luo, F. Zhang, C. Huang, X. Xiong, J. Chen, L. Wang, W. Gao, H. Ye, T. Wu, R. Zhou, and J. Zhan, “Aiot bench: Towards comprehensive benchmarking mobile and embedded device intelligence,” 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18), 2018.
  * [42] J. Zhan, L. Wang, W. Gao, and R. Ren, “Benchcouncil’s view on benchmarking ai and other emerging workloads,” arXiv preprint arXiv:1912.00572, 2019.
  * [43] “Pytorch.” http://pytorch.org.
  * [44] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [45] A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, and T. Mikolov, “Fasttext.zip: Compressing text classification models,” arXiv preprint arXiv:1612.03651, 2016.
  * [46] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein gan,” arXiv preprint arXiv:1701.07875, 2017.
  * [47] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in neural information processing systems, pp. 5998–6008, 2017.
  * [48] Z. Wojna, A. N. Gorban, D.-S. Lee, K. Murphy, Q. Yu, Y. Li, and J. Ibarz, “Attention-based extraction of structured information from street view imagery,” in 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 1, pp. 844–850, IEEE, 2017.
  * [49] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: Lessons learned from the 2015 mscoco image captioning challenge,” IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 4, pp. 652–663, 2017.
  * [50] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE international conference on computer vision, pp. 2223–2232, 2017.
  * [51] D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen, J. Chen, J. Chen, Z. Chen, M. Chrzanowski, A. Coates, G. Diamos, K. Ding, N. Du, E. Elsen, J. Engel, W. Fang, L. Fan, C. Fougner, L. Gao, C. Gong, A. Hannun, T. Han, L. V. Johannes, B. Jiang, C. Ju, B. Jun, P. LeGresley, L. Lin, J. Liu, Y. Liu, W. Li, X. Li, D. Ma, S. Narang, A. Ng, S. Ozair, Y. Peng, R. Prenger, S. Qian, Z. Quan, J. Raiman, V. Rao, S. Satheesh, D. Seetapun, S. Sengupta, K. Srinet, A. Sriram, H. Tang, L. Tang, C. Wang, J. Wang, K. Wang, Y. Wang, Z. Wang, Z. Wang, S. Wu, L. Wei, B. Xiao, W. Xie, Y. Xie, D. Yogatama, B. Yuan, J. Zhan, and Z. Zhu, “Deep speech 2: End-to-end speech recognition in english and mandarin,” in International conference on machine learning, pp. 173–182, 2016.
  * [52] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815–823, 2015.
  * [53] R.-L. Vieriu, S. Tulyakov, S. Semeniuta, E. Sangineto, and N. Sebe, “Facial expression recognition under a wide range of head poses,” in 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), vol. 1, pp. 1–7, IEEE, 2015.
  * [54] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Advances in neural information processing systems, pp. 91–99, 2015.
  * [55] X. He, L. Liao, H. Zhang, L. Nie, X. Hu, and T.-S. Chua, “Neural collaborative filtering,” in Proceedings of the 26th international conference on world wide web, pp. 173–182, International World Wide Web Conferences Steering Committee, 2017.
  * [56] C. Finn, I. Goodfellow, and S. Levine, “Unsupervised learning for physical interaction through video prediction,” in Advances in neural information processing systems, pp. 64–72, 2016.
  * [57] G. Toderici, D. Vincent, N. Johnston, S. Jin Hwang, D. Minnen, J. Shor, and M. Covell, “Full resolution image compression with recurrent neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5306–5314, 2017.
  * [58] X. Yan, J. Yang, E. Yumer, Y. Guo, and H. Lee, “Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision,” in Advances in Neural Information Processing Systems, pp. 1696–1704, 2016\.
  * [59] R. Nallapati, B. Zhou, C. Gulcehre, and B. Xiang, “Abstractive text summarization using sequence-to-sequence rnns and beyond,” arXiv preprint arXiv:1602.06023, 2016.
  * [60] M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu, “Spatial transformer networks,” in Advances in neural information processing systems, pp. 2017–2025, 2015.
  * [61] J. Tang and K. Wang, “Ranking distillation: Learning compact ranking models with high performance for recommender system,” in ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018.
  * [62] Z. Ming, C. Luo, W. Gao, R. Han, Q. Yang, L. Wang, and J. Zhan, “Bdgs: A scalable big data generator suite in big data benchmarking,” arXiv preprint arXiv:1401.5465, 2014.
  * [63] C. Olston, N. Fiedel, K. Gorovoy, J. Harmsen, L. Lao, F. Li, V. Rajashekhar, S. Ramesh, and J. Soyke, “Tensorflow-serving: Flexible, high-performance ml serving,” arXiv preprint arXiv:1712.06139, 2017.
  * [64] L. Hochstein and R. Moser, Ansible: Up and Running: Automating Configuration Management and Deployment the Easy Way.  ” O’Reilly Media, Inc.”, 2017.
  * [65] K. Hightower, B. Burns, and J. Beda, Kubernetes: up and running: dive into the future of infrastructure.  ” O’Reilly Media, Inc.”, 2017.
  * [66] A. JMeter, “Apache jmeter,” Online.(2016). http://jmeter. apache. org/-Visited, pp. 04–25, 2017.
  * [67] P. Webb, D. Syer, J. Long, S. Nicoll, R. Winch, A. Wilkinson, M. Overdijk, C. Dupuis, and S. Deleuze, “Spring boot reference guide,” Part IV. Spring Boot features, vol. 24, 2013.
  * [68] Y. Ni, D. Ou, S. Liu, X. Li, W. Ou, A. Zeng, and L. Si, “Perceive your users in depth: Learning universal user representations from multiple e-commerce tasks,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 596–605, ACM, 2018.
  * [69] C. Gormley and Z. Tong, Elasticsearch: the definitive guide: a distributed real-time search and analytics engine.  ” O’Reilly Media, Inc.”, 2015.
  * [70] “Restful api.” https://www.tensorflow.org/tfx/serving/api_rest.
  * [71] D. L. Mills, “Network time protocol (ntp),” Network, 1985.
  * [72] A. C. De Melo, “The new linux perf tools,” in Slides from Linux Kongress, vol. 18, 2010.
  * [73] “Nvidia profiling toolkit.” https://docs.nvidia.com/cuda/profiler-users-guide/index.html.
  * [74] W. Gao, J. Zhan, L. Wang, C. Luo, D. Zheng, F. Tang, B. Xie, C. Zheng, X. Wen, X. He, H. Ye, and R. Ren, “Data motifs: A lens towards fully understanding big data and ai workloads,” Parallel Architectures and Compilation Techniques (PACT), 2018 27th International Conference on, 2018.
  * [75] B. Xie, J. Zhan, X. Liu, W. Gao, Z. Jia, X. He, and L. Zhang, “Cvr: Efficient vectorization of spmv on x86 processors,” in 2018 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), 2018.
  * [76] “Eigen.” http://eigen.tuxfamily.org/.