# Bridging the Gap Between Domain-specific Frameworks and Multiple Hardware
Devices

Xu Wen  Institute of Computing Technology, Chinese Academy of Sciences and
University of Chinese Academy of SciencesBeijingChina<EMAIL_ADDRESS>,
Wanling Gao  Institute of Computing Technology, Chinese Academy of Sciences
and University of Chinese Academy of SciencesBeijingChina<EMAIL_ADDRESS>,  Lei Wang  Institute of Computing Technology, Chinese Academy of Sciences
and University of Chinese Academy of SciencesBeijingChina
<EMAIL_ADDRESS>and  Jianfeng Zhan  Institute of Computing Technology,
Chinese Academy of Sciences and University of Chinese Academy of
SciencesBeijingChina<EMAIL_ADDRESS>
###### Abstract.

The rapid development of domain-specific frameworks has presented us with a
significant challenge: The current approach of implementing solutions on a
case-by-case basis incurs a theoretical complexity of O(M$\times$N), thereby
increasing the cost of porting applications to different hardware platforms.
To address these challenges, we propose a systematic methodology that
effectively bridges the gap between domain-specific frameworks and multiple
hardware devices, reducing porting complexity to O(M+N). The approach utilizes
multi-layer abstractions. Different domain-specific abstractions are employed
to represent applications from various domains. These abstractions are then
transformed into a unified abstraction, which is subsequently translated into
combinations of primitive operators. Finally, these operators are mapped to
multiple hardware platforms. The implemented unified framework supports deep
learning, classical machine learning, and data analysis across X86, ARM,
RISC-V, IoT devices, and GPU. It outperforms existing solutions like scikit-
learn, hummingbird, Spark, and pandas, achieving impressive speedups: 1.1x to
3.83x on X86 servers, 1.06x to 4.33x on ARM IoT devices, 1.25x to 3.72x on
RISC-V IoT devices, and 1.93x on GPU. The source code is available at
https://github.com/BenchCouncil/bridger.git.

Deep Learning, Classical Machine Learning, Data Analysis, Runtime System

##  1\. Introduction

Different application domains exhibit distinct characteristics in terms of
data and computation. In deep learning (DL), tensors are prevalent, and
iterative processes driven by backpropagation are essential
(werbos1990backpropagation, ). Classical machine learning (CML) primarily
deals with arrays, although certain algorithms can handle tables and tensors
as well. CML also involves iterative processes, but they are not specifically
tied to backpropagation. Data analytics (DA) mainly revolves around tables and
arrays, with computations that do not necessarily require iteration.

To cater to the specific requirements of each domain, provide user-friendly
interfaces, and promote code reuse, current high-level applications often rely
on domain-specific frameworks. Some popular frameworks include TensorFlow
(10.5555/3026877.3026899, ), PyTorch (NEURIPS2019_bdbca288, ), and Mxnet
(chen2015mxnet, ) for DL; scikit-learn (10.5555/1953048.2078195, ), Spark
MLlib (10.5555/2946645.2946679, ), and XGBoost (chen2016xgboost, ) for CML;
and pandas (mckinney2011pandas, ), Spark (zaharia2010spark, ), and Dask
(rocklin2015dask, ) for DA. These frameworks enable developers to work
efficiently within their respective domains.

The rapid development of domain-specific frameworks has presented us with a
significant challenge: the current approach of implementing solutions on a
case-by-case basis incurs a theoretical complexity of O(M$\times$N), thereby
increasing the cost of porting applications to different hardware platforms.
As depicted in Fig. 1(a), we can observe the existence of numerous domain-
specific frameworks in the domains of DL, CML, and DA. However, these
frameworks often lack comprehensive support for the wide range of underlying
hardware options.

Frameworks like scikit-learn and pandas, which are widely utilized in CML and
DA, are primarily designed for hardware with mature software ecosystems such
as X86 and ARM. Consequently, they may not possess the necessary support and
optimizations for emerging hardware architectures like RISC-V, GPUs, and
Internet of Things (IoT) devices.

This gap poses challenges for both software and hardware utilization. From a
software perspective, the migration of applications to different hardware
platforms becomes significantly challenging and costly. On the hardware side,
emerging hardware devices face limitations in their development due to
inadequate software ecosystems.

(a) Current domain-specific frameworks lack support and optimizations for
diverse hardware devices. Case-by-case implementations have a complexity of
O(M$\times$N).

(b) We use domain-specific abstractions and unified abstractions to bridge
applications and hardware, reducing porting complexity from O(M$\times$N) to
O(M+N).

Figure 1. We propose a method to systematically bridge the gap between domain-
specific frameworks and multiple hardware devices.

Additionally, domain-specific frameworks exist in isolation, with the progress
made in one application domain unable to be shared with others, resulting in a
significant amount of duplicated work. DL has garnered significant attention
and investment, benefiting from relatively robust hardware support and
optimizations (mohammadi2018deep, ; debauche2020new, ; vrevca2020accelerating,
; zhang2019openei, ; chen2019eyeriss, ; tan2021efficient, ; niu2022gcd, ;
prakash2023cfu, ; yan2023hardware, ). However, CML and DA have not received as
much attention. They have limited hardware support and are unable to leverage
the advancements in the DL domain.

This paper proposes a systematic methodology for constructing a unified
framework that bridges the gap between multiple domain-specific frameworks and
diverse hardware devices. The methodology outlined in Fig. 1(b) demonstrates
how this approach reduces the complexity of porting applications to different
hardware platforms, achieving a complexity of O(M+N). It employs multi-layer
abstractions: various domain-specific abstractions are used to represent
applications from different domains, which are transformed into a unified
abstraction, then lowered to the combinations of primitive operators that are
finally mapped to multiple hardware platforms. It utilizes domain-specific
abstractions to support diverse application domains, addresses portability
issues through unified abstraction, leverages existing frameworks and
compilers to reduce engineering costs, and incorporates multi-level
optimizations to guarantee performance. A unified framework is implemented to
support high-level application domains, including DL, CML, and DA, and low-
level hardware devices, such as X86, ARM, RISC-V, IoT devices, and GPU.

This paper makes the following contributions:

  * •

We present a systematic methodology for constructing a unified framework that
effectively bridges the gap between domain-specific frameworks and hardware
devices. This methodology reduces the theoretical complexity of porting
applications from O(M$\times$N) to O(M + N).

  * •

We design a unified framework that supports three application domains
including DL, CML, and DA on various hardware devices such as X86, ARM,
RISC-V, IoT devices, and GPU.

  * •

We implement our work on top of TVM and LLVM, support a broader range of
hardware, and achieve a speedup of 1.1x to 3.83x on X86 servers, 1.06x to
4.33x on ARM IoT devices, 1.25x to 3.72x on RISC-V IoT devices, and 1.93x on
GPU, compared to existing solutions such as scikit-learn, hummingbird, Spark,
and pandas.

The remainder of the paper is organized as follows. Section 2 introduces the
challenges. Section 3 introduces the method. Section 4 details the design and
implementation. Section 5 showcases the experimental results. Section 6
discusses related work. Finally, we conclude in Section 7.

##  2\. Challenges

To facilitate the efficient execution of diverse applications from different
domains across multiple hardware platforms, three main challenges need to be
addressed: portability, performance, and expressiveness. Section 2.1
introduces portability, Section 2.2 introduces performance, Section 2.3
introduces expressiveness.

###  2.1. Portability

Portability refers to the ability to transfer a program from one hardware
environment to another (poole1975portability, ), and the primary factors
limiting portability are hardware differences such as architecture and
instruction sets. Efficiently supporting complex applications on diverse and
significantly different hardware has always been a hot topic in computer
science.

In the early days, developers primarily used assembly language to develop
programs on a specific hardware platform (booth1947coding, ;
wilkes1951preparation, ), without considering portability. Early compilers
(rutishauser1951automatische, ; backus1957fortran, ; 10.1145/321439.321440, )
could translate high-level languages into assembly code, but they were
specific to a particular hardware platform and lacked portability. With the
increasing variety of hardware types, the case-by-case implementation could no
longer meet the users’ demands, and the portability issue entered the
researchers’ view.

In the late 1970s, PQCC (leverett1980overview, ; wulf1980pqcc, ;
cattell1979code, ; brosgol1980tcolada, ) introduced a two-stage structure. The
frontend analyzes syntax and semantics to generate Intermediate Representation
(IR), and the backend reads the IR to generate executable code. PQCC allowed
the same code to run on multiple hardware platforms, ensuring portability.
Subsequent compilers such as GCC (stallman2003using, ) and LLVM
(lattner2004llvm, ) introduced a middle-end between the frontend and backend,
forming a three-stage structure. They support more programming languages and
hardware platforms and introduce more optimization techniques.

Looking back at the history of compiler development, we can find that the
introduction of the two-stage structure and unified IR marked significant
milestones. When it comes to supporting M programming languages on N hardware
platforms, implementing programming language on hardware case-by-case leads to
a complexity of O(M $\times$ N). By utilizing a unified IR, where various
programming languages can be translated into the IR and then used to generate
executable code for different hardware platforms, the complexity reduces to
O(M + N). The introduction of the IR bridges the gap between source code and
hardware, significantly reducing porting complexity and engineering costs, and
finally improving portability.

General-purpose compilers allow users to develop programs and generate
executable code for different hardware platforms, effectively addressing the
issue of portability. However, they lack domain-specific information and
require users to perform domain-specific optimizations. Additionally, users
need to implement algorithms from scratch at a low level, which can be quite
labor-intensive. With the rapid development of emerging application domains
such as DL, CML, and DA, users pay more attention to adapting high-level
applications and innovating algorithms, hoping to reuse underlying
implementations and optimizations. General-purpose compilers cannot fully meet
these user needs. As a result, many domain-specific frameworks have emerged.
They encapsulate algorithms specific to a particular domain, hiding the low-
level implementation details, and making it easier for users to utilize.
Consequently, these frameworks have gained popularity. However, this situation
raises the issue of portability once again.

Porting a framework to a particular hardware typically involves two methods:
leveraging high-performance libraries like BLAS (blackford2002updated, ) and
cuDNN (chetlur2014cudnn, ), or utilizing compilers such as LLVM
(lattner2004llvm, ). DL frameworks like TensorFlow and PyTorch use high-
performance libraries to support hardware like X86, ARM, and GPUs. However,
many hardware devices, such as numerous IoT devices, lack the necessary
libraries required by these frameworks. DL compilers like TVM support such
hardware through compilation, by generating LLVM IR and utilizing LLVM to
support them. However, CML and DL frameworks such as scikit-learn and pandas
primarily rely on libraries and are mainly fit for hardware with mature
ecosystems like X86 and ARM, while lacking support and optimizations for
emerging hardware like RISC-V, GPUs, and IoT devices.

###  2.2. Performance

Performance is also an important challenge, especially for latency-sensitive
and resource-constrained tasks. To ensure performance, multiple levels need to
be considered.

At the highest level, the characteristics of the application domain need to be
analyzed to fully explore optimization opportunities based on their features.
Take DL as an example. Many operators have data dependencies and their
mathematical properties ensure that their fusion does not affect the results,
leading to operator fusion optimizations (niu2021dnnfusion, ; zhao2022apollo,
; zheng2023chimera, ). Certain subgraphs in the computational graph can be
transformed equivalently, suitable for different scenarios, which gives rise
to graph substitutions replacing optimizations (jia2019taso, ; ma2020rammer, ;
fang2020optimizing, ). Data can have different layout formats, such as NCHW,
NHWC, and NCHWc, which are suitable for different workloads and hardware,
resulting in data layout transformation optimizations (liu2019optimizing, ;
xing2019dnnvm, ).

At the lowest level, it is crucial to fully leverage hardware features,
including processor architecture and instruction sets, parallelism, and memory
hierarchy. Hardware-related optimizations include loop transformation
(davidson1995aggressive, ; huang1999generalized, ; booshehri2013improving, ;
kennedy1993maximizing, ; fraboulet2001loop, ; mehta2014revisiting, ;
callahan1988compiling, ; mahlke1992effective, ), parallelization
(subhlok1993exploiting, ; lilja1994exploiting, ; reinders2007intel, ;
karlin2013exploring, ), vectorization (eichenberger2004vectorization, ;
nuzman2006multi, ; nuzman2006auto, ), among others.

In addition, many optimization techniques at the intermediate level are
unrelated to the application and hardware but are relatively more generic,
such as common subexpression elimination (cocke1970global, ;
potkonjak1996multiple, ) and dead code elimination (knoop1994partial, ;
xi1999dead, ). To ensure performance, optimization methods at all these levels
should not be overlooked.

High-performance libraries encapsulate common computations and provide user-
friendly programming interfaces after manual tuning. Many hardware vendors
also provide high-performance libraries targeting their hardware
(wang2014intel, ; armcomputelib, ; cublas, ; cutensor, ; chetlur2014cudnn, ).
These high-performance libraries cover low-level and intermediate-level
optimizations but do not involve domain-specific optimizations. Currently,
many domain-specific frameworks call high-performance libraries and supplement
them with domain-specific optimizations. The problem with high-performance
libraries is that, on the one hand, they do not support comprehensive
computations and often cannot meet the requirements of high-level application
domains. On the other hand, they have limited hardware support, lacking
optimizations for many emerging hardware platforms.

General-purpose compilers like GCC and LLVM cover low-level and intermediate-
level optimizations but do not involve domain-specific optimizations. They
require high-level frameworks to provide domain-specific optimizations.
Currently, these three application domains do not directly utilize general-
purpose compilers to support high-level applications but rather use them as
backends to efficiently support diverse hardware platforms. However, the
automatic optimizations of general-purpose compilers are not comprehensive,
and they require optimization-related information from high-level frameworks.

DL frameworks like TensorFlow and PyTorch have unified intermediate
abstractions that allow the reuse of many domain-specific optimizations. This
enables them to fully leverage domain-specific features. The intermediate and
low-level optimizations of these frameworks rely on high-performance
libraries, facing challenges in optimizing for emerging hardware. DL compilers
like XLA and TVM follow the same high-level optimizations but interface with
general-purpose compilers like LLVM at the lower level. They utilize these
compilers for intermediate and low-level optimizations, supporting a wider
range of hardware platforms compared to high-performance libraries and
achieving better performance on some hardware.

CML frameworks like scikit-learn and Spark MLlib lack intermediate
abstractions, and domain-specific optimizations are performed during algorithm
implementations. This makes it difficult to reuse optimizations for different
algorithms, which impacts performance and increases engineering costs.
Intermediate and low-level optimizations rely entirely on high-performance
libraries but are also limited by them.

DA frameworks like Spark and pandas have unified intermediate abstractions
that allow the reuse of domain-specific optimizations to fully leverage
domain-specific features. They also rely on high-performance libraries and are
therefore constrained by them. In summary, ensuring performance across
multiple hardware platforms is an important challenge, related to multi-level
optimizations.

###  2.3. Expressiveness

To support high-level applications, the framework needs to have strong
expressiveness, which involves clear and accurate data abstraction,
computation abstraction, and representation of data flow and computation
dependencies.

Data abstraction encompasses the comprehensive representation of the data used
in the applications and the effective organization and manipulation of this
data. Computation abstraction requires an accurate description of the
computations within the applications, breaking down complex applications into
simpler and more manageable computational units that can be combined. Data
flow and computation dependencies are commonly represented using graphs, with
the most common ones being data flow graphs (1676696, ) and control flow
graphs (10.1145/390013.808479, ).

Due to the variations in data and computations across different application
domains, their data and computation abstractions differ as well. We discuss
DL, CML, and DA separately. The data used in DL includes images, text, audio,
and more. Most of these data types are represented, stored, and processed
using tensors, which are the fundamental data abstraction for DL. In DL,
neural network models usually comprise multiple layers, each performing
various computational operations such as convolution, matrix multiplication,
and activation functions. These operations, known as operators, serve as the
fundamental computation abstraction. High-level applications are built by
combining these operators in different ways. Computational graphs, represented
as directed acyclic graphs (DAGs), are used to represent the data flow and
computation dependencies. In these graphs, nodes represent operators, and
edges represent tensors.

The data used in CML includes tables, text, images, and more. The data
abstraction in CML is relatively complex and includes scalars, matrices,
arrays, and tables. Applications in CML are composed of multiple models
organized in pipelines, including data preprocessing, linear models, trees,
forests, and more. These models are relatively complex but are still
considered as the minimum units for implementation and scheduling. Pipelines
are represented using DAGs, where nodes represent models and edges represent
different types of data.

DA primarily deals with data such as tables, text and time series. The data
abstraction in DA includes relational tables, scalars, and arrays.
Applications in DA are composed of pipelines that combine various operators,
including relational operators like selection, projection, and join, as well
as basic mathematical and statistical operators like matrix inverse and
covariance. These operators serve as the computation abstraction. Pipelines in
DA are represented using DAGs, where nodes represent operators and edges
represent data.

We find that the data flow and computation dependencies in these application
domains can all be represented using DAGs. However, the data and computation
abstractions differ significantly. The challenge lies in how to represent
applications from these different domains and how to represent cross-domain
applications effectively.

##  3\. Methodology

This section introduces the methodology to systematically bridge the gap
between domain-specific frameworks and multiple hardware devices, as shown in
Fig. 1(b). We use domain-specific abstractions and unified abstractions to
bridge applications and hardware, reducing porting complexity from
O(M$\times$N) to O(M+N).

###  3.1. Address Portability Issues through Intermediate Abstraction

Looking back at the development history of general-purpose compilers and the
ecosystems of the three application domains, it can be observed that the
primary method for addressing portability is through the introduction of
intermediate abstractions, bridging the gap between high-level application
domains and low-level hardware, thus reducing the porting costs from O(M
$\times$ N) to O(M + N). Based on a similar concept, we propose a unified
intermediate abstraction for the three application domains, which can
represent high-level algorithms and is then mapped to various low-level
hardware platforms.

###  3.2. Propose Domain-specific Abstractions and Unified Abstraction for
High-level Applications

Table 1. The differences in Classical Machine Learning, Deep Learning and Data Analysis |  Deep Learning |  Classical Machine Learning |  Data Analysis
---|---|---|---
Data Structure |  tensor |  scalar, array, table |  relational table, scalar, array
Data Type |  numeric |  numeric |  numeric, non-numeric
Computation Abstraction |  Operators as the smallest unit; Algorithms represented as the combinations of operators |  No computation abstraction; Case-by-case implementation; Large number of models |  Operators as the smallest unit; Small number of operators

The intermediate abstractions need to be able to represent applications from
different domains. We propose a two-layer abstraction: domain-specific
abstractions for interfacing with application domains, which are then
transformed into a unified abstraction for interfacing with hardware. The
unified abstraction can also be used to represent cross-domain applications.

First, we introduce the unified abstraction, which requires the exploration of
commonalities in three application domains. The biggest commonality among DL,
CML, and DA is that they can all be represented as DAGs to depict data flow.
This forms the basis for cross-domain pipelines and forms the establishment of
a unified abstraction. Therefore, we propose a unified abstraction based on
DAG.

The meanings of nodes and edges in the corresponding DAGs of the three domains
differ due to the differences in data and computation abstractions. To
accurately represent applications from the three domains, we propose domain-
specific abstractions for each domain. As shown in Table 1, for DL, data
mainly consists of numerical tensors and the computation abstraction treats DL
operators as the smallest unit, with algorithms represented as the
combinations of operators. We adopt the existing DL abstractions, using
tensors as the data abstraction and DL operators as the computation
abstraction.

For CML, data mainly consists of numerical scalars, arrays, and tables, all of
which can be converted to tensors directly. However, there is no suitable
computation abstraction. Models are implemented case by case, and there are a
large number of models. The problem lies in computation abstraction. We use
classical operator representation (COR) (wen2023cmlcompiler, ) as the
computation abstraction of CML. It represents CML models as combinations of
assignment, exchange, basic arithmetic, aggregation, comparison, and
conditional operators. In summary, we use tensors as the data abstraction and
COR as the computation abstraction for CML.

For DA, data structures include relational tables, scalars, and arrays, with
both numerical and non-numerical data types. The computation abstraction in DA
also treats operators as the smallest unit, and there are only a small number
of operators. The problem lies in data abstraction. We use TensorTable
(wen2024tensortable, ) as the data abstraction of DA, it stores data in a
tensor format and can represent different data structures and data types
mentioned above. In summary, we use TensorTable as the data abstraction and DA
operators as the computation abstraction for DA.

With the above approach, appropriate domain-specific abstractions are
constructed for different application domains. Subsequently, three domain-
specific abstractions are transformed into the unified abstraction. The data
abstraction of the unified abstraction includes tensor and TensorTable, while
the computation abstraction includes DL operators, COR, and DA operators.

###  3.3. Reuse Existing Frameworks and Compilers to Reduce Engineering Cost

We use domain-specific abstractions and unified abstractions to represent
high-level applications. However, to run multiple applications on different
hardware devices, these abstractions need to be mapped to the hardware. We
leverage existing frameworks and compilers to support multiple hardware
platforms, thereby reducing the engineering cost. Due to the better
portability of DL compilers compared with other domain-specific frameworks of
three domains, we reuse DL compilers to support hardware such as X86, ARM,
GPUs, and IoT devices. However, DL compilers also lack support for certain
hardware platforms like RISC-V. Therefore, we introduce a low-level code
generator to generate LLVM IR specifically for RISC-V and similar hardware,
leveraging LLVM’s support for these hardware. Additionally, DL compilers rely
on auto-tuning for optimizations, which incurs high time costs and are not
suitable for CML and DA. Therefore, we propose new optimization methods
specifically for them.

###  3.4. Use Multi-level Optimizations to Guarantee Performance

To improve performance, it is necessary to fully utilize the characteristics
of application domains and leverage hardware capabilities. Our work mainly
includes two-level optimizations: graph-level optimizations and operator-level
optimizations.

Graph-level optimization can be categorized into three types. Domain-specific
optimizations are tailored to the characteristics of each application domain
and are unique. To capture the domain-specific features, we propose domain-
specific abstractions for the three application domains and conduct domain-
specific optimization based on these abstractions, such as operator fusion and
data layout transformation for DL, sparse operator replacing and data type
rewriting for CML, as well as operator reorder and operator fusion for DA.
After domain-specific optimizations, three domain-specific abstractions are
transformed into a unified abstraction. The unified abstraction enables cross-
domain optimizations for mixed workloads from three domains, primarily
including cross-domain operator reorder and cross-domain operator fusion.
Subsequently, generic graph-level optimizations are applied to the unified
abstraction, such as constant propagation and common subexpression
elimination.

After graph-level optimizations are completed, operator-level optimizations
are performed. Operator-level optimization mainly consists of two types:
hardware-agnostic optimizations such as constant folding and copy propagation,
and hardware-specific optimizations such as loop transformation and
vectorization. All those optimizations work together to guarantee performance.

##  4\. The System Design and Implementation

This section illustrates the design and implementation. Section 4.1 shows the
design overview. Section 4.2 introduces domain-specific abstractions and
optimizations. Section 4.3 introduces unified abstraction and optimizations.
Section 4.4 introduces primitive operator set. Section 4.5 introduces code
generation and operator-level optimizations. Section 4.6 shows the
implementation.

###  4.1. Design Overview

Figure 2. Design Overview.

Fig. 2 shows the design overview. The frontend is used to read models and
pipelines from other frameworks and convert them to domain-specific
abstractions. Cross-domain applications that use multiple frameworks are
converted to many subgraphs and stitched together based on data dependencies.
Domain-specific abstractions are used to represent different application
domains, capturing their data and computation features. They are converted to
a DAG-based unified abstraction, which is then mapped to the combination of
primitive operators. Multi-level optimizations are performed for those
abstractions, including domain-specific optimizations for domain-specific
abstractions, cross-domain and generic graph-level optimizations for DAG-based
unified abstraction, as well as operator-level optimizations for primitive
operators. The backend is used to generate executable code. The runtime reads
executable code and input data, running on multiple hardware.

###  4.2. Domain-Specific Abstractions and Optimizations

We use domain-specific abstractions to represent high-level applications, all
of which are based on DAG, with different data and computation abstraction.
For DL, we use tensors as the data abstraction and DL operators as the
computation abstraction. For CML, we use tensors as the data abstraction and
classical operator representation (COR) (wen2023cmlcompiler, ) as the
computation abstraction. COR uses the combinations of six categories of
operators, namely assignment, swap, basic arithmetic, aggregation, comparison,
and conditional operators, to represent widely-used CML algorithms, such as
preprocessing algorithms, linear models, trees and forests, and SVMs. For DA,
we use TensorTable (wen2024tensortable, ) as the data abstraction and DA
operators as the computation abstraction. TensorTable represents relational
tables as a combination of tensors and dictionaries, by converting those non-
numerical data into numerical data and using auxiliary dictionaries to
preserve the mapping relations. TensorTable can also seamlessly encapsulate
other data structures such as scalars and arrays. DA operators are all
implemented based on TensorTable, including relational operators, covering
selection, projection, join, group by, and aggregation, as well as basic
mathematical and statistical operations.

We conduct domain-specific optimizations based on these abstractions to fully
utilize the features of each domain, as shown in Table 2. Domain-specific
optimizations for DL include operator fusion, data layout transformation,
graph substitutions replacing, and tensor recomputation. Domain-specific
optimizations for CML include sparse operator replacing, data type rewriting,
and redundant operator elimination. Domain-specific optimizations for DA
include operator reorder and operator fusion.

Table 2. Domain-specific optimizations for domain-specific abstractions Deep Learning |  Classical Machine Learning |  Data Analysis
---|---|---
Operator fusion, Data layout transformation, Graph substitutions replacing, Tensor recomputation |  Sparse operator replacing, Data type rewriting, Redundant operator elimination |  Operator reorder, Operator fusion

###  4.3. Unified Abstractions and Optimizations

After domain-specific optimizations, all three domain-specific abstractions
are transformed into a unified abstraction. It’s also based on DAG, with nodes
representing operators and edges denoting data. Its data abstraction includes
both tensor and TensorTable, and its computation abstraction includes DL
operators, COR, and DA operators.

As shown in Table 3, unified abstraction enables many optimizations, including
cross-domain optimizations and generic graph-level optimizations. Cross-domain
optimizations cater to mixed workloads from three domains, primarily including
cross-domain operator reoder and cross-domain operator fusion. Cross-domain
operator reorder changes the order of operators from different domains which
meet commutativity, to decrease computation. Cross-domain operator fusion
merges operators from different domains into one operator, to reduce
intermediate data. Generic graph-level optimizations include constant folding,
constant propagation, copy propagation, common subexpression elimination, dead
code elimination, and more.

Table 3. Graph-level optimizations for unified abstraction Cross-domain
Optimizations
---
Cross-domain operator reorder, Cross-domain operator fusion
Generic Optimizations
Constant folding, Constant propagation, Copy propagation, Common subexpression
elimination, Dead code elimination, etc.

###  4.4. Primitive Operator Set

After graph-level optimizations, the unified abstraction is mapped down to the
combinations of operators. DL encompasses over two thousand
operators111https://pytorch.org/get-started/pytorch-2.0/#primtorch-stable-
primitive-operators, while CML and DA each involve dozens of operators. The
cost of implementing and optimizing operators one by one is prohibitively
high. Therefore, we construct a primitive operator set consisting of eighty
operators, focusing on their implementations and optimizations. Algorithms
used by high-level applications can be represented as the combinations of
them. Subsequently, the primitive operator set is implemented, generating
executable code tailored for multiple hardware platforms.

The process of constructing the primitive operator set is as follows:

(1) We summarize all required operators in DL, CML, and DA, totaling around
2300 operators. (2) We eliminate long-tail operators that are rarely used and
those various overloads for the same operators, reducing to around 300
operators. (3) We retain operators that cannot be formed by combinations of
other operators or those whose performance is significantly lower when
combined, resulting in 80 operators, as the primitive operator set. The final
primitive operator set is shown in Table 4.

Table 4. Primitive operators Injective operators
---
mod, floor, ceil, clip, round, abs, negative, pow, log, sqrt, exp, sin, cos,
tan, not
Element-wise operators
add, subtract, multiply, divide, equal, not_equal, greater, greater_equal,
less, less_equal, all, any, and, or, xor, where
Reduction operators
max, min, sum, mean, topk, cumsum, cumprod
Index operators
nonzero, argmax, argmin, argsort, argwhere
Memory operators
reshape, copy, transpose, squeeze, take, cast, tile, reverse, slice, scatter,
split, flatten, gather, expand, concatenate
Key construct opertors
inner_key_construct, outer_key_construct, left_key_construct,
right_key_construct, cross_key_construct
Special computation operators
matmul, conv, max_pool, avg_pool, one_hot, resize, pad, dropout, unique, sort,
intersection, union, complement
Control operators
if_then_else, loop, while, scan

Primitive operators are primarily divided into eight categories: Injective
operators perform identical computations on each element within a single
tensor, including operations like modulus (mod), negation (negative), and
absolute value (abs), among others. Element-wise operators perform operations
between corresponding elements in two tensors, such as addition (add),
subtraction (subtract), multiplication (multiply), division (divide), and so
on. Reduction operators derive aggregated results from a single tensor across
one or multiple dimensions. Examples include maximum value (max), minimum
value (min), summation (sum), and mean computation (mean), among others. Index
operators compute each element within a single tensor and return the
corresponding index. Examples include returning indices for non-zero values
(nonzero), returning indices for maximum values (argmax), and indices for
minimum values (argmin), among others. Memory operators manage memory-related
actions, such as reshaping (reshape), copying data (copy), transposing
(transpose), and others. Key construct operators are used in join operations
to build key-value indices, including inner_key_construct to construct keys
for inner join, outer_key_construct to construct keys for outer join, and
others. Special computation operators encompass complex and crucial
computations like matrix multiplication (matmul) and convolution (conv). While
these operations can be expressed as the combinations of injective, element-
wise, and reduction operators, they are retained separately to ensure
optimizations, especially for hardware-specific optimizations. Control
operators represent control flow within computations, such as if condition
(if_then_else) and loop.

###  4.5. Code Generation and Operator-level Optimizations

For primitive operators, we adopt scheduling techniques proposed by Halide
(ragan2013halide, ) and TVM to describe and optimize them. The utilized
schedules are outlined in Table 5. Optimizations such as parallelization,
memory hierarchy, and vectorization are employed based on hardware
specifications to ensure performance. For hardware such as X86, ARM, and GPUs
that are well supported by TVM, the code generation part of TVM is directly
reused to generate executable code. To support more hardware platforms such as
RISC-V, we supplement code generation modules specifically for them, which can
generate LLVM IR that can be further optimized and translated into executable
code using LLVM. Parallelization is directly implemented using relevant APIs
like llvm::parallel. Vectorization is achieved through hardware-specific LLVM
intrinsics, for instance, $llvm.riscv.v$ for RISC-V vector extension
instructions.

Table 5. Schedules used in Primitive Operators Schedule |  Explanation
---|---
split |  Split one axis into multiple axes
tile |  Execute the computation tile by tile over two axes
fuse |  Merge multiple axes into one
reorder |  Change the orders of axes
compute_at |  Compute at certain axis
parallel |  Use multi-threading
vectorize/tensorize |  Use vectorization or tensorization instructions

Regarding cache-related optimizations, since many hardware cannot directly
manage various levels of cache, optimization is achieved by adjusting
parameters within the schedule, such as tile size and split size. We make
cache-related optimizations by solving a constrained optimization problem:

(1) $\displaystyle\mathop{argmin}\limits_{conf}(\mathop{max}\limits_{l\in
L}(\frac{DataMove_{l}}{BandWidth_{l}}))$ $\displaystyle\ .s.t.\ \forall l\in
L\ DataOccupy_{l}<Capacity_{l}$

The constraints ensure that the data occupation of each cache level is smaller
than its corresponding cache capacity. The optimization objective is to
minimize data movement time. The symbols are listed in Table 6.

Table 6. Symbols in Equation 1. Symbol |  Explanation
---|---
$conf$ |  The configurations in schedules
$DataMove_{l}$ |  The data movement of $l$-level cache
$DataOccupy_{l}$ |  The data occupation of $l$-level cache
$L$ |  Cache level
$BandWidth_{l}$ |  The bandwidth of $l$-level cache
$Capacity_{l}$ |  The capacity of $l$-level cache
Table 7. Operator-level Optimizations for Primitive Operators Hardware-
agnostic Optimizations
---
Inline, Constant folding, Constant propagation, Copy propagation, Common
subexpression elimination, Dead code elimination, etc.
Hardware-specific Optimizations
Loop unrolling, Loop fusion, Loop splitting, Blocking, Data alignment,
Parallelization, Vectorization, Register allocation, Instruction scheduling,
etc.

During code generation, we make operator-level optimizations, mainly
consisting of two types: hardware-agnostic optimizations and hardware-specific
optimizations, as shown in Table 7. Hardware-agnostic optimization includes
inlining, constant folding, constant propagation, copy propagation, common
subexpression elimination, dead code elimination, and more. Hardware-specific
optimization includes loop unrolling, loop fusion, loop splitting, blocking,
data alignment, parallelization, vectorization, tensorization, register
allocation, instruction scheduling, and more. Many graph-level optimizations
and operator-level optimizations share similar names and approaches, but they
are targeted at different entities (graphs and operators) and have significant
differences in implementations.

Finally, we deploy executable code on lightweight runtime to run DL, CML, and
DA applications, as well as cross-domain mixed applications, on multiple
hardware platforms.

###  4.6. Implementation

Figure 3. System Architecture.

The system architecture is shown in Fig. 3. The blue part represents the
components reused from TVM, the green part represents the components reused
from LLVM and CUDA, and the yellow part represents our contribution. At the
top level, there are application models and pipelines from other frameworks,
which are transformed into three domain-specific abstractions using APIs like
$from\\_pytorch$ and $from\\_sklearn$. After domain-specific optimizations,
the three domain-specific abstractions are transformed into a DAG-based
unified abstraction. The unified abstraction, after graph-level optimization,
is mapped down to the combinations of primitive operators. The primitive
operators are implemented using TVM, adopting the decoupled compute/schedule
principle (ragan2013halide, ; 10.5555/3291168.3291211, ), striving to reuse
code as much as possible while ensuring comprehensive optimization. Hardware-
agnostic and hardware-specific optimizations are performed at the operator
level. Subsequently, executable code is generated for multiple backends. For
hardware such as X86, ARM, and GPUs that are well supported by TVM, the code
generation part of TVM is directly reused to generate executable code. To
support more hardware platforms such as RISC-V, we supplement code generation
modules specifically for them, which can generate LLVM IR that can be further
optimized and translated into executable code using LLVM. We make cache-
related optimizations by solving a constrained optimization problem instead of
auto-tuning used by TVM, better fitting for CML and DA workloads. Finally, the
generated code runs on lightweight runtime which is implemented based on TVM
runtime and executes on multiple hardware platforms.

##  5\. Evaluation

This section summarizes the evaluation. Section 5.1 shows the experimental
setup. Section 5.2 evaluates the coverage of primitive operators. Section 5.3
evaluates the performance on diverse hardware.

###  5.1. Experimental Setup

We evaluate the performance on X86 server, ARM IoT device, RISC-V IoT device,
and GPU. We deploy a server node equipped with two Xeon E5-2620 V3 (Haswell)
CPUs, an Nvidia Titan RTX GPU, and 64 GB memory to conduct the experiments on
x86 server and GPU. Each CPU contains 6 physical cores with hyper-threading.
The GPU contains 4608 CUDA cores and 24 GB memory, CUDA version is 10.2. The
operating system used is Ubuntu 16.04. The ARM IoT device used is the
Raspberry Pi 4B, which features a quad-core Cortex-A72 CPU and 4GB memory. It
runs on the Raspbian 10 operating system. The RISC-V IoT device employed is
the VisionFive 2, equipped with a quad-core JH7110 CPU and 8GB memory. It runs
on the Debian 12 operating system. The software utilized includes TVM 0.15,
scikit-learn 1.1.2, Intel(R) Extension for Scikit-learn 2021.20220215.162512,
hummingbird 0.3.1, CMLCompiler 1.01, pandas 1.3.5, Spark 3.3.0, and MonetDB
11.39.17.

###  5.2. The Coverage of Primitive Operators

Figure 4. The coverage of primitive operators.

First, we analyze the coverage of the primitive operator set. To begin, we
analyze the coverage within the DL domain. A set of 103 widely-used DL models
is extracted from the ONNX model repository222https://github.com/onnx/models.
Among them, 101 models were entirely composed of primitive operators, while
only two models contained long-tail operators. Specifically, the ResNet-
preproc model has the $sequenc\\_map$ operator, and BiDAF model has the
$category\\_mapper$, $hardmax$, and $compress$ operators, which are long-tail
operators. Next, we analyze the coverage within the CML domain. We extract 47
high-frequency models shared between two widely used frameworks scikit-learn
and Spark MLlib. All these models can be entirely represented by primitive
operators, without any occurrences of long-tail operators. Finally, we analyze
the coverage within the DA domain. We analyze 22 DA pipelines from
TPC-H333https://www.tpc.org/tpch/. All of them can be fully represented by
primitive operators, without any long-tail operator occurrences. In summary,
as illustrated in Fig. 4, 98% of the 103 DL models, all 47 CML models, and all
22 DA pipelines can be represented by the primitive operators. This
demonstrates that the primitive operator set adequately covers the majority of
scenarios in DL, CML, and DA applications.

###  5.3. Performance

Figure 5. The performance on X86 server. The absent data means unsupported.

This section compares the performance on various hardware platforms against
existing frameworks. We test ten CML models and compare our work with sklearn,
intel sklearn, CMLCompiler, and hummingbird. Use YearPrediction dataset
(Dua:2019, ), which consists of 51,630 samples and 90 features. We test five
DA operators and compare our work with MonetDB, Spark, and pandas. Use BIXI
dataset (faghih2014land, ) at data scales of 1k, 10k, 100k, 1m, and 10m rows,
and show their geometric means. All experiments are repeated five times, and
the arithmetic mean is used as the final result.

Fig. 5 displays the performance on x86 server. Our work achieves the best
performance in 7 out of 10 tested CML models. CMLCompiler attains the best
performance in 3 models, while sklearn, intel sklearn, and Hummingbird do not
outperform in any algorithm. We achieve a 1.95x to 10.21x speedup compared
with sklearn across 10 models, with an average speedup of 3.83. We achieve a
1.2x to 8.18x speedup compared with intel sklearn across 10 models, with an
average speedup of 3.79. We achieve a 2.57x to 5.25x speedup compared with
Hummingbird across 10 models, with an average speedup of 3.24. This is
attributed to our better utilization of multi-core parallelism and SIMD
instructions. We achieve a 1.11x to 3.25x speedup compared with CMLCompiler
across 7 models, with an average speedup of 1.75. For algorithms like
Binarizer, Normalizer, MinMaxScaler, and RobustScaler, which involve
relatively simpler computations, where optimization space is limited, we do
not exhibit significant speedup compared to CMLCompiler, but the performance
difference remains within 5%. For more complex algorithms like
LogisticsRegression, SGDClassifier, and ExtraTreeClassifier, where a larger
optimization space exists, we showcase noticeable speedup due to the memory-
related optimizations.

Our work achieves the best performance in 2 out of 5 tested DA operators. We
achieve a 1.04x to 5.06x speedup compared with MonetDB across 4 operators,
with an average speedup of 2.38. We achieve a 3.34x to 108.37x speedup
compared with Spark across 5 operators, with an average speedup of 28.31. We
achieve a 1.46x to 1.58x speedup compared with pandas across 2 operators, with
an average speedup of 1.1. For the projection operator, we adjust the data
layout, thus exhibiting weaker performance compared to pandas. However, this
implementation ensures data locality, guaranteeing the performance of other
operators. For inner join and group by operators, we showcase a 7% and 20%
performance difference, respectively, which remains within an acceptable
range. Overall, we achieve a noticeable speedup compared to MonetDB and Spark,
and has its strengths compared to pandas.

Figure 6. The performance on ARM IoT device. The absent data means
unsupported.

Fig. 6 illustrates the performance of ARM IoT devices. Our work outperforms
across all 10 tested CML models. We achieve a 1.56x to 9.6x speedup compared
with sklearn, with an average speedup of 4.33. Since sklearn’s high-
performance library lacks optimizations for ARM IoT devices, our work obtains
a larger speedup on this device. We achieve a 1.13x to 2.98x speedup compared
with CMLCompiler, with an average speedup of 2.1. Intel sklearn and
Hummingbird do not support this device, their results are absent.

Our work achieves the best performance in 4 out of 5 tested DA operators. We
achieve a 1.04x to 1.32x speedup compared with pandas across 4 operators, with
an average speedup of 1.06. Except for the projection operator, which
sacrifices some performance to ensure data locality, our work performs equally
or better than pandas for the other operators. MonetDB and Spark do not
support this device, their results are absent.

Figure 7. The performance on RISC-V IoT device. The absent data means
unsupported.

Fig. 7 depicts the performance of RISC-V IoT devices. Our work outperforms
across all 10 tested CML models. We achieve a 1.93x to 10.23x speedup compared
with sklearn, with an average speedup of 3.72. We achieve a 1.02x to 4.32x
speedup compared with CMLCompiler, with an average speedup of 2.03. Because
TVM lacks optimization for RISC-V, CMLCompiler, reliant on TVM for multi-
hardware support, exhibits poor performance on RISC-V, with some algorithms
even slower than sklearn. However, our work uses the code generator to
optimize for RISC-V, resulting in a noticeable speedup compared to
CMLCompiler. Intel sklearn and Hummingbird do not support this device, their
results are absent.

Our work achieves the best performance in 4 out of 5 tested DA operators. We
achieve a 1.09x to 2.14x speedup compared with pandas across 4 operators, with
an average speedup of 1.25. MonetDB and Spark do not support this device,
their results are absent.

Figure 8. The performance on GPU. The absent data means unsupported.

Fig. 8 shows the performance on GPU. Our work achieves the best performance in
9 out of 10 tested CML models. We achieve a 1.16x to 3.47x speedup compared
with Hummingbird across 8 models, with an average speedup of 1.93. All other
frameworks cannot support GPU.

In summary, compared to state-of-the-practice and state-of-the-art frameworks,
our work supports a broader range of hardware and achieves noticeable speedup
on them, especially for devices lacking mature high-performance libraries.

##  6\. Related Work

This section summarizes some typical related works based on their supported
application domains, portability, performance, and methods.

Deep learning (DL) frameworks include TensorFlow (10.5555/3026877.3026899, ),
PyTorch (NEURIPS2019_bdbca288, ), MXNet (chen2015mxnet, ), and Keras
(gulli2017deep, ). They are specifically designed for DL and leverage unified
abstractions to make full use of domain-specific features and hardware,
resulting in good performance. These frameworks rely on high-performance
libraries such as Eigen (eigenweb, ), BLAS (blackford2002updated, ), and cuDNN
(chetlur2014cudnn, ) to support hardware platforms like X86, ARM, and GPUs.
However, they are unable to support hardware platforms that do not provide the
required libraries.

DL compilers include XLA (xla, ), Glow (rotem2018glow, ), Tensor Comprehension
(vasilache2018tensor, ), and TVM (10.5555/3291168.3291211, ). Similar to DL
frameworks, they focus on DL and ensure performance through multi-layer
abstractions. They integrate with general-purpose compilers like LLVM
(lattner2004llvm, ), using code generation to support a wider range of
hardware platforms, resulting in better portability.

Classical machine learning (CML) libraries include LibLinear
(fan2008liblinear, ), SVMlight (joachims1999svmlight, ), and LibSVM
(chang2011libsvm, ). They are dedicated to several CML models and achieve good
performance by implementing and optimizing algorithms from scratch. However,
their portability is limited.

CML frameworks include Scikit-learn (10.5555/1953048.2078195, ), Spark MLlib
(10.5555/2946645.2946679, ), RapidMiner (hofmann2016rapidminer, ), and Weka3
(frank2006weka, ). They implement and optimize algorithms at a higher level
case-by-case while leveraging high-performance libraries to support different
hardware platforms. Their portability and performance have significant
potential for improvement.

Data analysis (DA) frameworks include pandas (mckinney2011pandas, ), Dask
(rocklin2015dask, ), Hadoop (white2012hadoop, ), and Spark (zaharia2010spark,
). They have domain-specific abstractions, leveraging high-performance
libraries to support different hardware platforms. These frameworks exhibit
good performance with moderate portability.

There are also efforts to support multiple application domains through two
main approaches: extending existing frameworks and building from scratch for
specific hardware. Databricks (dai2019bigdl, ) attempts to support both CML
and DA in a DA framework, but it directly reuses existing abstractions, which
are not suitable for DL’s backpropagation and parameter iteration processes,
leading to redundant computations and additional overhead that severely affect
performance. Raven (park2022end, ) supports CML models in a DA framework,
while H2O (h2o_platform, ), Weka3 (lang2019wekadeeplearning4j, ), and KNIME
(fillbrunn2017knime, ) support DL models in CML frameworks. Hummingbird
(nakandala2020tensor, ) supports CML models on DL frameworks, but it lacks
domain-specific optimizations. They all rely on reusing existing abstractions
and invoking high-performance libraries, but cannot guarantee domain-specific
optimizations, resulting in performance deviations. NVIDIA supports all three
application domains on GPUs (chetlur2014cudnn, ; cuml, ; cudf, ). They achieve
good performance by implementing and optimizing each domain’s functionalities
from scratch but lack portability to other hardware platforms.

Our work simultaneously supports all three application domains and provides
support for a wider range of hardware platforms, ensuring both portability and
performance.

##  7\. Conclusion

This paper proposes a methodology for constructing a unified framework to
bridge the gap between domain-specific frameworks and diverse hardware
devices. The methodology leverages multi-layer abstractions, allowing
applications from different domains to be represented using specific
abstractions tailored to their respective fields. These abstractions are
subsequently transformed into a unified abstraction, which is further
translated into a minimum combination of primitive operators. Ultimately,
these operators finally mapped to multiple hardware platforms. This
methodology reduces the theoretical complexity of porting applications from
O(M$\times$N) to O(M + N). A unified framework is implemented, supporting
three application domains, including deep learning, classical machine
learning, and data analysis on various hardware devices such as X86, ARM,
RISC-V, IoT devices, and GPU. Compared to existing solutions such as scikit-
learn, hummingbird, Spark, and pandas, it supports a broader range of hardware
and achieves a speedup of 1.1x to 3.83x on X86 servers, 1.06x to 4.33x on ARM
IoT devices, 1.25x to 3.72x on RISC-V IoT devices, and 1.93x on GPU.

## References

  * [1] Arm compute library.  https://www.arm.com/technologies/compute-library, 2023.
  * [2] cublas: Basic linear algebra on nvidia gpus.  https://developer.nvidia.com/cublas, 2023.
  * [3] cutensor: Tensor linear algebra on nvidia gpus.  https://developer.nvidia.com/cutensor, 2023.
  * [4] Xla: Optimizing compiler for machine learning.  https://www.tensorflow.org/xla, 2023.
  * [5] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng.  Tensorflow: A system for large-scale machine learning.  In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, OSDI’16, page 265–283, USA, 2016. USENIX Association.
  * [6] Frances E. Allen.  Control flow analysis.  5(7):1–19, jul 1970.
  * [7] John W Backus, Robert J Beeber, Sheldon Best, Richard Goldberg, Lois M Haibt, Harlan L Herrick, Robert A Nelson, David Sayre, Peter B Sheridan, Harold Stern, et al.  The fortran automatic coding system.  In Papers presented at the February 26-28, 1957, western joint computer conference: Techniques for reliability, pages 188–198, 1957.
  * [8] L Susan Blackford, Antoine Petitet, Roldan Pozo, Karin Remington, R Clint Whaley, James Demmel, Jack Dongarra, Iain Duff, Sven Hammarling, Greg Henry, et al.  An updated set of basic linear algebra subprograms (blas).  ACM Transactions on Mathematical Software, 28(2):135–151, 2002\.
  * [9] Meisam Booshehri, Abbas Malekpour, and Peter Luksch.  An improving method for loop unrolling.  arXiv preprint arXiv:1308.0698, 2013.
  * [10] Andrew Donald Booth and Kathleen HV Britten.  Coding for arc.  1947\.
  * [11] Benjamin M Brosgol.  Tcolada and the” middle end” of the pqcc ada compiler.  ACM SIGPLAN Notices, 15(11):101–112, 1980.
  * [12] David Callahan and Ken Kennedy.  Compiling programs for distributed-memory multiprocessors.  The Journal of Supercomputing, 2:151–169, 1988.
  * [13] Roderic GG Cattell, Joseph M Newcomer, and Bruce W Leverett.  Code generation in a machine-independent compiler.  ACM SIGPLAN Notices, 14(8):65–75, 1979.
  * [14] Chih-Chung Chang and Chih-Jen Lin.  Libsvm: a library for support vector machines.  ACM transactions on intelligent systems and technology (TIST), 2(3):1–27, 2011.
  * [15] Tianqi Chen and Carlos Guestrin.  Xgboost: A scalable tree boosting system.  In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794, 2016.
  * [16] Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang.  Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems.  arXiv preprint arXiv:1512.01274, 2015.
  * [17] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy.  Tvm: An automated end-to-end optimizing compiler for deep learning.  In Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation, OSDI’18, page 579–594, USA, 2018. USENIX Association.
  * [18] Yu-Hsin Chen, Tien-Ju Yang, Joel Emer, and Vivienne Sze.  Eyeriss v2: A flexible accelerator for emerging deep neural networks on mobile devices.  IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 9(2):292–308, 2019.
  * [19] Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer.  cudnn: Efficient primitives for deep learning.  arXiv preprint arXiv:1410.0759, 2014.
  * [20] John Cocke.  Global common subexpression elimination.  In Proceedings of a symposium on Compiler optimization, pages 20–24, 1970.
  * [21] Jason Jinquan Dai, Yiheng Wang, Xin Qiu, Ding Ding, Yao Zhang, Yanzhang Wang, Xianyan Jia, Cherry Li Zhang, Yan Wan, Zhichao Li, et al.  Bigdl: A distributed deep learning framework for big data.  In Proceedings of the ACM Symposium on Cloud Computing, pages 50–60, 2019.
  * [22] Jack W Davidson and Sanjay Jinturkar.  An aggressive approach to loop unrolling.  Technical report, Citeseer, 1995.
  * [23] Olivier Debauche, Saïd Mahmoudi, Sidi Ahmed Mahmoudi, Pierre Manneback, and Frédéric Lebeau.  A new edge architecture for ai-iot services deployment.  Procedia Computer Science, 175:10–19, 2020.
  * [24] Dheeru Dua and Casey Graff.  UCI machine learning repository, 2017.
  * [25] Alexandre E Eichenberger, Peng Wu, and Kevin O’brien.  Vectorization for simd architectures with alignment constraints.  Acm sigplan notices, 39(6):82–93, 2004.
  * [26] Ahmadreza Faghih-Imani, Naveen Eluru, Ahmed M El-Geneidy, Michael Rabbat, and Usama Haq.  How land-use and urban form impact bicycle flows: evidence from the bicycle-sharing system (bixi) in montreal.  Journal of transport geography, 41:306–314, 2014.
  * [27] Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin.  Liblinear: A library for large linear classification.  the Journal of machine Learning research, 9:1871–1874, 2008.
  * [28] Jingzhi Fang, Yanyan Shen, Yue Wang, and Lei Chen.  Optimizing dnn computation graph using graph substitutions.  Proceedings of the VLDB Endowment, 13(12):2734–2746, 2020.
  * [29] Alexander Fillbrunn, Christian Dietz, Julianus Pfeuffer, René Rahn, Gregory A Landrum, and Michael R Berthold.  Knime for reproducible cross-domain analysis of life science data.  Journal of biotechnology, 261:149–156, 2017.
  * [30] Antoine Fraboulet, Karen Kodary, and Anne Mignotte.  Loop fusion for memory space optimization.  In Proceedings of the 14th international symposium on Systems synthesis, pages 95–100, 2001.
  * [31] Eibe Frank, Mark Hall, and Len Trigg.  Weka 3: Data mining software in java.  The University of Waikato: Hamilton, New Zealand, 2006.
  * [32] Gaël Guennebaud, Benoît Jacob, et al.  Eigen v3.  http://eigen.tuxfamily.org, 2010.
  * [33] Antonio Gulli and Sujit Pal.  Deep learning with Keras.  Packt Publishing Ltd, 2017.
  * [34] H2O.ai.  H2o: Scalable machine learning platform.  https://github.com/h2oai/h2o-3, 2022.
  * [35] Markus Hofmann and Ralf Klinkenberg.  RapidMiner: Data mining use cases and business analytics applications.  CRC Press, 2016.
  * [36] Jung-Chang Huang and Tau Leng.  Generalized loop-unrolling: a method for program speedup.  In Proceedings 1999 IEEE Symposium on Application-Specific Systems and Software Engineering and Technology. ASSET’99 (Cat. No. PR00122), pages 244–248. IEEE, 1999.
  * [37] Zhihao Jia, Oded Padon, James Thomas, Todd Warszawski, Matei Zaharia, and Alex Aiken.  Taso: optimizing deep learning computation with automatic generation of graph substitutions.  In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pages 47–62, 2019.
  * [38] Thorsten Joachims.  Svmlight: Support vector machine.  SVM-Light Support Vector Machine http://svmlight. joachims. org/, University of Dortmund, 19(4):25, 1999.
  * [39] Ian Karlin, Abhinav Bhatele, Jeff Keasler, Bradford L Chamberlain, Jonathan Cohen, Zachary DeVito, Riyaz Haque, Dan Laney, Edward Luke, Felix Wang, et al.  Exploring traditional and emerging parallel programming models using a proxy application.  In 2013 IEEE 27th International Symposium on Parallel and Distributed Processing, pages 919–932. IEEE, 2013.
  * [40] Kavi, Buckles, and Bhat.  A formal definition of data flow graph models.  IEEE Transactions on Computers, C-35(11):940–948, 1986.
  * [41] Ken Kennedy and Kathryn S McKinley.  Maximizing loop parallelism and improving data locality via loop fusion and distribution.  In International Workshop on Languages and Compilers for Parallel Computing, pages 301–320. Springer, 1993.
  * [42] Jens Knoop, Oliver Rüthing, and Bernhard Steffen.  Partial dead code elimination.  ACM Sigplan Notices, 29(6):147–158, 1994.
  * [43] Steven Lang, Felipe Bravo-Marquez, Christopher Beckham, Mark Hall, and Eibe Frank.  Wekadeeplearning4j: A deep learning package for weka based on deeplearning4j.  Knowledge-Based Systems, 178:48 – 50, 2019.
  * [44] Chris Lattner and Vikram Adve.  Llvm: A compilation framework for lifelong program analysis & transformation.  In International symposium on code generation and optimization, 2004\. CGO 2004., pages 75–86. IEEE, 2004.
  * [45] Bruce W Leverett, Roderic Geoffrey Galton Cattell, Steven O Hobbs, Joseph M Newcomer, Andrew Henry Reiner, Bruce R Schatz, and William A Wulf.  An overview of the production quality compiler-compiler project.  Computer, 13(8):38–49, 1980.
  * [46] David J Lilja.  Exploiting the parallelism available in loops.  Computer, 27(2):13–26, 1994.
  * [47] Yizhi Liu, Yao Wang, Ruofei Yu, Mu Li, Vin Sharma, and Yida Wang.  Optimizing $\\{$CNN$\\}$ model inference on $\\{$CPUs$\\}$.  In 2019 USENIX Annual Technical Conference (USENIX ATC 19), pages 1025–1040, 2019.
  * [48] Lingxiao Ma, Zhiqiang Xie, Zhi Yang, Jilong Xue, Youshan Miao, Wei Cui, Wenxiang Hu, Fan Yang, Lintao Zhang, and Lidong Zhou.  Rammer: Enabling holistic deep learning compiler optimizations with $\\{$rTasks$\\}$.  In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 881–897, 2020.
  * [49] Scott A Mahlke, David C Lin, William Y Chen, Richard E Hank, and Roger A Bringmann.  Effective compiler support for predicated execution using the hyperblock.  ACM SIGMICRO Newsletter, 23(1-2):45–54, 1992.
  * [50] Wes McKinney et al.  pandas: a foundational python library for data analysis and statistics.  Python for high performance and scientific computing, 14(9):1–9, 2011.
  * [51] Sanyam Mehta, Pei-Hung Lin, and Pen-Chung Yew.  Revisiting loop fusion in the polyhedral framework.  In Proceedings of the 19th ACM SIGPLAN symposium on Principles and practice of parallel programming, pages 233–246, 2014.
  * [52] Xiangrui Meng, Joseph Bradley, Burak Yavuz, Evan Sparks, Shivaram Venkataraman, Davies Liu, Jeremy Freeman, DB Tsai, Manish Amde, Sean Owen, Doris Xin, Reynold Xin, Michael J. Franklin, Reza Zadeh, Matei Zaharia, and Ameet Talwalkar.  Mllib: Machine learning in apache spark.  J. Mach. Learn. Res., 17(1):1235–1241, jan 2016.
  * [53] Mehdi Mohammadi, Ala Al-Fuqaha, Sameh Sorour, and Mohsen Guizani.  Deep learning for iot big data and streaming analytics: A survey.  IEEE Communications Surveys & Tutorials, 20(4):2923–2960, 2018\.
  * [54] Supun Nakandala, Karla Saur, Gyeong-In Yu, Konstantinos Karanasos, Carlo Curino, Markus Weimer, and Matteo Interlandi.  A tensor compiler for unified machine learning prediction serving.  In 14th $\\{$USENIX$\\}$ Symposium on Operating Systems Design and Implementation ($\\{$OSDI$\\}$ 20), pages 899–917, 2020.
  * [55] Wei Niu, Jiexiong Guan, Xipeng Shen, Yanzhi Wang, Gagan Agrawal, and Bin Ren.  Gcd 2: A globally optimizing compiler for mapping dnns to mobile dsps.  In 2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 512–529. IEEE, 2022.
  * [56] Wei Niu, Jiexiong Guan, Yanzhi Wang, Gagan Agrawal, and Bin Ren.  Dnnfusion: accelerating deep neural networks execution with advanced operator fusion.  In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, pages 883–898, 2021.
  * [57] Dorit Nuzman and Richard Henderson.  Multi-platform auto-vectorization.  In International Symposium on Code Generation and Optimization (CGO’06), pages 11–pp. IEEE, 2006.
  * [58] Dorit Nuzman, Ira Rosen, and Ayal Zaks.  Auto-vectorization of interleaved data for simd.  ACM SIGPLAN Notices, 41(6):132–143, 2006.
  * [59] NVIDIA.  cuml.  https://docs.rapids.ai/api/cuml/stable/, 2022.
  * [60] NVIDIA.  cuml.  https://docs.rapids.ai/api/cudf/stable/, 2022.
  * [61] Kwanghyun Park, Karla Saur, Dalitso Banda, Rathijit Sen, Matteo Interlandi, and Konstantinos Karanasos.  End-to-end optimization of machine learning prediction queries.  In Proceedings of the 2022 International Conference on Management of Data, pages 587–601, 2022.
  * [62] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [63] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay.  Scikit-learn: Machine learning in python.  J. Mach. Learn. Res., 12(null):2825–2830, nov 2011.
  * [64] Peter C Poole and William M Waite.  Portability and adaptabilty.  Software Engineering: An Advanced Course, pages 183–277, 1975.
  * [65] Miodrag Potkonjak, Mani B Srivastava, and Anantha P Chandrakasan.  Multiple constant multiplications: Efficient and versatile framework and algorithms for exploring common subexpression elimination.  IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 15(2):151–165, 1996.
  * [66] Shvetank Prakash, Tim Callahan, Joseph Bushagour, Colby Banbury, Alan V Green, Pete Warden, Tim Ansell, and Vijay Janapa Reddi.  Cfu playground: Full-stack open-source framework for tiny machine learning (tinyml) acceleration on fpgas.  In 2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 157–167. IEEE, 2023.
  * [67] Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman Amarasinghe.  Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines.  Acm Sigplan Notices, 48(6):519–530, 2013.
  * [68] James Reinders.  Intel threading building blocks: outfitting C++ for multi-core processor parallelism.  ” O’Reilly Media, Inc.”, 2007.
  * [69] Matthew Rocklin.  Dask: Parallel computation with blocked algorithms and task scheduling.  In Proceedings of the 14th python in science conference, volume 130, page 136. SciPy Austin, TX, 2015.
  * [70] Nadav Rotem, Jordan Fix, Saleem Abdulrasool, Garret Catron, Summer Deng, Roman Dzhabarov, Nick Gibson, James Hegeman, Meghan Lele, Roman Levenstein, et al.  Glow: Graph lowering compiler techniques for neural networks.  arXiv preprint arXiv:1805.00907, 2018.
  * [71] Heinz Rutishauser.  Über automatische rechenplanfertigung bei programmgesteuerten rechenmaschinen.  Zeitschrift Angewandte Mathematik und Mechanik, 31(8-9):255–255, 1951.
  * [72] Richard M Stallman et al.  Using the gnu compiler collection.  Free Software Foundation, 4(02), 2003.
  * [73] Jaspal Subhlok, James M Stichnoth, David R O’hallaron, and Thomas Gross.  Exploiting task and data parallelism on a multicomputer.  In Proceedings of the fourth ACM SIGPLAN symposium on Principles and practice of parallel programming, pages 13–22, 1993.
  * [74] Tianxiang Tan and Guohong Cao.  Efficient execution of deep neural networks on mobile devices with npu.  In Proceedings of the 20th International Conference on Information Processing in Sensor Networks (Co-Located with CPS-IoT Week 2021), pages 283–298, 2021.
  * [75] Nicolas Vasilache, Oleksandr Zinenko, Theodoros Theodoridis, Priya Goyal, Zachary DeVito, William S Moses, Sven Verdoolaege, Andrew Adams, and Albert Cohen.  Tensor comprehensions: Framework-agnostic high-performance machine learning abstractions.  arXiv preprint arXiv:1802.04730, 2018.
  * [76] Jure Vreča, Karl JX Sturm, Ernest Gungl, Farhad Merchant, Paolo Bientinesi, Rainer Leupers, and Zmago Brezočnik.  Accelerating deep learning inference in constrained embedded devices using hardware loops and a dot product unit.  IEEE Access, 8:165913–165926, 2020.
  * [77] Endong Wang, Qing Zhang, Bo Shen, Guangyong Zhang, Xiaowei Lu, Qing Wu, Yajuan Wang, Endong Wang, Qing Zhang, Bo Shen, et al.  Intel math kernel library.  High-Performance Computing on the Intel® Xeon Phi™: How to Fully Exploit MIC Architectures, pages 167–188, 2014.
  * [78] Xu Wen.  Tensortable: Extending pytorch for mixed relational and linear algebra pipelines.  BenchCouncil Transactions on Benchmarks, Standards and Evaluations, 4(1):100161, 2024.
  * [79] Xu Wen, Wanling Gao, Anzheng Li, Lei Wang, Zihan Jiang, and Jianfeng Zhan.  Cmlcompiler: A unified compiler for classical machine learning.  In Proceedings of the 37th International Conference on Supercomputing, pages 63–74, 2023.
  * [80] Paul J Werbos.  Backpropagation through time: what it does and how to do it.  Proceedings of the IEEE, 78(10):1550–1560, 1990.
  * [81] Tom White.  Hadoop: The definitive guide.  ” O’Reilly Media, Inc.”, 2012.
  * [82] Maurice V. Wilkes.  Computers then and now.  J. ACM, 15(1):1–7, jan 1968.
  * [83] Maurice Vincent Wilkes.  The Preparation of Programs for an Electronic Digital Computer: With special reference to the EDSAC and the Use of a Library of Subroutines.  Addison-Wesley Press, 1951.
  * [84] William Allan Wulf.  Pqcc: A machine-relative compiler technology.  Technical report, Carnegie-Mellon University, Department of Computer Science, 1980.
  * [85] Hongwei Xi.  Dead code elimination through dependent types.  In International Symposium on Practical Aspects of Declarative Languages, pages 228–242. Springer, 1999.
  * [86] Yu Xing, Shuang Liang, Lingzhi Sui, Xijie Jia, Jiantao Qiu, Xin Liu, Yushun Wang, Yi Shan, and Yu Wang.  Dnnvm: End-to-end compiler leveraging heterogeneous optimizations on fpga-based cnn accelerators.  IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 39(10):2668–2681, 2019.
  * [87] Zheyu Yan, Qing Lu, Weiwen Jiang, Lei Yang, X Sharon Hu, Jingtong Hu, and Yiyu Shi.  Hardware–software co-design of deep neural architectures: From fpgas and asics to computing-in-memories.  In Embedded Machine Learning for Cyber-Physical, IoT, and Edge Computing: Software Optimizations and Hardware/Software Codesign, pages 271–301. Springer, 2023.
  * [88] Matei Zaharia, Mosharaf Chowdhury, Michael J Franklin, Scott Shenker, and Ion Stoica.  Spark: cluster computing with working sets.  In Proceedings of the 2nd USENIX conference on Hot topics in cloud computing, 2010.
  * [89] Xingzhou Zhang, Yifan Wang, Sidi Lu, Liangkai Liu, Weisong Shi, et al.  Openei: An open framework for edge intelligence.  In 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS), pages 1840–1851. IEEE, 2019.
  * [90] Jie Zhao, Xiong Gao, Ruijie Xia, Zhaochuang Zhang, Deshi Chen, Lei Chen, Renwei Zhang, Zhen Geng, Bin Cheng, and Xuefeng Jin.  Apollo: Automatic partition-based operator fusion through layer by layer optimization.  Proceedings of Machine Learning and Systems, 4:1–19, 2022.
  * [91] Size Zheng, Siyuan Chen, Peidi Song, Renze Chen, Xiuhong Li, Shengen Yan, Dahua Lin, Jingwen Leng, and Yun Liang.  Chimera: An analytical optimizing framework for effective compute-intensive operators fusion.  In 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pages 1113–1126. IEEE, 2023.