# Architecting Intermediate Layers for Efficient Composition of Data
Management and Machine Learning Systems

Supun Abeysinghe, Fei Wang, Gregory Essertel, Tiark Rompf  Purdue
UniversityWest LafayetteIndianaUSA tabeysin, wang603, gesserte,
<EMAIL_ADDRESS>
(none)

###### Abstract.

Modern data analytics workloads combine relational data processing with
machine learning (ML). Most DBMS handle these workloads by offloading these ML
operations to external specialized ML systems. While both DBMS and ML systems
go to great lengths to optimize performance for their specific workloads,
significant performance is lost when used in combination, due to data movement
across system boundaries, conversions between incompatible internal data
formats, and the lack of cross system optimizations.

A key idea to remove these bottlenecks is to integrate existing data
manipulation systems with ML systems by building a common intermediate layer
(IR). Although this idea has been explored before (Weld, Delite), previous
such attempts require significant re-engineering of prior systems and still
fall short in achieving best-of-breed performance for individual tasks (e.g.,
SQL, Deep Learning). Specifically, they rely on re-implementing existing
systems using a generic set of operators and fail to match best-of-breed
individual performance due to the inability to recover high-level
optimizations from this generic IR through compiler analysis.

We present Flern, the first intermediate-layer integration between DB and ML
systems that are best-of-breed individually, competitive with the best
compiled query engines such as HyPer on comprehensive relational benchmarks
(TPC-H) and competitive with TensorFlow and PyTorch in state-of-the-art ML
models (e.g., DeepSpeech, SqueezeNet, Transformers) and also represents a new
state-of-the-art for integration. A key realization is to architect
intermediate layers based on generative programming capabilities, which
preserves high-level contextual information for cross optimizations and
enables the construction of a variety of complex structures and cross system
optimizations with minimal effort.

††copyright: none††journalyear: none††conference: ; ;

##  1\. Introduction

Today’s data analytics workloads often combine traditional relational
processing with modern machine learning (ML) operations. These two workload
types are significantly different from each other and have different domain-
specific properties that lead to the construction of independent specialized
systems (i.e., DBMS for relational processing and ML frameworks for ML
operations) (Schelter et al., 2018). These systems go to great lengths to
optimize their workloads by taking domain-specific properties into account
(e.g., query compilation, binding to low-level specialized kernels).

However, when complex data analytics pipelines are built by combining such
systems, their end-to-end performance becomes suboptimal. This can be due to
multiple reasons, including the inability to perform global optimizations
across system boundaries and the overhead of data movement and conversion at
these boundaries, often caused by incompatible data formats. For instance, a
simple relational query that uses an ML model inside Postgres (using PyTorch
for ML) can be more than 50x slower than a manual hand-optimized program
written for the same task.

Our work aims to address this key challenge: how can we efficiently manage
workloads that combine relational processing with ML? Our objectives are
threefold: 1) to eradicate costly overheads at system boundaries, 2) to
perform global optimizations on the entire computation, rather than treating
ML computations as black boxes, and 3) to achieve these goals without the need
to re-engineer systems from the ground up.

One approach is to add support for ML computations inside DBMS by extending
SQL engines with new operators (e.g., user-defined functions, iterative
computations, etc.) (Jasny et al., 2020; Schüle et al., 2021; Feng et al.,
2012; Chen et al., 2017; Schleich et al., 2019; Hellerstein et al., 2012).
However, these approaches have limited support for the types of models they
can run and do not support modern ML models (e.g., Transformers (Vaswani et
al., 2017)) due to lack of expressiveness. Moreover, the performance is not on
par with specialized ML systems (e.g., PyTorch, TensorFlow, etc.) mainly due
to the inability to incorporate domain-specific optimizations.

Another, more intrusive but also more powerful, approach is to build a common
intermediate layer across different systems. The intermediate layer must be
sufficiently generic to support all operations of disjoint systems. That is,
it should support all SQL/DataFrame operations (e.g., different types of
joins, aggregation queries, etc.) and all deep learning operators (e.g.,
automatic differentiation (AD), complex ML models, etc.). A line of work that
follows this idea of building common intermediate layers (Palkar et al., 2017;
Pirk et al., 2016; Sujeeth et al., 2014) achieves this by imposing a single,
fixed intermediate layer as a _one-size fits all_ consisting of a handful of
generic operators (e.g., _map_ , _reduce_ , etc.). This eliminates overheads
at system boundaries by generating a unified executable. While there is
considerable flexibility for optimization, this generic IR lacks vital
contextual information which makes it difficult to recover low-level
performance through compiler analysis. Moreover, all the high-level operators
(e.g., Joins, Tensor operations, etc.) of existing systems (or their front-
ends) should be re-implemented using these generic IR constructs which require
a significant engineering effort and can be challenging for complex operators
(e.g., in Section 2, we investigate a Hash Join implementation of such a
system and show how it differs from a classic textbook implementation).

Another approach is constructing high-level IRs comprising high-level DB and
ML ops. For instance, Raven (Park et al., 2022) extends ONNX (Developers,
2019), an IR for ML, with relational algebra operations to construct an IR
tailored for combined workloads. This strategy facilitates cross-workload
optimization, as it provides access to high-level abstractions and rich
contextual information, as opposed to using a generic, fixed IR. While this
approach can improve end-to-end performance by enabling cross-workload
optimizations, it still incurs significant overheads at system boundaries due
to data copying and format conversion. This is because, despite creating a
unified IR, separate runtime systems remain in use for DB (such as Spark SQL
(Armbrust et al., 2015) or SQL Server) and ML (like ONNXRuntime (developers,
2021)) execution.

In this work, we investigate the potential for achieving the best-of-both
worlds. That is, having the capability to perform general, reusable compiler
optimizations using a low-level common IR and generating a unified executable
that eliminates overheads at system boundaries and also, having access to
high-level abstractions and contextual information to add cross-system
optimizations. To achieve this goal, we leverage an existing relational query
compiler, Flare (Tahboub et al., 2018; Essertel et al., 2018), and an ML
compiler, Lantern (Wang et al., 2019; Wang et al., 2018), both of which are
built using generative programming (introduced in Section 2.2) (Rompf and
Odersky, 2010). Specifically, we integrate them at a higher-level to handle
combined relational and ML workloads.

Generative programming has demonstrated great effectiveness in architecting
large-scale systems for individual workloads with relatively smaller
engineering effort compared to highly engineered counterparts, while achieving
competitive performance (Tahboub et al., 2018; Essertel et al., 2018; Wei et
al., 2020, 2019). This generative approach allows programmers to develop their
systems using a high-level language with user-friendly features (e.g., high-
level type system, high-level data structures, abstractions such as classes,
interfaces, generics, etc.) while achieving native performance by translating
this high-level code to low-level native code (thus, _generative_) which does
not contain any of the high-level abstractions, hence, _abstraction without
regret_ (Rompf, 2012).

In this work, we investigate the ways to reconfigure such systems designed
around the idea of generative programming to efficiently compose multiple such
systems to handle complex workloads. We demonstrate this generative
programming approach enables efficient post-hoc integration of such
specialized systems with relatively smaller engineering effort. Specifically,
we build Flern, a query compiler that supports full ML capabilities, by
integrating Flare (Tahboub et al., 2018; Essertel et al., 2018), a state-of-
the-art query compiler and Lantern (Wang et al., 2019; Wang et al., 2018) an
ML framework, that are developed based on the same generative programming
technique called Lightweight Modular Staging (LMS) (Rompf and Odersky, 2010).
Flern is the first system that is best-of-breed individually (competitive with
state-of-the-art in-memory SQL engines like HyPer (Neumann, 2011) on full
relational benchmarks (Essertel et al., 2018) e.g., TPC-H and competitive with
TensorFlow and PyTorch in sophisticated deep learning models (Wang et al.,
2019) (Wang et al., 2018) e.g., DeepSpeech (Amodei et al., 2016a), SqueezeNet
(Iandola et al., 2016b), Transformer (Vaswani et al., 2017)) and also the new
state-of-the-art for integration.

Figure 1.  Previous approaches directly map the user program to a generic IR,
lacking necessary contextual information for high-level global optimizations.
In contrast, Flern preserves high-level abstractions and vital contextual
information, enabling seamless introduction of cross-optimizations.

Contributions This paper’s primary intellectual contribution lies in its
examination of the constraints inherent to existing specialized code
generation methods when dealing with multi-paradigm workloads, specifically DB
and ML. Then, we present an approach for tackling this challenge of system
integration in general: architect systems based on generative programming, so
that they can be adapted more effectively and at a lower engineering cost, and
provide evidence of the approach’s effectiveness by building Flern. Our
specific contributions are summarized as follows:

  * •

We analyze the limitations of existing approaches to build common intermediate
layers and discuss why generative programming emerges as a superior choice for
integrating systems efficiently with minimal re-engineering cost (Section 2)

  * •

We demonstrate how generative programming enables efficient post-hoc
integration of prior systems by combining two state-of-the-art systems in DB
and ML domains. Specifically, we illustrate how our approach can be utilized
to eliminate overheads at system boundaries and implement cross-optimizations,
all while maintaining the best-of-breed performance of individual systems.
(Section 3)

  * •

We present two sets of benchmarks where we first evaluate the performance
impact of each introduced cross-optimization, followed by a comparison of the
performance against state-of-the-art baselines. This comparison demonstrates
that our approach (1) either outperforms or achieves competitive results in
isolated tasks (i.e., data manipulation and ML independently), and (2)
delivers state-of-the-art performance—demonstrating up to an order of
magnitude speedups when applied to combined tasks (Section 4).

In Section 5, we present an analysis of related work in this domain. Finally,
in Section 6, we draw conclusions and discuss potential future research
directions. A preliminary demonstration paper outlining the initial conception
and operation of our proposed system was previously published (Essertel et
al., 2019). In this current work, we add further optimizations on top of the
initial integration, and conduct a comprehensive experimental evaluation,
comparing our system against well-established baselines.

##  2\. Motivation

###  2.1. Why Intermediate Layer Integration?

Specialized ML systems commonly employ various techniques to achieve optimal
performance, such as the usage of high-level abstractions (e.g., Tensor) that
make it relatively easy to add domain-specific optimizations (e.g., Tensor
transformations). To achieve performance parity with these specialized ML
systems during mixed DB and ML workloads, the capability to incorporate these
optimizations becomes vital. However, this task can prove challenging when ML
operations are incorporated into DBMS using divergent abstractions and
methodologies (e.g., extended SQL execution engines, user-defined functions,
etc.). The key hurdle is the intricacy involved in adapting the existing
optimization strategies to these different settings.

⬇

def join(expr1, expr2, d1_keys, d2_keys, keys_type, ..):

""" Computes a join on two tables """

weld_obj = WeldObject(encoder_, decoder_)

df1_var = weld_obj.update(expr1)

if isinstance(expr1, WeldObject):

df1_var = expr1.obj_id

weld_obj.dependencies[df1_var] = expr1

df2_var = weld_obj.update(expr2)

if isinstance(expr2, WeldObject):

df2_var = expr2.obj_id

weld_obj.dependencies[df2_var] = expr2

#Some String manipulations constructing holes of the template elided

weld_template = """

let df2_join_table = result(

for(

groupmerger[%

|b, i, e| merge(b, {%

)

);

result(for(

appender,

|b, i, e|

for(

lookup(df2_join_table, %

b,

|b2, i2, e2| merge(b, {%

)

))"""

weld_obj.weld_code = weld_template %

return weld_obj

Figure 2. Join Implementation of Weld (for accelerating Pandas). The operators
emit code for IR construction as blocks of strings. This kind of manipulation
of code in stringified form is generally error-prone, relatively harder to
maintain and implement.

An alternative strategy is to integrate a DBMS with an existing ML system
already furnished with essential domain-specific optimizations. Such
integration potentially curtails the need for substantial re-engineering
efforts. A case in point is the PL/Python plugin (Postgre, [n.d.]), which
facilitates the execution of Python-based arbitrary procedures within
PostgreSQL. As a result, any ML framework featuring Python interfaces, such as
PyTorch (Paszke et al., 2019) or TensorFlow (Abadi et al., 2016), can be
employed within the database for ML operation implementation. In this way, the
DBMS essentially gains full access to a fully-fledged ML system’s capabilities
for effectively managing combined relational and ML workloads.

This approach, however, encounters a significant drawback. Both systems tend
to operate as separate entities, which leads to data copying, format
conversions at system boundaries, and hinders cross-system optimizations. For
instance, in the Postgres scenario, data must be transferred from the database
to the Python environment, which then needs to be converted into PyTorch’s
tensor data format. Once the ML computation concludes, these tensors have to
be transformed back into PL/Py objects, copied back to Postgres, and finally
converted into the original record format. For example, when executing a
three-layer neural network model on the NYC-Taxi dataset (Kaggle, 2017) using
this approach, the total execution time comes to 638 seconds whereas time
spent on doing actual ML computations comes to around 6 seconds, underlining
the substantial overheads involved (examined in detail in Section 4.4.2).

To mitigate or even eradicate these overheads, we turn our attention to query
compilers (Essertel et al., 2018; Tahboub et al., 2018; Neumann, 2011). These
compilers accelerate query execution speed by generating specialized, low-
level code tailored to a particular query. Generally, these systems build an
IR for the incoming query, conduct a series of optimizations on the IR, and
then generate code from the optimized IR. This generated code is compiled by a
generic compiler and executed to produce the final result. One strategy to
eliminate the data movement overheads across systems in the combined workload
setting is to combine the systems at this IR level. This entails lowering both
the query compiler and the ML computations to a single common IR, enabling
global optimizations and minimizing data movement overheads by carefully
analyzing the IR and generating a unified executable.

###  2.2. Why Generative Programming?

⬇

case class HashJoinOp(left: Op, right:Op)(/* elided */) extends Op {

// HashMap specialized for corresponding Schemas

val map = LinkedHashMap(keySchema, left.schema)

def exec(callback: Record => None) = {

// Store records from left operator tree in the HashMap

left.exec { tuple =>

map.update(leftHash(tuple), tuple)

}

// Retrive records from right subtree and join

right.exec { tuple =>

for (lTuple <- map(rightHash(rTuple)) if joinCond(lTuple, rTuple))

callback(lTuple ++ rTuple)

}

}

}

Figure 3. Hash-Join Implementation of Flare which is a query compiler based on
generative programming. The implementation contains the operator logic
implemented using a high-level programming language (Scala) as opposed to
emitting blocks of stringified code and has access to all the features of the
host language (e.g., type system, abstractions, etc.).

Delite (Sujeeth et al., 2014) and Weld (Palkar et al., 2017) are prior systems
that follow the strategy of developing common IRs across multiple systems,
which allows for the generation of unified executables. Both of these methods
share a similar characteristic in utilizing a fixed, generic intermediate
layer composed of a limited set of generic operators that front-ends use to
implement their functionalities. This offers substantial flexibility as any
optimizations added to this generic IR layer can be reused by any front-end.
However, this flexibility tends to come with notable trade-offs. Firstly, the
generic nature of the IR means that most of the contextual information
originating from high-level abstractions such as tensors or DataFrame will be
lost at the IR level. For example, if a 2D tensor is transposed twice
consecutively, it is relatively easy to determine that this results in a no-op
if we have access to contextual information about the data (i.e., that it is a
2D tensor), but this would require compiler analysis of the looping structures
in the generic IR. Secondly, implementing complex operators using these
minimal IR constructs can pose a challenge. This is because their actual
implementations may significantly deviate from their textbook versions.

For instance, Weld defines a minimal IR that encapsulates the structure of
prevalent data-parallel algorithms, complemented by a runtime API that allows
disjoint libraries to construct Weld IR fragments. These host libraries need
to fully re-implement their operators to emit the necessary IR fragments that
reflect the desired operator logic. The backend then accumulates these emitted
IR fragments into a unified global IR. This combined IR undergoes several
stages of optimization and drives the process of generating optimized code via
LLVM (Lattner and Adve, 2004)

Figure 2 shows the Weld implementation of the Join operator (Palkar et al.,
2017). Notably, the implementation deviates substantially from a conventional
textbook implementation and consists of a stringified code template for the
Join operator containing placeholders that are filled based on function
parameters. However, since this kind of template expansion is limited by the
fact that the code is manipulated as strings, it is either impossible or
challenging to use features of high-level programming languages, such as type-
checking, on these code templates. As a result, these code templates often
pose challenges in implementation and maintenance. Consequently, systems
adopting this method usually support a restricted set of cases (e.g., a single
type of join, or a limited set of ML operations) due to the engineering effort
required and the complexity involved. Therefore, while there’s potential for
improvement in end-to-end performance, these approaches often either lack
support or fail to reach best-of-breed performance for specific workloads in
comprehensive benchmarks (such as the full TPC-H benchmark).

In contrast, generative programming is fine-grained, supports lots of
programmatic control, and leverages rich contextual information. Take
Lightweight Modular Staging (LMS) (Rompf and Odersky, 2010), for example. It’s
a generative programming framework in Scala that is based on the idea of
multi-stage programming (Taha and Sheard, 2000) (commonly known as staging).
Multi-stage programming is a paradigm that equips developers with the ability
to write generic programs using higher degrees of abstractions without
compromising on runtime efficiency (Taha, 2007). The high-level idea behind
staging is to delay computations of certain operations to a later stage,
thereby generating code for such operations with the information known in the
current stage. An illustrative example of LMS in action is depicted in the
following code snippet. Note that the implementation closely mirrors
programming in a conventional language, compared to resorting to manipulating
large blocks of stringified code templates.

⬇

def power(b: Rep[Int], n: Int):Rep[Int] =

if (n == 0) 1

else if (n %

power(b, n/2) * power(b, n/2)

else b * power(b, n-1)

def main(args: Rep[Array[String]]) =

println(power(args(0).toInt, 7))

LMS is type-driven, and utilizes Rep types to denote values that will be
computed in the subsequent stage, and therefore, should be computed by the
generated code (Rep represents next-stage expressions). All operations,
including language constructs such as control flow on conventional non-Rep
expressions (for instance, Int), are evaluated at the present stage. In
contrast, all Rep-typed expressions contribute to code generation. In the
given code snippet, b is of a Rep-type, denoting it is not known in the
current stage, while n is of a standard type, thus, its value is known. This
allows for partial evaluation of the program based on n, generating code for
later stage execution. Moreover, LMS is capable of directly generating
equivalent low-level C code for the partially evaluated Scala code (for
performance reasons). Below is the resultant C code.

⬇

int main(int argc, char** argv) = {

int x0 = atoi(argv[0]); // args(0).toInt

int x1 = x0 * (x0 * x0); // x1 = x0 ** 3

printf("%

// x0 * ((x0 ** 3)* (x0 ** 3)) = x0 ** 7

}"

A notable observation is that the original source code, written in Scala,
closely mirrors typical Scala code, except for the type annotation which
indicates the variables that needs to be staged. This pattern generally holds
true across most instances, implying that a developer can transform a standard
Scala program into a staged program primarily by altering the types.

Under the hood, LMS maintains an extensible graph-like IR to capture the
constructs and operations of the staged program. It features numerous IR-level
optimizations (including loop fusion, common sub-expression elimination, loop
unrolling, and function inlining), and the IR’s extensible nature allows
library developers to conveniently add their domain-specific optimizations
(Rompf et al., 2013). In this context, we can perceive LMS as enabling us to
utilize the entirety of Scala as a macro language in the construction of the
intermediate layer. This consequently simplifies the implementation of
intricate data structures and algorithms required for sophisticated data
manipulations and comprehensive deep learning functionalities. Furthermore,
LMS has the ability to generate low-level code that does not contain any high-
level abstractions utilized in the original Scala program (e.g., Tensor,
DataFrame), thereby eliminating potential runtime penalties tied to these
abstractions.

Prior work has demonstrated the effectiveness of generative programming as a
systems-building technique that dramatically simplifies the construction of
high-performance systems that have a high degree of internal variability and
need specialization to achieve performance (e.g., Flare (Essertel et al.,
2018), LB2 (Tahboub et al., 2018)). For example, Figure 3 shows the
implementation of HashJoin operator in LB2. This implementation equates to
writing the hash join algorithm for a straightforward query interpreter,
essentially following textbook pseudo-code, using a high-level programming
language, in this case, Scala. The developer has full access to high-level
data structures (e.g., LinkedHashMap) specialized for the given schemas,
abstractions (e.g., Record), and the Scala type system as opposed to writing
brittle stringified code templates.

In this work, we demonstrate that in the case of composing such systems by
constructing common IRs, these properties of generative programming are useful
for adapting system boundaries with minimal overhead and implementing key
global optimizations that span across system boundaries. It permits the
implementation of such optimizations using varied abstraction levels (see
Figure 1) while utilizing a high-level programming language, with complete
access to contextual information associated with high-level operators.
Essentially, this implies that cross-optimizations can be implemented using
high-level abstractions like Tensor and DataFrame, as opposed to mandating all
global optimizations to be carried out on a generic IR. In Section 3, we
illustrate how these generative programming properties can be harnessed to
seamlessly combine a generative programming-based query compiler with a ML
framework incurring minimal overheads at system boundaries and how we can
implement cross system operations and optimizations to achieve state-of-the-
art performance in combined DB and ML workloads.

##  3\. Flern Overview

In this section, we illustrate how the benefits of generative programming can
be harnessed to seamlessly integrate two pre-established systems from the
realms of DB (relational) and ML (tensor) for the processing of combined
workloads. Specifically, we build Flern using Flare (Essertel et al., 2018),
an SQL query compiler, and Lantern (Wang et al., 2019; Wang et al., 2018; Wang
and Rompf, 2018), an ML framework (Figure 5). Both of these systems have been
developed based on the same generative programming approach, Lightweight
Modular Staging (LMS) (Rompf and Odersky, 2010). This demonstrates how Flern
can benefit from the distinct strengths of these two systems, which each offer
full functionality in their respective domains, such as comprehensive SQL
support and the capability to handle modern deep learning models. Moreover,
both Flare and Lantern deliver performance that competes effectively with the
best-in-class systems in their respective fields.

⬇

# Flern: ML in-charge: training models by querying DB for data

for epoch in range(num_epochs):

for batch, target in sql("select ... from t1 join t2 ..."):

model.train(batch, target)

# Flern: DB in-charge: using ML models in SQL queries

sql.register_udf("classifier", model)

sql("select p, classifier(xs) from r;")

Figure 4. Flern’s main use cases: ML model training with DB data retrieval
using SQL queries, and DB query evaluation with trained ML models in
relational queries.

Flare: Flare is built on top of LB2 (Tahboub et al., 2018), a high-level query
compiler developed using generative programming techniques. LB2 implements
relational operators in a way that is similar to a simple query interpreter
and converts it to a query compiler using Futamura projections and partial
evaluation (Futamura, 1999). Flare operates on the optimized query plans
generated by Spark’s query optimizer (Catalyst (Armbrust et al., 2015))
derived from user queries and performs query compilation and runtime native
code generation. This code generation strategy of Flare achieves orders of
magnitude speedups over Spark and other RDBMS and achieves competitive
performance with state-of-the-art in-memory query compilers like Hyper
(Neumann, 2011) in full relational benchmarks (e.g., TPC-H) (Essertel et al.,
2018).

Lantern: Lantern (Wang et al., 2018; Wang et al., 2019; Wang and Rompf, 2018)
is a differentiable programming framework that handles automatic
differentiation via delimited continuations (Fischer, 1993), and code
generation via LMS (Rompf and Odersky, 2010). Delimited continuations allow
Lantern to support ML models with in-graph control-flows such as conditionals,
loops, and functions. LMS reifies the computation graph (after automatic
differentiation) for code generation that utilizes various BLAS and neural
network kernel libraries for multiple hardware platforms (such as CPU and
GPU). Lantern performs competitively with existing deep learning frameworks
(e.g., PyTorch and TensorFlow) in modern deep learning models.

Figure 4 illustrates the two main scenarios where Flern can be utilized.
Generally, combined DB and ML workloads can be classified into two major
categories. The first category encompasses cases where ML models are trained
using data obtained through DB queries. This also includes end-to-end data
science pipelines constructed using data manipulation tools (e.g., Pandas
(McKinney et al., 2010), Spark) and ML frameworks (e.g., PyTorch, TensorFlow).
In these scenarios, the ML system repeatedly retrieves data in batches from
the data manipulation system for model training. The second category involves
instances where queries utilizing pre-trained ML models are executed on a DB.
For instance, an SQL query may contain one or more pre-trained ML classifiers
as functions.

Figure 5. Flern’s architecture: Seamlessly integrating Flare and Lantern at a
common IR level and adding cross-system optimizations at the boundary. It
generates a unified executable for combined DB and ML workloads.

Recall that our primary design objective is to extend DBMS with ML
capabilities with minimal reconfigurations to existing systems, ensuring they
maintain their individual performance while improving their combined
performance. Flare and Lantern achieves competitive performance with state-of-
the-art systems for their respective workloads. Hence, one straightforward
approach would be to operate these two systems independently and transfer data
between them using a commonly used data format, such as CSV. This approach
wouldn’t require significant modifications to either system and could be
implemented by introducing a CSV exporter and importer at the respective
system boundaries. While this approach would maintain the performance of
individual systems, it introduces a substantial overhead due to the necessity
of exporting data in a chosen format, loading this data, parsing the loaded
data, etc. (as evaluated in Section 4.2).

A modest improvement on this naive implementation could involve using a binary
format to decrease the overhead associated with data formatting and parsing,
and to facilitate communication over memory (e.g., using pipes or shared
memory). However, communication overheads persist due to the separation of
runtime environments. Furthermore, in scenarios where complex relational
queries are mixed with tensor computations (e.g., ML Classifiers as UDFs),
this data movement may need to occur multiple times in both directions,
necessitating careful modifications to both systems to handle such
interactions with external systems.

###  3.1. Adapting the Boundaries of Flare and Lantern

A crucial characteristic of both Flare and Lantern is that they are built upon
the same generative programming paradigm - Lightweight Modular Staging (LMS).
As discussed in Section 2, a key benefit of this generative strategy is that
these high-performance systems are implemented in Scala, a feature-rich high-
level high-level programming language. Consequently, we can combine them
simply by importing them as Scala libraries, granting us access to the
functionalities of both systems.

However, instead of combining them using the conventional method of
transferring data across respective library functions, which incurs data
movement and conversion overheads, we propose bringing the two systems
together at a common layer and constructing a single IR graph at the LMS
level. More specifically, we devise a shared layer that supports the union of
operators from both systems while preserving all the transformations and
optimizations inherent to each individual workload.

The feasibility of this approach owes much to the modular architecture of both
systems (and systems architected in Scala in general). Specifically,
functionalities are enclosed in Scala traits as discrete modules, and these
more compact modules are combined using class compositions with mixins
(Odersky and Zenger, 2005) to construct a larger system. For instance, Flare
defines traits such as CompileProject, CompileJoin, and CompileAggregate to
manage the operator logic for Project, Join, and Aggregate operations. These
smaller building blocks are then combined into FlareOps. Lantern also employs
a similar design strategy. Thus, by leveraging this modular structure, we can
integrate the functionalities of both systems using a similar mechanism. We
then tailor the LMS backend so that it builds a single IR graph containing
operations from both workloads.

Despite their compatibility, the two systems operate using distinct high-level
abstractions, and they are not aware of each other’s constructs (e.g., Flare
does not recognize the Tensors in Lantern, and vice versa with Record, Buffer,
etc., in Flare). As a result, we must facilitate conversions between these
high-level abstractions at the boundaries of each system. However, a
distinctive trait of generative programming-based code generation is that
these high-level abstractions dissolve into native data structures in the
generated code (for instance, Tensor and Buffer transform into native C
arrays). Consequently, even though we carry out conversions between these
abstractions in the high-level code, these conversions effectively become
simple variable assignments in the generated code. This characteristic ensures
that there is no adverse impact on performance during runtime.

Thanks to the inherent extensibility of LMS (Rompf, 2016; Rompf et al., 2013;
Rompf and Odersky, 2010), implementing these conversions is a relatively
straightforward task. For instance, to transform a Buffer into a Tensor, we
can define a toTensor method that establishes a new node type (e.g., to-
tensor) in the IR. Subsequently, we can implement the code generation logic
for this newly established node, which translates to a trivial variable
assignment in the generated code. Alternatively, considering that both Buffer
and Tensor are constructed from the same lower-level abstraction
(Rep[Array[T]]), the conversion can be performed at that level. As an example,
Figure 6 (Line 11) illustrates how records are seamlessly transformed into a
tensor (also represented as an array) by simply assigning the variable.

While this approach might appear to be straightforward in its implementation,
its simplicity belies its effectiveness. It efficiently eliminates data
movement and conversion overheads at system boundaries—a significant
bottleneck in existing tools and a challenging problem as highlighted in
Section 1. Crucially, it maintains the individual optimizations of each
respective system, ensuring the preservation of individual best-of-breed
performance. In addition, by constructing a single IR for the entire program
at the LMS level, all the IR-level optimizations, including Dead Code
Elimination (DCE), code motion, and loop fusion, are performed globally.
Moreover, this also enables further global optimizations by downstream
general-purpose compilers (GCC or LLVM) during compilation of the unified
generated code.

Our approach does more than simply eradicate the overheads at system
boundaries—it also creates opportunities for further cross-system
optimizations that can considerably enhance performance. For instance, we can
avoid the need to materialize large amounts of intermediate data generated by
data processing systems for ML models, by fusing the data processing loops
with ML loops. In the following sections, we delve into several of these key
cross optimizations, all of which significantly improved Flern’s end-to-end
performance. A common trait among these implementations is the utilization of
high-level abstractions such as Tensor and Buffer to carry out cross-system
operations and optimizations (Figure 1).

###  3.2. Optimizing GPU Data Movement

Given the superior efficiency of GPUs in executing tensor computations, larger
ML workloads are typically processed on these units rather than CPUs. However,
this necessitates the transfer of pertinent data, including model inputs and
weights, from the host (CPU) to the GPU. In this section, we highlight two key
optimizations designed to enhance the efficiency of this data transfer
process, and we elucidate how we integrated these improvements into Flern.

By default, any memory allocations made by the host are pageable. Nonetheless,
GPUs cannot directly access this pageable memory. Thus, when data from such a
pageable host array needs to be transferred, the GPU driver initially
allocates a temporary page-locked or “pinned” host array. Subsequently, the
data is copied to this freshly allocated pinned array before being transferred
to the GPU memory. In Flern, we are aware of which data buffers require
transfer to the GPU. This enables us to eliminate the extra data copying step
by directly allocating the corresponding data buffers (used in data
manipulation) in pinned memory. We add this functionality to Flern by adding a
new operator that can create a Buffer with underlying data (i.e., the C array
in the generated code) directly allocated in pinned memory. Then, for any
Buffer that we invoke toTensor() followed by a toGPU(), we allocate the
original buffer in pinned memory.

Allocating host arrays in pinned memory enables the asynchronous transfer of
data to the GPU. Typically, when two separate or naively integrated systems
are employed for data manipulation and ML, the GPU remains idle until the data
processing phase concludes. A primary benefit of generating a single code for
the entire task is the capability to fuse host-to-device data copying
operations with data manipulation operators. Consequently, data can be
transferred to the GPU concurrently with the production of output from data
manipulation. This allows us to leverage asynchronous memory copying
mechanisms (e.g., cudaMemCpyAsync), to overlap data copying and data
processing.

###  3.3. Running ML UDFs Efficiently

⬇

1 /* select p, classifier(xs) from r; */

2 int main() {

3 struct r_record* data = /* load data */

4 long record_count = /* data count */

5

6 float *w1 = /* classifier weigths*/

7 float *b1 = /* classifier bias*/

8 float *w2 = /* classifier weigths*/

9

10 for(long i = 0; i < record_count; i++) {

11 float *tensor = data[i]->xs; // conversion

12 // ML Computation

13 float *y1 = gemm_kernel(tensor, w1, ...);

14 for(int j = 0; j < 4; j++) {

15 y1[j] += b1[j];

16 }

17 float *y2 = gemm_kernel(y1, w2, ...);

18 if (*y2 > 0.5) {

19 printf("%

20 } else {

21 printf("%

22 }

23 }

24 }

Figure 6. Generated code for a simple query that uses an ML classifier
(variables renamed, some parts elided to improve clarity). Here, xs is an
array of data and the classifier computes the output by doing two matrix
multiplications. This code looks exactly like a manually written version of
the query and performs the combined DB + ML workload.

Typically, data processing systems do not natively support ML operators.
Consequently, executing queries involving ML computations often requires
integrating calls to external systems like PyTorch or TensorFlow for ML
functionality. However, these external functions remain opaque to the data
management system, hampering optimization efforts. Furthermore, they introduce
substantial overheads in the form of data serialization and invocation, which
are required to perform computations outside the DBMS runtime. This process
often entails significant complexities and inefficiencies when traversing
system boundaries.

With Flern, we can access a comprehensive set of ML operations directly within
the DBMS. This allows Flern to execute queries involving ML operations as
internal functions. The advantages of conducting ML computations internally
rather than relying on external systems are manifold. For instance, these
internal ML UDFs are susceptible to optimizations, facilitating the seamless
integration of UDF computations with the rest of the tasks and enabling
efficient optimization during code generation (e.g., inlining UDF invocations,
code motion, etc.). Furthermore, it allows UDFs to undergo additional global
optimizations at the hands of downstream compilers, like GCC. Importantly,
since all execution takes place within the same runtime, the overhead
associated with invoking the UDF becomes negligible, leading to significantly
improved performance.

Consider the following simple query with a UDF.

⬇

SELECT x, classifierUDF(y) FROM r;

Below is a depiction of how an end-user can define such a UDF in Flern using
Scala. Users can create their UDFs as standard Scala functions, utilizing
functionalities from both Flare (for DB ops) and Lantern (for ML ops). In the
provided code snippet, the user defines a UDF that applies a pre-trained model
to an input value. The function’s signature should align with the number of
arguments passed to the UDF within the SQL query. When the user registers the
UDF, it is added to a collection known as udfMap, making it readily available
for use within the Flern environment.

⬇

def classifierUDF(a: Value) = {

val inputTensor = a.toTensor()

// run the pre-trained model

// (can do any arbitrary tensor computation here)

val output = model(inputTensor)

output.toValue

}

In order to incorporate this functionality into Flern, we establish a new
operator, ScalaUDF, which generates code for invoking UDFs as standard
function calls. Subsequently, the relevant methods in Flare are overridden (as
shown below) to use this newly introduced UDF operator when UDFs are invoked.
This method retrieves the corresponding function for the udfMap and executes
it with the appropriate parameters. The generation of code for the function
invocation is handled by the LMS under the hood. Figure 6 provides a depiction
of what the generated code would look like for the above query.

⬇

override def compilerExpr(/* elided */) = {

case ScalaUDF(function, dataType, children, ...) =>

// extract the function from udfMap

val extractedFunc = udfMap(function)

// extract the UDF arguments

// (need to recursively call compilerExpr because the arguments can be
subqueries)

val values = children map { compileExpr(_)(rec:_*) }

// call the function

extractedFunc(values:_*)

case _ =>

super.compilerExpr(/* elided */)

}

This approach, however, encounters a significant performance impediment due to
Flare’s query execution model, which operates on a single record at a time,
utilizing a modified version of the data-centric model (Tahboub et al., 2018).
This is not optimal for executing relatively large UDFs since the overheads of
kernel launches can overwhelm the performance. As a remedy, we introduced the
VectorizedUDF operator that launches kernels for batches of data, rather than
individual instances, thereby amortizing the kernel launch overhead. This
implementation mirrors the above-described strategy, but instead of processing
single Records of data, it handles data in batches.

Figure 7. Adding a separate thread pool to run ML UDFs on GPU. These threads
retrieve records as they are produced, accumulate them and run on the GPU
while data processing is ongoing.

While CPUs suffice for running smaller ML models as UDFs, many modern deep
learning models (e.g., a transformer-based sentiment classifier for text
data), demand more computational power, rendering CPU-based computation
insufficient. Consequently, GPU support becomes crucial for executing these
UDFs. We could use the Lantern GPU backend to run VectorizedUDF kernels, but
this approach impedes overall performance as CPU data manipulation is halted
until the output for the current batch is retrieved. Also, each data
processing thread independently and concurrently initiates GPU kernel launches
and data transfers with other CPU threads, leading to inefficient data
transfer and execution.

To address these issues, we have incorporated a new thread pool, having one
thread per GPU, distinct from the data processing threads (as depicted in
Figure 7). This innovative setup accumulates records as they are generated and
then batches all smaller transfers and kernel invocations together, thereby
efficiently amortizing the overhead.

###  3.4. Supporting other Popular Front Ends

For the effective acceleration of existing end-to-end data science pipelines
with Flern, it’s crucial to provide support for widely used front ends, such
as Pandas, Spark, PyTorch, TensorFlow, among others, as highlighted by
(Kaggle, 2018b). Existing tools, like Snek-LMS (Decker et al., 2019), allow
for language virtualization and source code transformations for Python by
implementing a multi-stage programming framework akin to LMS. This framework
transforms Python code into an IR which can then generate code in another
language. For example, lightly annotated PyTorch code can be translated into
an S-Expression representation that is then converted into Lantern.

Building on this concept, we could create a translator that generates Flare
code from Python source code. A more efficient alternative, however, is to
utilize the existing front ends of Spark. With support for various front-end
APIs such as PySpark and Koalas (which offers a Pandas-like interface), Spark
leverages its Catalyst optimizer to construct optimized query plans and its
core backend for query execution. Since Flare can operate solely on these
optimized query plans built by Spark, we can extract the query plans generated
by these high-level front ends and employ Flare’s runtime for execution. With
Lantern and Flare’s ability to support these front ends, Flern has the
capability to accelerate end-to-end data science pipelines written using these
user-friendly APIs, requiring only minimal modifications to the original code.

##  4\. Experimental Evaluation

We evaluate the performance of Flern in two phases. First, we evaluate the
impact of the main global optimizations we have detailed in Section 3, such as
minimizing data movement overheads, streamlining UDF execution, and
overlapping ML computations (e.g., GPU) with data manipulations. Second, we
benchmark Flern’s performance against established systems. A distinct feature
of Flern is its ability to rival the performance of top-tier systems in
individual workloads. As such, we begin by evaluating Flern’s standalone
performance across a variety of representative data manipulation and machine
learning benchmarks, comparing it with the leading systems designed for those
specific tasks. It should be noted that, for respective individual workloads,
Flern generates code that is almost identical to that of Flare and Lantern.
This similarity allows it to replicate previously published experimental
results. Consequently, we have omitted any results where Flern simply
reproduces the benchmark performance previously published for Flare and
Lantern.

For data manipulation workloads, Flern demonstrates the capability to execute
the complete TPC-H benchmark (Council., [n.d.]), achieving competitive
performance with cutting-edge query compilers like HyPer (Neumann, 2011),
reproducing the results in (Essertel et al., 2018; Tahboub et al., 2018). In
this paper, we include an additional data science-oriented benchmark and
compare against more baselines. For ML workloads, Flern exhibits the ability
to run sophisticated ML models such as ResNet (He et al., 2016), DeepSpeech2
(Amodei et al., 2016b), and SqueezeNet (Iandola et al., 2016a), matching the
performance of leading-edge frameworks like PyTorch and TensorFlow,
reproducing the results in (Wang et al., 2019; Wang et al., 2018). In this
paper, we include experimental results for a transformer-based (Vaswani et
al., 2017) model.

Subsequently, we analyze the performance of the combined tasks. This includes
relational queries that combine ML computations and end-to-end data science
pipelines where data manipulation is followed by ML training. Our analysis
highlights how superficial integrations between task-specific libraries, such
as Spark for data manipulation and PyTorch for deep learning, can result in
added overheads and inhibit potential global optimizations, leading to
suboptimal end-to-end performance. The primary objective of our experiment
section is to substantiate Flern’s comprehensive performance across both types
of workloads and to validate its capacity to match, if not surpass, the best-
of-breed performance in representative benchmarks.

###  4.1. Benchmark Environment

For CPU only workloads, we used a NUMA machine with 4 sockets, 24 Intel(R)
Xeon(R) Platinum 8168 cores per socket, and 750GB RAM per socket (3 TB total).
For GPU enabled workloads, we conducted experiments on a NUMA machine with 2
sockets, 12 Intel(R) Xeon(R) Gold 6126 cores per socket, and 100GB RAM per
socket (200 GB total), a GPU cluster with four NVIDIA GeForce GTX 1080 Ti 11
GB GPUs. Both servers ran Ubuntu 18.04.4 LTS as the OS.

We use the following versions of libraries/frameworks in the experiments (and
the respective versions). Python 3.7.6, Pandas 1.0.3, PyTorch 1.5.0,
TensorFlow 2.2.0, NumPy 1.18.1, Weld 0.4.0, Dask 2.15.0, CuDF 0.14, Spark
2.4.5, Postgres 12.8, Python-DuckDB 0.8.0, Python-Polars 0.17.12 and GCC
7.5.0. Each reported result is an average of five runs after removing the
lowest and highest value. Unless otherwise specified, all the tools are
running in multi-threaded mode with the default number of threads which in
most cases equal to the number of logical cores in the system.

###  4.2. Effects of Key Optimizations

(a)

(b)

(c)

(d)

Figure 8. Performance impact of global optimizations; (a) using shared data
buffers to minimize data duplication (CPU), (b) using pinned memory and
performing asynchronous data movement to GPU (c) vectorizing the UDFs (CPU)
(d) running the UDFs asynchronous (in GPU)

In this section, we evaluate the performance impact of the key optimization
steps we have incorporated. We used the NYC-Taxi dataset from Kaggle (Kaggle,
2017), which is used for training ML models to predict the duration of taxi
trips based on certain information such as pickup and drop-off locations, time
of day, and so on. The ML model utilized for this task is a three-layer fully
connected neural network, featuring ReLu activations (Agarap, 2018) between
each layer.

Minimizing Data Movement Overhead: We begin by evaluating the impact of
minimizing data movement overhead at system boundaries. Figure 8(a) shows the
performance across three scenarios. One common approach to combining systems
is to export the output from one system and import it from the other. Such a
naive integration between Flare and Lantern, employing a standard data format
(in this case, CSV), results in substantial data movement overhead, accounting
for over 80% of the execution time. This is because such formats are not
specifically designed for data transfer, causing both the writer and reader to
spend considerable time on redundant tasks such as formatting and parsing
data. An improvement to this approach is to utilize a binary format,
recognized by both systems, significantly accelerating the data movement
process (approximately 2.8 times faster). On the other hand, as discussed in
Section 3.1, in Flern, high-level abstractions like Tensors and FlatBuffers
translate into native data structures (i.e., C Arrays) in the generated code,
enabling the systems to coexist within the same memory address space and
leverage shared data buffers, thereby minimizing unnecessary data duplication.
This strategy effectively eradicates the data movement overhead between the
systems, resulting in over a 6x speedup compared to the basic integration.

Optimized Host to Device Data Movement: In the next experiment, we evaluate
the impact of improving the data movement from CPU memory to GPU memory.
Figure 8(b) depicts the results for this micro-benchmark. The first bar
represents the sequential case, where the data manipulation segment of the
workload is completed first, followed by the transfer of the produced data to
the GPU. In this scenario, data transfer consumes nearly 30% of the total
running time. With the optimized approach, data is moved in batches as it’s
being produced, rather than waiting for the completion of the entire data
manipulation phase. As a result, data movement is overlapped with data
manipulation computations, effectively increasing efficiency. Furthermore,
extra data copying is eliminated by allocating relevant buffers in page-locked
(i.e., pinned) memory, as explained in Section 3.2. By overlapping memory
transfers and streamlining the copying process, we achieve a 40% reduction in
the end-to-end running time.

Optimized UDF Execution Our next set of experiments evaluates the performance
of executing ML models (or any tensor computations) as internal UDFs in Flern,
as detailed in Section 3.3. Figure 8(c) illustrates the notable performance
improvements of executing UDFs in a vectorized approach, which yields an
execution time more than 140x faster than the serial approach. This
significant difference can be attributed to the kernel launch
overheads—specifically, the cost of launching the cblas kernels in this case
being amortized across the records in a batch during vectorized execution.
Conversely, in the serial case, each record incurs these costs individually.

Figure 8(d) shows the execution time for conducting the same experiment on
GPUs. We have excluded the results for the single-instance version due to its
markedly inferior performance relative to the vectorized versions. The first
bar in Figure 8(d) represents the scenario in which the UDF call is blocking
and separately invoked by each data processing thread. This means that every
thread independently initiates data transfers (between CPU and GPU) and all
kernel invocations. As a result, it leads to a larger number of smaller data
transfers and kernel invocations (due to limited GPU memory), which, adds up
to substantial launch overheads. Flern addresses this by incorporating a
distinct worker thread pool (as illustrated in Figure 7), where all data
movement and kernel invocations are collectively managed. In this setting, the
dedicated ML threads are capable of moving data and invoking kernels for large
data batches, thereby reducing overall kernel launch overheads. This
optimization results in a 5x speedup compared to the initial scenario.

###  4.3. Evaluating Individual Workloads

In this section, we evaluate Flern’s performance across separate individual
workloads. As previously detailed in Section 3, Flern preserves all
optimizations inherent to the individual systems. The purpose of the
experiments conducted in this section is to verify the sustained state-of-the-
art performance of individual workloads in the absence of interactions with
other systems.

(a)

(b)

Figure 9. Performance comparison for individual workloads (a) ML (b) DB. (a)
Per epoch execution time for Transformer model (Vaswani et al., 2017) on
WMT’14 Machine Translation dataset (Elliott et al., 2016) (b) Running Time (s)
for executing multiple (nested) join queries, followed by a _group by_ and an
_aggregation_ (vertical axis is in log-scale).

####  4.3.1. Data Manipulation

We evaluate the performance of data manipulation using the Favorita Dataset
from Kaggle (Kaggle, 2018a). The dataset consists of six tables representing
information about sales of a retail store, and the largest table contains 125
million tuples. We perform a natural join across the six tables, followed by a
_group by_ and an _aggregation_ query. This benchmark covers a representative
data manipulation workload since we perform joins across large relations and
performing aggregated queries on grouped relations.

Figure 9(b) shows the results from this benchmark. Pandas is the go-to
framework among data scientists due to its user-friendly API and popularity
(Kaggle, 2018b). However, when used in medium-scale workloads (¿1GB), the
performance of Pandas becomes inferior due to the lack of multi-threaded
execution, which results in the longest execution time for this benchmark.
Dask, in comparison, enhances Pandas’ performance by manipulating chunks of
Pandas DataFrames and executing them concurrently (Dask Development Team,
2016). Despite this improvement over Pandas, Dask’s execution time remains
substantially higher than the other systems.

Spark, widely acknowledged as the de facto standard for big data processing
(Salloum et al., 2016), is known to exhibit subpar performance when utilized
for medium-sized workloads (Essertel et al., 2018), yielding lower performance
compared to other faster systems. Despite implementing query compilation
techniques that superficially resemble those of Flern, Spark’s performance
lags notably in this benchmark. This gap, as delineated by Essertel et al.
(2018), can be attributed to the inherent limitations of Spark’s query
compilation approach, such as the granularity of code generation.

CuDF performs the entirety of its data manipulation computations on GPUs with
minimal reliance on CPU. While CuDF outperforms Spark, it exhibits a longer
runtime than Flern. This gap in performance can be attributed to the overheads
related to data spilling (from GPU to CPU), a consequence of limited GPU
memory, as well as the varying internal algorithms utilized by different
platforms (for example, Hash Join versus Sort-merge Join). However, for a
thorough and accurate comparison, it would be necessary to conduct an
extensive performance analysis, factoring in hardware costs, given CuDF’s
primary utilization of GPUs. Such an analysis, while valuable, lies outside
the purview of this paper.

DuckDB (Raasveldt and Mühleisen, 2019) and Polars (Developers, 2022), both
relatively recent data analytics frameworks, are specifically designed for
fast single-node, parallel execution. Much like SQLite (Hipp, 2020), DuckDB is
fashioned as an embedded analytical DBMS, employing a highly efficient
vectorized engine for query execution. For this benchmark, Flern displays
competitive performance compared to these systems, achieving significant
speedups - 9.6x against Polars and 1.44x against DuckDB. The observed speedup
can be attributed to Flern’s ability to generate specialized code, effectively
eliminating any runtime interpretive overhead.

Considering Flern’s ability to replicate the previously published competitive
performance against leading-edge systems like HyPer (Neumann, 2011), along
with its impressive results on new benchmarks, it can be confidently stated
that Flern is capable of achieving performance on par with the state-of-the-
art systems for data manipulation workloads.

####  4.3.2. Machine Learning

In the preceding section, we demonstrated how Flern significantly outperforms
the performance of many commonly utilized data processing libraries and
systems within the data science community. In this section, we shift our focus
to evaluate Flern’s capabilities when executing state-of-the-art deep learning
models. Specifically, we trained a transformer-based machine translation model
(Vaswani et al., 2017) on the WMT’14 Multimodal Translation dataset (Elliott
et al., 2016). The model mirrors the configuration of the original
Transformer-base model (Vaswani et al., 2017), employing 6 encoder and decoder
layers and an embedding size of 256, while parameters are updated using the
Adagrad optimizer.

Figure 9(a) provides a comparison of the per-iteration running time for Flern
against PyTorch. Given the similarity of the internal CUDA kernels used by
both systems, their performance is almost equivalent. This experiment’s key
finding illustrates that Flern can execute these cutting-edge deep learning
models with competitive performance, a capability that has not been
demonstrated by prior works developing common intermediate layer integrations
(Palkar et al., 2017; Sujeeth et al., 2013).

###  4.4. Evaluating Combined Workloads

To assess the performance of Flern in combined workloads, we present three
sets of experiments. Firstly, we execute an augmented User-Defined Function
(UDF) that incorporates a tensor computation within it. Secondly, a set of
experiments where a complete ML model is used as a UDF, executed on both CPU
and GPU. Lastly, we implement an end-to-end pipeline where an ML model is
trained using the data obtained via a relational query.

####  4.4.1. Enhanced UDFs

In this section, we evaluate the performance of Flern in comparison with Weld,
a system that bears superficial similarities to our own. We selected a
benchmark from one of Weld’s openly available implementations in which a
scalar value (a crime index) is computed via dot product involving a chosen
subset of columns (crime-related data) and a predefined constant vector. This
benchmark showcases a use case where multiple libraries are jointly employed.
Specifically, in the case of Weld, the data manipulation portion is handled by
Grizzly, a Pandas accelerator based on Weld, and the dot product is handled by
Weld-NumPy. Similarly, Flern utilizes Lantern for dot product operations.

(a)

(b)

Figure 10. Execution time for performing a dot product as a UDF in an SQL
query. (a) shows the end-to-end execution time whereas (b) compares the
execution time of the generated code of Weld and Flern.

Figure 10(a) shows the end-to-end execution time, including time for data
loading, code generation, and compilation (for both Weld and Flern).
Intriguingly, Weld’s end-to-end time surpasses that of Pandas. This can be
primarily attributed to Weld’s reliance on Pandas for data loading and the
time expended on marshaling and demarshaling data between the host language
and Weld’s runtime. Figure 10(b) provides a glimpse into the actual runtime of
the code generated by Weld and Flern, excluding data loading and any other
communication overheads. Weld’s generated code has been optimized through
several passes, including loop fusion, vectorization, loop unrolling, and
others, as well as LLVM level optimizations (Palkar et al., 2018). Similarly,
Flern’s generated code undergoes a multitude of optimizations at various
levels, from domain-specific ones to code specialization and low-level
compiler optimizations. Notably, Flern’s resultant code exhibits a 2.5x
speedup compared to Weld. This suggests that while both systems construct a
common IR across systems and generate low-level code, the methodology employed
to architect such IRs can have impacts on the final performance.

####  4.4.2. Running ML Classifiers as UDFs

In this section, we assess the performance of executing ML classifiers as UDFs
in relational queries, comparing Flern’s performance with Spark, Postgres, and
DuckDB with which PyTorch serves as the ML framework. We note that the
publicly available Weld implementation does not support the workloads in this
section (and Section 4.4.3).

In the case of Spark, it employs Apache Arrow (Richardson et al., 2022), a
common in-memory columnar data format for data transition between JVM and
Python environments. Arrow minimizes data serialization and transfer costs,
thereby cutting down the communication overhead across system boundaries. This
strategy considerably outpaces the conventional method of executing the UDF as
an external function on a per-record basis (i.e., scalar UDF), with
serialization/deserialization at system boundaries (the numbers for this case
are not reported). For Postgres, we employed the PL/Python plugin and
implemented the ML operations using PyTorch. We have adopted a similar
approach for DuckDB as Spark, leveraging Arrow to efficiently transfer data to
PyTorch in a vectorized manner. Specifically, we used the provided utilities
to convert the relation into batches of records, which were then streamed to
PyTorch for performing ML inference.

This experiment reuses the same dataset and model from Section 4.2 for the CPU
scenario and employs a Transformer-based sentiment classifier on the Yelp
Reviews Dataset (Yelp, [n.d.]) (1 million reviews) for the GPU scenario.
Specifically, for sentiment analysis, we calculate the aggregate number of
positive and negative reviews based on restaurant star ratings (the query is
displayed below). The sentiment classifier contains word and positional
embeddings (Vaswani et al., 2017), six encoder layers, succeeded by a three-
layer fully connected layer with ReLu (Agarap, 2018) activations.

⬇

SELECT

stars,

Sum(CASE WHEN sentiment < 0.5 THEN 1 ELSE 0 END) AS negative,

Sum(CASE WHEN sentiment >= 0.5 THEN 1 ELSE 0 END) AS positive

FROM

(SELECT stars, classifier(review) AS sentiment FROM review NATURAL JOIN
business)

GROUP BY stars

(a)

(b)

Figure 11. Performance comparison for (a) Running a 3-layer Neural Network
(Regression) UDF on CPU (b) Running a Transformer-based Sentiment Classifier
UDF on GPU

Figure 11(a) shows the execution time for the CPU scenario, in which Flern
showcases speedups exceeding 50x against Postgres, approximately 12x over
Spark, and around 4x over DuckDB. We conducted numerous micro-benchmarks to
uncover the reason behind this substantial speedup. For the Postgres+PT,
Spark+PT, and DuckDB+PT scenarios, we ran the corresponding workloads on each
individual system (i.e., Postgres, Spark, DuckDB, and PyTorch separately) to
gauge the extent of overheads introduced due to the integration. Figure 11(a)
emphasizes the time portion allocated for actual computation.

These overheads primarily stem from data copying and format conversion
overheads occurring at system boundaries. Specifically, for Spark, this
comprises the cost of converting Spark DataFrames to Arrow, transitioning data
from one execution environment to another, converting Arrow data to Pandas
DataFrames, and so on. Similar data copying happens in the Postgres and DuckDB
scenarios as well. Such overheads account for more than 90% of the total
execution time for Postgres and Spark, and exceed 70% for DuckDB. As Flern
generates a unified piece of low-level code with the UDFs inlined and fused
with the data processing loops, none of these overheads exist. Furthermore,
these UDFs are transparent to the compiler and undergo further global
optimizations.

Figure 11(b) presents the execution time for the GPU scenario. Interestingly,
the Postgres implementation runs faster than Spark. This is likely due to a
lack of coordination among the Spark workers when executing computations on
the GPU. Specifically, each worker independently dispatches computations to
the GPU, leading to multiple workers launching kernels for smaller data
batches (constrained by the limited GPU memory) simultaneously. This results
in an increase in kernel launches, smaller but more frequent data transfers
between host and device, and consequently, an increase in launch overheads,
which becomes the dominant factor. As our micro-benchmarks in Section 4.2
indicated, such lack of coordination can significantly impede performance.
Conversely, in the case of Postgres, a single PyTorch worker is initiated by
the DB to handle all GPU computations, thereby minimizing such kernel launch
overheads. However, this approach has the drawback of requiring the complete
set of records to be ready before invoking the ML computation. DuckDB
outperforms Spark due to its highly efficient vectorized query engine tuned
for single node workloads. As outlined in Section 3.2 (Figure 7), Flern
employs a separate thread pool that batches ML computations from data
processing workers, which mitigates the kernel launch overheads. This,
combined with the elimination of system boundary crossing overheads, makes
Flern notably faster than the other systems.

It’s also important to highlight that while the cross-system communication
overhead is confined to a single occurrence for the two queries chosen for our
experiments in this section, scenarios may arise where this communication
recurs multiple times within a single query, multiplying the impact of
aforementioned cross-system overheads. Seamless integration in the style of
Flern would be even more advantageous in such scenarios.

####  4.4.3. End-to-end Training

Figure 12.  Execution time (ms) for an end-to-end ML training pipeline
consisting of querying a relational source, performing a join, and training an
ML model. Each bar segment in the chart denotes the time duration dedicated to
specific subtasks (such as data reading and performing joins, etc.) in each
case.

In this section, we assess the performance of Flern in the context of training
an ML model using data obtained through a relational query. We use the
Favorita dataset (Kaggle, 2018a) from Kaggle, identical to the one used in
Section 4.3.1. The task involves training a model to predict sales numbers
accurately for a large grocery chain. This particular task has also been
utilized as a benchmark in previous related studies (Shaikhha et al., 2020;
Schleich et al., 2019). The dataset contains multiple relational tables,
including related information such as daily oil prices, item-specific data
(for instance, whether an item was promoted), transaction histories, and more.
Similar to Shaikhha et al. (2020), we initially conduct a natural join
operation across all the relations, subsequently selecting all numerical
columns. We then train a neural network comprising three layers, with ReLU
(Agarap, 2018) activations used in the first two layers. We compute the Mean
Squared Error (MSE) loss, calculate gradients concerning the loss, and
leverage Stochastic Gradient Descent (SGD) to optimize the model parameters.
For this benchmark, the ML computations are run on the GPU. We run this
benchmark on a selection of the most widely used and high-performance systems
in the DB and ML domains.

The results of this benchmark are summarized in Figure 12. The performance
mirrors the pattern observed in Section 4.3.1 for data manipulation
benchmarks, attributable to the same identified limitations. CuDF was unable
to complete the experiment due to insufficient GPU memory. Given that Flare
performs runtime native code generation, there is no _direct_ way of
integrating it with a library like PyTorch, making data movement a bottleneck
in this context. In contrast, Flern, which establishes an end-to-end compiled
data path for executing the combined workload, outperforms all other
approaches. Flern achieves a speedup exceeding 13x compared to the original
Pandas version and manages around a 3.2x speedup over a basic integration
between Flare and PyTorch. DuckDB and Polars, with their highly efficient
parallel query engines, come closest to achieving the performance of Flern.
However, Flern still outperforms them by a factor of approximately 3x,
primarily due to its tight integration with the ML system.

##  5\. Related Work

Weld (Palkar et al., 2017; Palkar et al., 2018) is a common runtime
specifically targeting data-intensive applications with a main focus on
physical data movement optimizations. Weld provides a unified runtime for
multiple library front ends such as Pandas (McKinney et al., 2010) and NumPy
(van der Walt et al., 2011) by implementing the corresponding operations using
the Weld API that constructs Weld IR fragments. Delite (Sujeeth et al., 2014,
2013) is a compiler framework for implementing embedded domain-specific
languages (DSLs). The Distributed Multiloop Language (DMLL) (Brown et al.,
2016) performs cross optimizations on data processing and ML operations. Lara
(Kunft et al., 2019, 2016) is a DSL that supports both collection processing
and ML operations, and accompanies an associated IR that represents the
combined workload. Despite these systems’ capability to enhance performance in
combined workloads, their performance either falls short in isolated tasks,
failing to reach state-of-the-art or they lack the comprehensive functionality
required to run extensive benchmark suites such as full TPC-H or modern ML
models. Moreover, these systems necessitate the ground-up construction of
individual systems or substantial modifications to pre-existing systems,
leading to increased developmental complexity and effort.

Split Annotations (Palkar and Zaharia, 2019) identifies these limitations and
presents an approach that can combine existing systems without modification.
Specifically, they treat library functions as black boxes and adds a cross-
function data pipelining layer to curb data movement overheads. Nonetheless,
this approach sacrifices the benefits of building common IRs and performing
code generation, such as cross-system operator fusion.

Numerous prior works have investigated the approaches to integrate ML
computations into DB systems to circumvent the expensive data movement
overheads (Jasny et al., 2020; Hellerstein et al., 2012; Feng et al., 2012;
Schüle et al., 2021; Karanasos et al., 2020). MADLib (Hellerstein et al.,
2012) and Bismarck (Feng et al., 2012), for instance, encapsulate ML
computations as user-defined functions (UDFs) within the DBMS. Furthermore,
factorized ML approaches efficiently perform ML operations on multi-table data
by pushing ML computations to the normalized relations, thus eliminating the
necessity to materialize extensive join results (Kumar et al., 2015a, b;
Olteanu and Schleich, 2016a; Schleich et al., 2016; Schleich et al., 2019;
Olteanu and Schleich, 2016b; Chen et al., 2017; Li et al., 2019). However,
these approaches tend to be algorithm-specific and either do not support the
full range of modern ML models or lean on external systems like TensorFlow
(Abadi et al., 2016) for more complex models (Developers, 2023).

Raven (Karanasos et al., 2020; Park et al., 2022) extends ONNX IR (Developers,
2019) by incorporating relational algebra operations, thus creating a unified
IR for the combined workload and facilitating cross-optimizations. Similarly,
Umbra (Neumann and Freitag, 2020; Rieger et al., [n.d.]) integrates compiled
TorchScript (Developers, 2020; Paszke et al., 2019) modules as relational
algebra operations in the query plan. Despite the considerable flexibility
these approaches provide, they still rely on disparate systems for actual
execution, which leads to overheads at the interfaces between these systems.

Kläbe et al. (2023) delve into a variety of approaches for in-DB ML. Our
approach mostly aligns with their proposed ‘native operator’ approach, which
they perceive as the most performance-optimized route. Although they postulate
that this path might necessitate substantial engineering effort, Flern
actualizes this design using an existing ML framework, thereby circumventing
the need for implementing these operators from scratch.

Many DBMS and cloud vendors offer in-DB ML support. These include, Google
BigQuery ML (Google, 2018), Microsoft SQL Server ML (Microsoft, [n.d.]),
Redshift-ML (Amazon, 2021), Postgres (via PL/Python (Postgre, [n.d.])), Oracle
(Oracle, [n.d.]), Vertica-ML (Fard et al., 2020). Their main focus is to
provide a user interface designed for handling queries that incorporate ML
computations, although the execution of these computations is often delegated
to external systems (Kläbe et al., 2023).

##  6\. Conclusions

Modern data analytics workloads frequently intertwine DB and ML computations.
Yet, the integration of ML capabilities with DBMSs (or vice versa) poses
significant challenges due to the contrasting nature of these workloads. While
individual ML systems and DBMSs are optimized for their specific tasks, the
performance often diminishes significantly when these systems are integrated
naively. A promising strategy to mitigate these overheads involves
architecting individual systems first and then integrating them at a common
intermediate layer, facilitating the generation of a unified executable for
the combined workload.

This paper introduces Flern, the first such intermediate layer that achieves
best-of-breed performance in both individual _and_ combined workloads. Our
approach, based on generative programming, demonstrates how common
intermediate layers enable efficient integration between existing systems
without incurring substantial re-engineering costs. We posit that this paper
outlines a principled methodology to address this challenge in general:
architect systems based on generative programming so that they can be adapted
more effectively and at a lower engineering cost.

## References

  * (1)
  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016\.  TensorFlow: A System for Large-Scale Machine Learning. In _OSDI_. USENIX Association, 265–283.
  * Agarap (2018) Abien Fred Agarap. 2018\.  Deep Learning using Rectified Linear Units (ReLU).  _CoRR_ abs/1803.08375 (2018).
  * Amazon (2021) Amazon. 2021.  Redshift ML.  https://aws.amazon.com/redshift/features/redshift-ml/ [Online; accessed 27-April-2023].
  * Amodei et al. (2016a) Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Erich Elsen, Jesse H. Engel, Linxi Fan, Christopher Fougner, Awni Y. Hannun, Billy Jun, Tony Han, Patrick LeGresley, Xiangang Li, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Sheng Qian, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Chong Wang, Yi Wang, Zhiqian Wang, Bo Xiao, Yan Xie, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. 2016a.  Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin. In _ICML_ _(JMLR Workshop and Conference Proceedings, Vol. 48)_. JMLR.org, 173–182.
  * Amodei et al. (2016b) Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. 2016b.  Deep speech 2: End-to-end speech recognition in english and mandarin. In _International conference on machine learning_. PMLR, 173–182.
  * Armbrust et al. (2015) Michael Armbrust, Reynold S. Xin, Cheng Lian, Yin Huai, Davies Liu, Joseph K. Bradley, Xiangrui Meng, Tomer Kaftan, Michael J. Franklin, Ali Ghodsi, and Matei Zaharia. 2015\.  Spark SQL: Relational Data Processing in Spark. In _SIGMOD Conference_. ACM, 1383–1394.
  * Brown et al. (2016) Kevin J. Brown, HyoukJoong Lee, Tiark Rompf, Arvind K. Sujeeth, Christopher De Sa, Christopher R. Aberger, and Kunle Olukotun. 2016\.  Have abstraction and eat performance, too: optimized heterogeneous computing with parallel patterns. In _CGO_. ACM, 194–205.
  * Chen et al. (2017) Lingjiao Chen, Arun Kumar, Jeffrey F. Naughton, and Jignesh M. Patel. 2017.  Towards Linear Algebra over Normalized Data.  _Proc. VLDB Endow._ 10, 11 (2017), 1214–1225.
  * Council. ([n.d.]) The Transaction Processing Council. [n.d.].  TPC-H Version 2.15.0.
  * Dask Development Team (2016) Dask Development Team. 2016\.  _Dask: Library for dynamic task scheduling_.  https://dask.org https://dask.org.
  * Decker et al. (2019) James M Decker, Dan Moldovan, Andrew A Johnson, Guannan Wei, Vrittant Bhardwaj, Gregory Essertel, and Fei Wang. 2019.  Snek: Overloading Python Semantics via Virtualization.  (2019).  https://www.cs.purdue.edu/homes/rompf/papers/decker-preprint201907.pdf.
  * Developers (2023) MADLib Developers. 2023\.  Apache MADlib: Deep Learning.  https://madlib.apache.org/docs/latest/group__grp__dl.html.
  * Developers (2019) ONNX Developers. 2019\.  ONNX: Open Neural Network Exchange.  https://github.com/onnx/onnx.
  * developers (2021) ONNX Runtime developers. 2021\.  ONNX Runtime.  https://onnxruntime.ai/.
  * Developers (2020) PyTorch Developers. 2020\.  TorchScript.  https://pytorch.org/docs/stable/jit.html.
  * Developers (2022) Polars Developers. 2022\.  Polars.  https://github.com/pola-rs/polars.
  * Elliott et al. (2016) Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016.  Multi30K: Multilingual English-German Image Descriptions. In _VL@ACL_. The Association for Computer Linguistics.
  * Essertel et al. (2019) Grégory Essertel, Ruby Y Tahboub, Fei Wang, James Decker, and Tiark Rompf. 2019\.  Flare & Lantern: Efficiently swapping horses midstream.  _Proceedings of the VLDB Endowment_ 12, 12 (2019), 1910–1913.
  * Essertel et al. (2018) Grégory M. Essertel, Ruby Y. Tahboub, James M. Decker, Kevin J. Brown, Kunle Olukotun, and Tiark Rompf. 2018.  Flare: Optimizing Apache Spark with Native Compilation for Scale-Up Architectures and Medium-Size Data. In _OSDI_. USENIX Association, 799–815.
  * Fard et al. (2020) Arash Fard, Anh Le, George Larionov, Waqas Dhillon, and Chuck Bear. 2020.  Vertica-ml: Distributed machine learning in vertica database. In _Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data_. 755–768.
  * Feng et al. (2012) Xixuan Feng, Arun Kumar, Benjamin Recht, and Christopher Ré. 2012.  Towards a unified architecture for in-RDBMS analytics. In _SIGMOD Conference_. ACM, 325–336.
  * Fischer (1993) Michael J. Fischer. 1993\.  Lambda-Calculus Schemata.  _LISP Symb. Comput._ 6, 3-4 (1993), 259–288.
  * Futamura (1999) Yoshihiko Futamura. 1999\.  Partial Evaluation of Computation Process - An Approach to a Compiler-Compiler.  _High. Order Symb. Comput._ 12, 4 (1999), 381–391.
  * Google (2018) Google. 2018.  BigQuery ML.  https://cloud.google.com/bigquery/docs [Online; accessed 27-April-2023].
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Hellerstein et al. (2012) Joseph M. Hellerstein, Christopher Ré, Florian Schoppmann, Daisy Zhe Wang, Eugene Fratkin, Aleksander Gorajek, Kee Siong Ng, Caleb Welton, Xixuan Feng, Kun Li, and Arun Kumar. 2012\.  The MADlib Analytics Library or MAD Skills, the SQL.  _CoRR_ abs/1208.4165 (2012).
  * Hipp (2020) Richard D Hipp. 2020\.  SQLite.  https://www.sqlite.org/index.html
  * Iandola et al. (2016a) Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. 2016a.  SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and¡ 0.5 MB model size.  _arXiv preprint arXiv:1602.07360_ (2016).
  * Iandola et al. (2016b) Forrest N. Iandola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. 2016b.  SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size.  _CoRR_ abs/1602.07360 (2016).
  * Jasny et al. (2020) Matthias Jasny, Tobias Ziegler, Tim Kraska, Uwe Röhm, and Carsten Binnig. 2020.  DB4ML - An In-Memory Database Kernel with Machine Learning Support. In _SIGMOD Conference_. ACM, 159–173.
  * Kaggle (2017) Kaggle. 2017.  NYC Taxi Trip Duration.  https://www.kaggle.com/c/nyc-taxi-trip-duration [Online; accessed 21-April-2023].
  * Kaggle (2018a) Kaggle. 2018a.  Favorita Grocery Sales Forecasting.  https://www.kaggle.com/c/favorita-grocery-sales-forecasting [Online; accessed 21-April-2023].
  * Kaggle (2018b) Kaggle. 2018b.  Kaggle Developer Survey.  www.kaggle.com/headsortails/what-we-do-in-the-kernels-a-kaggle-survey-story.
  * Karanasos et al. (2020) Konstantinos Karanasos, Matteo Interlandi, Fotis Psallidas, Rathijit Sen, Kwanghyun Park, Ivan Popivanov, Doris Xin, Supun Nakandala, Subru Krishnan, Markus Weimer, Yuan Yu, Raghu Ramakrishnan, and Carlo Curino. 2020\.  Extending Relational Query Processing with ML Inference. In _CIDR_. www.cidrdb.org.
  * Kläbe et al. (2023) Steffen Kläbe, Stefan Hagedorn, and Kai-Uwe Sattler. 2023\.  Exploration of Approaches for In-Database ML.  (2023).
  * Kumar et al. (2015a) Arun Kumar, Mona Jalal, Boqun Yan, Jeffrey F. Naughton, and Jignesh M. Patel. 2015a.  Demonstration of Santoku: Optimizing Machine Learning over Normalized Data.  _Proc. VLDB Endow._ 8, 12 (2015), 1864–1867.
  * Kumar et al. (2015b) Arun Kumar, Jeffrey F. Naughton, and Jignesh M. Patel. 2015b.  Learning Generalized Linear Models Over Normalized Data. In _SIGMOD Conference_. ACM, 1969–1984.
  * Kunft et al. (2016) Andreas Kunft, Alexander Alexandrov, Asterios Katsifodimos, and Volker Markl. 2016.  Bridging the gap: towards optimization across linear and relational algebra. In _Proceedings of the 3rd ACM SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond_. 1–4.
  * Kunft et al. (2019) Andreas Kunft, Asterios Katsifodimos, Sebastian Schelter, Sebastian Breß, Tilmann Rabl, and Volker Markl. 2019.  An Intermediate Representation for Optimizing Machine Learning Pipelines.  _Proc. VLDB Endow._ 12, 11 (2019), 1553–1567.
  * Lattner and Adve (2004) Chris Lattner and Vikram S. Adve. 2004.  LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation. In _CGO_. IEEE Computer Society, 75–88.
  * Li et al. (2019) Side Li, Lingjiao Chen, and Arun Kumar. 2019.  Enabling and Optimizing Non-linear Feature Interactions in Factorized Linear Algebra. In _SIGMOD Conference_. ACM, 1571–1588.
  * McKinney et al. (2010) Wes McKinney et al. 2010\.  Data structures for statistical computing in python. In _Proceedings of the 9th Python in Science Conference_ , Vol. 445. Austin, TX, 51–56.
  * Microsoft ([n.d.]) Microsoft. [n.d.].  SQL Server ML.  https://learn.microsoft.com/en-us/sql/machine-learning/sql-server-machine-learning-services?view=sql-server-ver16 [Online; accessed 27-April-2023].
  * Neumann (2011) Thomas Neumann. 2011\.  Efficiently Compiling Efficient Query Plans for Modern Hardware.  _Proc. VLDB Endow._ 4, 9 (2011), 539–550.
  * Neumann and Freitag (2020) Thomas Neumann and Michael J Freitag. 2020.  Umbra: A Disk-Based System with In-Memory Performance.. In _CIDR_.
  * Odersky and Zenger (2005) Martin Odersky and Matthias Zenger. 2005.  Scalable component abstractions. In _OOPSLA_. ACM, 41–57.
  * Olteanu and Schleich (2016a) Dan Olteanu and Maximilian Schleich. 2016a.  F: Regression Models over Factorized Views.  _Proc. VLDB Endow._ 9, 13 (2016), 1573–1576.
  * Olteanu and Schleich (2016b) Dan Olteanu and Maximilian Schleich. 2016b.  Factorized Databases.  _SIGMOD Rec._ 45, 2 (2016), 5–16.
  * Oracle ([n.d.]) Oracle. [n.d.].  Oracle Machine Learning.  https://docs.oracle.com/en/database/oracle/oracle-database/18/dmcon/neural-network.html [Online; accessed 27-April-2023].
  * Palkar et al. (2018) Shoumik Palkar, James J. Thomas, Deepak Narayanan, Pratiksha Thaker, Rahul Palamuttam, Parimarjan Negi, Anil Shanbhag, Malte Schwarzkopf, Holger Pirk, Saman P. Amarasinghe, Samuel Madden, and Matei Zaharia. 2018\.  Evaluating End-to-End Optimization for Data Analytics Applications in Weld.  _Proc. VLDB Endow._ 11, 9 (2018), 1002–1015.
  * Palkar et al. (2017) Shoumik Palkar, James J. Thomas, Anil Shanbhag, Malte Schwarzkopf, Saman P. Amarasinghe, and Matei Zaharia. 2017.  A Common Runtime for High Performance Data Analysis. In _CIDR_. www.cidrdb.org.
  * Palkar and Zaharia (2019) Shoumik Palkar and Matei Zaharia. 2019.  Optimizing data-intensive computations in existing libraries with split annotations. In _SOSP_. ACM, 291–305.
  * Park et al. (2022) Kwanghyun Park, Karla Saur, Dalitso Banda, Rathijit Sen, Matteo Interlandi, and Konstantinos Karanasos. 2022.  End-to-end Optimization of Machine Learning Prediction Queries. In _SIGMOD Conference_. ACM, 587–601.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library. In _NeurIPS_. 8024–8035.
  * Pirk et al. (2016) Holger Pirk, Oscar R. Moll, Matei Zaharia, and Sam Madden. 2016\.  Voodoo - A Vector Algebra for Portable Database Performance on Modern Hardware.  _Proc. VLDB Endow._ 9, 14 (2016), 1707–1718.
  * Postgre ([n.d.]) Postgre. [n.d.].  PL/Python: Python Procedural Language.  https://www.postgresql.org/docs/current/plpython.html [Online; accessed 27-April-2023].
  * Raasveldt and Mühleisen (2019) Mark Raasveldt and Hannes Mühleisen. 2019.  DuckDB: an Embeddable Analytical Database. In _SIGMOD Conference_. ACM, 1981–1984.
  * Richardson et al. (2022) Neal Richardson, Ian Cook, Nic Crane, Jonathan Keane, Romain François, Jeroen Ooms, and Apache Arrow. 2022.  _arrow: Integration to ’Apache’ ’Arrow’_.  https://github.com/apache/arrow/, https://arrow.apache.org/docs/r/.
  * Rieger et al. ([n.d.]) Maximilian Rieger, Moritz Sichert, and Thomas Neumann. [n.d.].  Integrating Deep Learning Frameworks into Main-Memory Databases.  ([n. d.]).
  * Rompf (2012) Tiark Rompf. 2012\.  _Lightweight Modular Staging and Embedded Compilers - Abstraction without Regret for High-Level High-Performance Programming_.  Ph.D. Dissertation. EPFL, Switzerland.
  * Rompf (2016) Tiark Rompf. 2016\.  Reflections on LMS: exploring front-end alternatives. In _SCALA@SPLASH_. ACM, 41–50.
  * Rompf and Odersky (2010) Tiark Rompf and Martin Odersky. 2010.  Lightweight modular staging: a pragmatic approach to runtime code generation and compiled DSLs. In _GPCE_. ACM, 127–136.
  * Rompf et al. (2013) Tiark Rompf, Arvind K. Sujeeth, Nada Amin, Kevin J. Brown, Vojin Jovanovic, HyoukJoong Lee, Manohar Jonnalagedda, Kunle Olukotun, and Martin Odersky. 2013.  Optimizing data structures in high-level programs: new directions for extensible compilers based on staging. In _POPL_. ACM, 497–510.
  * Salloum et al. (2016) Salman Salloum, Ruslan Dautov, Xiaojun Chen, Patrick Xiaogang Peng, and Joshua Zhexue Huang. 2016\.  Big data analytics on Apache Spark.  _Int. J. Data Sci. Anal._ 1, 3-4 (2016), 145–164.
  * Schelter et al. (2018) Sebastian Schelter, Felix Bießmann, Tim Januschowski, David Salinas, Stephan Seufert, and Gyuri Szarvas. 2018.  On Challenges in Machine Learning Model Management.  _IEEE Data Eng. Bull._ 41, 4 (2018), 5–15.
  * Schleich et al. (2016) Maximilian Schleich, Dan Olteanu, and Radu Ciucanu. 2016\.  Learning Linear Regression Models over Factorized Joins. In _SIGMOD Conference_. ACM, 3–18.
  * Schleich et al. (2019) Maximilian Schleich, Dan Olteanu, Mahmoud Abo Khamis, Hung Q. Ngo, and XuanLong Nguyen. 2019\.  A Layered Aggregate Engine for Analytics Workloads. In _SIGMOD Conference_. ACM, 1642–1659.
  * Schüle et al. (2021) Maximilian E. Schüle, Harald Lang, Maximilian Springer, Alfons Kemper, Thomas Neumann, and Stephan Günnemann. 2021.  In-Database Machine Learning with SQL on GPUs. In _SSDBM_. ACM, 25–36.
  * Shaikhha et al. (2020) Amir Shaikhha, Maximilian Schleich, Alexandru Ghita, and Dan Olteanu. 2020.  Multi-layer optimizations for end-to-end data analytics. In _CGO_. ACM, 145–157.
  * Sujeeth et al. (2014) Arvind K. Sujeeth, Kevin J. Brown, HyoukJoong Lee, Tiark Rompf, Hassan Chafi, Martin Odersky, and Kunle Olukotun. 2014\.  Delite: A Compiler Architecture for Performance-Oriented Embedded Domain-Specific Languages.  _ACM Trans. Embed. Comput. Syst._ 13, 4s (2014), 134:1–134:25.
  * Sujeeth et al. (2013) Arvind K. Sujeeth, Tiark Rompf, Kevin J. Brown, HyoukJoong Lee, Hassan Chafi, Victoria Popic, Michael Wu, Aleksandar Prokopec, Vojin Jovanovic, Martin Odersky, and Kunle Olukotun. 2013.  Composition and Reuse with Compiled Domain-Specific Languages. In _ECOOP_ _(Lecture Notes in Computer Science, Vol. 7920)_. Springer, 52–78.
  * Taha (2007) Walid Taha. 2007\.  A Gentle Introduction to Multi-stage Programming, Part II. In _GTTSE_ _(Lecture Notes in Computer Science, Vol. 5235)_. Springer, 260–290.
  * Taha and Sheard (2000) Walid Taha and Tim Sheard. 2000.  MetaML and multi-stage programming with explicit annotations.  _Theor. Comput. Sci._ 248, 1-2 (2000), 211–242.
  * Tahboub et al. (2018) Ruby Y. Tahboub, Grégory M. Essertel, and Tiark Rompf. 2018.  How to Architect a Query Compiler, Revisited. In _SIGMOD Conference_. ACM, 307–322.
  * van der Walt et al. (2011) Stéfan van der Walt, S. Chris Colbert, and Gaël Varoquaux. 2011.  The NumPy Array: A Structure for Efficient Numerical Computation.  _Comput. Sci. Eng._ 13, 2 (2011), 22–30.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need. In _NIPS_. 5998–6008.
  * Wang et al. (2018) Fei Wang, James M. Decker, Xilun Wu, Grégory M. Essertel, and Tiark Rompf. 2018\.  Backpropagation with Callbacks: Foundations for Efficient and Expressive Differentiable Programming. In _NeurIPS_. 10201–10212.
  * Wang and Rompf (2018) Fei Wang and Tiark Rompf. 2018.  A Language and Compiler View on Differentiable Programming. In _ICLR (Workshop)_. OpenReview.net.
  * Wang et al. (2019) Fei Wang, Daniel Zheng, James M. Decker, Xilun Wu, Grégory M. Essertel, and Tiark Rompf. 2019.  Demystifying differentiable programming: shift/reset the penultimate backpropagator.  _Proc. ACM Program. Lang._ 3, ICFP (2019), 96:1–96:31.
  * Wei et al. (2020) Guannan Wei, Oliver Bracevac, Shangyin Tan, and Tiark Rompf. 2020.  Compiling symbolic execution with staging and algebraic effects.  _Proc. ACM Program. Lang._ 4, OOPSLA (2020), 164:1–164:33.
  * Wei et al. (2019) Guannan Wei, Yuxuan Chen, and Tiark Rompf. 2019.  Staged abstract interpreters: fast and modular whole-program analysis via meta-programming.  _Proc. ACM Program. Lang._ 3, OOPSLA (2019), 126:1–126:32.
  * Yelp ([n.d.]) Yelp. [n.d.].  Yelp Open Dataset.  https://www.yelp.com/dataset [Online; accessed 21-April-2023].