# A Survey of Multi-Tenant
Deep Learning Inference on GPU

Fuxun Yu†, Di Wang‡, Longfei Shangguan‡, Minjia Zhang‡, Chenchen Liu§, Tolga
Soyata†, Xiang Chen†
†George Mason University, ‡Microsoft, §University of Maryland, Baltimore
County

###### Abstract

Deep Learning (DL) models have achieved superior performance. Meanwhile, the
computing hardware like NVIDIA GPUs also demonstrated strong computing scaling
trends with 2$\times$ throughput and memory bandwidth for each generation.
With such strong computing scaling of GPUs, multi-tenant deep learning
inference by co-locating multiple DL models onto the same GPU become widely
deployed to improve resource utilization, enhance serving throughput, and
reduce energy cost, etc. However, achieving efficient multi-tenant DL
inference is challenging which requires thorough full-stack system
optimization. Previous surveys either target at summarizing single tenant deep
learning inference optimizations, or only focus on certain single optimization
layer alone, such as graph-level, kernel-level, etc. This survey aims to
summarize and categorize the emerging challenges and optimization
opportunities for multi-tenant DL inference on GPU. By overviewing the entire
optimization stack, summarizing the multi-tenant computing innovations, and
elaborating the recent technique advances, we hope that this survey could shed
light on new optimization perspectives and motivate novel works in future
large-scale DL inference system optimization.

##  I Introduction

DL Application and Computing Trends Deep Learning (DL) models have achieved
superior performance in cognitive tasks like vision, speech and language
domain, and have been adopted in medical analysis, machine translation,
product recommendation, _etc._ The momentum of DL-based intelligence has
appealed millions of users and created a wide-spectrum of cloud & edge
applications like VR/AR games, intelligent robots and vehicles, large-scale
recommendation systems, and even metaverse applications [12], many of which
are featured with multi-modality, multi-tasking and substantial task
complexity, as shown in Figure 1 (a).

The emergence of such massive DL applications motivates the adoption of DL
accelerators, especially GPUs, in both cloud and edge. According to the report
[31], GPUs accounted for 85% of the $2.98B cloud data center accelerator
market in 2018. The edge hardware market, with the emerging smart
manufacturing, surveillance applications, is also projected to grow from $920M
in 2021 to $2,080M by 2026 and the edge GPUs are also taking a steady growth
to more than 50% market share with Nvidia Jetson, TX2, Xavier, Orin, etc.

Within such trends, the capacity of recent generations of GPUs demonstrates
exponential growing speed.111 Detailed scaling statistics of GPU capacities
could be found in [37]. From K80, P40, and P100 to recent T40, V100, A100
architectures, GPUs maintain a trend of doubling performance. The last
generation of V100 [7] offers 120 Tera floating point operations per second
(TFLOPS) and 900 GB/s memory bandwidth, and the numbers further increase to
312 TFLOPS and 1.6TB/s memory bandwidth for the newer A100 [6]. A100 reports
the ResNet50 [14] inference speed of 36,436 images/second, showing the
computing capacity that overwhelms the limited needs from conventional single
DL model execution schemes. Therefore, with such scaling trends in both
application complexity and GPU capacity, single model execution cannot fulfill
the needs of application scenarios nor fully utilizing the GPUs.

Multi-Tenant DL Inference, as shown in Figure 1 (b), is one promising solution
to the aforementioned scenarios with multi-modality and multi-tasking needs by
running mixed DL model workloads simultaneously on one powerful GPU to improve
the utilization, throughput, and power efficiency, etc.

There are recently many emerging works that tackle the multi-tenant DL
inference optimization on GPUs. These works usually take single optimization
point of view drawing from traditional single-model optimization experiences,
e.g., either from the DL model scheduling perspective [44, 10, 2], or the GPU
resource management perspective [9, 38, 19]. However, achieving the ult-most
efficiency for multi-tenant DL computing is more challenging as it needs to
thoroughly consider the differences between single vs. multi-tenant DL
inference, and requires multi-layer DL optimization or full-stack co-
optimization. As so, there is a great need for a systematical review of
opportunities and challenges on multi-tenant DL inference optimization.

Our survey is the first work that thoroughly analyzes the multi-tenant GPU
scheduling problem, summarizes the major differences in single- vs. multi-
tenant computing optimization, and reveals the emerging opportunities and
potential benefits of multi-tenant DL inference on GPUs. To ease the
understanding, our work also draws some experience from pevious DL computing
stacks, and compares single vs multi-tenant DL inference on GPU from a
hierarchical perspective.

Figure 1: The Emerging Trend of Multi-Tenant DL Computing on GPU.

Single vs Multi-Tenant: A Hierarchical Comparison. Traditional single-tenant
DL compilers (Figure 1 (c)) already include multi-layer optimization:
algorithm-level compression [45, 29, 46], graph-level rewriting [15, 49, 17],
runtime scheduling [10, 34] and kernel tuning [3, 16, 11, 40], etc. However,
optimizations targeting at single-tenant are usually ill-fitted for multi-
tenant inference with following examples.

From the top graph level, multi-tenant DL inference workloads represented in
directed acyclic graphs (DAGs) come with significantly higher volume of multi-
model operators than single model. This incurs many non-mergeable operators
and exposes much larger scheduling space. Another example in the lower kernel
level is TVM [3]. As one of the most competitive kernel auto-tuning framework,
TVM comes with a built-in assumption of single-tenant execution setting, the
tuning configuration of which aims to saturate all SMs and memory bandwidth of
the GPU. Such assumption and the single-tenant targeted tuning however becomes
unsuitable for multi-kernel concurrent execution with partial resource
available for each kernel. According to [8], the maximum throughput gap
comparing single-tenant vs. multi-tenant tuned configurations for the same
computing kernel could reach $5\times$ difference.

Further down to the GPU hardware, multi-tenant DL inference requires many
scheduling support to address problems like inter-tenant interference, such as
dedicated GPU hardware primitives for resource partitioning, isolation and
allocation, etc. With the current trends towards multi-tenant applications,
GPU vendors like Nvidia have recently released many new features including
Multi-Stream [22], Multi-Process Service (MPS) [25], Multi-Instance GPU (MIG)
[24] and virtual GPUs (vCS) [26] to support both runtime scheduling and
resource management, which exposes new research opportunities for multi-tenant
DL scheduling and optimization.

With such differences considered, this work will adopt such a view of
optimization stack to thoroughly analyze the multi-tenant challenges and
introduce emerging works.

By reviewing the emerging challenges, opportunities, and research works on
multi-tenant DL computing on the GPU, we hope this survey could motivate more
design and innovations in this promising new domain. The remaining paper is
organized as follows: Section II introduces the novel challenges and
opportunities in multi-tenant GPU computing stack and a high-level overview of
current vendor GPU support. Section III summarizes recent research works for
multi-tenant computing in detail. We then give our vision and insights in
Section IV. Section V concludes this paper.

##  II Challenges & Opportunities
for Multi-Tenant Computing on GPU

In this section, we first characterize the major differences between single-
vs. multi-tenant DL computing optimization through the full DL computing
stack. We then introduce the recently-released GPU features, such as Stream,
MPS, MIG, which provide important fundamental backend support for multi-tenant
computing optimization.

###  II-A Challenges for Multi-Tenant DL Computing

Traditional DL computing optimization in full stack often expands in 1
service-level orchestration [36], 2 graph-level optimization [15, 43], 3
runtime-level scheduling [10], 4 kernel-level tuning [3, 39] and 5 resource-
level management [9, 19]. Although there are many previous works for computing
optimization in these difference levels, multi-tenant computing shows dramatic
characteristics that make these methods ill-fitted. According the the same
optimization stack, we summarize the major differences in Table I and analyze
the computing challenges in Figure 2.

TABLE I: Challenges for Multi-Tenant Optimization.

Full Optimization Stack Single-Tenant Multi-Tenant Co-location No Yes 1
Service-level Interference No High Interference 2 Graph-level DAG(s) Mostly
Seq. Seq. + Parallel Parallelism Limited Extensive 3 Runtime-level Complexity
Low High Resource Usage Exclusive Shared 4 Kernel-level Tuning Objective 100%
util. $x$% partial util. 5 Resource-level Management No Resource Partition

1

Service-level: AI-centric cloud services handle millions of service queries
simultaneously [41]. With the massive computing capacity of GPUs, multiple DL
queries could be strategically co-located for efficient concurrent execution,
which is one key difference between multi-tenant GPU computing versus
traditional CPU multi-tasking. By allowing the resource sharing among
concurrent DL workloads, the service providers could potentially improve the
GPU resource utilization and reduce cost of ownership (COO) like
infrastructure and power cost especially for large-scale data centers [24].

However, the challenges remains for strategic co-location like that the inter-
tenant interference [19] could happen and degrade the quality of service such
as service-level objectives (SLA) of tail latency and throughput. This could
become worse with increased number of co-located workloads and degrade the
overall serving throughput. Therefore, there are many recent service-level
orchestration works [19, 42, 5] that design different heuristic-based,
modeling-based or prediction-based mechanisms to conduct strategic co-location
for efficient multi-tenant computing on GPUs.

Figure 2: Multi-Tenant DL Computing Challenges.

2

Graph-level: DNNs with many operators are commonly represented as directed
acyclic graphs (DAGs), which use nodes to represent operators and edges to
represent the data flow and dependency [15]. Single-model DAGs are usually
sequential with limited parallelism like VGG, ResNets, MobileNets and
EfficietNets, which have only one or two branches and thus exposes small
scheduling space [20].

By contrast, multi-tenant DL computing with multiple parallel DAGs usually
have extensive inter-operator parallelism, as shown in Figure 2, which enables
more flexible inter-operator scheduling among tenants [44]. Certain challenges
exist in such graph scheduling such as the increased complexity in larger
number of operators and scheduling space, and the more complex GPU resource
contention analysis, etc.

3

Runtime-level: Previously due to the limited inter-operator parallelism, only
a few works [10, 2] touch upon runtime-level scheduling such as certain multi-
branch models like Inception, NasNets and Transformers. These works leverage
certain GPU runtime primitives (_e.g._ , Nvidia multi-stream [22]) for
concurrent operator scheduling, many of which however incur large runtime
overheads. For example, multi-stream synchronization forces all streams to
wait/stall until the last stream finishes its workloads [22]. Multi-tenant
scheduling tends to suffer more from such overheads with the increased number
of operators and scheduling complexity. Due to the increased attention in GPU
multi-tenant scheduling, GPU vendors have recently released a series of
important features such as CUDA graphs [27] to address such scheduling
overheads.

4

Kernel-level: Kernel configurations such as loop tiling, thread blocking,
memory colasing, _etc._ could significantly influence each operator’s
computing efficiency. Previous single-tenant kernel-level works like TVM [3]
and TF-XLA [39] try to find the best configuration that can saturate the GPU
resource, i.e., exclusive resource usage. However, as multi-tenant DNNs share
the underlying resource, kernels optimized for single-tenant settings can
easily become sub-optimal for multi-tenant scenarios. Recently, there are
certain works that show multi-tenant DL computing should optimize kernel
configurations according to its available resource ratio during practical
execution [8], which shows a 5$\times$ throughput difference.

5

Resource-level: To achieve adaptive multi-tenant resource partitioning and
provisioning, it asks for both strategic design and hardware support. The
first challenge for such adaptive resource provisioning lies in the DL
workload dynamics [44]. Multiple DL models with different deep structures have
highly non-stable computing/memory requirements, making the inter-model
resource sharing and competition highly dynamic and thus hard to determine the
optimal resource partitioning. On the other hand, adaptive resource management
requires the flexible GPU reconfiguration capability. Although there are
certain adaptive resource provisioning features (_e.g._ , Nvidia multi-process
service, multi-instance GPU [25, 24]) that supports resource partitioning, the
reconfiguration process requires non-negligible time (_e.g._ , several $ms$),
which is a major limitation of the recent resource scheduling works [9, 19].

Figure 3: Multi-Stream, MPS and MIG Illustration.

###  II-B Emerging Multi-Tenant Computing Opportunities

Previously, one important reason that hinders the adoption and development of
multi-tenant DL computing on GPU is the insufficient hardware scheduling
mechanisms. But with the increasing attention in this topic, GPU vendors like
Nvidia release certain new GPU multi-tenant features to support multi-tenant
DL inference, which provides great opportunities for multi-tenant scheduling.
The multi-tenant GPU scheduling features could be categorized into two major
types: software-level and hardware-level support.

1

Software Approach: The very first GPU multi-tasking feature is the Multi-
Stream mechanism [22] supported in the Fermi GPU architecture (Figure 3 (a)).
As a software-based programming model, a stream can contain a sequence of
issued operations that execute on the GPU. Operations in different streams
could run concurrently and share the underlying GPU resources like SMs [22].
The similar feature Hyper-Q [21] is introduced in the Kepler GPU architecture
(2013) that expands previous 16-way to 32-way hardware kernel queues for
higher concurrency support. Along with the concurrency support, the CUDA
library also releases certain scheduling APIs like DeviceSync, StreamWait,
etc. to support more fine-grained scheduling capability [23]. These software-
level APIs provide valuable multi-tenant GPU scheduling mechanisms, based on
which many recent works have started to explore the fine-grained DL operator-
level scheduling techniques [44, 10].

TABLE II: GPU Support for Multi-Tenant Computing.

Mechanisms Stream MPS MIG Partition Type No Logical Physical Max Partition
Unlimited 48 7 SM Isolation No By Percentage Yes Mem BW Isolation No No Yes
Performance QoS No Partial Yes Reconfiguration Dynamic Process Launch When
Idle

TABLE III: Recent Works on Multi-Tenant Computing Optimization (JCT: job completion time, SLA: service-level agreement). Ref. | Hardware | Perspective | Algorithm/Strategy | Improvement/Achievement
---|---|---|---|---
Inter-Aware [19] | GPU | DL Service-level Orchestration |  |  $\bullet$ ML-based Interference Predictor
---
$\bullet$ Proactive Query Scheduler
|  $\bullet$ Reducing Job Interference
---
$\bullet$ Enhancing Serving Throughout
Irina [42] | GPU | DL Service-level Orchestration |  |  $\bullet$ Online Query Scheduler
---
$\bullet$ Heuristic-based Preemption
$\bullet$ Concurrent Execution & Batching
$\bullet$ Reducing Client-Side JCT
PREMA [5] | NPU | DL Service-level Orchestration |  |  $\bullet$ Online Query Scheduler
---
$\bullet$ Heuristic-based Preemption
|  $\bullet$ Reduced High-Priority Job JCT
---
$\bullet$ Maintaining Low-Priority SLA
Runtime-Aware [44] | GPU | Graph & Runtime-level Scheduling |  |  $\bullet$ Multi-Model DAG Rewriting
---
$\bullet$ ML-based Scheduling Search
$\bullet$ Multi-Stream Runtime Scheduling
$\bullet$ Reduced Inference Latency
Spatial-Tune [8] | GPU | Kernel-level Auto-Tuning |  |  $\bullet$ MPS-based Resource Allocation
---
$\bullet$ Partial-Resource Kernel Tuning
|  $\bullet$ Enhanced Kernel Performance
---
$\bullet$ Reduced Inter-kernel Interference
GSlice [9] | GPU | Resource-level Management |  |  $\bullet$ MPS-based Resource Partitioning
---
$\bullet$ Adaptive Batching
|  $\bullet$ Enhanced Serving Throughput
---
$\bullet$ Maintaining SLA
Spatial-Partition [4] | GPU | Resource-level Management |  |  $\bullet$ MPS-based Resource Partitioning
---
$\bullet$ Interference-aware Scheduling
|  $\bullet$ Enhanced Serving Throughput
---
$\bullet$ Maintaining SLA
MIG-Serving [38] | GPU | Resource-level Management |  |  $\bullet$ MIG-based Resource Reconfiguration
---
$\bullet$ Fast & Slow Query Scheduling
|  $\bullet$ Enhanced Serving Throughput
---
$\bullet$ Maintaining SLA
Planaria [13] |  | Systolic
---
Arrays
Resource-level Management |  $\bullet$ Architecture Reconfiguration |  |  $\bullet$ Enhanced Serving Throughput
---
$\bullet$ Reduced Energy Consumption

2

Hardware Approach: Besides the software support, NVIDIA recently also releases
advanced hardware-level resource management mechanisms to support flexible
resource allocation, isolation and virtualization. These resource management
methods can be categorized into two types: logical and physical. Multi-Process
Service (MPS) [25] is a logical resource partitioning mechanism (Figure 3 (b))
that allows user to partition the streaming multi-processors (SMs) and
allocate them to different processes, for example, 30%, 70% to two concurrent
processes. Such partition is done by the software-based process-to-SM mapping
scheduling and thus considered logical. Notably, although MPS enables logical
SM partitioning, other GPU resources like memory bandwidth are not partitioned
and thus MPS cannot fully avoid the inter-process resource competition and
interference. To address this, the recently introduced Multi-Instance GPU
(MIG) [24] on Ampere architecture enables physical partitioning of both SMs
and memory bandwidths through dedicated GPU architecture design (Figure 3
(c)). Such physical partition ensures fully isolated resources, and thus no
interference can happen between different processes. MIG support splitting one
A100 GPU into seven fully isolated GPU instances. Meanwhile, it provides
certain reconfiguration capability when the GPU is fully or partial idle. For
example, one A100 could be split into three instances with the ratio of 4:2:1
and then reconfigured to be 3:3:1, etc [24]. More detailed comparison of
Stream, MPS, MIG could be found in Table II.

##  III Multi-Tenant Computing Optimization:
Design and Innovations

Based on the former challenges and opportunities, we review the emerging works
tackling the multi-tenant scheduling optimization from different perspectives.
We summarize these works in Table III. From a top-down view, these works are
categorized into several levels, i.e., from DL service-level orchestration,
graph & runtime-level scheduling, to kernel-level auto-tuing and then GPU
resource-level management.

###  III-A Service-level Orchestration

DL service-level orchestration is an important feature in large-scale data
centers to improve the GPU utilization. As the top-most scheduling level, such
orchestration usually regards one service query as the basic scheduling unit.
This reduces the scheduling complexity as there is no need to consider the
intra-DNN model structure details (operators and graphs). One example is the
Microsoft Deep Learning Inference Service (DLIS) system [36]. The service
orchestrator characterizes different models’ resource requirements and then
strategically places one or multiple queries onto hosts through the service
router. Therefore, it could maximize the served queries per second (QPS) while
ensure little inter-query interference so as to maintain similar tail latency.

However, designing a proper co-location strategy or system is a non-trivial
task. For example, one challenging factor is the serving dynamics, i.e.,
undetermined arrival rates and/or distribution of incoming DL queries,
different RNN/LSTMs queries with varied inputs and control states. Distinct
from static workloads that we can get the full information, such dynamic
scenarios require us to either utilize historical data or predict the future
workload dynamics. PREMA [5] proposed a predictive multi-task DNN scheduling
algorithm that combines off-line records and online token-based job scheduling
to determine the best multi-tenant co-location strategy.

Another challenge in multi-tenant co-location is how to accurately predict the
inter-model resource interference. This is a critical factor in ensuring QoS
such as tail latency. [19] trained a ML-based latency degradation predictor
under co-location using offline-profiled hardware-level features such as SM
and DRAM usage, PCIE read/write BW, buffer usage, etc. Then the latency
degradation predictor is used to evaluate the model placement’s potential
influence for each query.

However, these works have certain scalability issues as they mostly targeted
at static model types, hardware types, etc., which may not be suitable for
dynamic workloads. Meanwhile, as each DNN can have many operators (_e.g._ ,
layers) that have fluctuated resource consumption, such coarse-grained
scheduling (with one entire query as the basic unit) may suffer from resource
under-utilization/contention occasionally and thus still hinders the QoS.

###  III-B Graph and Runtime-level Scheduling

Graph and runtime-level scheduling could help address one of the
aforementioned challenge of coarse-grained granularity by enabling more fine-
grained scheduling, _e.g._ , the DNN operators. This could be done by
leveraging the GPU software-level support such as the multi-stream mechanism
and scheduling APIs. For example, [44] propose an ML-based scheduling strategy
for multi-tenant DNN exeuction acceleration. It first abstracts multiple DNN’s
computation graph with all operators into a global intermediate representation
(IR), which enables flexible resource sharing between different tenants so as
to improve the utilization. To find the optimal concurrent operator execution
strategy in the huge scheduling space, they design a ML-based auto-search
method by defining three main factors: scheduling search space, profiling-
guided latency cost model, and the ML search algorithm. Based on offline
profiling records, the search algorithm could find the best scheduling for
optimal GPU utilization and throughput.

Such graph and runtime-level operator scheduling could usually achieve better
performance due to the fine-grained design, but they also face more
scalability issues, _e.g._ , when the number of co-located workloads increase
to very large. Meanwhile, it also applies to static or known multi-tenant
workload only, which cannot address dynamic model types.

###  III-C Resource-Level Management

Besides the aforementioned works, another optimization perspective to solve
the inter-tenant inference is to conduct fine-grained resource managing [9,
13]. For example, spatial partitioning and allocation of GPU resources to
different DL workloads could isolate different jobs’ resource (_e.g._ , stream
multiprocessors (SMs), memory bandwidths), thus avoiding the job interference
in the hardware resource level. However, as we introduced before, achieving
fine-grained resource partitioning is non-achievable until recently GPU
vendors release a series of resource sharing and partitioning support like
multi-streams, multi-process services (MPS [25]) and multi-instance GPU (MIG
[24]). Most recent resource-level management works are built upon these
technologies.

For example, GSlice [9] uses MPS to conduct adaptive SM partitioning for
different DNN jobs. They design a self-learning method to dynamically adjust
the GPU resource allocation ratio for each workload and thus avoid
interference among co-located DL workloads and maximize the throughput. [4]
utilizes similar spatial partitioning mechanism by MPS while additionally
combines temporal scheduling strategies. MIG-Serving [38] is the most recent
work that adopts the newly-released MIG feature on A100 to achieve spatial
resource management for multi-tenant scheduling.

However, such spatial resource partitioning solutions also have a intrinsic
limitation that is the inflexible re-configuration when the workloads change
and requires resource partitioning adjustment. For GPUs, re-configuring the
resource partitioning requires certain amount of time (_e.g._ , tens of $ms$
or more), which can be even larger than one DL inference workloads’ processing
time. Therefore, re-configuring frequently is not practical and thus limits
such solutions’ performance when facing dynamic workloads. [9] tries to reduce
the stall caused by reconfiguration time of MPS by utilizing a standby/shadow
process. However, the minimum time for switching one partitioning
configuration to another one still cost several seconds, which is still non-
negligible in online serving.

###  III-D Potential Directions for Remaining Challenges

1

ML-based Prediction and Online Learning: To address the problem of service
dynamics, using ML-based predictive model (_e.g._ , reinforcement learning,
LSTM, _etc._) is one promising direction, which can potentially predict the
future queries trend and guide the overall scheduling. The ML-based model can
be initially trained offline by historical serving records. During the online
serving process, active learning and continual learning [30, 28] using the
latency/throughput as feedback can be potentially utilized to improve the
predictive accuracy and the scheduling effectiveness consistently.

Another way of leveraging ML-based prediction is to conduct light-weight
modeling to predict the latency degradation under different multi-model and
hardware combinations so that the scheduler can make better decision regarding
the latency SLA constraints. For example, the work [19] built a ML model to
predict the latency of multi-model inference cases on different machines. As
the effectiveness of the final scheduling solution highly depends on the
modeling accuracy, the scalability and generality issue across hardware/model
types needs to be addressed, which can be also very challenging.

Figure 4: The future trends to a larger scale DL system.

2

Software-Hardware Co-Scheduling: The software and hardware scheduling could be
complementary to provide both high job scheduling flexibility and strict
resource isolation. There are some recent works that adopt such a temporal-
spatial combined perspective. [4] uses MPS to conduct resource partitioning
and then implements a heuristic-based task scheduler to find the appropriate
mapping between the DNN queries and gpu partitions. In addition to that,
software-hardware scheduling could also be leveraged to alleviate certain re-
configuration overhead. For example, it’s potential to conduct software-based
scheduling within a partitioned GPU slice, _e.g._ , combining multi-stream
with MIG. In this way, fine-grained scheduling could be achieved without re-
partitioning the entire GPU, avoiding the reconfiguration overhead.

##  IV Towards Large-Scale DL Computing:
Vision and Insights

###  IV-A Architecture Design with “Full Stack in the Loop”

The fast development of multi-tenant DL computing brings many challenges for
the system stack optimizations. Besides for the GPU only, this also enlightens
the other DL-oriented hardware architecture designers (_e.g._ , TPU, chiplet,
neuromorphic and quantum-based accelerators) to optimize for flexibility and
agility facing a rapidly changing DL application landscape. Specifically, one
important future trend is the “full stack in the loop”, i.e., to remove the
boundaries in the vertical DL system stack and conduct full-stack integration
to strive for both optimal performance and flexibility.

One example in this line of efforts is the DL compiler renovation by tvm unity
[32], as shown in Figure 4 (a). Current DL computing stack conducts separate
layer-wise optimization (graph-runtime-kernel-resource) and single directional
top-down deployment. This however prohibits feedback loops and cross-layer
interactions for SW/HW co-compiling based on model workloads and hardware
characteristics. Unifying the abstraction between layers would thus greatly
facilitate the new full-stack optimization as a loop, not only for multi-
tenant computing, but also for future wider DL application.

Another example is the increasing attention in the versatile and flexible
chiplet-based SW/HW co-design [33, 48] that uses multi-chip-modules (MCMs).
Compared to traditional large monolithic die, such MCM combines smaller
chiplets into a larger system and substantially reduces fabrication and design
costs. However, it requires thorough application awareness to optimize the
chip design and overall performance. As so, such chiplet modules could also
greatly benefit from the full-stack-in-the-loop architecture of DL computing.

###  IV-B The Future Large-Scale DL System Landscape

Multi-tenant DL computing is a natural generalization result due to the
significant computing scaling trend of GPU. However, recently another model
scaling trend is observed, that is, designing and training super-scale AI
models for general intelligence. For example, the recent SOTA giant AI model
Megatron-NLG [35] has reached 530 billions of parameters and requires tens of
GPUs to conduct multi-node distributed inference. If we take such model
scaling into consideration, even more new DL computing modes could be observed
and enrich the future DL & system landscape.

We describe the future large-scale DL system landscape by using a taxonomy in
Figure 4 (b). Using Instance (I) to denote one DNN model and Device (D) to
denote the hardware, traditional DL system mostly comes within the Single
Instance Single Device (SISD) domain and only constitute the top-left quarter
in the full spectrum. Multi-tenant computing emerges as the Multiple Instances
Single Device (MISD) with the computing scaling trend, as we summarized in
this servery.

Whereas diagonally, with the model scaling trend, the Single Instance Multiple
Devices (SIMD) interaction mode also emerges and is attracting more attention
such as distributed inference for super-scale giant models [18, 47, 50]
including language, recommendation models, etc. Finally, Multiple Instances
Multiple Devices (MIMD) computing would eventually combine all these modes and
become a practical needs for future DL-centric data center optimization.

##  V Conclusion

Deep Learning-based intelligence creates a wide-spectrum of applications
featured with substantial complexity such as multi-modality and multi-tasking.
GPU is one major type of DL accelerators that democratized such complex DL
applications in both cloud and edge. Meanwhile, its gen-by-gen capacity
demonstrates exponential scaling. With such application complexity and GPU
capacity scaling, multi-tenant DL computing is emerging as an effective
computing paradigm on GPU to enhance the throughput, resource utilization, and
power efficiency. This survey summarizes and categorizes the emerging
challenges and optimization opportunities for multi-tenant DL inference on GPU
following a hierarchical comparison with traditional single-tenant
optimization. We hope that this survey could shed lights on new perspectives
and novel works in future large-scale DL system optimization.

## References

  * [1]
  * Bai et al. [2021] Yang Bai, Xufeng Yao, Qi Sun, and Bei Yu. 2021\.  AutoGTCO: Graph and Tensor Co-Optimize for Image Recognition with Transformers on GPU. In _2021 IEEE/ACM International Conference On Computer Aided Design (ICCAD)_.
  * Chen et al. [2018] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. 2018\.  $\\{$TVM$\\}$: An automated end-to-end optimizing compiler for deep learning. In _13th $\\{$USENIX$\\}$ Symposium on Operating Systems Design and Implementation ($\\{$OSDI$\\}$ 18)_. 578–594.
  * Choi et al. [2021] Seungbeom Choi, Sunho Lee, Yeonjae Kim, Jongse Park, Youngjin Kwon, and Jaehyuk Huh. 2021\.  Multi-model Machine Learning Inference Serving with GPU Spatial Partitioning.  _arXiv preprint arXiv:2109.01611_ (2021).
  * Choi and Rhu [2020] Yujeong Choi and Minsoo Rhu. 2020.  Prema: A predictive multi-task scheduling algorithm for preemptible neural processing units. In _2020 IEEE International Symposium on High Performance Computer Architecture (HPCA)_.
  * Choquette et al. [2021] Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. 2021.  Nvidia a100 tensor core gpu: Performance and innovation.  _IEEE Micro_ 41, 2 (2021), 29–35.
  * Coorporation [2017] NVIDIA Coorporation. 2017\.  _NVIDIA Tesla V100 GPU Architecture_.  Technical Report.  http://www.nvidia.com/object/volta-architecture.
  * Dhakal et al. [2020a] Aditya Dhakal, Junguk Cho, Sameer G Kulkarni, KK Ramakrishnan, and Puneet Sharma. 2020a.  Spatial Sharing of GPU for Autotuning DNN models.  _arXiv preprint arXiv:2008.03602_ (2020).
  * Dhakal et al. [2020b] Aditya Dhakal, Sameer G Kulkarni, and KK Ramakrishnan. 2020b.  Gslice: controlled spatial sharing of gpus for a scalable inference platform. In _Proceedings of the 11th ACM Symposium on Cloud Computing_. 492–506.
  * Ding et al. [2020] Yaoyao Ding, Ligeng Zhu, Zhihao Jia, Gennady Pekhimenko, and Song Han. 2020.  IOS: Inter-Operator Scheduler for CNN Acceleration.  _arXiv preprint arXiv:2011.01302_ (2020).
  * Dumoulin and Visin [2016] Vincent Dumoulin and Francesco Visin. 2016.  A guide to convolution arithmetic for deep learning.  _arXiv preprint arXiv:1603.07285_ (2016).
  * Fernandez [2022] Peter Fernandez. 2022\.  Facebook, Meta, the metaverse and libraries.  _Library Hi Tech News_ (2022).
  * Ghodrati et al. [2020] Soroush Ghodrati, Byung Hoon Ahn, Joon Kyung Kim, Sean Kinzer, Brahmendra Reddy Yatham, Navateja Alla, Hardik Sharma, Mohammad Alian, Eiman Ebrahimi, Nam Sung Kim, et al. 2020\.  Planaria: Dynamic architecture fission for spatial multi-tenant acceleration of deep neural networks. In _2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_. IEEE, 681–697.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Jia et al. [2019] Zhihao Jia, Oded Padon, James Thomas, Todd Warszawski, Matei Zaharia, and Alex Aiken. 2019\.  TASO: optimizing deep learning computation with automatic generation of graph substitutions. In _Proceedings of the 27th ACM Symposium on Operating Systems Principles_. 47–62.
  * Lavin and Gray [2016] Andrew Lavin and Scott Gray. 2016.  Fast algorithms for convolutional neural networks. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4013–4021.
  * Li et al. [2020] Ao Li, Bojian Zheng, Gennady Pekhimenko, and Fan Long. 2020\.  Automatic horizontal fusion for GPU kernels.  _arXiv preprint arXiv:2007.01277_ (2020).
  * Lui et al. [2021] Michael Lui, Yavuz Yetim, Özgür Özkan, Zhuoran Zhao, Shin-Yeh Tsai, Carole-Jean Wu, and Mark Hempstead. 2021.  Understanding capacity-driven scale-out neural recommendation inference. In _2021 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)_. IEEE, 162–171.
  * Mendoza et al. [2021] Daniel Mendoza, Francisco Romero, Qian Li, Neeraja J Yadwadkar, and Christos Kozyrakis. 2021.  Interference-Aware Scheduling for Inference Serving. In _Proceedings of the 1st Workshop on Machine Learning and Systems_. 80–88.
  * Niu et al. [2021] Wei Niu, Jiexiong Guan, Yanzhi Wang, Gagan Agrawal, and Bin Ren. 2021.  DNNFusion: accelerating deep neural networks execution with advanced operator fusion. In _Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation_. 883–898.
  * NVIDIA [2013] NVIDIA. 2013.  Hyper-Q.  https://developer.download.nvidia.com/compute/DevZone/C/html_x64/6_Advanced/simpleHyperQ/doc/HyperQ.pdf.
  * NVIDIA [2015] NVIDIA. 2015.  CUDA Multi-Streams.  https://developer.nvidia.com/blog/gpu-pro-tip-cuda-7-streams-simplify-concurrency/.
  * NVIDIA [2020a] NVIDIA. 2020a.  CUDA Programming Guide.  https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html.
  * NVIDIA [2020b] NVIDIA. 2020b.  NVIDIA Multi Instance GPU (MIG).  https://docs.nvidia.com/datacenter/tesla/mig-user-guide/.
  * NVIDIA [2020c] NVIDIA. 2020c.  NVIDIA Multi Process Service (MPS).  https://docs.nvidia.com/deploy/pdf/CUDA-Multi-Process-Service-Overview.pdf.
  * NVIDIA [2020d] NVIDIA. 2020d.  NVIDIA Virtual Compute Server.  https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/solutions/resources/documents1/Technical-Brief-Multi-Instance-GPU-NVIDIA-Virtual-Compute-Server.pdf.
  * NVIDIA [2021] NVIDIA. 2021.  CUDA Graphs.  https://developer.nvidia.com/blog/cuda-graphs/.
  * Parisi et al. [2019] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. 2019.  Continual lifelong learning with neural networks: A review.  _Neural Networks_ 113 (2019), 54–71.
  * Qin et al. [2018] Zhuwei Qin, Fuxun Yu, Chenchen Liu, and Xiang Chen. 2018\.  Functionality-oriented convolutional filter pruning.  _arXiv preprint arXiv:1810.07322_ (2018).
  * Ren et al. [2021] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. 2021\.  A survey of deep active learning.  _ACM Computing Surveys (CSUR)_ 54, 9 (2021), 1–40.
  * Reports [2021] Market Reports. 2021\.  Global Data Center Accelerator Market Size, Status and Forecast 2020-2025.  https://www.mynewsdesk.com/brandessence/pressreleases/data-center-accelerator-market-size-2021-cagr-38-dot-7-percent-3112488.
  * Sampson et al. [2022] Adrian Sampson, Tianqi Chen, and Jared Roesch. 2022\.  Apache TVM Unity: a vision for the ML software and hardware ecosystem.  https://tvm.apache.org/2021/12/15/tvm-unity.
  * Shao et al. [2019] Yakun Sophia Shao, Jason Clemons, Rangharajan Venkatesan, Brian Zimmer, Matthew Fojtik, Nan Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, et al. 2019\.  Simba: Scaling deep-learning inference with multi-chip-module-based architecture. In _Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture_. 14–27.
  * Shen et al. [2021] Haichen Shen, Jared Roesch, Zhi Chen, Wei Chen, Yong Wu, Mu Li, Vin Sharma, Zachary Tatlock, and Yida Wang. 2021.  Nimble: Efficiently compiling dynamic neural networks for model inference.  _Proceedings of MLSys_ (2021).
  * Smith et al. [2022] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022\.  Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model.  _arXiv preprint arXiv:2201.11990_ (2022).
  * Soifer et al. [2019] Jonathan Soifer, Jason Li, Mingqin Li, Jeffrey Zhu, Yingnan Li, Yuxiong He, Elton Zheng, Adi Oltean, Maya Mosyak, Chris Barnes, et al. 2019\.  Deep learning inference service at microsoft. In _2019 $\\{$USENIX$\\}$ Conference on Operational Machine Learning (OpML 19)_. 15–17.
  * Sun et al. [2019] Yifan Sun, Nicolas Bohm Agostini, Shi Dong, and David Kaeli. 2019\.  Summarizing CPU and GPU design trends with product data.  _arXiv preprint arXiv:1911.11313_ (2019).
  * Tan et al. [2021] Cheng Tan, Zhichao Li, and et al. 2021.  Serving DNN Models with Multi-Instance GPUs: A Case of the Reconfigurable Machine Scheduling Problem.  _arXiv:2109.11067_ (2021).
  * TensorFlow [2020] TensorFlow. 2020\.  TensorFlow XLA (Accelerated Linear Algebra).  https://www.tensorflow.org/xla.
  * Vanholder [2016] Han Vanholder. 2016\.  Efficient inference with tensorrt. In _GPU Technology Conference_ , Vol. 1. 2.
  * Wesolowski et al. [2021] Lukasz Wesolowski, Bilge Acun, Valentin Andrei, Adnan Aziz, Gisle Dankel, Christopher Gregg, Xiaoqiao Meng, Cyril Meurillon, Denis Sheahan, Lei Tian, et al. 2021\.  Datacenter-Scale Analysis and Optimization of GPU Machine Learning Workloads.  _IEEE Micro_ 41, 5 (2021), 101–112.
  * Wu et al. [2020] Xiaorui Wu, Hong Xu, and Yi Wang. 2020.  Irina: Accelerating DNN Inference with Efficient Online Scheduling. In _4th Asia-Pacific Workshop on Networking_. 36–43.
  * Yang et al. [2021] Yichen Yang, Phitchaya Phothilimthana, Yisu Wang, Max Willsey, Sudip Roy, and Jacques Pienaar. 2021\.  Equality saturation for tensor graph superoptimization.  _Proceedings of Machine Learning and Systems_ 3 (2021), 255–268.
  * Yu and et al. [2021] Fuxun Yu and et al. 2021\.  Automated Runtime-Aware Scheduling for Multi-Tenant DNN Inference on GPU. In _Proceedings of the 40th IEEE International Conference on Computer Aided Design (ICCAD)_.
  * Yu et al. [2020a] Fuxun Yu, Chenchen Liu, Di Wang, Yanzhi Wang, and Xiang Chen. 2020a.  AntiDote: Attention-based Dynamic Optimization for Neural Network Runtime Efficiency. In _2020 Design, Automation & Test in Europe Conference & Exhibition (DATE)_.
  * Yu et al. [2020b] Fuxun Yu, Zhuwei Qin, Di Wang, Ping Xu, Chenchen Liu, Zhi Tian, and Xiang Chen. 2020b.  Dc-cnn: computational flow redefinition for efficient cnn through structural decoupling. In _2020 Design, Automation & Test in Europe Conference & Exhibition (DATE)_. IEEE, 1097–1102.
  * Zhao et al. [2018] Zhuoran Zhao, Kamyar Mirzazad Barijough, and Andreas Gerstlauer. 2018.  Deepthings: Distributed adaptive deep learning inference on resource-constrained iot edge clusters.  _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ 37, 11 (2018), 2348–2359.
  * Zheng et al. [2020b] Hao Zheng, Ke Wang, and Ahmed Louri. 2020b.  A versatile and flexible chiplet-based system design for heterogeneous manycore architectures. In _2020 57th ACM/IEEE Design Automation Conference (DAC)_. IEEE, 1–6.
  * Zheng et al. [2020a] Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen, et al. 2020a.  Ansor: Generating high-performance tensor programs for deep learning. In _14th $\\{$USENIX$\\}$ Symposium on Operating Systems Design and Implementation ($\\{$OSDI$\\}$ 20)_. 863–879.
  * Zhou et al. [2019] Li Zhou, Hao Wen, Radu Teodorescu, and David HC Du. 2019\.  Distributing deep neural networks with containerized partitions at the edge. In _2nd $\\{$USENIX$\\}$ Workshop on Hot Topics in Edge Computing (HotEdge 19)_.


LM Source |  | LM Max
---
SHAP Value
(Opposite)
abide | 1.0 | 1.0 | 0.0 | 0.003838 | XLex | NaN | NaN | NaN | NaN | NaN
abo | 1.0 | 1.0 | 0.0 | 0.000976 | XLex | NaN | NaN | NaN | NaN | NaN
aboard | 2.0 | 2.0 | 0.0 | 0.122640 | XLex | NaN | NaN | NaN | NaN | NaN
abolition | 1.0 | 1.0 | 0.0 | 0.006430 | XLex | NaN | NaN | NaN | NaN | NaN
abroad | 3.0 | 3.0 | 0.0 | 0.039073 | XLex | NaN | NaN | NaN | NaN | NaN
writeoff | NaN | NaN | NaN | NaN | NaN | 1 | negative | 0 | LM | 0
writeoffs | NaN | NaN | NaN | NaN | NaN | 1 | negative | 0 | LM | 0
wrongful | NaN | NaN | NaN | NaN | NaN | 1 | negative | 0 | LM | 0
wrongfully | NaN | NaN | NaN | NaN | NaN | 1 | negative | 0 | LM | 0
wrongly | NaN | NaN | NaN | NaN | NaN | 1 | negative | 0 | LM | 0

##  Appendix B

TABLE B.1: F1 score obtained across different experiments using the XLex methodology based on the RoBERTa transformer model. The highest values are represented in bold. Source of the lexicon | Normalized | Evaluation set | F1 on whole dataset |  | F1 only on the part of the dataset for
---
which the LM-based model has an answer
|  |  | LM | XLex |  | XLex+
---
LM
| LM on
---
LM
| XLex on
---
LM
| (XLex+LM)
---
on LM
nasdaq | Yes | financial_phrase_bank | 0.287 | 0.451 | 0.714 | 0.688 | 0.448 | 0.731
nasdaq | Yes | fiqa_labeled_df | 0.297 | 0.457 | 0.473 | 0.789 | 0.739 | 0.802
nasdaq | Yes | fpb_fiqa | 0.295 | 0.434 | 0.466 | 0.726 | 0.437 | 0.739
nasdaq | Yes | sem_eval | 0.305 | 0.664 | 0.721 | 0.752 | 0.704 | 0.775
nasdaq | No | financial_phrase_bank | 0.287 | 0.457 | 0.72 | 0.688 | 0.446 | 0.726
nasdaq | No | fiqa_labeled_df | 0.297 | 0.437 | 0.454 | 0.789 | 0.739 | 0.802
nasdaq | No | fpb_fiqa | 0.295 | 0.426 | 0.459 | 0.726 | 0.433 | 0.738
nasdaq | No | sem_eval | 0.305 | 0.672 | 0.713 | 0.752 | 0.717 | 0.759
fiqa_fpb_sentfin_neutral | Yes | financial_phrase_bank | 0.287 | 0.451 | 0.689 | 0.688 | 0.474 | 0.731
fiqa_fpb_sentfin_neutral | Yes | fiqa_labeled_df | 0.297 | 0.434 | 0.456 | 0.789 | 0.752 | 0.827
fiqa_fpb_sentfin_neutral | Yes | fpb_fiqa | 0.295 | 0.424 | 0.448 | 0.726 | 0.458 | 0.751
fiqa_fpb_sentfin_neutral | Yes | sem_eval | 0.305 | 0.434 | 0.476 | 0.752 | 0.676 | 0.775
fiqa_fpb_sentfin_neutral | No | financial_phrase_bank | 0.287 | 0.448 | 0.694 | 0.688 | 0.463 | 0.731
fiqa_fpb_sentfin_neutral | No | fiqa_labeled_df | 0.297 | 0.416 | 0.439 | 0.789 | 0.747 | 0.827
fiqa_fpb_sentfin_neutral | No | fpb_fiqa | 0.295 | 0.425 | 0.448 | 0.726 | 0.457 | 0.748
fiqa_fpb_sentfin_neutral | No | sem_eval | 0.305 | 0.434 | 0.48 | 0.752 | 0.66 | 0.775
financial_phrase_bank | Yes | fiqa_labeled_df | 0.297 | 0.409 | 0.447 | 0.789 | 0.688 | 0.84
financial_phrase_bank | Yes | fpb_fiqa | 0.295 | 0.4 | 0.43 | 0.726 | 0.428 | 0.739
financial_phrase_bank | Yes | sem_eval | 0.305 | 0.416 | 0.445 | 0.752 | 0.462 | 0.767
financial_phrase_bank | No | fiqa_labeled_df | 0.297 | 0.43 | 0.456 | 0.789 | 0.692 | 0.802
financial_phrase_bank | No | fpb_fiqa | 0.295 | 0.413 | 0.439 | 0.726 | 0.436 | 0.738
financial_phrase_bank | No | sem_eval | 0.305 | 0.416 | 0.444 | 0.752 | 0.467 | 0.767
TABLE B.2: MCC obtained across different experiments using the XLex methodology based on the RoBERTa transformer model. The highest values are represented in bold. Source of the lexicon | Normalized | Evaluation set | MCC on whole dataset |  | MCC only on the part of the dataset for
---
which the LM-based model has an answer
|  |  | LM | XLex |  | XLex+
---
LM
| LM on
---
LM
| XLex on
---
LM
| (XLex+LM)
---
on LM
nasdaq | Yes | financial_phrase_bank | 0.192 | 0.362 | 0.453 | 0.489 | 0.364 | 0.539
nasdaq | Yes | fiqa_labeled_df | 0.216 | 0.372 | 0.432 | 0.594 | 0.492 | 0.615
nasdaq | Yes | fpb_fiqa | 0.209 | 0.306 | 0.418 | 0.519 | 0.312 | 0.524
nasdaq | Yes | sem_eval | 0.265 | 0.332 | 0.477 | 0.567 | 0.417 | 0.601
nasdaq | No | financial_phrase_bank | 0.192 | 0.383 | 0.471 | 0.489 | 0.359 | 0.531
nasdaq | No | fiqa_labeled_df | 0.216 | 0.313 | 0.374 | 0.594 | 0.492 | 0.615
nasdaq | No | fpb_fiqa | 0.209 | 0.281 | 0.399 | 0.519 | 0.301 | 0.525
nasdaq | No | sem_eval | 0.265 | 0.349 | 0.465 | 0.567 | 0.437 | 0.578
fiqa_fpb_sentfin_neutral | Yes | financial_phrase_bank | 0.192 | 0.392 | 0.434 | 0.489 | 0.46 | 0.539
fiqa_fpb_sentfin_neutral | Yes | fiqa_labeled_df | 0.216 | 0.303 | 0.371 | 0.594 | 0.523 | 0.659
fiqa_fpb_sentfin_neutral | Yes | fpb_fiqa | 0.209 | 0.286 | 0.376 | 0.519 | 0.375 | 0.54
fiqa_fpb_sentfin_neutral | Yes | sem_eval | 0.265 | 0.308 | 0.466 | 0.567 | 0.355 | 0.593
fiqa_fpb_sentfin_neutral | No | financial_phrase_bank | 0.192 | 0.376 | 0.443 | 0.489 | 0.412 | 0.539
fiqa_fpb_sentfin_neutral | No | fiqa_labeled_df | 0.216 | 0.249 | 0.32 | 0.594 | 0.499 | 0.659
fiqa_fpb_sentfin_neutral | No | fpb_fiqa | 0.209 | 0.288 | 0.373 | 0.519 | 0.373 | 0.531
fiqa_fpb_sentfin_neutral | No | sem_eval | 0.265 | 0.31 | 0.481 | 0.567 | 0.322 | 0.593
financial_phrase_bank | Yes | fiqa_labeled_df | 0.216 | 0.25 | 0.363 | 0.594 | 0.375 | 0.683
financial_phrase_bank | Yes | fpb_fiqa | 0.209 | 0.245 | 0.351 | 0.519 | 0.303 | 0.528
financial_phrase_bank | Yes | sem_eval | 0.265 | 0.296 | 0.422 | 0.567 | 0.386 | 0.581
financial_phrase_bank | No | fiqa_labeled_df | 0.216 | 0.306 | 0.398 | 0.594 | 0.385 | 0.615
financial_phrase_bank | No | fpb_fiqa | 0.209 | 0.284 | 0.383 | 0.519 | 0.325 | 0.529
financial_phrase_bank | No | sem_eval | 0.265 | 0.296 | 0.418 | 0.567 | 0.4 | 0.581
TABLE B.3: F1 score obtained across different experiments using the XLex methodology based on the FinBERT transformer model. The highest values are represented in bold. Source of the lexicon | Normalized | Evaluation set | F1 on whole dataset |  | F1 only on the part of the dataset for
---
which the LM-based model has an answer
|  |  | LM | XLex |  | XLex+
---
LM
| LM on
---
LM
| XLex on
---
LM
| (XLex+LM)
---
on LM
nasdaq | Yes | financial_phrase_bank | 0.287 | 0.421 | 0.432 | 0.688 | 0.441 | 0.694
nasdaq | Yes | fiqa_labeled_df | 0.297 | 0.497 | 0.505 | 0.789 | 0.802 | 0.836
nasdaq | Yes | fpb_fiqa | 0.295 | 0.448 | 0.458 | 0.726 | 0.477 | 0.74
nasdaq | Yes | sem_eval | 0.305 | 0.684 | 0.693 | 0.752 | 0.774 | 0.783
nasdaq | No | financial_phrase_bank | 0.287 | 0.389 | 0.408 | 0.688 | 0.419 | 0.696
nasdaq | No | fiqa_labeled_df | 0.297 | 0.498 | 0.5 | 0.789 | 0.815 | 0.823
nasdaq | No | fpb_fiqa | 0.295 | 0.428 | 0.442 | 0.726 | 0.464 | 0.74
nasdaq | No | sem_eval | 0.305 | 0.676 | 0.692 | 0.752 | 0.766 | 0.791
fiqa_fpb_sentfin_neutral | Yes | financial_phrase_bank | 0.287 | 0.708 | 0.724 | 0.688 | 0.673 | 0.702
fiqa_fpb_sentfin_neutral | Yes | fiqa_labeled_df | 0.297 | 0.522 | 0.537 | 0.789 | 0.765 | 0.823
fiqa_fpb_sentfin_neutral | Yes | fpb_fiqa | 0.295 | 0.462 | 0.484 | 0.726 | 0.458 | 0.75
fiqa_fpb_sentfin_neutral | Yes | sem_eval | 0.305 | 0.69 | 0.741 | 0.752 | 0.715 | 0.798
fiqa_fpb_sentfin_neutral | No | financial_phrase_bank | 0.287 | 0.696 | 0.711 | 0.688 | 0.673 | 0.702
fiqa_fpb_sentfin_neutral | No | fiqa_labeled_df | 0.297 | 0.518 | 0.53 | 0.789 | 0.765 | 0.811
fiqa_fpb_sentfin_neutral | No | fpb_fiqa | 0.295 | 0.461 | 0.483 | 0.726 | 0.459 | 0.751
fiqa_fpb_sentfin_neutral | No | sem_eval | 0.305 | 0.696 | 0.742 | 0.752 | 0.724 | 0.806
financial_phrase_bank | Yes | fiqa_labeled_df | 0.297 | 0.486 | 0.511 | 0.789 | 0.471 | 0.814
financial_phrase_bank | Yes | fpb_fiqa | 0.295 | 0.461 | 0.486 | 0.726 | 0.453 | 0.739
financial_phrase_bank | Yes | sem_eval | 0.305 | 0.432 | 0.488 | 0.752 | 0.452 | 0.814
financial_phrase_bank | No | fiqa_labeled_df | 0.297 | 0.476 | 0.496 | 0.789 | 0.479 | 0.802
financial_phrase_bank | No | fpb_fiqa | 0.295 | 0.448 | 0.477 | 0.726 | 0.446 | 0.74
financial_phrase_bank | No | sem_eval | 0.305 | 0.429 | 0.488 | 0.752 | 0.452 | 0.806
TABLE B.4: MCC obtained across different experiments using the XLex methodology based on the FinBERT transformer model. The highest values are represented in bold. Source of the lexicon | Normalized | Evaluation set | MCC on whole dataset |  | MCC only on the part of the dataset for
---
which the LM-based model has an answer
|  |  | LM | XLex |  | XLex+
---
LM
| LM on
---
LM
| XLex on
---
LM
| (XLex+LM)
---
on LM
nasdaq | Yes | financial_phrase_bank | 0.192 | 0.315 | 0.352 | 0.489 | 0.414 | 0.489
nasdaq | Yes | fiqa_labeled_df | 0.216 | 0.497 | 0.525 | 0.594 | 0.604 | 0.673
nasdaq | Yes | fpb_fiqa | 0.209 | 0.375 | 0.415 | 0.519 | 0.451 | 0.526
nasdaq | Yes | sem_eval | 0.265 | 0.417 | 0.452 | 0.567 | 0.565 | 0.613
nasdaq | No | financial_phrase_bank | 0.192 | 0.279 | 0.326 | 0.489 | 0.378 | 0.493
nasdaq | No | fiqa_labeled_df | 0.216 | 0.503 | 0.516 | 0.594 | 0.632 | 0.648
nasdaq | No | fpb_fiqa | 0.209 | 0.334 | 0.381 | 0.519 | 0.425 | 0.526
nasdaq | No | sem_eval | 0.265 | 0.415 | 0.465 | 0.567 | 0.546 | 0.624
fiqa_fpb_sentfin_neutral | Yes | financial_phrase_bank | 0.192 | 0.454 | 0.494 | 0.489 | 0.418 | 0.5
fiqa_fpb_sentfin_neutral | Yes | fiqa_labeled_df | 0.216 | 0.565 | 0.608 | 0.594 | 0.53 | 0.648
fiqa_fpb_sentfin_neutral | Yes | fpb_fiqa | 0.209 | 0.394 | 0.47 | 0.519 | 0.386 | 0.54
fiqa_fpb_sentfin_neutral | Yes | sem_eval | 0.265 | 0.39 | 0.511 | 0.567 | 0.43 | 0.636
fiqa_fpb_sentfin_neutral | No | financial_phrase_bank | 0.192 | 0.441 | 0.478 | 0.489 | 0.424 | 0.5
fiqa_fpb_sentfin_neutral | No | fiqa_labeled_df | 0.216 | 0.557 | 0.592 | 0.594 | 0.53 | 0.624
fiqa_fpb_sentfin_neutral | No | fpb_fiqa | 0.209 | 0.398 | 0.472 | 0.519 | 0.392 | 0.543
fiqa_fpb_sentfin_neutral | No | sem_eval | 0.265 | 0.407 | 0.517 | 0.567 | 0.451 | 0.648
financial_phrase_bank | Yes | fiqa_labeled_df | 0.216 | 0.456 | 0.531 | 0.594 | 0.465 | 0.637
financial_phrase_bank | Yes | fpb_fiqa | 0.209 | 0.382 | 0.468 | 0.519 | 0.361 | 0.524
financial_phrase_bank | Yes | sem_eval | 0.265 | 0.297 | 0.483 | 0.567 | 0.371 | 0.653
financial_phrase_bank | No | fiqa_labeled_df | 0.216 | 0.429 | 0.489 | 0.594 | 0.5 | 0.615
financial_phrase_bank | No | fpb_fiqa | 0.209 | 0.346 | 0.443 | 0.519 | 0.341 | 0.523
financial_phrase_bank | No | sem_eval | 0.265 | 0.288 | 0.483 | 0.567 | 0.388 | 0.648

## References

  * [1] M. Hasan, J. Popp, J. Oláh _et al._ , “Current landscape and influence of big data on finance,” _Journal of Big Data_ , vol. 7, no. 1, pp. 1–17, 2020.
  * [2] I. Goldstein, C. S. Spatt, and M. Ye, “Big data in finance,” _The Review of Financial Studies_ , vol. 34, no. 7, pp. 3213–3225, 2021.
  * [3] N. Mohamed and J. Al-Jaroodi, “Real-time big data analytics: Applications and challenges,” in _2014 international conference on high performance computing & simulation (HPCS)_. IEEE, 2014, pp. 305–310.
  * [4] V. Ravi and S. Kamaruddin, “Big data analytics enabled smart financial services: opportunities and challenges,” in _Big Data Analytics: 5th International Conference, BDA 2017, Hyderabad, India, December 12-15, 2017, Proceedings 5_. Springer, 2017, pp. 15–39.
  * [5] M. Cao, R. Chychyla, and T. Stewart, “Big data analytics in financial statement audits,” _Accounting Horizons_ , vol. 29, no. 2, pp. 423–429, 2015.
  * [6] J. Smailović, M. Grčar, N. Lavrač, and M. Žnidaršič, “Predictive sentiment analysis of tweets: A stock market application,” in _Human-Computer Interaction and Knowledge Discovery in Complex, Unstructured, Big Data: Third International Workshop, HCI-KDD 2013, Held at SouthCHI 2013, Maribor, Slovenia, July 1-3, 2013. Proceedings_. Springer, 2013, pp. 77–88.
  * [7] A. Derakhshan and H. Beigy, “Sentiment analysis on stock social media for stock price movement prediction,” _Engineering Applications of Artificial Intelligence_ , vol. 85, pp. 569–578, 2019.
  * [8] R. Ren, D. D. Wu, and T. Liu, “Forecasting stock market movement direction using sentiment analysis and support vector machine,” _IEEE Systems Journal_ , vol. 13, no. 1, pp. 760–770, 2018.
  * [9] R. Yang, L. Yu, Y. Zhao, H. Yu, G. Xu, Y. Wu, and Z. Liu, “Big data analytics for financial market volatility forecast based on support vector machine,” _International Journal of Information Management_ , vol. 50, pp. 452–462, 2020.
  * [10] F.-T. Tsai, H.-M. Lu, and M.-W. Hung, “The effects of news sentiment and coverage on credit rating analysis,” 2010.
  * [11] S. Gül, Ö. Kabak, and I. Topcu, “A multiple criteria credit rating approach utilizing social media data,” _Data & Knowledge Engineering_, vol. 116, pp. 80–99, 2018.
  * [12] H.-M. Lu, F.-T. Tsai, H. Chen, M.-W. Hung, and S.-H. Li, “Credit rating change modeling using news and financial ratios,” _ACM Transactions on Management Information Systems (TMIS)_ , vol. 3, no. 3, pp. 1–30, 2012.
  * [13] D. Zhang, W. Xu, Y. Zhu, and X. Zhang, “Can sentiment analysis help mimic decision-making process of loan granting? a novel credit risk evaluation approach using gmkl model,” in _2015 48th Hawaii International Conference on System Sciences_. IEEE, 2015, pp. 949–958.
  * [14] B. Yoon, Y. Jeong, and S. Kim, “Detecting a risk signal in stock investment through opinion mining and graph-based semi-supervised learning,” _IEEE Access_ , vol. 8, pp. 161 943–161 957, 2020.
  * [15] J. R. McColl-Kennedy, M. Zaki, K. N. Lemon, F. Urmetzer, and A. Neely, “Gaining customer experience insights that matter,” _Journal of service research_ , vol. 22, no. 1, pp. 8–26, 2019.
  * [16] L. Ziora, “The sentiment analysis as a tool of business analytics in contemporary organizations,” _Studia Ekonomiczne_ , vol. 281, pp. 234–241, 2016.
  * [17] H. Mili, I. Benzarti, M.-J. Meurs, A. Obaid, J. Gonzalez-Huerta, N. Haj-Salem, and A. Boubaker, “Context aware customer experience management: A development framework based on ontologies and computational intelligence,” _Sentiment Analysis and Ontology Engineering: An Environment of Computational Intelligence_ , pp. 273–311, 2016.
  * [18] X. Tian, J. S. He, and M. Han, “Data-driven approaches in fintech: a survey,” _Information Discovery and Delivery_ , 2021.
  * [19] C.-C. Chen, H.-H. Huang, and H.-H. Chen, “Fintech applications,” in _From Opinion Mining to Financial Argument Mining_. Springer, 2021, pp. 73–87.
  * [20] H. Liu, I. Chatterjee, M. Zhou, X. S. Lu, and A. Abusorrah, “Aspect-based sentiment analysis: A survey of deep learning methods,” _IEEE Transactions on Computational Social Systems_ , vol. 7, no. 6, pp. 1358–1375, 2020.
  * [21] F. Benedetto and A. Tedeschi, “Big data sentiment analysis for brand monitoring in social media streams by cloud computing,” _Sentiment Analysis and Ontology Engineering: An Environment of Computational Intelligence_ , pp. 341–377, 2016.
  * [22] D. Alessia, F. Ferri, P. Grifoni, and T. Guzzo, “Approaches, tools and applications for sentiment analysis implementation,” _International Journal of Computer Applications_ , vol. 125, no. 3, 2015.
  * [23] M. Taboada, J. Brooke, M. Tofiloski, K. Voll, and M. Stede, “Lexicon-based methods for sentiment analysis,” _Computational linguistics_ , vol. 37, no. 2, pp. 267–307, 2011.
  * [24] S. Taj, B. B. Shaikh, and A. F. Meghji, “Sentiment analysis of news articles: a lexicon based approach,” in _2019 2nd international conference on computing, mathematics and engineering technologies (iCoMET)_. IEEE, 2019, pp. 1–5.
  * [25] M. Wankhade, A. C. S. Rao, and C. Kulkarni, “A survey on sentiment analysis methods, applications, and challenges,” _Artificial Intelligence Review_ , vol. 55, no. 7, pp. 5731–5780, 2022.
  * [26] T. Loughran and B. McDonald, “When is a liability not a liability? textual analysis, dictionaries, and 10-ks,” _The Journal of finance_ , vol. 66, no. 1, pp. 35–65, 2011.
  * [27] E. Boiy and M.-F. Moens, “A machine learning approach to sentiment analysis in multilingual web texts,” _Information retrieval_ , vol. 12, pp. 526–558, 2009.
  * [28] A. Sharma and S. Dey, “A comparative study of feature selection and machine learning techniques for sentiment analysis,” in _Proceedings of the 2012 ACM research in applied computation symposium_ , 2012, pp. 1–7.
  * [29] S. Malviya, A. K. Tiwari, R. Srivastava, and V. Tiwari, “Machine learning techniques for sentiment analysis: A review,” _SAMRIDDHI: A Journal of Physical Sciences, Engineering and Technology_ , vol. 12, no. 02, pp. 72–78, 2020.
  * [30] M. Neethu and R. Rajasree, “Sentiment analysis in twitter using machine learning techniques,” in _2013 fourth international conference on computing, communications and networking technologies (ICCCNT)_. IEEE, 2013, pp. 1–5.
  * [31] L. Zhang, S. Wang, and B. Liu, “Deep learning for sentiment analysis: A survey,” _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , vol. 8, no. 4, p. e1253, 2018.
  * [32] A. Yadav and D. K. Vishwakarma, “Sentiment analysis using deep learning architectures: a review,” _Artificial Intelligence Review_ , vol. 53, no. 6, pp. 4335–4385, 2020.
  * [33] N. C. Dang, M. N. Moreno-García, and F. De la Prieta, “Sentiment analysis based on deep learning: A comparative study,” _Electronics_ , vol. 9, no. 3, p. 483, 2020.
  * [34] D. Tang, B. Qin, and T. Liu, “Deep learning for sentiment analysis: successful approaches and future challenges,” _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , vol. 5, no. 6, pp. 292–303, 2015.
  * [35] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [36] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [37] K. Mishev, A. Gjorgjevikj, I. Vodenska, L. T. Chitkushev, and D. Trajanov, “Evaluation of sentiment analysis in finance: from lexicons to transformers,” _IEEE Access_ , vol. 8, pp. 131 662–131 682, 2020.
  * [38] X. S. Huang, F. Perez, J. Ba, and M. Volkovs, “Improving transformer optimization through better initialization,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 4475–4483.
  * [39] S. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” _arXiv preprint arXiv:1705.07874_ , 2017.
  * [40] T. Loughran and B. McDonald, “Textual analysis in accounting and finance: A survey,” _Journal of Accounting Research_ , vol. 54, no. 4, pp. 1187–1230, 2016.
  * [41] P. C. Tetlock, “Giving content to investor sentiment: The role of media in the stock market,” _The Journal of finance_ , vol. 62, no. 3, pp. 1139–1168, 2007.
  * [42] C. Dougal, J. Engelberg, D. Garcia, and C. A. Parsons, “Journalists and the stock market,” _The Review of Financial Studies_ , vol. 25, no. 3, pp. 639–679, 2012.
  * [43] U. G. Gurun and A. W. Butler, “Don’t believe the hype: Local media slant, local advertising, and firm value,” _The Journal of Finance_ , vol. 67, no. 2, pp. 561–598, 2012.
  * [44] L. Dodevska, V. Petreski, K. Mishev, A. Gjorgjevikj, I. Vodenska, L. Chitkushev, and D. Trajanov, “Predicting companies stock price direction by using sentiment analysis of news articles,” in _Proceedings of the 15th Annual International Conference on Computer Science and Education in Computer Science_ , 2019, pp. 37–42.
  * [45] T. Loughran and B. McDonald, “Measuring readability in financial disclosures,” _the Journal of Finance_ , vol. 69, no. 4, pp. 1643–1671, 2014.
  * [46] S. Krishnamoorthy, “Sentiment analysis of financial news articles using performance indicators,” _Knowledge and Information Systems_ , vol. 56, no. 2, pp. 373–394, 2018.
  * [47] P. J. Stone, D. C. Dunphy, and M. S. Smith, “The general inquirer: A computer approach to content analysis.” 1966.
  * [48] S. M. Mohammad, S. Kiritchenko, and X. Zhu, “Nrc-canada: Building the state-of-the-art in sentiment analysis of tweets,” _arXiv preprint arXiv:1308.6242_ , 2013.
  * [49] D. T. Vo and Y. Zhang, “Don’t count, predict! an automatic approach to learning sentiment lexicons for short text,” in _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , 2016, pp. 219–224.
  * [50] F. Viegas, M. S. Alvim, S. Canuto, T. Rosa, M. A. Gonçalves, and L. Rocha, “Exploiting semantic relationships for unsupervised expansion of sentiment lexicons,” _Information Systems_ , vol. 94, p. 101606, 2020.
  * [51] T. Bos and F. Frasincar, “Automatically building financial sentiment lexicons while accounting for negation,” _Cognitive Computation_ , pp. 1–19, 2021.
  * [52] H. Saif, Y. He, M. Fernandez, and H. Alani, “Adapting sentiment lexicons using contextual semantics for sentiment analysis of twitter,” in _European Semantic Web Conference_. Springer, 2014, pp. 54–63.
  * [53] N. Kaji and M. Kitsuregawa, “Building lexicon for sentiment analysis from massive collection of html documents,” in _Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL)_ , 2007, pp. 1075–1083.
  * [54] H. Kanayama and T. Nasukawa, “Fully automatic lexicon expansion for domain-oriented sentiment analysis,” in _Proceedings of the 2006 conference on empirical methods in natural language processing_ , 2006, pp. 355–363.
  * [55] N. Kaji and M. Kitsuregawa, “Automatic construction of polarity-tagged corpus from html documents,” in _Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions_ , 2006, pp. 452–459.
  * [56] W. L. Hamilton, K. Clark, J. Leskovec, and D. Jurafsky, “Inducing domain-specific sentiment lexicons from unlabeled corpora,” in _Proceedings of the conference on empirical methods in natural language processing. conference on empirical methods in natural language processing_ , vol. 2016. NIH Public Access, 2016, p. 595.
  * [57] O. Araque, G. Zhu, and C. A. Iglesias, “A semantic similarity-based perspective of affect lexicons for sentiment analysis,” _Knowledge-Based Systems_ , vol. 165, pp. 346–359, 2019.
  * [58] W. Zhao, T. Joshi, V. N. Nair, and A. Sudjianto, “Shap values for explaining cnn-based text classification models,” _arXiv preprint arXiv:2008.11825_ , 2020.
  * [59] K. E. Mokhtari, B. P. Higdon, and A. Başar, “Interpreting financial time series with shap values,” in _Proceedings of the 29th Annual International Conference on Computer Science and Software Engineering_ , 2019, pp. 166–172.
  * [60] X. Xiaomao, Z. Xudong, and W. Yuanfang, “A comparison of feature selection methodology for solving classification problems in finance,” in _Journal of Physics: Conference Series_ , vol. 1284, no. 1. IOP Publishing, 2019, p. 012026.
  * [61] M. Rizinski, H. Peshov, K. Mishev, L. T. Chitkushev, I. Vodenska, and D. Trajanov, “Ethically responsible machine learning in fintech,” _IEEE Access_ , vol. 10, pp. 97 531–97 554, 2022.
  * [62] E. Kokalj, B. Škrlj, N. Lavrač, S. Pollak, and M. Robnik-Šikonja, “Bert meets shapley: Extending shap explanations to transformer-based classifiers,” in _Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation_ , 2021, pp. 16–21.
  * [63] S. Consoli, L. Barbaglia, and S. Manzan, “Fine-grained, aspect-based sentiment analysis on economic and financial lexicon,” _Knowledge-Based Systems_ , vol. 247, p. 108781, 2022.
  * [64] A. Moreno-Ortiz, J. Fernández-Cruz, and C. P. C. Hernández, “Design and evaluation of sentiecon: A fine-grained economic/financial sentiment lexicon from a corpus of business news,” in _Proceedings of The 12th Language Resources and Evaluation Conference_ , 2020, pp. 5065–5072.
  * [65] M. Yekrangi and N. Abdolvand, “Financial markets sentiment analysis: Developing a specialized lexicon,” _Journal of Intelligent Information Systems_ , vol. 57, pp. 127–146, 2021.
  * [66] J. Fang and B. Chen, “Incorporating lexicon knowledge into svm learning to improve sentiment classification,” in _Proceedings of the workshop on sentiment analysis where AI meets psychology (SAAIP 2011)_ , 2011, pp. 94–100.
  * [67] R. Catelli, S. Pelosi, and M. Esposito, “Lexicon-based vs. bert-based sentiment analysis: A comparative study in italian,” _Electronics_ , vol. 11, no. 3, p. 374, 2022.
  * [68] A. H. Huang, H. Wang, and Y. Yang, “Finbert: A large language model for extracting information from financial text,” _Contemporary Accounting Research_ , vol. 40, no. 2, pp. 806–841, 2023.
  * [69] P. Malo, A. Sinha, P. Takala, O. Ahlgren, and I. Lappalainen, “Learning the roles of directional expressions and domain concepts in financial news analysis,” in _2013 IEEE 13th International Conference on Data Mining Workshops_. IEEE, 2013, pp. 945–954.
  * [70] K. Cortis, A. Freitas, T. Daudert, M. Huerlimann, M. Zarrouk, S. Handschuh, and B. Davis, “Semeval-2017 task 5: Fine-grained sentiment analysis on financial microblogs and news.” Association for Computational Linguistics (ACL), 2017.
  * [71] S. Mazzanti, “Shap values explained exactly how you wished someone explained to you,” https://towardsdatascience.com/shap-explained-the-way-i-wish-someone-explained-it-to-me-ab81cc69ef30, 2020, [Online; accessed 05-July-2021].
  * [72] S. Lundberg, “SHapley Additive exPlanations,” https://github.com/slundberg/shap, 2018, [Online; accessed 29-Jan-2023].
  * [73] M. Honnibal and I. Montani, “spaCy meets Transformers: Fine-tune BERT, XLNet and GPT-2,” https://explosion.ai/blog/spacy-transformers, 2019, [Online; accessed 10-February-2023].
  * [74] N. Lathia, “When is a neural net too big for production?” https://neal-lathia.medium.com/when-is-a-neural-net-too-big-for-production-4315452193ef, 2019, [Online; accessed 10-March-2023].

|  Maryan Rizinski received the B.S. and M.S. degrees in electrical
engineering and information technologies from University Ss. Cyril and
Methodius in Skopje, where he is a Ph.D. candidate in computer science. He is
currently an engineering manager at Bosch, with over ten years of industry
experience leading globally-distributed software engineering teams. His
expertise spans multiple aspects of the software project lifecycle management,
from planning, requirement gathering, and analysis, estimations to driving
delivery, rollout, and troubleshooting for international customers. Throughout
his professional career, he has managed the implementation of Internet of
Things (IoT) and fiber-optics infrastructure projects and has been mentoring
and consulting startup IT companies. He is also a lecturer of computer science
at Boston University’s Metropolitan College, where he is teaching and
facilitating networking and data science classes. His doctoral research
focuses on novel approaches for using machine learning (ML) and natural
language processing (NLP) in the financial industry and other related areas.
His research aims to enable more accurate decision-making and address
fundamental problems of improving the explainability of deep-learning models
and addressing ML-related ethical challenges in finance applications. His past
research interests focused on computer networking, wireless communications,
and new Internet and IoT architectures.
---|---
|  Hristijan Peshov received the Bachelor of Science degree in software
engineering and information systems from the Faculty of Computer Science and
Engineering, Saints Cyril and Methodius University in Skopje, in 2022. He also
works as a Software Engineer. His research interests include data science,
machine learning, explainable AI, natural language processing, and network
analysis.
---|---
|  Kostadin Mishev received the bachelor’s degree in informatics and computer
engineering and the master’s degree in computer networks and e-technologies
degree from Saints Cyril and Methodius University, Skopje, in 2013 and 2016,
respectively, where he is currently pursuing the Ph.D. degree. He is also a
Teaching and a Research Assistant with the Faculty of Computer Science and
Engineering, Saints Cyril and Methodius University. His research interests
include Data Science, Natural Language Processing, Semantic Web, Web
technologies, and Computer Networks.
---|---
|  Milos Jovanovik is an Associate Professor at the Faculty of Computer
Science and Engineering, at the Ss. Cyril and Methodius University in Skopje.
He’s also a Senior R&D Knowledge Graphs Engineer at OpenLink Software, London,
UK. He obtained his Ph.D. in 2016 at the Ss. Cyril and Methodius University in
Skopje, in the field of Computer Science and Engineering, with a doctoral
thesis in the domain of Linked Data. He has published over 50 scientific
papers and has participated in over 30 research projects on international and
national levels. His main research interests include Knowledge Graphs, Linked
Data, Open Data, and Data Science.
---|---
|  Dimitar Trajanov (Member, IEEE) received a Ph.D. degree in computer
science. He is a Full professor at the Faculty of Computer Science and
Engineering - ss. Cyril and Methodius University – Skopje and Visiting
Research Professor at Boston University. From March 2011 until September 2015,
he was the founding Dean of the Faculty of Computer Science and Engineering,
and in his tenure, the faculty became the largest technical Faculty in
Macedonia. Dimitar Trajanov is the leader of the Regional Social Innovation
Hub, established in 2013 as a cooperation between UNDP and the Faculty of
Computer Science and Engineering. Dimitar Trajanov is the author of more than
200 journal and conference papers and seven books. He has been involved in
more than 70 research and industry projects, of which in more than 40 projects
as a project leader. His research interests include Data Science, Machine
Learning, NLP, FinTech, Semantic Web, e-commerce, Technology for Development,
ESG, and Climate Change.
---|---