# Extending Memory Capacity in Consumer Devices with
Emerging Non-Volatile Memory: An Experimental Study

Geraldo F. Oliveira1 Saugata Ghose2 Juan Gómez-Luna1 Amirali Boroumand3 ,
Alexis Savery3 Sonny Rao4 Salman Qazi3 Gwendal Grignou3 and  Rahul Thakur3
Eric Shiu4 Onur Mutlu1 1ETH Zürich 2University of Illinois Urbana-Champaign
3Google 4Rivos

###### Abstract.

DRAM scalability is becoming a limiting factor to the available memory
capacity in consumer devices. As a potential solution, manufacturers have
introduced emerging non-volatile memories (NVMs) into the market, which can be
used to increase the memory capacity of consumer devices by augmenting or
replacing DRAM. In this work, we provide the first analysis of the impact of
extending the main memory space of consumer devices using off-the-shelf NVMs.
We equip real web-based Chromebook computers with the Intel Optane solid-state
drive (SSD), which contains state-of-the-art low-latency NVM, and use the NVM
as swap space. We analyze the performance and energy consumption of the
Optane-equipped Chromebooks, and compare this with (i) a baseline system with
double the amount of DRAM than the system with the NVM-based swap space; and
(ii) a system where the Intel Optane SSD is naively replaced with a state-of-
the-art NAND-flash-based SSD. Our experimental analysis reveals that while
Optane-based swap space provides a cost-effective way to alleviate the DRAM
capacity bottleneck in consumer devices, naive integration of the Optane SSD
leads to several system-level overheads, mostly related to (1) the Linux block
I/O layer, which can negatively impact overall performance; and (2) the off-
chip traffic to the swap space, which can negatively impact energy
consumption. To reduce the Linux block I/O layer overheads, we tailor several
system-level mechanisms (i.e., the I/O scheduler and the I/O request
completion mechanism) to the currently-running application’s access pattern.
To reduce the off-chip traffic overhead, we leverage an operating system
feature (called Zswap) that allocates some DRAM space to be used as a
compressed in-DRAM cache for data swapped between DRAM and the Intel Optane
SSD, significantly reducing energy consumption caused by the off-chip traffic
to the swap space. We conclude that emerging NVMs are a cost-effective
solution to alleviate the DRAM capacity bottleneck in consumer devices, which
can be further enhanced by tailoring system-level mechanisms to better
leverage the characteristics of our workloads and the NVM.

consumer devices, DRAM, emerging technologies, experimental characterization,
I/O systems, memory capacity, memory systems, non-volatile memory, quality of
service, solid-state drives, storage systems, system performance, tail
latency, user experience, web browsers

††copyright: none

##  1\. Introduction

R1.1The number and diversity of consumer devices (e.g., smartphones, tablets,
Chromebooks (chromebook, ), and wearable devices) are growing rapidly
(emarketer2016slowing, ; reddi2018two, ; armqualcomm2014, ; halpern2016mobile,
; CanalysN72, ). The number of consumer devices has surpassed the number of
desktop computers (reddi2018two, ). For example, web-based computers, such as
Chromebooks, account for 58% of all computer shipments to schools in the
United States (heater2017chromebook, ). These devices have different design
constraints than traditional computers due to their limited area, power
dissipation restrictions, and target market. Therefore, it is essential to
guarantee low cost per device while maintaining good performance and high-
quality user experience.

One critical component of consumer devices is the main memory (typically
consisting of DRAM (dennard1968dram, ; mutlu2014memorybook, )), which is used
not only as the working memory space but also as storage for least-recently-
used (i.e., cold) memory blocks (i.e., as swap space (bovet2005understanding,
; tanenbaum1997operating, ; lecturevirtualmemory, )). As consumer devices grow
in sophistication, many target applications handle increasing amounts of data
and require larger main memory capacity to avoid significant performance
issues (halpern2016mobile, ; badr2020mocktails, ; boroumand2018google, ;
mohan2017storage, ; amiraliphd, ; nelsonsize, ; lebeck2020end, ).
Unfortunately, it is becoming increasingly challenging to increase DRAM
capacity inside consumer devices due to the worsening reliability, cost, and
performance issues as manufacturers scale DRAM technology to higher storage
capacity levels (mutlu2013memory, ; mutlu2015research, ; kim2014flipping, ;
mutlu2019rowhammer, ; kim2020revisiting, ; mutlu2015main, ; kang2014co, ;
hong2010memory, ; kanev_isca2015, ; mutlu2017rowhammer, ; ghose2018your, ;
liu2013experimental, ; frigo2020trrespass, ; liu2012raidr, ; patel2017reach, ;
qureshi2015avatar, ; mandelman2002challenges, ; khan2014efficacy, ;
khan2016parbor, ; khan2017detecting, ; lee2015adaptive, ; lee2017design, ;
chang2017understandingphd, ; chang2017understandingsigmetrics, ;
chang2016understanding, ; chang2014improving, ; meza2015revisiting, ;
david2011memory, ; deng2011memscale, ; yauglikcci2022understanding, ;
orosa2021deeper, ; hassan2021uncovering, ).

As a potential solution to the DRAM scalability challenge, manufacturers have
introduced emerging non-volatile memories (NVMs) into the market, which can be
used to expand the memory capacity of consumer devices by augmenting or
replacing DRAM (lee2009architecting, ; qureshi2009scalable, ; lee2010phase, ;
lee2010phasecacm, ; kultursay2013evaluating, ; zhou2009durable, ;
wong2010phase, ; meza2012case, ; meza2013case, ; song2020improving, ;
song2021aging, ; song2019enabling, ; atwood2018pcm, ; bock2011analyzing, ;
burr2008overview, ; du2013bit, ; ferreira2010increasing, ; jiang2012fpb, ;
jiang2013hardware, ; kannan2016energy, ; qureshi2011pay, ;
qureshi2010improving, ; qureshi2010morphable, ; sebastian2017temporal, ;
wang2015exploit, ; yue2013accelerating, ; zhou2012writeback, ;
zhou2013writeback, ; yoon2013techniques, ; DAC-2009-DhimanAR, ; wang2013low, ;
chen2010advances, ; diao2007spin, ; hosomi2005novel, ; raychowdhury2009design,
; akinaga2010resistive, ; wong2012metal, ; yang2013memristive, ;
kund2005conductive, ; bondurant1990ferroelectronic, ). However, these NVM-
based devices are still slower than DRAM (harris2020ultra, ; wu2021storage, ;
lee2010phasecacm, ; lee2010phase, ; lee2009architecting, ; wang2019panthera, ;
salkhordeh2019analytical, ; yoon2012row, ; meza2012enabling, ). For example,
the state-of-the-art Intel Optane SSD (solid-state drive) (h10, ), which is a
low-latency NVM-based SSD device (i.e., an SSD device that uses NVM as its
primary persistent storage media), has an access latency that is two orders of
magnitude slower than that of DRAM (izraelevitz2019basic, ;
psaropoulos2019bridging, ) (but it is still one order of magnitude faster than
traditional NAND-flash-based SSDs (harris2020ultra, ; lee2019asynchronous, ;
zhang2018performance, ; chien2018characterizing, ; yang2020exploring, ;
hady2017platform, ; wu2019exploiting, ; imamura2018reducing, ; wu2021storage,
)), while providing a better cost-per-byte ($1.50 per GB (optanePrice, ) vs.
$5 per GB for DRAM (dramcost, )). Previous works propose two different ways to
integrate an NVM device into state-of-the-art computers in order to alleviate
DRAM scalability issues. The first method uses the device as byte-addressable
main memory, which directly replaces DRAM. In this method, the system can
access the NVM device directly using load and store instructions
(peng2019system, ; metzler2015prototyping, ; hassan2015energy, ;
chauhan2016nvmove, ; kultursay2013evaluating, ; lee2009architecting, ;
lee2010phase, ; qureshi2009scalable, ; zhou2009durable, ; DAC-2009-DhimanAR, ;
yoon2014efficient, ; yoon2012row, ; li2017utility, ; meza2012enabling, ;
zhang2021chameleondb, ; bae20182b, ; kim2018optimized, ; wu2021storage, ). The
second method uses the NVM as a block storage device that replaces a NAND-
flash-based SSD (cai2017error, ; luo2018improving, ; luo2018heatwatch, ;
cai2017vulnerabilities, ; luo2016enabling, ; cai2015read, ; lecturenand, ) or
a magnetic hard drive, using the same access protocol and interface
(harris2020ultra, ; lee2019asynchronous, ; zhang2018performance, ;
chien2018characterizing, ; yang2020exploring, ; hady2017platform, ;
wu2019exploiting, ; ke2018lirs, ; liu2020nvm, ; imamura2018reducing, ;
han2020splitkv, ; papagiannis2020optimizing, ; jia2020flash, ; bae20182b, ;
wu2019towards, ). Both methods alleviate DRAM scalability issues by taking
advantage of the higher density and lower cost-per-byte that NVMs offer over
DRAM. However, _entirely replacing DRAM_ with NVM in consumer devices imposes
large system integration and design challenges (e.g., due to the high write
access latency and limited endurance of NVM (lee2009architecting, ;
lee2010phase, ; lee2010phasecacm, )).

Integrating emerging NVM-based SSDs in consumer devices can open up new
opportunities for memory management. Traditional desktop and enterprise
computers employ a swap space (bovet2005understanding, ;
tanenbaum1997operating, ; lecturevirtualmemory, ; lebeck2020end, ) to increase
the total main memory space available in the system beyond the capacity of
available DRAM. In such systems, the system moves cold memory blocks present
in DRAM to a swap space, which usually exists in a high-latency and high-
capacity storage device (e.g., NAND-flash-based SSD, magnetic hard drive).
However, consumer devices usually do _not_ employ a swap space
(zhong2014building, ; kim2015cause, ; liu2017non, ; zhong2017building, ;
kim2019analysis, ; zhu2017smartswap, ; kim2018comparison, ; zhong2014dr, ;
kim2017application, ; kim2019ezswap, ; kim2020maintaining, ; guo2015mars, ;
liang2020acclaim, ), since accessing a storage device directly impacts system
performance and user experience due to the device’s high access latency. Since
emerging NVM-based SSDs have an order of magnitude lower access latency than
the commonly-used fast storage devices, i.e.,  NAND-flash-based SSDs, they
have the potential to enable the use of a swap space in consumer devices.
Recent works (zhong2014building, ; kim2015cause, ; liu2017non, ;
zhong2017building, ; kim2019analysis, ; zhu2017smartswap, ; kim2018comparison,
; zhong2014dr, ) propose extending the total main memory space available to
applications by using NVM as swap space for DRAM in mobile systems. However,
no prior work analyzes the implications of enabling a _real_ NVM-based swap
space in _real_ consumer devices.

In this work, we provide the _first_ analysis of the impact of extending the
main memory space of consumer devices using off-the-shelf NVMs. We
_extensively_ examine system performance and energy consumption when the NVM
device is used as swap space for DRAM main memory to effectively extend the
main memory capacity. Our empirical analyses lead us to several observations
and insights that can be useful for the design of future systems and NVMs.

For our experimental evaluation, we equip real web-based Chromebook computers
(chromebook, ) with the Intel Optane SSD (h10, ). Our target workloads are
interactive applications, with a focus on the Google Chrome (chrome, ) web
browser. We choose such workloads for two reasons. First, in interactive
applications, the system needs to respond to user inputs at a target output
latency to provide a satisfactory user experience. Second, in Chromebooks, the
Chrome browser serves as the main interface to execute services for the user.
We compare the performance and energy consumption of interactive workloads
running on our Chromebook with NVM-based swap space, where the Intel Optane
SSD capacity is used as swap space to extend main memory capacity, against two
state-of-the-art systems: (i) a baseline system with double the amount of DRAM
than the system with the NVM-based swap space, which resembles current
consumer devices but has high manufacturing cost due to the large DRAM
capacity and relatively high cost-per-bit of DRAM; and (ii) a system where the
Intel Optane SSD is naively replaced with a state-of-the-art (yet slower) off-
the-shelf NAND-flash-based SSD, which we use as a swap space of equivalent
size as the NVM-based swap space. The NAND-flash-based SSD provides a cheap
alternative to extend the main memory space, but it can penalize system
performance due to its high access latency. We use a memory capacity pressure
test (memorypressure, ) to measure the impact of the new NVM swap space on
user tasks that consist of loading, scrolling, and switching between Chrome
browser tabs. We measure how the NVM device increases the 99th-percentile
latency (i.e., tail latency) of each task and the total number of Chrome tabs
that the user can open without discarding old tabs. We divide our system
evaluation, analysis, and optimization into two major parts: (1) _Evaluating
NVM for Consumer Devices_ , and (2) _System Optimization_.

Evaluating NVM for Consumer Devices. In the first part of our work, we compare
the baseline system with the system where we extend the main memory space with
the Intel Optane SSD and the system where we extend the main memory space with
the NAND-flash-based SSD. R1.2We make four major observations. First, we
observe that extending the main memory space with the Intel Optane SSD
improves the average performance of interactive workloads (measured as the
latency of switching across Chrome browser tabs) compared to the baseline
system with twice the amount of DRAM. The NVM-based swap space enables the
system to leverage a larger aggregate main memory space than the baseline
system while also reducing system cost. However, extending the main memory
space with the Intel Optane SSD increases the number of violations of the
application’s target output latency by 2.6$\times$ (on average) once the
memory traffic between DRAM and the Intel Optane SSD exceeds a threshold,
which happens under high system load (e.g., a large number of opened Chrome
browser tabs). Since the Intel Optane SSD is integrated to the system via the
high-latency off-chip bus, constantly moving data between DRAM and the Intel
Optane SSD directly impacts browser performance. Second, we observe that
accessing the Intel Optane SSD through the power-hungry off-chip bus
_significantly_ increases energy consumption. We mitigate this issue by
allocating some DRAM space to be used as a compressed in-DRAM cache for data
swapped between DRAM and the Intel Optane SSD, which reduces the number of
accesses to the Intel Optane SSD by up to 2.11$\times$ and, as a result,
improves energy efficiency. Third, extending the main memory space even with
the slow NAND-flash-based SSD provides performance benefits compared to the
baseline system. However, due to the high access latency of the NAND-flash-
based SSD, the number of violations of the application’s target output latency
increases compared to the baseline system and the system with the Intel Optane
SSD’s NVM-based swap space. Fourth, we observe that the Linux block I/O layer
becomes a major source of performance overhead when the main memory space is
extended using NVM, primarily due to (i) I/O scheduling bottlenecks created by
the mismatch between our workloads’ I/O access patterns and the default I/O
scheduling policy; and (ii) overheads related to the asynchronous operation of
the I/O request completion mechanism. We mitigate some of these overheads by
proposing two system optimizations that can better leverage the
characteristics of our workloads and the NVM.

System Optimization. In the second part of our work, we mitigate some of the
system-level overheads we identify in the first part of our work by proposing
two system optimizations that can better leverage the characteristics of our
workloads and the NVM. First, we employ different I/O schedulers that better
match our workloads’ I/O access patterns, which improves performance. Second,
we change the default asynchronous I/O request completion model to a hybrid
I/O request completion model that adaptively switches from synchronous to
asynchronous operation. The baseline asynchronous I/O request completion model
entails non-trivial overheads (e.g., latency overheads of interrupt and
context switch). The hybrid I/O request completion model partially avoids
these overheads by allowing the process to synchronously wait for the
completion of I/O requests for a determined period of time. As a result, in
the best case, the process needs to wait for only the low access latency of
NVM, instead of incurring the large latency overheads of the asynchronous I/O
request completion model.

We make the following key contributions in this work:

  * •

We perform the first experimental analysis of the impact of using off-the-
shelf non-volatile memory (NVM) for swap space in a real consumer device. Our
studies highlight how a state-of-the-art NVM-based Intel Optane SSD can be
used effectively to extend the total main memory space available to
interactive applications such as the Google Chrome web browser.

  * •

We demonstrate that using a state-of-the-art off-the-shelf NVM-based SSD as
swap space can improve the performance of interactive applications compared to
increasing DRAM capacity, but that naively integrating the NVM-based SSD leads
to system-level overheads. These overheads primarily arise from the Linux
block I/O layer and the off-chip traffic to the SSD.

  * •

We identify two system optimizations that can mitigate some of the system-
level overheads that occur when using an NVM-based SSD as swap space. Both of
these optimizations adapt system-level mechanisms to application and runtime
behavior, and improve the overall performance of the system.

##  2\. Background and Motivation

We provide the background and motivation required to understand the main
components of our experimental setup and evaluated workloads. First, we
discuss how the memory system impacts the performance of the Google Chrome web
browser (chrome, ) (Section 2.1). Second, we investigate how users interact
with consumer devices and how this interaction contributes to memory capacity
pressure (Section 2.2). Third, we explain the main characteristics of the
Intel Optane SSD device (intel2018900p, ) (Section 2.3).

###  2.1. Google Chrome Web Browser in Consumer Devices

The web browser is one of the main applications of consumer devices. Due to
its significance, several web browser applications are present in many mobile
benchmark suites (gutierrez2011full, ; huang2014moby, ;
pandiyan2013performance, ; boroumand2018google, ). The Google Chrome web
browser (chrome, ), which has over a billion active users and the largest
share of the mobile browsing market (popper2017google, ; googleshare, ), is
one of the most relevant web browsers available. Therefore, we investigate
Google Chrome performance in this work.

Chrome performance can be defined based on three key metrics: (i) the time it
takes to load a web page; (ii) the smoothness of scrolling a web page (i.e.,
whether or not the user can perceive discontinuous movements or jumps when
moving up/down inside a web page, measured in frames per second); and (iii)
how quickly the browser can switch between web pages in different browser tabs
(i.e., its _tab switch latency_). Loading a new web page and scrolling through
a web page are highly compute-intensive operations (boroumand2018google, ),
since the main operations that Chrome performs during their execution are
rendering (blink, ) and rasterization (skia, ), respectively. In this work, we
are primarily interested in evaluating the impact of the swap space on Chrome
performance. Therefore, the most relevant metric for our analysis is the tab
switch latency, since switching between a recently-opened web page and a
previously-opened web page (i.e., a web page that is open in an inactive tab)
will likely result in a page fault when the system runs out of memory.

In Chrome, each tab represents a single process associated with the web page
displayed in the tab, which improves reliability and security
(reis2009isolating, ; barth2008security, ). When the user switches between
tabs, the browser executes two main tasks. First, it executes a context-switch
between the currently-opened tab and the requested tab. Second, it executes a
load operation of the requested web page, which involves loading data frames
related to the requested web page from memory and rendering the requested web
page. The latency from the time a user clicks a web page to the time the web
page is rendered on the screen is crucial since it impacts user satisfaction.
This time is mostly dominated by how fast the system can load the data frames
related to the requested web page from memory or disk. However, storing a
large number of web pages has become a challenge for consumer devices for two
main reasons. First, the total size of a single web page has been growing in
recent years due to the increased use of images, JavaScript, and video in web
pages (httparchieve, ). Second, users tend to keep many web pages open
concurrently during web browsing, leading to many open tabs. This results in a
demand for larger memory space required to keep the web page in physical
memory in modern systems.

To understand the impact of the number of open web pages and memory
consumption, we evaluate how many web pages it is possible to open in a system
with $8\text{\,}\mathrm{GB}\text{/}$ of DRAM before the system runs out of
physical memory. Figure 1 shows the memory profile of a test that continually
opens new Chrome web pages during one hour of execution. We observe that the
system runs out of physical memory (i.e., free memory) by opening only 30 web
pages (within the course of almost 10 minutes). When this happens, the system
enters a _memory capacity pressure_ state, leading to increased swap activity
(as shown by the increasing red line in Figure 1) and, consequently,
performance degradation (not shown in Figure 1). We conclude that there is a
clear need to provide more memory capacity to support more concurrently-open
web pages and, hence, better user experience in consumer (i.e., mobile)
devices.

Figure 1. System memory usage while loading 30 Chrome web pages.

A traditional way of expanding the memory space in modern systems is to enable
page swapping (bovet2005understanding, ; tanenbaum1997operating, ;
lecturevirtualmemory, ). However, mobile devices usually disable page swapping
due to the large performance penalty and user experience degradation imposed
on the system by high-latency storage devices (zhong2014building, ;
kim2015cause, ; liu2017non, ; zhong2017building, ; kim2019analysis, ;
zhu2017smartswap, ; kim2018comparison, ; zhong2014dr, ; kim2017application, ;
kim2019ezswap, ; kim2020maintaining, ; guo2015mars, ; liang2020acclaim, ).
Instead, a kernel module contiguously verifies the memory space and terminates
processes to make room for incoming memory requests. This approach is called “
_low memory killer_ ” (rientjes2010oom, ; collins2011android, ; lebeck2020end,
). Recently, Google enabled a swap alternative for its mobile devices (i.e.,
Google Pixel smartphones (pixel, ) and Google Chromebooks (chromebook, )). In
these devices, the operating system (OS) enables an in-DRAM compressed swap
space, called _ZRAM_ (jennings2013transparent, ). When enabling ZRAM in the
system, the OS reserves a fraction of the DRAM space to be used as a swap
device. Pages are compressed before being moved from the working region in
main memory to ZRAM (i.e., _swapped out_), and decompressed before being moved
from ZRAM to the working region in main memory (i.e., _swapped in_). By using
compression in ZRAM, the system can increase the capacity of the swap space by
the compression ratio (e.g., by 3:1 (shiu2015system, ; shiu2017driving, ;
pekhimenko2013linearly, )).

###  2.2. The Impact of Main Memory Capacity Pressure on Consumer Devices

R2/1AWe design an experiment to understand the impact of main memory capacity
pressure on consumer devices. Our experiment aims to characterize: (1) how
users utilize a web browser; (2) how often users suffer from high response
latencies from interactive workloads; and (3) how often users push the system
into a state of memory capacity pressure. For this purpose, we distributed
Chromebook devices111R2/1BThe Chromebook devices we use for our experiment
consist of off-the-shelf Chromebook devices with
$8\text{\,}\mathrm{GB}\text{/}$ DRAM capacity, of which
$4\text{\,}\mathrm{GB}\text{/}$ are reserved to enable an in-DRAM compressed
swap space.R2/1C to 114 different users at Google, whom we asked to perform
their daily activities using the Chromebook devices. We picked the users
randomly from a major division at the company that employs thousands of
people. We monitored their activity by periodically collecting the following
system information over a period of three months: R2/2

  * •

_Number of Chrome tabs opened:_ We recorded the number of open tabs across all
Chrome windows. Data samples were reported every 5 minutes. In total, we
collected 19,487 data samples.

  * •

_Tab switch latency:_ We collected the tab switch latencies for each tab
switch the user performed during their activity. Data samples were reported at
each individual tab switch. In total, we collected 62,243 data samples.

  * •

_Memory capacity pressure level:_ We periodically (i.e., every five seconds)
sampled the current state of the memory to determine which of the following
three memory capacity pressure levels the memory was currently experiencing:
_no memory capacity pressure_, _moderate memory capacity pressure_, and
_critical memory capacity pressure_. The memory capacity pressure level is
defined as follows. First, Chrome calculates the amount of $fill$ memory, as
$fill=1-\frac{(mem\\_free+\frac{swap\\_free}{RAM\\_vs\\_swap\\_weight})}{(mem\\_total+\frac{swap\\_total}{RAM\\_vs\\_swap\\_weight})}$;
where mem_free is the amount of main memory space currently free, mem_total is
the total amount of main memory space (free and occupied), swap_free is the
total amount of memory space in the swap device that is currently free,
swap_total is the total amount of swap space, and $RAM\\_vs\\_swap\\_weight$
accounts for the relative ease (considering memory access latency) of
allocating RAM directly versus having to swap its contents out first. This
parameter has a default value of “4”, which the operating system empirically
has defined. If _fill_ ranges between 60% and 95%, the system is under
moderate memory capacity pressure. If _fill_ is greater than or equal to 95%,
it is under critical memory capacity pressure (MemoryCo67, ). In total, we
collected 1,571,701 data samples.

Figure 2(a) shows how many Chrome tabs users opened during our experiments. We
observe that users had up to 20 Chrome tabs open in 68% of the samples; 21 to
40 Chrome tabs open in 18% of the samples; 41 to 80 Chrome tabs open in 10% of
the samples; and 81 to 160 Chrome tabs open in 4% of the samples (no user had
more than 160 tabs open at any time). Even though 4% is a relatively low
number of occurrences, it represents 710 sample points where the users kept a
large number of Chrome tabs open.

(a) Distribution of the number of open tabs.

(b) Distribution of the number of tabs switch latencies.

Figure 2. Distribution of the number of open tabs and tab switch latencies.

Figure 2(b) shows the distribution of the tab switch latency the users
experienced. We observe that users experienced a tolerable latency (i.e., a
tab switch latency less than $250\text{\,}\mathrm{ms}\text{/}$222We use
$250\text{\,}\mathrm{ms}\text{/}$ for the tolerable tab switch latency, since
a rule of thumb in the web performance community is to provide visual feedback
in under $250\text{\,}\mathrm{ms}\text{/}$ to keep the user engaged
(grigorikPerf, ; lohrImpatient, ).) in 67.3% of our samples. However, the
users experienced unacceptable latency from the system (i.e., a tab switch
latency greater than or equal to $250\text{\,}\mathrm{ms}\text{/}$) in 32.7%
of the samples. The tab switch latency was larger than 1 second for 20.59% of
the samples. We periodically collected the memory capacity pressure level of
the system to verify that the high tab switch latency was due to high memory
capacity pressure. Our experiment shows that 35.8% of our samples, the system
experiences moderate to critical memory capacity pressure (563,143 data
samples in moderate memory capacity pressure and 42 data samples in critical
memory capacity pressure from a total 1,571,701 data samples).

With this experiment, we conclude that real users, for a considerable fraction
of their usage time of the Chrome web browser, often push the system to points
that induce moderate to critical memory capacity pressure, leading to large
and often unacceptable response times for interactive workloads.

###  2.3. Intel Optane SSD

In the past several decades, various works (lee2009architecting, ;
qureshi2009scalable, ; lee2010phase, ; lee2010phasecacm, ;
kultursay2013evaluating, ; zhou2009durable, ; wong2010phase, ; meza2012case, ;
meza2013case, ; song2020improving, ; song2021aging, ; song2019enabling, ;
atwood2018pcm, ; bock2011analyzing, ; burr2008overview, ; du2013bit, ;
ferreira2010increasing, ; jiang2012fpb, ; jiang2013hardware, ;
kannan2016energy, ; qureshi2011pay, ; qureshi2010improving, ;
qureshi2010morphable, ; sebastian2017temporal, ; wang2015exploit, ;
yue2013accelerating, ; zhou2012writeback, ; zhou2013writeback, ;
yoon2013techniques, ; DAC-2009-DhimanAR, ; wang2013low, ; chen2010advances, ;
diao2007spin, ; hosomi2005novel, ; raychowdhury2009design, ;
akinaga2010resistive, ; wong2012metal, ; yang2013memristive, ;
kund2005conductive, ; bondurant1990ferroelectronic, ) have investigated how to
employ novel data storage technologies (e.g., phase-change memory, PCM
(wong2010phase, ; lee2010phase, ; lee2009architecting, ; qureshi2009scalable,
; lee2010phasecacm, ; zhou2009durable, ; atwood2018pcm, ; bock2011analyzing, ;
burr2008overview, ; du2013bit, ; ferreira2010increasing, ; jiang2012fpb, ;
jiang2013hardware, ; kannan2016energy, ; qureshi2011pay, ;
qureshi2010improving, ; qureshi2010morphable, ; sebastian2017temporal, ;
wang2015exploit, ; yue2013accelerating, ; zhou2012writeback, ;
zhou2013writeback, ; song2020improving, ; song2019enabling, ;
yoon2013techniques, ; DAC-2009-DhimanAR, ; meza2012case, ; song2021aging, );
spin-transfer torque magnetic RAM, STT-MRAM (wang2013low, ;
kultursay2013evaluating, ; chen2010advances, ; diao2007spin, ;
hosomi2005novel, ; raychowdhury2009design, ; meza2012case, ); metal-oxide
resistive RAM, ReRAM (akinaga2010resistive, ; wong2012metal, ;
yang2013memristive, ; Chi2016, ; song2018graphr, ; song2017pipelayer, ;
yao2017face, ; hu2016dot, ); conductive bridging RAM, CBRAM
(kund2005conductive, ; gopalan2011demonstration, ; jana2015conductive, ;
cha2020conductive, ); ferroelectric RAM, FeRAM (bondurant1990ferroelectronic,
; scott1989ferroelectric, ; scott2007applications, ; mikolajick2001feram, ))
to build fast non-volatile memories. Intel and Micron recently announced the
first widely-available commercial NVM device based on the 3D XPoint non-
volatile memory technology (webb20163d, ), called Intel Optane (intel2018900p,
). Intel provides two different memory devices based on Optane: (1) the Intel
Optane SSD (intel2018900p, ), and (2) the Intel Optane DC Persistent DIMM
(cutress2016intel, ). The key difference between these two devices is their
system interface. For the Intel Optane SSD, the device has a system interface
similar to current NAND-based flash memory devices, where the system
communicates to the device via the PCIe bus (specification2002pci, ). This
configuration provides one order of magnitude lower latency than traditional
NAND-flash-based SSDs (harris2020ultra, ; lee2019asynchronous, ;
zhang2018performance, ; chien2018characterizing, ; yang2020exploring, ;
hady2017platform, ; wu2019exploiting, ; imamura2018reducing, ; wu2021storage,
). For the Intel Optane DC Persistent DIMM, the device is integrated into the
system with a DIMM-based interface, similar to DRAM devices. The system
directly accesses the device using load/store requests at the byte granularity
(izraelevitz2019basic, ; patil2019performance, ; gill2019single, ;
wu2020lessons, ; weiland2019early, ; shanbhag2020large, ;
mironov2019performance, ; peng2019system, ; yang2020empirical, ;
benson_perma_2022, ; xiang2022characterizing, ). This configuration provides a
much lower access latency, on the order of hundreds of nanoseconds (around
$169\text{\,}\mathrm{ns}\text{/}$ for sequential reads (izraelevitz2019basic,
; yang2020empirical, ; benson_perma_2022, ; xiang2022characterizing, )), but
comes at a high cost, 5$\times$ the cost of the Intel Optane SSD in dollars-
per-bit (optanePrice, ; IntelOpt84, ).

Even though the Intel Optane DC Persistent DIMM can provide significant
benefits for future systems due to its performance characteristics, there are
several challenges to solve before leveraging such devices in future systems,
including: (1) the need for system mechanisms for proper data placement
between DRAM and the Optane DIMM device (bittman2020twizzler, ;
wang2019panthera, ; salkhordeh2019analytical, ; li2017utility, ; yoon2012row,
; meza2012enabling, ), (2) difficulties in fabricating printed circuit boards
(PCBs) for mobile platforms that can accommodate the Optane DIMM, and (3)
accommodating the high cost-per-bit of the Intel Optane DIMM in cost-sensitive
mobile systems. Even though prior works (zhong2014building, ; kim2015cause, ;
liu2017non, ; zhong2017building, ; zhu2017smartswap, ; kim2018comparison, )
propose, using simulation models, to integrate byte-addressable NVM devices as
swap space for mobile systems, we choose the Intel Optane SSD for our studies
for two main reasons. First, due to the manufacturing difficulties, including
high manufacturing cost, and open system-level challenges that need to be
solved before integrating the Optane DIMM as swap space in consumer devices.
Second, since the goal of our work is to evaluate the performance implications
of emerging NVM devices in _real_ consumer devices.

##  3\. Experimental Setup and Methodology

In this work, we characterize the performance of interactive workloads running
on consumer devices. Our target device is the Google Chromebook web-based
computer. We use the Asus Chromebox 3 (chromebox, ) for our experiments, as it
is not physically possible to integrate the Intel Optane SSD module in regular
Chromebook due to its limited PCIe lanes. The Asus Chromebox runs the same
operating system as the Chromebook device (ChromeOS (wright2009ready, )), and
has a similar hardware configuration. The device is equipped with a 7th-
generation Intel Core i3-7100U processor (i3, ),
$8\text{\,}\mathrm{GB}\text{/}$ DDR4 memory (skhynixddr4, ), and a
$32\text{\,}\mathrm{GB}\text{/}$ NAND-flash-based SSD (transcendssd, ).
ChromeOS uses up to 50% of the DRAM capacity (i.e.,
$4\text{\,}\mathrm{GB}\text{/}$) to enable an in-DRAM compressed swap space
called ZRAM, capable of holding up to $12\text{\,}\mathrm{GB}\text{/}$ of
compressed data (assuming a 3:1 compression ratio (shiu2015system, ;
shiu2017driving, ; pekhimenko2013linearly, )).333We use a 3:1 ZRAM compression
rate as an empirically-evaluated upper bound observed by prior works
(shiu2015system, ; shiu2017driving, ; pekhimenko2013linearly, ). In practice,
ZRAM compression rate varies with the pages getting swapped out. We observe an
average ZRAM compression rate of 1.14:1, with a maximum of 3:1, and a minimum
0.001:1 from our analysis. R1/3AWe modify the system by (1) removing the in-
DRAM compressed swap space and including an Intel Optane SSD module, which the
system uses as the swap device for DRAM; and (2) reducing the DRAM size to
$4\text{\,}\mathrm{GB}\text{/}$, to hold the non-swap-space DRAM capacity
constant. We use the Intel Optane H10 (h10, ) module for our experiments. It
contains a $16\text{\,}\mathrm{GB}\text{/}$ Intel Optane SSD device and a
$256\text{\,}\mathrm{GB}\text{/}$ Intel QLC 3D-NAND-flash-based SSD. We modify
the Intel Optane H10 firmware to avoid using the NAND-flash-based SSD during
our experiments.444R1/3BWe selected the Optane H10 module for our experiments
because it was the only Optane device in stock at the time we performed the
studies. Based on the technical specifications (h10, ; m10, ), the H10 module
combines the Intel Optane M10 module (m10, ) (i.e., an M.2 module containing
only the Intel Optane SSD) with a QLC 3D-NAND-flash-based SSD. In our initial
tests, we did _not_ observe any performance impact on the raw performance of
the H10 module with our modified firmware. Our modified firmware only disables
the QLC 3D-NAND-flash-based SSD in the H10 module.

We compare the performance and energy consumption of interactive workloads
running on our Chromebook using three system configurations, as Table 1
describes:

  * •

Baseline: a baseline system with 8 GB of DRAM. $4\text{\,}\mathrm{GB}\text{/}$
are used as main memory, which is uncompressed, and the other
$4\text{\,}\mathrm{GB}\text{/}$ are used as an in-DRAM compressed swap space
(ZRAM), which can house up to $12\text{\,}\mathrm{GB}\text{/}$ of actual data,
assuming a 3:1 compression ratio (shiu2015system, ; shiu2017driving, ;
pekhimenko2013linearly, );

  * •

Optane: a system with $4\text{\,}\mathrm{GB}\text{/}$ of main memory, and
$16\text{\,}\mathrm{GB}\text{/}$ of Intel Optane SSD swap space;

  * •

NANDFlash: a system with $4\text{\,}\mathrm{GB}\text{/}$ of main memory, and
$16\text{\,}\mathrm{GB}\text{/}$ of NAND-flash-based SSD swap space.

Table 1. Evaluated system configurations.

Swap Space Configurations
---
Configuration | DRAM Capacity | Swap Space Device | Swap Space Size | Effective Memory Capacity
Baseline | $8\text{\,}\mathrm{GB}\text{/}$ | In-DRAM Compressed Swap Space (ZRAM (jennings2013transparent, )) | $12\text{\,}\mathrm{GB}\text{/}$ | $16\text{\,}\mathrm{GB}\text{/}$
Optane | $4\text{\,}\mathrm{GB}\text{/}$ | Intel Optane SSD (H10 Module) (h10, ) | $16\text{\,}\mathrm{GB}\text{/}$ | $20\text{\,}\mathrm{GB}\text{/}$
NANDFlash | $4\text{\,}\mathrm{GB}\text{/}$ | NVMe NAND-flash-based SSD | $16\text{\,}\mathrm{GB}\text{/}$ | $20\text{\,}\mathrm{GB}\text{/}$
Common System Parameters
| Asus Chromebox (chromebox, ); 7th-generation Intel Core i3-7100U processor
(i3, );
Hardware Setup | DDR4 main memory (skhynixddr4, ); $32\text{\,}\mathrm{GB}\text{/}$ NAND-flash-based SSD (skhynixddr4, ) for storage
| Operating System: ChromeOS (wright2009ready, ); kernel version 4.14
Software Setup | Test Automation Tool: Chromium Project’s memory capacity pressure test (memorypressure, )

One of the main obstacles we faced during our analysis was creating the
correct experimental setup. This is challenging for three main reasons, mostly
related to the lack of a standard benchmark suite for consumer devices
(badr2020mocktails, ) and the lack of automation tools for real-world
experiments on mobile devices like Chromebooks:

_Challenge 1: Executing real-world workloads._ Popular interactive workloads
for mobile devices are proprietary (e.g., social networks such as Facebook
(facebook, ) and Instagram (instagram, ), messengers such as WhatsApp
(whatsapp, ) and Telegram (telegram, ), document readers such as Adobe Acrobat
Reader (adobe, ), games such as Minecraft (minecraft, )), and their source
code is not openly available. This limits the scope of our analysis, since we
can only analyze such applications as a black box, often making it unclear
which specific system resources an application uses and why. Prior works
(gutierrez2011full, ; huang2014moby, ; pandiyan2013performance, ;
guthaus2001mibench, ; lee1997mediabench, ) put effort into creating benchmark
suites for mobile applications. However, they are outdated and often include
only a small number of kernels from a small set of applications.

_Challenge 2: Automating execution and enabling reproducibility._ There is a
lack of tools for automating the execution of mobile workloads
(badr2020mocktails, ). This is critical when evaluating an entire system,
since experiments need to be executed multiple times to reduce system-level
noise (e.g., due to OS tasks, uncontrollable network response times). Without
automation, it is difficult to launch and execute applications in an automatic
and easily reproducible manner, deal with network traffic, and mimic user
interactions with the system, for example.

_Challenge 3: Stressing the main memory capacity._ As prior works show
(lebeck2020end, ) and as we observe in our analysis, running a single
application is usually not enough to stress the main memory capacity and
create swap activity, which we aim to study in this work. This happens because
many popular interactive applications have a memory footprint of only a few
hundreds of megabytes (lebeck2020end, ), which fit within the main memory of
the device. Even though we could mix an increasingly large number of different
applications until we place the system under memory capacity pressure, this
procedure would be hard to automate, since each application requires different
user interactions, and generating random combinations of workloads could
dramatically change our analysis.

To overcome these challenges, we rely on the infrastructure that the open-
source Chromium project (chromium, ) provides to automate the execution of
web-based processes. Specifically, we use the Chromium project’s open-source
memory capacity pressure test (memorypressure, ) to evaluate the system. The
test has three phases. In the first phase (memory pressure), the test opens
multiple Chrome tabs in Chrome until the first tab discard occurs (i.e., when
Chrome terminates the process associated with an open tab). In the second
phase (cold switch), the test opens the least-recently-used tabs (called cold
tabs) to induce page faults. In the third phase (heavy load), the test
executes tab switches to measure system performance under heavy memory
capacity pressure. A tab discard happens when Chrome observes that the system
is running out of memory. To calculate the amount of available memory, Chrome
computes
$available\\_mem=available\\_RAM+num\\_swap\\_pages/RAM\\_vs\\_swap\\_weight$,
where $num\\_swap\\_pages$ defines the number of available (free and non-
defective) page slots in all active swap areas, and $RAM\\_vs\\_swap\\_weight$
accounts for the relative ease of allocating RAM directly versus having to
swap its contents out first.

Using the memory capacity pressure test in our analysis allows us to overcome
all three main challenges we discuss above. We mitigate the first challenge
(i.e., executing real-world workloads) by using the open-source and commonly-
used Chrome web browser as our primary workload driver. Doing so brings two
main advantages for our experimental setup. First, we can fully understand
Chrome’s internal structure since it is an open-source tool, and can
understand the system resources it demands using different profiling tools
(e.g., _perf_ profiler (perf, )). Second, we create an environment where the
system executes distinct tasks concurrently, since the user utilizes Chrome as
the primary interface to execute different services, and Chrome creates a new
process for each new Chrome tab. We mimic a multiprocess system that runs
different workloads and stresses different segments of the system, by opening
different Chrome tabs that execute different services. These services include
Google web services (e.g., YouTube (youtube, ), Google Maps (maps, ), Google
Sheets (sheets, ), Google Docs (docs, )), Facebook (facebook, ), and Twitter
(twitter, ), each of which demands different computational sources. For
example, when loading YouTube as one of our Chrome tabs, the newly-created
process executes tasks related to web browsing (e.g., texture tiling, color
blitting) and YouTube-related tasks, such as video decoding, locally
(chromiummedia, ). We profile our system using the _perf_ profiling tool while
executing our memory capacity pressure test, to characterize the workloads and
tasks executed by our setup that are _unrelated_ to web browsing tasks. A non-
exhaustive list of workloads executed in our evaluation setup is:

  * •

Video decoding using the FFmpeg (ffmpegDo29, ) and libvpx (TheWebMP86, )
libraries, which are used to execute the VP8/VP9 (grange2016vp9, ) video
decoder;

  * •

Web Media Player (chromiummedia, ) to support HTML5 video playback
(HTMLStan14, ), including audio/video decoders supported by the Mojo System
API (Mojodocs82, ) and the Video Acceleration API (VaAPI61, );

  * •

Audio rendering utilizing Chromium’s audio rendering API (chromiummedia, );

  * •

GIF decoding/encoding using the SkGifCodec (srccodec76, );

  * •

V8 JavaScript engine (V8JavaSc93, ).

We observe that our experimental setup includes two of the workloads also
evaluated by prior work on consumer devices (boroumand2018google, ) (Chrome
web browser, VP9 video decoding). In addition, our setup includes several
workloads not covered by prior work (boroumand2018google, ) (e.g., Web Media
Player, audio rendering, GIF decoding/encoding, V8 JavaScript engine) that are
commonly employed in consumer devices. Our system setup is sufficient for our
study, since our goal is not to provide optimizations for a particular
workload, but rather to understand the impact of new memory technology in a
real system while running real applications.

We mitigate the second challenge (i.e., the lack of tools for automation and
reproducibility) by leveraging the memory capacity pressure test’s capability
to load a new Chrome tab, scroll through a tab, and perform tab switches
across open tabs without user’s intervention. This is possible since the test
utilizes ChromeOS’ Tast integration-testing framework (TastTast15, ). The
framework provides APIs that allow the test code to interact with elements of
the user interface through the chrome.automation library (chromeau55, ).

We mitigate the third challenge (i.e., stressing the memory system to induce
enough swap activity) by launching enough Chrome tabs until the system
experiences moderate to critical memory capacity pressure (see Section 2.2).
We can easily control the amount of memory capacity pressure in the system
since we can easily predict the memory footprint of opening a new Chrome tab.
As we discuss in Section 2.2, we quantitatively analyze a range of memory
capacity pressure conditions that are experienced during real user activity in
a user study across 114 users.

Metrics. We use two key metrics throughout our analysis in this paper: (i)
_tab count_ , which is the number of Chrome tabs our memory capacity pressure
test can open before a tab discard happens; and (ii)  _tab switch latency_ ,
which is the latency of switching across different tabs that are already open.
The tab count metric provides an indication of the memory capacity pressure
the system can support. The tab switch latency metric is relevant and
important for two main reasons. First, it directly impacts the user experience
by affecting the response time to the user. Second, the tab switch latency
metric provides a clear indication of the performance impact on our
interactive workloads of moving memory blocks back and forth between main
memory and the swap space. By switching to a previously-opened tab, whose
memory pages have been moved from main memory to the swap space, we force the
system to load data from the swap space. Then, the latency associated with the
data movement from the swap space to main memory is accounted for in the final
tab switch latency.

##  4\. Evaluating Intel Optane SSD for Consumer Devices

In this section, we evaluate the performance implications of employing an
Intel Optane SSD in consumer devices. First, we evaluate the impact on Chrome
web browser performance of reducing DRAM size while using the Intel Optane SSD
as swap space for DRAM (Section 4.1). Second, we leverage a compressed in-DRAM
cache to reduce tail latency and energy consumption in a system equipped with
the Intel Optane SSD (Section 4.2). Third, we evaluate the impact of replacing
the Intel Optane SSD with a cheaper state-of-the-art NAND-flash-based SSD
(Section 4.3).

Throughout the evaluations we conduct in this section, we push the system to a
critical memory capacity pressure state by opening as many Chrome tabs as
possible (i.e., until the first tab discard happens). We evaluate such an
extreme state since our goal in this section is to fully understand the
benefits and drawbacks of employing the Intel Optane SSD device in our system.
In addition, we report and analyze the impact of the Intel Optane SSD device
as swap space at moderate memory capacity pressure states, where the system
has fewer Chrome tabs open.

###  4.1. Effect of NVM as a Swap Space

We evaluate (i) the number of Chrome tabs the memory capacity pressure test
can open before a tab discard happens, and (ii) the tab switch latency for the
baseline (i.e., the system with $8\text{\,}\mathrm{GB}\text{/}$ of DRAM and
ZRAM as the swap device) and the Optane (i.e., the system with
$4\text{\,}\mathrm{GB}\text{/}$ of DRAM and the Intel Optane SSD as the swap
device) configurations. During our analysis, we observe that the number of
open tabs by the Optane configuration is 24% larger than the number of open
tabs by the baseline configuration our memory capacity pressure test (164 vs.
132 open tabs for the Optane configuration and the baseline configuration,
respectively). We observe that this increase in the number of open tabs in the
Optane configuration is due to the increase in total memory space provided by
the Optane configuration. In the Optane configuration, the total memory space
available is $4\text{\,}\mathrm{GB}\text{/}$ of DRAM plus
$16\text{\,}\mathrm{GB}\text{/}$ of swap (thus,
$20\text{\,}\mathrm{GB}\text{/}$ of effective main memory space). In the
baseline configuration, this value is 25% lower, since even though the system
has $8\text{\,}\mathrm{GB}\text{/}$ of DRAM, up to 50% of DRAM space is
reserved for the in-DRAM compressed swap space (i.e., ZRAM). With an up to 3:1
compression ratio (shiu2015system, ; shiu2017driving, ;
pekhimenko2013linearly, ), the total memory space in the baseline
configuration becomes $4\text{\,}\mathrm{GB}\text{/}$ of DRAM plus
$12\text{\,}\mathrm{GB}\text{/}$ of swap space (thus,
$16\text{\,}\mathrm{GB}\text{/}$ of effective main memory space).

R2/3Figure 3 shows the tab switch latency distribution for the baseline and
Optane configurations. The figure depicts the tab switch ID, i.e., an
identifier for a given sorted tab switch latency (x-axis) and the sorted tab
switch latency (y-axis, in logarithmic scale). We draw two observations.

Figure 3. Tab switch latency distribution: Baseline (ZRAM) vs. Optane.

First, we observe that the tab switch latency of the baseline configuration is
lower than that of the Optane configuration when the number of tab switches is
lower than 130. This is because the Optane configuration has half of the DRAM
space as the baseline configuration. As a result, the Optane configuration
experiences more page faults than the baseline configuration. In our
experiments, the baseline configuration experiences 451 page faults until 130
tab switches, which are serviced by the ZRAM space. In contrast, the Optane
configuration experiences 10$\times$ the page faults of the baseline until it
hits 130 tab switches, which significantly reduces Optane performance at low
tab switch counts. It is important to highlight that there are fewer page
faults for the baseline configuration than for the Optane configuration when
the number of tab switches is low, because the physical memory space the OS
reserves for ZRAM is _not_ statically allocated. Initially, during execution,
the baseline configuration enjoys a larger memory space than the Optane
configuration since the baseline configuration has
$8\text{\,}\mathrm{GB}\text{/}$ of physical memory available as working
memory. This leads to a lower number of page faults experienced by the
baseline configuration than by the Optane configuration when the tab count
(and therefore the number of tab switches) is low. However, as swap activity
increases with larger number of open tabs, the OS allocates physical memory
for the compressed swap space utilized by the baseline. The OS allocates
physical memory for the ZRAM swap space until the total swap space size
reaches a predefined threshold ($4\text{\,}\mathrm{GB}\text{/}$ of physical
DRAM space in our setup). After the maximum swap space threshold is reached,
the baseline configuration can leverage _only_ the remaining
$4\text{\,}\mathrm{GB}\text{/}$ of physical memory as working memory, which
leads to a much larger number of page faults.

Second, after 130 tab switches, the Optane configuration provides a lower tab
switch latency than the baseline configuration. That is due to how ZRAM
operates once the system runs out of memory: Chrome pages need to be
constantly swapped into/out of the ZRAM space. This incurs high CPU overhead
and data movement between DRAM regions since the processor needs to frequently
(1) find cold pages to move from main memory to ZRAM (i.e., swap pages into
the ZRAM space), which requires the processor to execute data compression
operations; and (2) find and move requested cold pages from ZRAM to main
memory in case of a page fault (i.e., swap pages out of the ZRAM space), which
requires the processor to execute data decompression operations. In contrast,
the Optane configuration greatly alleviates CPU usage caused by
compression/decompression activities. When memory pages are swapped into/out
of the Optane-based swap space, the processor needs to _only_ issue
asynchronous read/write requests to the Intel Optane SSD device. We observe
that by eliminating the CPU time the system would spend on
compression/decompression activities, the page fault latency in the Optane
configuration is 35% lower than in the baseline ZRAM configuration, which
leads to lower tab switch latencies in the Optane configuration. We conclude
that the Optane configuration provides better performance than the ZRAM
configuration for high-enough tab counts.

Even though using the Intel Optane SSD as a swap space provides benefits for
average tab switch latency (especially at high tab counts), it can also harm
tail latency performance. Figure 4 shows the distribution of tab switches with
latency larger than $250\text{\,}\mathrm{ms}\text{/}$ for both the baseline
and Optane configurations. We make two observations. First, the fraction of
tab switches with a latency larger than $250\text{\,}\mathrm{ms}\text{/}$ for
the baseline configuration is 7.1%, on average during the execution of the
memory capacity pressure test. In contrast, in the Optane configuration, the
fraction of high-latency tab switches is 18.4%, on average. Thus, in the
Optane configuration, the fraction of high-latency tab switches is 2.6$\times$
those in the baseline configuration. Second, for low tab counts (until 60
tabs), the percentage of high-latency tab switches for the Optane
configuration is (i) the _same_ as in the baseline configuration for 1–20 tab
counts (0% vs. 0% for the Optane and baseline configurations, respectively);
and (ii) _slightly_ larger than the baseline configuration for 21–40 and 40–61
tab counts (high-latency tab switches make up 5.5% vs. 2% for 21–40 tab
counts, and 6.5% vs. 3% for 41–60 tab counts, for the Optane and baseline
configurations, respectively). However, after 61 tabs, the number of tab
switches with high latency increases significantly for the Optane
configuration. To understand the reason of this increase in tab switches with
high latency in the Optane configuration, we analyze the page fault latency
distribution during the execution of our memory capacity pressure next.

Figure 4. High-latency tab switch distribution: Baseline (ZRAM) vs. Optane.

R1/4Figure 5 shows the page fault latency distribution for all page faults in
the baseline and Optane configurations during the execution of our memory
capacity pressure test. We make two observations. First, when analyzing the
page fault latency, we observe that at least 98% of the page faults have a
response latency of less than $10\text{\,}\mathrm{\SIUnitSymbolMicro
s}\text{/}$ for both configurations (Figure 5, left). Second, when examining
the tail of the latency distribution (Figure 5, right), the Optane
configuration has 4.43$\times$ the number of page faults of the baseline
(i.e., the number of page faults with a latency larger than
$10\text{\,}\mathrm{\SIUnitSymbolMicro s}\text{/}$ in the Optane configuration
is 4.43$\times$ that of the baseline configuration). We conclude that the
increase in high-latency tab switches for the Optane configuration (in Figure
4) is due to less than 2% of the page faults, with these page faults having a
high response latency (more than $10\text{\,}\mathrm{\SIUnitSymbolMicro
s}\text{/}$).

Figure 5. Page fault latency distribution.

Figure 6 compares the impact of the baseline and Optane configurations on
average memory subsystem energy consumption and swap traffic (i.e., the total
number of bytes swapped into/out of the swap space). To model Intel Optane
energy consumption, we assume that the device uses PCM-based memory cells, as
previous work suggests (choe2017intel, ). Then, we gather the read/write
energy consumption of PCM-based memory devices and DRAM, as reported by
previous work (chen2012energy, ), to model the energy consumption of our
system, i.e.,
$4.4\text{\,}\mathrm{pJ}\text{/}$/$2.47\text{\,}\mathrm{pJ}\text{/}$ read
energy-per-bit and
$5.5\text{\,}\mathrm{pJ}\text{/}$/$14.03\text{\,}\mathrm{pJ}\text{/}$
(set)–$19.73\text{\,}\mathrm{pJ}\text{/}$ (reset) write energy-per-bit for
DRAM/PCM.555We evaluate _only_ the energy of the memory subsystem in our
analyses. This does _not_ include the energy consumed by the processor. In the
figure, we normalize both metrics to the baseline values (y = 1.0 in the
plot). We observe that the Optane configuration increases energy consumption
and swap traffic by 69.5$\times$ and 37.2$\times$, respectively, compared to
the baseline configuration. This is due to two main reasons. First, writing
one bit to Optane consumes up to 3.6$\times$ the energy of writing one bit to
DRAM (chen2012energy, ). Even though reading one bit from Optane consumes 56%
less energy than reading one bit from DRAM (chen2012energy, ), in our
analysis, we find that the majority (i.e., 54%) of the accesses to the Intel
Optane SSD are write accesses. Second, the Optane configuration generates much
more swap activity than the baseline, since the Optane configuration has half
the DRAM size of the baseline configuration. We conclude that using the Intel
Optane SSD as a swap space to DRAM severely penalizes system energy
consumption.

Figure 6. Energy and swap traffic. Y-axis values are normalized to the
baseline configuration.

Summarizing our findings, the Optane configuration provides benefits compared
to the baseline system, since it enables a large number of tab switches due to
an increase in the main memory space. However, it has significant drawbacks in
terms of tail latency, system energy and swap traffic, compared to the
baseline with double the amount of DRAM. Most of these downsides come from the
large number of accesses, especially write accesses, to the Intel Optane SSD.
To solve these problems, we next examine multiple techniques that can (i)
reduce the swap traffic in the Optane configuration (Section 4.2) and (ii)
improve overall system performance for the Optane configuration (Section 5).

###  4.2. Reducing Tail Latency by Enabling a Compressed RAM Cache

The Intel Optane SSD can improve overall performance for consumer devices
since it enables an extended memory space. However, as we show in Section 4.1,
it can also negatively impact tail latency and system energy consumption due
to the need to issue high-latency and power-hungry I/O requests to access the
device. In addition, NVM devices such as the Intel Optane SSD suffer from
limited endurance (lee2009architecting, ; lee2010phase, ; lee2010phasecacm, ).
As a result, the large number of write operations caused by page swapping can
degrade system reliability.

To overcome these issues introduced by the Intel Optane SSD, we aim to reduce
the number of accesses to it. To this end, we augment the Optane configuration
with Zswap (zolnierkiewicz2013efficient, ), an in-DRAM swap cache used to
store compressed cold-pages. Zswap takes memory pages that are in the process
of being swapped out from main memory to the swap device (i.e., the Intel
Optane SSD) and attempts to compress them into a dynamically-allocated DRAM-
based memory pool. When a page fault happens, the OS checks if the requested
memory page is stored in the Zswap cache (i.e., Zswap load hit); if the page
is not in the Zswap cache, the OS loads the memory page from the swap space
(i.e., Zswap load miss). In case of a Zswap load hit, the OS (1) loads and
decompresses the memory page stored in the Zswap cache, and (2) services the
page fault request by writing the decompressed memory page into DRAM. The
motivation behind Zswap is to trade CPU cycles for a potential reduction in
I/O requests. Zswap can improve performance if the read requests are serviced
faster from the in-DRAM compressed swap cache than from the swap
device.666Even though both ZRAM and Zswap use an in-DRAM compressed memory
space to operate, they are fundamentally different mechanisms. While ZRAM is
an in-DRAM compressed _swap space_ , Zswap is an in-DRAM _swap cache_. The
system sees ZRAM as a swap device and Zswap as a cache for memory pages
swapped into/out of the swap device. We enable Zswap only in the Optane and
NANDFlash configurations, since the goal of enabling Zswap is to reduce I/O
traffic to _off-chip swap devices_ (i.e., the Intel Optane SSD and the NAND-
flash-based SSD in our experiments). We indicate Zswap-enabled configurations
explicitly throughout the paper (e.g., the Optane configuration with Zswap is
labeled _Optane+Zswap_).

To fully leverage Zswap in the Optane configuration, we first need to tailor
two important parameters related to Zswap execution. First, as explained in
Section 2, a tab discard happens when Chrome observes that the system is
running out of memory, which is computed based on the RAM_vs_swap_weight
parameter (i.e., the relative ease of allocating DRAM pages directly versus
having to swap its contents out first; see Section 3). This parameter is
empirically defined for ZRAM as 4. Similarly, we empirically evaluate which
value RAM_vs_swap_weight enables the memory capacity pressure test to open
more tabs before the first tab discard happens. We run tests varying the
RAM_vs_swap_weight value from 1 to 8, and we observe that RAM_vs_swap_weight
equals to 1 provides the largest number of open tabs for the Zswap
configuration. Second, we need to understand the impact of the maximum Zswap
cache size (i.e., max_pool_size) on the tab switch latency. To this end, we
run the memory capacity pressure test while varying the max_pool_size from 0%
to 50% of the total DRAM size. As expected, we observe that the tab switch
latency increases with the max_pool_size. We find that setting max_pool_size
to 20% of the total DRAM capacity provides a good balance between reduction in
I/O traffic and tab switch latency.

Figure 7 compares the total number of open tabs before a tab discard happens
in the baseline and Optane configurations, and when we enable Zswap in our
Optane configuration (labeled Optane+Zswap). We observe that by enabling
Zswap, the number of open tabs reduces by 12% compared to the Optane
configuration without Zswap enabled. By enabling Zswap, we effectively reduce
the total memory space available, thus causing a discard to happen sooner.

Figure 7. Number of open tabs: Baseline vs. Optane vs. Optane with Zswap
enabled (Optane+Zswap).

Figure 8(a) shows the impact of enabling Zswap on the tab switch latency.
Enabling Zswap maintains a similar overall tab switch latency as the Optane
configuration without Zswap. Figure 8(b) shows the impact that Zswap has on
high-latency tab switches. We make three observations. First, on average
across all tab counts, enabling Zswap leads to a modest increase of 4% on the
number of tab switches with latency larger than
$250\text{\,}\mathrm{ms}\text{/}$ (up to 29% for tab counts of 101–120).
Second, the large difference in the fraction of tabs with high latency tab
switches between Optane and Optane+Zswap configurations happens when swap
activity increases, and the swap cache gets full. When the swap cache gets
full, the system needs to (1) free an entry in the swap cache, (2) decompress
the selected entry and evict it back to the swap device, and (3) compress the
new page and insert it in the swap cache. These operations represent the
worst-case latency for a Zswap operation. Third, tab counts of 21–40 and 41–60
have a similar fraction of tab switches whose latencies are unacceptable in
both the Optane and Optane+Zswap configurations. Enabling Zswap reduces the
number of high-latency tab switches for tab counts of 21–40, resulting in a
similar fraction of high-latency tabs as the baseline. For 21–40 tabs, the
number of high latency tab switches is 2%, 5.5%, and 3%, for the baseline,
Optane, and Optane+Zswap configurations, respectively. We conclude that when a
page request hits in the Zswap cache, the system can provide a lower tab
switch latency.

(a) Tab switch latency distribution.

(b) High-latency tab switch distribution.

Figure 8. Tab switch latency: Baseline vs. Optane vs. Optane+Zswap.

To fully understand the impact of enabling Zswap in the Optane configuration,
we monitor the CPU utilization of the system for both Optane and Optane+Zswap
configurations. For this, we execute the memory capacity pressure test for
both configurations and monitor the CPU utilization with the vmstat tool
(tanaka2005monitoring, ) for one hour. Figure 9 depicts the CPU utilization
for both configurations. We make two observations. First, we observe that,
when we disable Zswap (Figure 9a), the CPU spends part of the execution time
waiting for I/O operations to complete. Towards the end of the execution of
the memory capacity pressure test, the I/O waiting time increases
dramatically, since the swap activity also greatly increases. Second, we
observe that by enabling Zswap, the system spends a smaller fraction of its
time waiting for I/O operations, as Figure 9b shows. On the other hand, it
also spends a larger fraction of its time on kernel activity due to Zswap
compression/decompression execution, which reduces the fraction of time spent
on user activity and penalizes Chrome browser performance. We conclude that
the increase in kernel activity caused by Zswap cache
compression/decompression operations is the _primary_ cause of the increase in
the fraction of high-latency tab switches in the Optane+Zswap configuration.

Figure 9. System execution time breakdown during a memory capacity pressure
test.

To evaluate the effectiveness of the Zswap cache, we analyze the Zswap cache
behavior during the execution of the memory capacity pressure test. Figure
10(a) shows the number of load hits, load misses, and total loads that the
Zswap cache services. Figure 10(b) shows the hit rate of the Zswap cache over
the same time. We make the two observations based on Figure 10. First, the
Zswap cache provides a high hit rate of 97%, on average during the execution
of the memory capacity pressure test. In fact, we see in Figure 10(b) that the
hit rate is close to 100% from 16 minutes to 30 minutes. Second, we observe
that the hit rate drops from 100% at 30 minutes to 91% at 60 minutes. We
observe that at 30 minutes, the Zswap cache gets full, which requires the
system to evict old pages frequently. However, even during high memory
capacity pressure, the Zswap cache maintains a significantly high hit rate. We
conclude that Zswap cache is an effective cache for the swap space.

(a) Zswap cache load hits and load misses.

(b) Zswap cache hit rate.

Figure 10. Zswap cache performance.

Figure 11 shows the distribution of the compression and decompression
latencies in the Zswap cache during the execution of the memory capacity
pressure test. In the figure, dashed lines represent the average
compression/decompression latency, and solid lines represent the maximum
compression/decompression latency. We make two observations. First, the
decompression latency, which is on the critical path of Chrome execution when
a page fault happens, is $3.9\text{\,}\mathrm{\SIUnitSymbolMicro s}\text{/}$
on average during the execution of the memory capacity pressure test (minimum
of $1.5\text{\,}\mathrm{\SIUnitSymbolMicro s}\text{/}$, and maximum of
$42.6\text{\,}\mathrm{\SIUnitSymbolMicro s}\text{/}$). We observe that 98.7%
of the decompression requests have a latency of less than
$10\text{\,}\mathrm{\SIUnitSymbolMicro s}\text{/}$. Second, the compression
latency is larger, with an average latency of
$12.1\text{\,}\mathrm{\SIUnitSymbolMicro s}\text{/}$ (minimum of
$1.5\text{\,}\mathrm{\SIUnitSymbolMicro s}\text{/}$, and maximum of
$138.2\text{\,}\mathrm{\SIUnitSymbolMicro s}\text{/}$). As a comparison, the
Intel Optane SSD read latency is $22.4\text{\,}\mathrm{\SIUnitSymbolMicro
s}\text{/}$ on average (minimum of $9.0\text{\,}\mathrm{\SIUnitSymbolMicro
s}\text{/}$, and maximum of $5380\text{\,}\mathrm{\SIUnitSymbolMicro
s}\text{/}$). We conclude that Zswap is an effective caching mechanism for the
swap space, since it provides significantly lower access latency than directly
accessing the swap device.

Figure 11. Zswap cache compression/decompression latency distribution. Dashed
(solid) lines represents average (maximum) compression/decompression latency.

We also study the system energy savings that the Zswap cache provides. Figure
12(a) shows the energy savings when Zswap is enabled for the Optane
configuration. We make three observations from the figure. First, we observe
that enabling Zswap reduces overall system energy consumption by 2$\times$.
Second, we observe that the majority of the energy is spent on write requests.
To understand these energy results, we analyze in Figure 12(b) the amount of
memory swap-in and swap-out activity during the execution of the test. As
shown in Figure 12(b), with Zswap cache enabled, swap-in and swap-out activity
reduces by 2.06$\times$ and 2.11$\times$, respectively. This large reduction
in swap activity directly translates to a reduction in energy consumption.
Third, even with Zswap enabled, the Optane+Zswap configuration consumes
34.75$\times$ the energy of the baseline configuration. This large increase in
energy consumption is due to (i) an increase in swap activity since the
Optane+Zswap configuration enables significantly more tab switches than the
baseline and (ii) the high energy cost of write operations to the Optane
device when swapping out pages (chen2012energy, ). We conclude that the Zswap
cache greatly reduces energy consumption due to a large reduction of swap
traffic. However, the Optane+Zswap configuration still significantly increases
the energy consumption compared to the baseline. We expect that such energy
can be further reduced by employing techniques to reduce the energy cost of
write operations on NVM devices (guo2018latency, ; choi2017nvm, ;
swami2016secret, ; chen2012energy, ).

(a) Effect of Zswap cache on system energy.

(b) Effect of Zswap cache on swap traffic.

Figure 12. Effect of Zswap cache on system energy and swap traffic.

R2/2/1Lifetime Analysis. One characteristic of NVM devices is their limited
write endurance, i.e., a memory cell in the Optane device becomes unreliable
beyond a certain number of writes. Therefore, we evaluate how Optane’s limited
write endurance affects the lifetime of our system when employing the Intel
Optane SSD as a swap space. We compare the Intel Optane SSD lifetime (in
years) when executing Chrome tab switching and scrolling activities in a
system with and without Zswap. To do so, we adopt the lifetime model in
(lee2009architecting, ), which estimates the lifetime of a memory module
driven by the access patterns observed in our Chrome workload. We assume a
conservative Optane cell endurance of $10^{6}$ writes (optaneendurance, )
(i.e., the same cell endurance of PCM-based memory cells (chang2016improving,
; aghaei2014prolonging, ; ferreira2010increasing, )) and an _optimistic_ wear-
leveling mechanism that evenly distributes write requests across all cells of
the Intel Optane media (which the Intel Optane SSD is reported to implement
(IntelOp22:online, )). Our model shows that the lifetime of the Optane
configuration without (with) Zswap enabled when executing our Chrome workload
is 4.5 (8.3) years. Such an expected lifetime can be hard to obtain in
practice because it is unlikely that the wear-leveling algorithm will be able
to distribute _all_ writes across the Optane device _equally_. However, prior
works propose more realistic wear-leveling mechanisms that can achieve up to
53% of the lifetime of an optimistic wear-leveling mechanism
(qureshi2009enhancing, ). Therefore, employing such a wear-leveling mechanism
can still guarantee a high lifetime for our Optane-based system without (with)
Zswap enabled of 2.4 (4.4) years.

Summarizing our findings, the Zswap cache is an effective caching mechanism
that reduces the swap traffic and system energy consumption when utilizing the
Intel Optane SSD as a swap space. These benefits come at the cost of a small
increase in the number of high-latency tab switches and a small decrease in
open tab count.

###  4.3. Effect of Using Different NVM Devices

In the previous sections, we show that enabling Intel Optane SSD as swap space
for consumer devices can provide significant benefits due to the extended main
memory space it provides. However, it is essential to understand if we can
achieve similar results using cheaper state-of-the-art NAND-flash-based SSDs
in place of the Intel Optane SSD. We aim to study whether state-of-the-art
NAND-flash-based SSDs that are already widely used (e.g., Micron (micronslc, )
and Transcend (transcendssd, ) NAND-flash-based SSDs) can provide similar
benefits for our workloads as we have observed using the Intel Optane SSD in
Sections 4.1 and 4.2.

There are three major differences between the Intel Optane SSD and a
traditional NAND-flash-based SSD: (1) lower access latency in the Intel Optane
SSD, (2) higher endurance in the Intel Optane SSD, and (3) higher cost of the
Intel Optane SSD device. First, as previous works (harris2020ultra, ;
lee2019asynchronous, ; zhang2018performance, ; chien2018characterizing, ;
yang2020exploring, ; hady2017platform, ; wu2019exploiting, ;
imamura2018reducing, ; wu2021storage, ) show, performing a
$4\text{\,}\mathrm{kB}\text{/}$ random read using the Intel Optane SSD is
approximately 6$\times$ faster than using a traditional NAND-flash-based SSD.
Second, the Intel Optane SSD can provide 10$\times$ the endurance of a
traditional NAND-flash-based SSD (optane_pe_cycles, ). Third, a traditional
NAND-flash-based SSD is approximately 3$\times$ cheaper than the Intel Optane
SSD ($0.50/GB  (optanePrice, ) vs. $1.5/GB  (ssdprice, ), respectively).

We compare the number of open Chrome tabs and the tab switch latency when
using the Intel Optane SSD (Optane configuration) versus an NVMe NAND-flash-
based SSD as the swap device (NANDFlash configuration). We choose a
$16\text{\,}\mathrm{GB}\text{/}$ M.2 NVMe NAND-flash-based SSD for our
experiments. We also evaluate the effect of enabling Zswap when using the
NAND-flash-based SSD (NANDFlash+Zswap).

R1/6Figure 13 compares the number of open Chrome tabs under five
configurations (baseline, Optane, Optane+Zswap, NANDFlash, NANDFlash+ZSwap).
We make two observations. First, the NANDFlash configuration enables 14% more
open tabs than the Optane configuration. This is due to the RAM_vs_swap_weight
parameter. This kernel parameter defines the effort (in terms of swap
activity) that the system will demand to allocate more memory. For our Optane
configuration, we empirically choose the RAM_vs_swap_weight value that gives
the best trade-off regarding swap activity, number of tabs open, and tab
switch latency. However, since we use the NANDFlash configuration only as a
reference, we utilize the default RAM_vs_swap_weight value that the kernel
suggests. Thus, even though the kernel can allocate more memory in the
NANDFlash configuration than in the Optane configuration, more Chrome tabs
result in higher swap activity. Second, when enabling a Zswap cache using 20%
of the DRAM capacity in the NANDFlash configuration (the same capacity as in
the Optane+Zswap configuration, we evaluate in Section 4.2), the number of
open tabs reduces by 12%, due to the decrease in the available DRAM space.

Figure 13. Number of open tabs: Baseline vs. Optane vs. NANDFlash.

Figure 14 shows the tab switch latency distribution (Figure 14(a)) and the
distribution of high-latency tab switches (Figure 14(b)) for all five
configurations. We make two observations from the figure. First, we observe
(in Figure 14(a)) that the high access latency of the NAND-flash-based SSD
leads to a significant increase in the tab switch latency in the NANDFlash
(NANDFlash+Zswap) configuration compared to the Optane (Optane+Zswap)
configuration. The average tab switch latency of the NANDFlash
(NANDFlash+Zswap) configuration is 3.6$\times$ (10$\times$) that of the Optane
(Optane+Zswap) configuration. Second, the number of high-latency tab switches
in the NANDFlash configuration (Figure 14(b)) increases by 35% compared to the
baseline configuration, and by 39% compared to the Optane configuration. For
high tab counts, the fraction of tabs with high latency is as much as 70% (for
100–120 tabs) in the NANDFlash+Zswap configuration. We conclude that the
NANDFlash configuration provides benefits compared to the baseline
configuration, but it is unable to approach the performance of the Optane
configuration.

(a) Tab switch latency distribution. Dashed lines represent Zswap enabled.

(b) High-latency tab switch distribution.

Figure 14. Tab switch latency: Baseline vs. Optane vs. NANDFlash.

Summarizing our findings, using a state-of-the-art NAND-flash-based SSD to
extend the main memory capacity of the system improves performance compared to
the baseline, considering the number of additional open tabs the NANDFlash
configuration provides. However, it cannot achieve similar performance as
using the Intel Optane SSD to extend the main memory capacity due to the high
access latency of the NAND-flash-based SSD, which leads to a significantly
larger number of high-latency tab switches than the Intel Optane SSD. Thus,
the Intel Optane SSD can lead to a better user experience than a state-of-the-
art NAND-flash-based SSD.

##  5\. System Optimization

Using the Intel Optane SSD as swap space allows our system to enjoy an
extended memory space, which translates to an average latency improvement for
our workload at the cost of larger tail latencies. However, longer tail
latencies are usually not acceptable. Thus, it is essential to reduce tail
latency (i.e., the 99th-percentile latency in our tab switch latency
distributions) for interactive workloads since it affects how the user
experiences the system.

The goal of this section is to analyze the primary sources of latency
overheads that impact the tail latency in the system when we make use of the
Intel Optane SSD as swap space. We extensively profile the system when
executing the Google Chrome web browser to identify performance bottlenecks
caused by the added swap device. We observe that the Linux block I/O layer
increases both the average and the 99th-percentile latency for Chrome’s page
faults significantly, mostly due to I/O scheduling issues and queuing delays,
and overheads related to the I/O request completion mechanism. To solve these
issues, we tune the system parameters related to the Linux block I/O layer,
aiming to improve the 99th-percentile latency for tab switches.

In this section, we limit the number of open tabs in our experiments to 50, as
50 tabs are enough to generate moderate memory capacity pressure in our system
and thus examine tail latencies.

###  5.1. Profiling the Chrome Browser

We extensively profile the Google Chrome browser’s activity while running the
memory capacity pressure test when the Intel Optane SSD is employed as swap
space. We use the perf profiling tool (de2010new, ) to collect the execution
time breakdown of each Chrome tab (including kernel activity). Figure 15 shows
a simplified execution breakdown of one of the representative Chrome tabs that
demonstrates long-latency switching times. We observe from the figure that the
tab spends more than 96% of its execution time on kernel modules that manage
I/O requests (i.e., the do_page_fault kernel function, which issues I/O
requests in case of a page fault operation; and the blk_mq_complete_request
kernel function, which receives and completes the processing of I/O requests
issued to the swap device). We observe that (i) 50% of the execution time is
spent on issuing a block I/O request due to page faults, and (ii) 46% of the
execution time is spent on processing the requested page once the data is
received from the swap device. The remaining 4% of the execution time is spent
on Chrome’s internal processes and other kernel calls. Therefore, we conclude
that the block I/O layer is the _primary_ source of the tail latency overhead.

Figure 15. perf results for a Chrome tab.

###  5.2. Linux Block I/O Layer

The block I/O layer is the Linux kernel layer responsible for managing block
I/O devices (e.g., magnetic hard drives, SSDs) (bovet2005understanding, ;
chen2012energy, ). It is a key system component since accessing block I/O
devices involves issuing high-latency and power-hungry operations to the block
device. Therefore, the block I/O layer is highly optimized to ensure low
latency and high throughput from block devices. Figure 16 illustrates the
primary operations the block I/O layer performs when an application or the
kernel issues an I/O request (e.g., read from a file, page fault). The block
I/O layer works in three main steps.

Figure 16. Linux block I/O layer.

R1/7  Figure 17. Q2C, Q2D, and D2C latency distribution for (i) Chrome
processes, (ii) kswapd, and (iii) Chrome processes+kswapd (_All_) during the
execution of the memory capacity pressure test. Error bars depict the minimum
and maximum data point values, and a bubble depicts average value of each
category.

First, when the block I/O layer receives a block I/O request, it queues the
request in a request queue (called _software queue_), which is unique per CPU.
Second, it attempts to merge and sort requests based on the sector number to
avoid costly seek operations in the device. Third, it schedules the requests
using an I/O scheduler. The scheduled request is stored in a dispatch queue
(called _hardware dispatch queue_). Requests stored in the hardware dispatch
queue are issued to the device driver and eventually reach the block device.
Once the device completes executing the request, it sends the response back to
the block I/O layer, which finalizes the execution of the request by either
(1) waking up the requester, in case an interrupt-based (IRQ) I/O request
completion mechanism is employed (bovet2005understanding, ); or (2) forwarding
the response to the requester, in case a polling-based I/O request completion
mechanism is employed (bovet2005understanding, ). Recent Linux kernels have
adopted a multi-queue (MQ) block I/O layer (bjorling2013linux, ), since modern
block devices (e.g., NVMe devices (tavakkol2018flin, )) can execute many
requests concurrently (bjorling2013linux, ; tavakkol2018mqsim, ;
tavakkol2018flin, ). The MQ block I/O layer employs one request queue per CPU
core for each block device.

To monitor and analyze the performance implications of the block I/O layer in
the system, we make use of the blktrace tool (brunelle2007blktrace, ).
blktrace monitors the activity of the block I/O layer, and provides detailed
timing information for each major operation. We analyze three important
timings that the tool reports: _(i) queue-to-device ( Q2D)_, the time from
when a block I/O request enters the block I/O layer to the time the request is
issued to the block device, including queuing, merging, and scheduling (❶ in
Figure 16); _(ii) device-to-completion ( D2C)_, the time it takes the block
device to complete the request (❷ in Figure 16); and _(iii) queue-to-
completion ( Q2C)_, the total end-to-end time for a block I/O request to
complete, i.e., Q2C = Q2D \+ D2C (❸ in Figure 16).

During the execution of the memory capacity pressure test, we profile the
block I/O layer using blktrace. We analyze the latencies for I/O requests that
Chrome processes and the kernel memory management unit (kswapd) issue. Figure
17 shows the Q2C, Q2D, and D2C latencies during the execution of the test. The
end-to-end block I/O latency (i.e., Q2C latency) is
$1.80\text{\,}\mathrm{ms}\text{/}$, on average across both Chrome processes
and kswapd read and write I/O requests (min. of
$0.0085\text{\,}\mathrm{ms}\text{/}$, max. of
$414.11\text{\,}\mathrm{ms}\text{/}$). The Q2D latency is
$0.03\text{\,}\mathrm{ms}\text{/}$, on average (min. of
$0.00087\text{\,}\mathrm{ms}\text{/}$, max. of
$413.80\text{\,}\mathrm{ms}\text{/}$). The D2C latency is
$1.78\text{\,}\mathrm{ms}\text{/}$, on average (min. of
$0.00723\text{\,}\mathrm{ms}\text{/}$, max. of
$46.10\text{\,}\mathrm{ms}\text{/}$). We make three observations from the
reported latencies. First, Chrome processes (chrome, Chrome_IOThread, and
CompositorTileW) issue high-latency I/O requests (up to
$414.11\text{\,}\mathrm{ms}\text{/}$). Most of these requests are read
requests caused by page faults. Second, kswapd is responsible for issuing a
majority of the write requests to the I/O device.777The kswapd process issues
_only_ write I/O requests since its goal is to free memory by reclaiming
inactive memory pages. If an inactive memory page is dirty, the kswapd process
writes this dirty page to swap space. Third, for high-latency I/O requests,
most of the I/O latency comes from the block I/O layer rather than from the
swap device. While the device latency (i.e., D2C) is at most
$46.10\text{\,}\mathrm{ms}\text{/}$, the latency of queuing and scheduling
requests in the block I/O layer (i.e., the Q2D latency) dominates the
execution time of high-latency I/O requests (as observed by the high maximum
Q2D latency of $413.80\text{\,}\mathrm{ms}\text{/}$).

Based on this analysis, we conclude that the block I/O layer operations are
the primary bottleneck in high-latency block I/O requests. To alleviate this
bottleneck, we investigate how two system optimizations impact block I/O
latencies and, consequently, Chrome performance. We investigate the effect of
(1) different block I/O schedulers (Section 5.3) and (2) different I/O request
completion mechanisms (Section 5.4) on system’s performance.

###  5.3. Optimization 1: Block I/O Schedulers

Figure 18. Four Linux block I/O layer schedulers.

The block I/O layer of the Linux kernel provides four different multi-queue
I/O schedulers. These schedulers vary in complexity, and their aim and ability
to consider and exploit different properties of different block devices.
Therefore, it is essential to tailor the system to make use of the I/O
scheduler that matches the requirements and characteristics of the Intel
Optane SSD.

We first briefly explain each of the four I/O schedulers. The four I/O
schedulers are called None (bjorling2013linux, ), Kyber (blkmqKyb12, ), MQ-
Deadline (mqdeadline, ), and budget-fair queuing (BFQ)
(bovet2005understanding, ). Figure 18 illustrates the operation of the four
I/O schedulers. The None I/O scheduler (❶ in Figure 18) is the simplest I/O
scheduler. It employs a simple first-in first-out (FIFO) request queue.
Therefore, it does _not_ reorder requests. The request queue includes both
read and write I/O requests. Due to its simplicity, the None I/O scheduler
incurs low overhead, but does not guarantee any quality-of-service.

The Kyber I/O scheduler (❷ in Figure 18) maintains two separate request
queues, one for synchronous block I/O read requests, and another for
asynchronous block I/O write requests. It prioritizes requests in the read
queue over those in the write queue, unless a write request has been
outstanding for too long, i.e., it times out by reaching the target write
access latency (the default target write access latency for the Kyber I/O
scheduler is $10\text{\,}\mathrm{ms}\text{/}$ (blkmqKyb12, )). As such, it is
also a simple I/O scheduler that aims to provide better service to read
requests.

The MQ-Deadline I/O scheduler (❸ in Figure 18) employs three different request
queues: read FIFO, write FIFO, and a sorted FIFO. The sorted FIFO maintains
read and write I/O requests that are sorted by the sector number they are to
access. The scheduler prioritizes I/O requests from the sector-sorted FIFO
unless any request in either the read or write queue is about to violate its
service deadline. The default deadline is $500\text{\,}\mathrm{ms}\text{/}$
for read I/O requests, and $5\text{\,}\mathrm{s}\text{/}$ for write I/O
requests.

The BFQ I/O scheduler (❹ in Figure 18) is the most complex I/O scheduler among
the four, which leads to high scheduling overhead. The BFQ I/O scheduler
guarantees fairness across processes by distributing the throughput of the
block I/O device proportionally to each process via an indirectly assigned
weight value. The BFQ I/O scheduler employs an I/O request queue per process
and assigns an I/O budget per I/O request queue. It assigns I/O budgets,
measured in number of sectors, proportionally to a process’s I/O activity.
This way, I/O-bound processes with sequential I/O requests are assigned a
large I/O budget, while processes with short and sporadic I/O requests are
assigned a small I/O budget. The BFQ I/O scheduler uses a variant of the
worst-case fair weighted fair queuing+ (WF2Q+) scheduling algorithm
(bennett1997hierarchical, ) to select an I/O request queue to be serviced
(typically the I/O request queue with the lowest I/O budget). The selected I/O
request queue is prioritized, and its I/O requests are exclusively serviced
until its I/O budget finishes. As a result, the BFQ I/O scheduler guarantees a
fraction of the device throughput to each process. The Linux kernel uses the
BFQ I/O scheduler by default, since it usually provides high system
responsiveness and fairness, even though its scheduling decision overhead is
higher than the other three I/O schedulers.

Figure 19(a) shows the tab switch latency for the Optane configuration under
the four different I/O schedulers. We observe that, on average, the four
different I/O schedulers provide similar tab switch latencies. The average tab
switch latency for the None, Kyber, MQ-Deadline, and BFQ I/O schedulers is
$116\text{\,}\mathrm{ms}\text{/}$, $119\text{\,}\mathrm{ms}\text{/}$,
$120\text{\,}\mathrm{ms}\text{/}$, and $118\text{\,}\mathrm{ms}\text{/}$,
respectively. However, when we examine the fraction of high-latency tab
switches in Figure 19(b), we observe that the fraction of high-latency tab
switches increases significantly with open tab count for the default BFQ I/O
scheduler. The BFQ I/O scheduler leads to the largest number of high-latency
tab switches when the system has 41–50 tabs open, while the Kyber I/O
scheduler provides the lowest number of high-latency tab switches for the same
tab-count range, reducing the fraction of high-latency tab switches by 44%
compared to the BFQ I/O scheduler. Therefore, to reduce Chrome’s tail latency,
the Kyber I/O scheduler can potentially be a better I/O scheduler than the
default BFQ I/O scheduler.

(a) Tab switch latency distribution. Horizontal dashed lines represent average
values.

(b) High-latency tab switch distribution.

Figure 19. Tab switch latency: Optane with different I/O schedulers.

We further analyze the tab switch latency distribution in Figure 20 for the
four I/O schedulers. The figure shows the tab switch latency percentiles
(along the x-axis) and the corresponding tab switch latency (y-axis),
normalized to the values when the system employs the default BFQ I/O
scheduler. We make three observations. First, at the tail (99th-percentile)
latency, the None, Kyber, and MQ-Deadline I/O schedulers significantly reduce
the tab switch latency compared to the default BFQ I/O scheduler, by 29%, 35%,
and 18%, respectively. Second, the BFQ I/O scheduler provides the lowest tab
switch latency for latency percentiles outside the tail. The None/Kyber/MQ-
Deadline I/O schedulers slightly increase BFQ’s tab switch latency by
3%/3/%/10%, 4%/4%/4%, and 5%/4%/5% at the 50th-, 60th-, and 70th-percentile
latencies, respectively. Third, when moving closer to the latency percentiles
at the tail (i.e., from the 80th-percentile latency to the 95th-percentile
latency), we observe that the None, Kyber, and MQ-Deadline I/O schedulers all
provide lower tab switch latencies than the BFQ I/O scheduler. By employing
the Kyber/MQ-Deadline I/O schedulers, the tab switch latency reduces by 3%/4%
and 7%/6% compared to the BFQ I/O scheduler for the 80th- and 90th-percentile
latencies. At the 95th-percentile latency, the None I/O scheduler reduces the
tab switch latency by 12% compared to the BFQ I/O scheduler. Therefore, we
conclude that even though the default BFQ I/O scheduler provides good
performance overall (except at the tail), alternative I/O schedulers can
greatly improve tail latency performance, thereby improving user experience.

Figure 20. Normalized tab switch latency of four schedulers, categorized
across different latency percentiles. Y-axis is normalized to the default BFQ
I/O scheduler.

To further understand the I/O schedulers’ impact on Chrome browser
performance, we analyze the Q2C latency (i.e., the end-to-end I/O request
latency) for each I/O scheduler. Figure 21 depicts the end-to-end I/O request
(i.e., Q2C) latency percentiles (along the x-axis) and the corresponding Q2C
latency (y-axis), normalized to the Q2C latency values for the BFQ I/O
scheduler. We make three observations. First, the None I/O scheduler reduces
Q2C latency in a majority of the latency percentiles. However, such a
reduction does _not_ directly translate to tab switch latency improvements (as
seen in Figure 20). This happens because the None I/O scheduler does not
enforce any ordering among read and write I/O requests. Since (i) Chrome
mostly issues read I/O requests during the execution of our test (as Figure 17
shows) and (ii) the Intel Optane SSD internally handles read and write
requests equally (as characterized by prior work (wu2019towards, )), the None
I/O scheduler delays the execution of Chrome’s read I/O operations by
executing write I/O requests, which hurts Chrome’s performance. Second, the
Kyber I/O scheduler provides the best Q2C reduction for the 99th-percentile
latency, which directly translates to better tab switch latency (Figure 20).
The Kyber I/O scheduler improves Chrome’s performance since it better fits
Chrome’s I/O request characteristics by: (i) reducing the average I/O queuing,
merging, and scheduling latencies (i.e., Q2D latencies) by up to 3.7$\times$
that of the BFQ I/O scheduler (not shown); and (ii) prioritizing read I/O
requests over writes.  Third, even though Kyber’s Q2C latency is larger than
BFQ’s Q2C latency (Figure 21) at the 90th-percentile latency, Kyber’s Q2C
latency for read requests is slightly lower (by 15%) than BFQ’s Q2C latency
for read requests (not shown), which translates to a faster tab switch latency
for Kyber at the 90th-percentile latency. However, at the 95th-percentile
latency, Kyber’s Q2C latencies for both read and write requests are larger
than BFQ’s Q2C latencies. At the 99th-percentile latency, the system’s high
memory capacity pressure highlights the high overhead of the BFQ I/O
scheduler, leading to a significant increase in BFQ’s Q2C latency for both
read and write requests. BFQ’s Q2C latency increases by 99% from the 95th-
percentile latency to the 99-percentile latency. In contrast, Kyber’s Q2C
latency increases by only 19% when moving from the 95th-percentile latency to
the 99-percentile latency. We conclude that the Kyber I/O scheduler can reduce
tail latency for the Chrome browser since it (i) provides low overhead I/O
scheduling decisions for already critical I/O requests while (ii) matching the
access pattern of our workload by prioritizing read accesses over writes.

Figure 21. Q2C latency of four I/O schedulers. Y-axis is normalized to the
default BFQ I/O scheduler.

R2/4BEnergy Analysis. Figure 22 compares the impact of the different I/O
schedulers (x-axis) on the average memory subsystem energy consumption (which
includes the energy consumption of main memory and swap space) for the Optane
configuration (y-axis; normalized to the baseline BFQ I/O scheduler). We use
the same energy model described in Section 4.1 for our analysis. We make two
key observations from the figure. First, we observe from the figure that all
four I/O scheduler mechanisms achieve a _similar_ memory subsystem energy
consumption during the execution of our test. Second, the Kyber I/O scheduler
slightly _increases_ energy consumption by 6.9%, while the None I/O scheduler
slightly _decreases_ energy consumption by 6.3% compared to the baseline BFQ
I/O scheduler. This is due to an increase in the number of write I/O requests
the Kyber I/O scheduler produces compared to the None I/O scheduler. Recall
that the Kyber I/O scheduler uses dedicated queues for read and write quest
and dispatch write requests using a pre-defined threshold, while the None I/O
scheduler uses a single queue for read and write requests and dispatch both
read/write requests using a first-in-first-out approach. This leads to the
Kyber I/O scheduler prioritizing more write requests from background kernel
processes than the None I/O scheduler and the baseline BFQ mechanism. In
contrast, the None I/O scheduler serves the processes that generate an I/O
request first (in our case, Chrome processes issuing read I/O requests due to
moderate-to-high swap activity). We conclude that since the available I/O
schedulers mostly target improving the throughput of block I/O devices, they
achieve a similar memory subsystem energy consumption.

R2/4Bfig

Figure 22. Energy consumption: Optane with different I/O schedulers. Y-axis is
normalized to the default BFQ I/O scheduler.

We conclude that, we can reduce especially the block I/O tail latency by
employing different I/O schedulers (e.g., Kyber) from the default Linux block
I/O scheduler. However, the high latency overhead related to managing I/O
requests is still large in comparison to the actual device time (as Figure 17
shows). Therefore, we evaluate a second optimization in Section 5.4.

###  5.4. Optimization 2: Interrupts vs. Polling Based I/O Request Completion

Another key component of the Linux block I/O layer that directly impacts I/O
performance is the I/O request completion mechanism. There are two main I/O
request completion mechanisms in current Linux systems: (1) interrupt-based
(i.e., IRQ-based) I/O request completion (bovet2005understanding, ) and (2)
polling-based I/O request completion (yang2012poll, ). Interrupt-based I/O
request completion employs an asynchronous operation model. When a process
issues a block I/O request, the OS puts the process to sleep and context
switches to another process. When the I/O response arrives, the device driver
receives an interrupt and wakes up the sender process. In contrast, polling-
based I/O request completion employs a synchronous operation model. When a
process issues a block I/O request, the process continuously polls in the CPU
waiting for the I/O request to complete (i.e., instead of going to sleep, the
process continually executes CPU instructions to check the current status of
the I/O request).

While interrupt-based I/O request completion may incur large system overheads
due to context switching, polling-based I/O request completion imposes a high
CPU load to the system. Previous work (le2017latency, ) proposes a hybrid I/O
request completion mechanism, which targets fast NVM devices. In this hybrid
I/O request completion mechanism, when a process issues an I/O request, the OS
puts the process to sleep, similar to the IRQ mode. However, to remove the
context switch latency from the critical path of I/O request completion, the
OS wakes up the process after some predefined sleep delay time t. Then, the
process polls the I/O request queue for completion of its request until the
response arrives from the block device. The hybrid I/O request completion
mechanism can improve performance compared to polling since it reduces the
number of CPU cycles spent on polling.

The hybrid I/O request completion mechanism works in two modes: (1)  _fixed
latency_ , where the user sets the sleep delay time t to a specific latency;
and (2)  _adaptive latency_ , where the OS dynamically sets the sleep delay
time t by attempting to estimate when the I/O request will complete
(Queuesys14:online, ). In the adaptive latency mode, the OS monitors the
completion time of the different types of I/O requests, and then utilizes half
of the average of the I/O request completion time for a particular I/O request
type as the sleep delay time t for future I/O requests of that type. Based on
this estimation, the OS puts the process that issues I/O requests to sleep
before entering a polling loop. The adaptive latency mode is enabled by
setting t to 0.

We evaluate how different I/O request completion mechanisms impact the Chrome
browser performance. The OS employs the interrupt-based I/O request completion
mechanism by default. However, Intel recommends enabling the Hybrid technique
when using the Intel Optane SSD for enterprise computing (hybridoptane, ).
Previous work (le2017latency, ) advocates that fast NVM-based devices can
benefit from the polling-based I/O request completion mechanism, since the
context switch latency incurred during interrupt-based I/O request completion
can be larger than the device access latency. We evaluate all three I/O
request completion mechanisms to identify the best-performing one to use for
Chrome with an Intel Optane SSD used as swap space. In the Hybrid I/O request
competition mechanism, we evaluate the adaptive latency mode (by setting t =
0) and fixed latency mode, where we evaluate two values of t
($2\text{\,}\mathrm{\SIUnitSymbolMicro s}\text{/}$ and
$4\text{\,}\mathrm{\SIUnitSymbolMicro s}\text{/}$).

Figure 23 shows the tab switch latency distribution (Figure 23(a)) and the
distribution of high-latency tab switches for the three I/O completion
mechanisms (Figure 23(b)) for the Optane configuration with Zswap enabled,
when we employ three different I/O request completion mechanisms: (1)
interrupt-based (IRQ); (2) polling-based (Polling); and (3) Hybrid with t = 0
(i.e., adaptive latency mode), t = $2\text{\,}\mathrm{\SIUnitSymbolMicro
s}\text{/}$, and t = $4\text{\,}\mathrm{\SIUnitSymbolMicro s}\text{/}$. We
make three observations. First, the interrupt-based mechanism provides the
lowest average tab switch latency, with an average tab switch latency
reduction of 60% compared to the Polling mechanism, which provides the highest
average tab switch latency; and 11% compared to the Hybrid (t = 2) mechanism,
which provides the second-best average tab switch latency (Figure 23(a)).
Second, on average across tab counts, the interrupt-based I/O request
completion mechanism leads to the lowest number of high-latency tab switches,
with only 2.3% of tab switches being high latency, versus 3.9%/4.8%/7.8%/3.2%
from the Polling, Hybrid (t=0), Hybrid (t=2), and Hybrid (t=4) mechanisms,
respectively (Figure 23(b)). Third, we observe that the Polling mechanism
eliminates high-latency tab switches when 11–30 tabs are opened. With only a
small number of open Chrome tabs (1–10 tabs), Chrome issues few I/O requests
due to the system’s low swap traffic (as Figure 1 shows). In such case, the
Polling mechanism increases the number of high-latency tab switches compared
to the interrupt-based I/O completion mechanism since, in the event of an I/O
request, the system cannot context switch to Chrome tab, which will likely not
issue an I/O request, thus increasing tab switch latency. On the other hand,
when the number of open Chrome tabs increases, and consequently the swap
traffic (Figure 1), the system can sometimes leverage idle CPU time to wait
for the completion of Chrome’s I/O requests, reducing I/O request latency. We
conclude that, on average, maintaining the interrupt-based I/O request
completion mechanism is still the best approach for consumer devices. However,
enabling polling can sometimes be a good alternative.

(a) Tab switch latency distribution. Horizontal dashed lines represent average
values.

(b) High-latency tab switch latency distribution.

Figure 23. Tab switch latency: Optane with different I/O completion
mechanisms.

Figure 24 shows the tab switch latency distribution for the three different
I/O completion mechanisms. We make two observations. First, the interrupt-
based (IRQ) I/O completion mechanism provides the lowest tab switch latency of
the evaluated mechanisms up to the 95th-percentile latency. On average, it
reduces tab switch latency by 12%, 14%, 20%, and 17% compared to the Polling,
Hybrid (t=0), Hybrid (t=2), and Hybrid (t=4) mechanisms, respectively. Second,
when examining the 99th-percentile latency, the Hybrid (t=0) mechanism reduces
the tab switch latency by 7% compared to the IRQ mechanism. This happens
because at high memory capacity pressure, most of the processes are waiting
for I/O. As a result, the OS has few opportunities (if any) to switch in a
process that can make forward progress, and there is therefore little to no
performance cost in keeping the waiting processes awake to poll the CPU and
eliminating the context switch overhead that they would incur with IRQ. We
conclude that the Hybrid I/O request completion mechanism is a good solution
to reduce tail latency for Chrome when using an Intel Optane SSD.

Figure 24. Tab switch latency distribution: different I/O completion
mechanisms.

R2/4CEnergy Analysis. Figure 25 compares the impact of the different I/O
completion mechanisms (x-axis) on the average memory subsystem energy
consumption for the Optane configuration (y-axis; normalized to the baseline
IRQ completion mechanism). We use the same energy model described in Section
4.1 for our analysis. The figure shows that the different I/O completion
mechanisms have little to no impact on the average memory subsystem energy
consumption. This is because such mechanisms do _not_ employ any optimization
targeting the reduction of I/O traffic or prioritization of I/O requests.

R2/4Cfig

Figure 25. Energy consumption: Optane with different I/O completion
mechanisms. Y-axis is normalized to the default IRQ I/O completion mechanism.

We conclude that (1) the interrupt-based I/O request completion mechanism
provides the best average performance for the Chrome web browser, but (2) at
the tail latency (i.e., 99th-percentile latency), the hybrid I/O request
completion mechanism can further reduce I/O request latency. We, therefore,
believe that there needs to be further research into new I/O request
completion mechanisms that provide both the best average and tail performance.

##  6\. Key Takeaways

To summarize, our experimental analysis reveals that extending the main memory
space by using the Intel Optane SSD as NVM-based swap space for DRAM provides
a cost-effective way to alleviate DRAM scalability issues. However, naively
integrating the Intel Optane SSD into the system leads to several system-level
overheads that can negatively impact overall performance and energy
efficiency. We mitigate such overheads by examining and evaluating system
optimizations driven by our analyses.

We provide the following six key takeaways from our empirical analyses:

  1. (1)

Effect of Intel Optane SSD as swap space (Section 4.1). Reducing DRAM size and
extending the main memory space with the Intel Optane SSD as swap space
provides benefits for the Chrome browser, since it can (a) increase the number
of open tabs, and (b) reduce system cost. However, it also leads to an
increase in the number of tab switches with high latency compared to the
baseline.

  2. (2)

Reducing tail latency by enabling Zswap (Section 4.2). Zswap is a good
mechanism to reduce I/O traffic introduced by the Intel Optane SSD, at the
cost of a small increase in tab switch latency at large tab counts. The Zswap
cache reduces system energy by 2$\times$ (compared to the Intel Optane SSD
without Zswap enabled), at the cost of increasing the high-latency tab
switches by 4% and reducing the number of open tabs by 12%.

  3. (3)

Effect of using different NVM devices (Section 4.3). A state-of-the-art NAND-
flash-based SSD provides benefits over both the baseline and the Intel Optane
SSD. Importantly, it enables more Chrome tabs to be open. These benefits come
due to the larger effective main memory capacity provided by the state-of-the-
art NAND-flash-based SSD over the baseline configuration. Unfortunately, these
benefits come at the cost of higher tab switch latencies, compared to both the
baseline and Optane configurations, due to the much longer device latencies of
NAND flash memory. These large tab switch latencies degrade user experience.
Taking both performance and user experience into account, emerging NVM-based
SSDs such as the Intel Optane SSD are quite promising to employ in consumer
devices, providing performance benefits without the undesirable user
experience trade-offs incurred by NAND-flash-based SSDs.

  4. (4)

System bottlenecks caused by NVMs (Section 5.1). The Linux block I/O layer is
a key system bottleneck when the Intel Optane SSD is used as swap space. We
can mitigate some of the overheads caused by the block I/O layer by (a)
employing an I/O scheduler that meets the requirements of the application’s
access pattern and (b) using different I/O request completion mechanisms.

  5. (5)

Optimization 1: block I/O schedulers (Section 5.3). We can reduce tab switch
latency by changing the default BFQ I/O scheduler in the system that uses the
Intel Optane SSD as swap space. We reduce 95th- and 99th-percentile latencies
by employing the None and the Kyber I/O schedulers, respectively, as those I/O
schedulers reduce I/O scheduling overheads and fit the I/O access pattern of
the Chrome web browser.

  6. (6)

Optimization 2: interrupt- vs. polling-based I/O request completion (Section
5.4). On average, the interrupt-based I/O request completion mechanism
provides the best performance for the system with the Intel Optane SSD device.
However, the Hybrid I/O request completion mechanism can help reduce 99th-
percentile latency for block I/O requests.

Based on our analysis, we conclude that there is a large optimization space to
be explore in order to _efficiently_ adopt emerging NVMs in consumer devices.
For example, we believe that one of the main issues the system suffers from
when executing interactive workloads is that scheduling decisions made by the
OS do not consider the response time expected by the workload. Exposing such
information to the OS could reduce tail latency and allow the scheduler to
take action according to the needs of a particular workload (e.g., by
prioritizing the workload with the shorter or more urgent response deadlines).
We leave the design, implementation, and evaluation of such ideas for future
work.

###  6.1. Overall Limitations of the Technology

R2/4DEven though employing the Intel Optane SSD as a swap space can lead to
several benefits in terms of cost and performance, it can also impact overall
system energy consumption and lifetime. We provide the following two key
takeaways from our empirical analyses that highlight the limitations of NVM-
based swap space in consumer devices:

  1. (1)

Effect of Intel Optane SSD as swap space on energy consumption. Integrating
Intel Optane SSD as a swap space increases average memory subsystem energy
consumption to 69.5$\times$ that of the baseline ZRAM-based system
configuration (Section 4.1; Figure 6). This happens due to the higher swap
activity enabled by the Optane-based swap space (Section 4.1). Such an
increase in energy consumption can be mitigated by employing a Zswap cache,
which reduces the increase caused by the Optane-based swap space to
34.75$\times$ that of the baseline (Section 4.2; Figure 12(a)). Unfortunately,
tuning the block I/O scheduler (Section 5.3) and I/O completion mechanism
(Section 5.4) do _not_ lead to significant energy savings for the Optane
configuration, since such optimizations primarily target improving the
throughput and latency of I/O operations, rather than reducing energy
consumption.

  2. (2)

Effect of Intel Optane SSD as swap space on system lifetime. The Intel Optane
SSD, as an NVM-based device, suffers from limited write endurance, which can
impact the lifetime of the system. Based on our analysis (Section 4.2), we
observe that it would take an Optane-based system (without Zswap) running our
Chrome web browser 4.5 years to experience a write-endurance failure. Enabling
Zswap increases the lifetime of the Optane-based system to 8.3 years.

Many prior works (lee2009architecting, ; song2020improving, ;
yoon2013techniques, ; choi2017nvm, ; guo2018latency, ; yavits2020wolfram, ;
aghaei2014prolonging, ; chang2016improving, ; chen2012age, ;
cheng2016efficient, ; fan2014wl, ; han2015enhanced, ; im2014differentiated, ;
joo2010energy, ; liu2014application, ; qureshi2009enhancing, ;
qureshi2011practical, ; zhou2009durable, ) aim to reduce the impact of
emerging NVMs on overall system energy consumption and lifetime. The great
majority of such works aim to (i) reduce the number of write operations the
system issue to the NVM device using techniques such as caching
(lee2009architecting, ; yoon2013techniques, ), write-aware data mapping and
data allocation algorithms (song2020improving, ; yoon2013techniques, ;
choi2017nvm, ), and data compression (guo2018latency, ); and (ii) distribute
write operations across NVM cells using diverse wear-leveling techniques
(yavits2020wolfram, ; aghaei2014prolonging, ; chang2016improving, ;
chen2012age, ; cheng2016efficient, ; fan2014wl, ; han2015enhanced, ;
im2014differentiated, ; joo2010energy, ; liu2014application, ;
qureshi2009enhancing, ; qureshi2011practical, ; zhou2009durable, ). We believe
such approaches can be employed to mitigate the limitations of NVMs in
consumer devices. We leave such analyses for future work.

##  7\. Related Work

To our knowledge, this is the first work that (i) comprehensively analyzes the
impact of extending the main memory space of consumer devices using _real_
off-the-shelf emerging NVM-based SSDs, and (ii) proposes practical system-
level optimizations that can mitigate the tail latency of interactive
workloads when employing emerging NVM-based SSDs in the system. We discuss the
large body of related work on NVM using four broad categories.

Enabling NVM-Based Swap Space for Mobile Devices. Several past works
(zhong2014building, ; kim2015cause, ; liu2017non, ; zhong2017building, ;
kim2019analysis, ; zhu2017smartswap, ; kim2018comparison, ) investigate how to
efficiently enable swap-based NVMs for mobile devices. Unlike our work, these
past works do not utilize _real_ NVM devices to evaluate their mechanisms or
their system-level implications on a _real_ mobile system. Thus, it is not
fully clear if their results and insights can be easily translated to a real
system employing a real NVM device. We briefly describe the key mechanisms
proposed by each of these works.

Two prior works (zhong2014building, ; liu2017non, ) propose to improve swap
performance and the lifetime of byte-addressable NVM devices being used as
swap space in smartphones. These works emulate swapping behavior by creating a
swap area inside DRAM (similar to our ZRAM configuration). CAUSE
(kim2015cause, ) is a hybrid memory architecture for mobile devices that
leverages application access patterns to allocate memory either in DRAM or in
NVM within a hybrid DRAM-NVM memory architecture. Similarly, Kim et al.
(kim2019analysis, ) employ an NVM-based swap space for Android devices, which
leverages hot/cold data to manage swap activity between DRAM and NVM. Zhong et
al. (zhong2017building, ) aim to reduce write endurance issues related to NVM
devices by identifying and swapping cold pages from DRAM to NVM-based swap
space in smartphones. SmartSwap (zhu2017smartswap, ) predicts the most-rarely-
used applications to be dynamically swapped to a flash-memory-based swap space
ahead of time. Kim et al. (kim2018comparison, ) compare two swap space
organizations for mobile devices: (1) a hierarchical swap architecture, where
NVM-based swap space is used as a cache for a larger flash-memory-based swap
space; and (2) a hybrid swap architecture, where both NVM and flash devices
are used as a single-level swap space. As part of this work, the authors
propose SPP-CLOCK (kim2018comparison, ), a mechanism to identify hot/cold data
to manage swap activity.

We believe that many of the mechanisms proposed by these prior works can be
adapted to be employed in our system to further improve performance and
lifetime. We leave such studies to future works.

Improving Block I/O Latency for Fast NVMe Devices. Previous works
(lee2019asynchronous, ; lee2020case, ; tavakkol2018flin, ; oh2020h, ;
shin2014path, ; vuvcinic2014dc, ; zhang2018flashshare, ; liu2022towards, )
propose several techniques to mitigate block I/O latencies for fast NVMe
devices. These techniques include software (lee2019asynchronous, ; oh2020h, ;
shin2014path, ; vuvcinic2014dc, ; zhang2018flashshare, ; liu2022towards, ) and
hardware solutions (tavakkol2018flin, ; lee2020case, ) to provide lower I/O
access latency (lee2019asynchronous, ; vuvcinic2014dc, ; zhang2018flashshare,
), page fault handling (lee2020case, ), and I/O scheduling (tavakkol2018flin,
; oh2020h, ; liu2022towards, ). Even though these techniques are promising
solutions to reduce the high block I/O latencies, they require substantial
changes in the hardware and the software stack, which are outside the scope of
this work, but can also be used in our proposed system.

Another body of work (caulfield2012providing, ; kim2016nvmedirect, ;
scargall2020introducing, ; meza2013case, ; yang2017spdk, ; OpenMPDK, ;
peter2015arrakis, ; kim2017user, ; kwon2017strata, ) aims to completely remove
the block I/O layer from the system by providing programming models that
enable the user to directly access data from fast NVMe devices. Even though
this is a promising solution, it involves several challenges such as code
refactoring and security, which can be a promising direction for future work.

Real NVM Devices in Real Systems. Since the release of the Intel Optane SSD,
various works (wu2019exploiting, ; zhang2018performance, ; wu2017early, ;
jia2020flash, ; han2020splitkv, ; ke2018lirs, ; chien2018characterizing, ;
liu2020nvm, ; lu2021case, ) have experimentally shown that the Intel Optane
SSD can improve performance, energy, and cost for different workloads (e.g.,
databases (wu2019exploiting, ; zhang2018performance, ; lu2021case, ), high-
performance computing (wu2017early, ), key-value stores (jia2020flash, ;
han2020splitkv, ), machine learning (ke2018lirs, ; chien2018characterizing, ),
query processing (liu2020nvm, )). Our work differs from these works since we
(i) target a different family of workloads (i.e., interactive consumer
workloads, and in particular the Google Chrome web browser) and (ii) employ
the Intel Optane SSD as an extension of main memory, instead of as a separate
storage device.

Hybrid DRAM-NVM Memory Systems. A large body of works (lee2009architecting, ;
qureshi2009scalable, ; lee2010phase, ; lee2010phasecacm, ;
kultursay2013evaluating, ; zhou2009durable, ; DAC-2009-DhimanAR, ;
wang2019panthera, ; salkhordeh2019analytical, ; li2017utility, ; yoon2012row,
; meza2012enabling, ; singh2022sibyl, ; oh2015sqlite, ; li2022multi, ;
raybuck2021hemem, ) propose to use NVMs as an alternative technology to DRAM,
where the NVM completely replaces DRAM as the main memory device
(lee2009architecting, ; qureshi2009scalable, ; lee2010phase, ;
lee2010phasecacm, ; kultursay2013evaluating, ; zhou2009durable, ), or is
incorporated into the memory hierarchy alongside DRAM to create a hybrid
DRAM–NVM memory system (DAC-2009-DhimanAR, ; wang2019panthera, ;
salkhordeh2019analytical, ; li2017utility, ; yoon2012row, ; meza2012enabling,
; oh2015sqlite, ; raybuck2021hemem, ). Unlike our work, these works either (1)
use NVMs as part of main memory, and not as swap space, which increases the
complexity of the memory architecture; or (2) mainly leverage simulation
infrastructures to evaluate their proposals, and thus, do not examine _real_
NVM devices and their implications on real systems with real measurement data.

##  8\. Conclusion

R2/4EIn this paper, we comprehensively evaluate the performance implications
of leveraging real emerging NVMs as an extension of main memory space in real
consumer devices, while targeting interactive workloads. We employ a state-of-
the-art NVM-based SSD device (i.e., the Intel Optane SSD) as swap space for
DRAM, which increases the effective main memory capacity in our system. We
observe that using the Intel Optane SSD can improve the average and tail
latency performance of the Chrome web browser, compared to a baseline system
with double the amount of DRAM, and to a system where a state-of-the-art NAND-
flash-based SSD is used for the swap space. We identify that the Linux block
I/O layer becomes a major source of performance overhead when the main memory
space is extended using NVM, primarily due to (i) I/O scheduling bottlenecks;
and (ii) overheads related to the asynchronous operation of the I/O request
completion mechanism. We mitigate some of these overheads by proposing two
system optimizations that can better leverage the characteristics of our
workloads and the NVM. We also evaluate the limitations of real emerging NVMs
in consumer devices and conclude that real systems need to employ solutions to
mitigate the issues associated with energy increase and lifetime degradation
NVM devices introduce. We conclude that emerging NVMs are a cost-effective
solution to alleviate the DRAM capacity bottleneck in consumer devices. We
hope that the results of our study can inspire and drive novel hardware and
software optimizations in future NVM-based computing systems.

## Acknowledgments

We thank the SAFARI Research Group members for valuable feedback and the
stimulating intellectual environment they provide. We acknowledge support from
the SAFARI Research Group’s industrial partners, especially ASML, Facebook,
Google, Huawei, Intel, Microsoft, and VMware. We acknowledge support from the
Semiconductor Research Corporation and the ETH Future Computing Laboratory.
This research started at Google, during Geraldo F. Oliveira’s internship, and
continued as a successful collaboration between Google and SAFARI since then.

## References

  * (1) Google LLC, “Chromebook,” https://www.google.com/chromebook/.
  * (2) eMarketer, “Slowing Growth Ahead for Worldwide Internet Audience,” 2016.
  * (3) V. J. Reddi, H. Yoon, and A. Knies, “Two Billion Devices and Counting,” _IEEE Micro_ , 2018.
  * (4) ARM and Qualcomm, “Enabling the Next Mobile Computing Revolution with Highly Integrated ARMv8-A Based SoCs,” White Paper, 2014.
  * (5) M. Halpern, Y. Zhu, and V. J. Reddi, “Mobile CPU’s Rise to Power: Quantifying the Impact of Generational Mobile CPU Design Trends on Performance, Energy, and User Satisfaction,” in _HPCA_ , 2016.
  * (6) Canalys, “Chromebooks Lead PC Revival in Q1 2021 With 275% Growth,” https://rb.gy/jm7xu, 2021.
  * (7) B. Heater, “As Chromebook Sales Soar in Schools, Apple and Microsoft Fight Back,” _TechCrunch_ , 2017.
  * (8) R. H. Dennard, “Field-Effect Transistor Memory,” U.S. Patent 3,387,286, 1968.
  * (9) Y. Kim and O. Mutlu, “Memory Systems,” in _Computing Handbook, Third Edition: Computer Science and Software Engineering_. Taylor & Francis, 2014.
  * (10) D. P. Bovet and M. Cesati, _Understanding the Linux Kernel: From I/O Ports to Process Management_ , 3rd ed. O’Reilly Media, Inc., 2005.
  * (11) A. S. Tanenbaum and A. S. Woodhull, _Operating Systems: Design and Implementation_. Prentice Hall Englewood Cliffs, 1997.
  * (12) O. Mutlu, “Lecture Notes for Digital Design and Computer Architecture – Lecture 23b: Virtual Memory,” https://rb.gy/qzj7r, 2020.
  * (13) M. Badr, C. Delconte, I. Edo, R. Jagtap, M. Andreozzi, and N. E. Jerger, “Mocktails: Capturing the Memory Behaviour of Proprietary Mobile Architectures,” in _ISCA_ , 2020.
  * (14) A. Boroumand, S. Ghose, Y. Kim, R. Ausavarungnirun, E. Shiu, R. Thakur, D. Kim, A. Kuusela, A. Knies, P. Ranganathan _et al._ , “Google Workloads for Consumer Devices: Mitigating Data Movement Bottlenecks,” in _ASPLOS_ , 2018.
  * (15) J. Mohan, D. Purohith, M. Halpern, V. Chidambaram, and V. J. Reddi, “Storage on Your Smartphone Uses More Energy Than You Think,” in _HotStorage_ , 2017.
  * (16) A. Boroumand, “Practical Mechanisms for Reducing Processor-Memory Data Movement in Modern Workloads,” Ph.D. dissertation, Carnegie Mellon University, 2020.
  * (17) R. Nelson, “The Size of iPhone’s Top Apps Has Increased by 1,000% in Four Years,” https://sensortower.com/blog/ios-app-size-growth, 2017.
  * (18) N. Lebeck, A. Krishnamurthy, H. M. Levy, and I. Zhang, “End the Senseless Killing: Improving Memory Management for Mobile Operating Systems,” in _USENIX ATC_ , 2020.
  * (19) O. Mutlu, “Memory Scaling: A Systems Architecture Perspective,” in _IMW_ , 2013.
  * (20) O. Mutlu and L. Subramanian, “Research Problems and Opportunities in Memory Systems,” _SUPERFRI_ , 2015.
  * (21) Y. Kim, R. Daly, J. Kim, C. Fallin, J. H. Lee, D. Lee, C. Wilkerson, K. Lai, and O. Mutlu, “Flipping Bits in Memory Without Accessing Them: An Experimental Study of DRAM Disturbance Errors,” in _ISCA_ , 2014.
  * (22) O. Mutlu and J. S. Kim, “RowHammer: A Retrospective,” _TCAD_ , 2019.
  * (23) J. S. Kim, M. Patel, A. G. Yağlıkçı, H. Hassan, R. Azizi, L. Orosa, and O. Mutlu, “Revisiting RowHammer: An Experimental Analysis of Modern DRAM Devices and Mitigation Techniques,” in _ISCA_ , 2020.
  * (24) O. Mutlu, “Main Memory Scaling: Challenges and Solution Directions,” in _More than Moore Technologies for Next Generation Computer Design_ , 2015.
  * (25) U. Kang, H.-S. Yu, C. Park, H. Zheng, J. Halbert, K. Bains, S. Jang, and J. S. Choi, “Co-Architecting Controllers and DRAM to Enhance DRAM Process Scaling,” in _The Memory Forum_ , 2014.
  * (26) S. Hong, “Memory Technology Trend and Future Challenges,” in _IEDM_ , 2010.
  * (27) S. Kanev, J. P. Darago, K. Hazelwood, P. Ranganathan, T. Moseley, G.-Y. Wei, and D. Brooks, “Profiling a Warehouse-Scale Computer,” in _ISCA_ , 2015.