# An Attention Self-supervised Contrastive Learning based Three-stage Model
for Hand Shape Feature Representation in Cued Speech

###### Abstract

Cued Speech (CS) is a communication system for deaf people or hearing impaired
people, in which a speaker uses it to aid a lipreader in phonetic level by
clarifying potentially ambiguous mouth movements with hand shape and
positions. Feature extraction of multi-modal CS is a key step in CS
recognition. Recent supervised deep learning based methods suffer from noisy
CS data annotations especially for hand shape modality. In this work, we first
propose a self-supervised contrastive learning method to learn the feature
representation of image without using labels. Secondly, a small amount of
manually annotated CS data are used to fine-tune the first module. Thirdly, we
present a module, which combines Bi-LSTM and self-attention networks to
further learn sequential features with temporal and contextual information.
Besides, to enlarge the volume and the diversity of the current limited CS
datasets, we build a new British English dataset containing 5 native CS
speakers. Evaluation results on both French and British English datasets show
that our model achieves over 90% accuracy in hand shape recognition.
Significant improvements of 8.75% (for French) and 10.09% (for British
English) are achieved in CS phoneme recognition correctness compared with the
state-of-the-art.

Index Terms: Cued Speech, Self-supervised contrastive learning, Self-attention
network, Hand shape recognition

##  1 Introduction

Cued Speech (CS) [1] conveys the visual form of spoken language through
different hand shapes (representing consonants) and hand positions
(representing vowels) near the mouth. Accurate feature extraction (including
lip, hand position and hand shape) plays an important role in automatic CS
recognition.

Currently, lip and hand position feature extraction has achieved good results.
For example, Liu _et al._ [2], [3] proposed a CLNF based inner lip feature
extraction model and an ABMM based CS hand position detector obtaining a good
performance. Besides, YOLOv5 [4] performs well in real-time object detection
which can be used to detect hand positions in CS. However, there are still
some challenges which limit a good hand shape feature extractor. Firstly, the
movements of lips and hands are asynchronous in CS [5], [6] while most of the
proposed hand shape annotation methods rely on audio based segmentation [7].
This causes noisy hand shape annotation for CS data. Secondly, since the hand
shape changes quickly when it moves, for example, the same hand shape with
fast moving, variable rotations, occlusion and motion blur causing different
appearance, it is difficult to extract accurate and robust hand shape features
[8]. Indeed, most previous works [3], [9], [10] ignored the variability and
complexity of hand shape in CS. Thirdly, the limited CS dataset brings great
limitations to the deep learning based methods of CS hand shape feature
extraction. At present, the published CS datasets merely include a single
speaker French dataset [5] and a single speaker British English dataset [11].

To address the above mentioned problems and improve the performance of CS hand
shape feature representation, in this work, we propose a three-stage hand
shape feature extraction model. More precisely, in the first stage, a self-
supervised contrastive learning module [12] is explored to learn image
structure information. In the second stage, we manually annotate a small
amount (10% of the training set) of CS data to fine-tune the module, and
obtain more accurate hand shape features. In the third stage, we combine Bi-
LSTM [13] and self-attention networks (SANs) [14] to obtain temporal and
contextual information of hand shapes. In addition, we establish a new British
English dataset with 390 sentences recorded by five native speakers, which is
the first multi-speaker British English CS dataset. We validate our model in
multi-lingual and multi-speaker scenarios (on French and British English
datasets), achieving better results than state-of-the-art (SOTA).

In summary, our contributions are three-fold:

  * •

A new multi-speaker British English CS dataset is built to solve the problem
of CS hand shape recognition for the first time.

  * •

A three-stage hand shape feature extraction model based on self-supervised
contrastive learning and self-attention mechanism is proposed to model spatial
and temporal features of CS hand shape.

  * •

Experimental results in multi-lingual and multi-speaker scenarios show that
our model is superior to SOTA in hand shape feature extraction and CS
recognition.

##  2 Related Work

Figure 1: Overview of the proposed model. S1, S2 and S3 are three stages.

Hand shape feature extraction plays an important role in CS recognition. In
the early works, the classical methods extracted hand shape features with the
artifices on the hand. Burger _et al._ [15] let the speaker wear black gloves
to obtain accurate hand segmentation. Noureddine _et al._ [16] placed blue
marks on the fingers of the speaker to obtain the coordinates of fingers, so
as to extract the accurate position and shape of the hand. However, both the
speaker’s clothing color and background color will affect the accuracy of hand
segmentation.

Recently, [3] proposed a new scheme without any artifices, which used a GMM
based foreground extraction for hand detection. Then, Convolutional Neural
Networks (CNNs) were employed to extract features from hand regions, feeding
them to an HMM-GMM classifier. Similarly, with skin color detection, [9] used
a motion-based Kalman filter to track the hand and employed 3D-CNNs to extract
spatio-temporal features from hand region sequence. However, these methods are
easily affected by skin color, illumination and noisy annotation data, which
can easily lead to inaccurate hand positions.

Hand shape feature extraction is also active in other fields, such as gesture
recognition [17] and sign language recognition [8]. Current gesture
recognition algorithms attempt to extract hand features such as skin color,
appearance, motion, skeleton and deep learning based detection [18]. However,
skin color based [19] and motion based methods [3], [20] were greatly affected
by illumination changes and background color, while the hand moves around the
face in CS. Appearance based methods [21], [22] modeled visual appearance by
extracting image features, and compared these parameters with the features
extracted from the input image. But in CS, the same hand shape may present
very different appearances, while different hand shapes may present similar
appearances. In addition, it is difficult to get accurate hand key points from
RGB images. Therefore, it is not suitable to apply skeleton based methods
[23], [24] to CS. Deep learning based methods [25], [26] used supervised
learning scheme, which was easily affected by noisy data.

##  3 Methodology

Our model architecture mainly includes three stages (see Figure 1): (i) self-
supervised contrastive learning based hand shape feature representation
module, (ii) a small amount of manually annotated data based fine-tuning
module, and (iii) self-attention based sequential hand shape feature learning
module.

###  3.1 Self-supervised contrastive learning based hand shape feature
representation

Firstly, we perform a random transformation on input images, including
cropping, horizontal flipping, color distortions, gray level distortions, and
Gaussian blur. Then, we get two related images at the same instance, denoted
by $x_{i}^{j}(j=1,2)$. Next, ResNet18 is used as the basic encoder to extract
the representation vectors $h_{i}^{j}$ from the augmented data, where

$h_{i}^{j}=f(x_{i}^{j})=ResNet(x_{i}^{j}).$ (1)

Then, feature projection is used to improve the quality of the feature
representation $h_{i}^{j}$, thus mapping the feature representation to the
space where contrastive loss can be applied. Two non-linear fully connected
(FC) layers are used for mapping to obtain new feature representations
$z_{i}^{j}$ which is defined as

$z_{i}^{j}=g(h_{i}^{j})=W^{(2)}\sigma(W^{(1)}h_{i}^{j}),$ (2)

where $\sigma$ is the ReLU activation function. $W^{(1)}$ and $W^{(2)}$ are
learnable weight matrices. Then, given a set of samples $\\{x_{k}\\}$ and an
image $x_{i}$, the goal of the contrastive learning is to detect the most
similar image $x_{j}$ from $\\{x_{k}\\}_{k\neq i}$.

We input $N$ samples into this module and get $2N$ data representations. For a
positive pair, the other $2(N-1)$ augmented examples within a batch are
regarded as negative examples. The cosine similarity $sim(\cdot)$ is used to
calculate the probability of similarity between any two feature
representations. After that, a contrastive loss function is applied to
calculate the loss of positive pairs, which is defined as

$\ell_{i,j}=-\log{\frac{\exp(sim(z_{i},z_{j})/t)}{\sum_{k=1}^{2N}1_{[k\neq
i]}\exp(sim(z_{i},z_{k})/t)}},$ (3)

where $1_{[k\neq i]}$ is an indicator function and its value is 1 iff $k\neq
i$. $t$ represents the temperature parameter. We calculate the loss of the sum
of all positive pairs in a batch and take the average.

###  3.2 Manually annotated data based fine-tuning

We fine-tune the model with a small amount of manually annotated data (10% of
the training set) to obtain more accurate hand shape features. In the feature
projection, to map feature representations to the same space, some information
of hand shapes are discarded. Therefore, only the basic encoder network in the
first module is used for fine-tuning. We directly feed hand ROI to the pre-
trained ResNet18, obtaining the hand shape features and feeding them to this
module. Two non-linear FC layers are used as the classification network in
this module.

###  3.3 Self-attention based sequential hand shape feature learning

Based on the above two modules, we can extract accurate spatial hand shape
features. However, CS hand shape is a video sequence, thus contextual
information is also important. To learn temporal and sequential information of
hand shapes, we combine Bi-LSTM and multi-head SANs. Among them, Bi-LSTM
models the global information of a sequence to extract features with temporal
and contextual information. SANs learn the internal dependence of the
sequence, so as to better model the internal structure of the sequence and
emphasize the important information. In order to make full use of the features
learned from Bi-LSTM and SANs, we use a short-cut connection to add their
outputs. Let $C=\\{c_{1},c_{2},...,c_{N}\\}$ be the input sequence, the
representation of the encoder is defined as:

$s_{output}=s^{p}+s^{q},$ (4) $s^{p}=U(C),\text{ and }s^{q}=V(s^{p}),$ (5)

where $p$ and $q$ represent the layer number of the encoder, respectively. $U$
and $V$ represent the Bi-LSTM and SANs, respectively. $s_{output}$ represents
the obtained sequential hand shape features.

##  4 Evaluations

###  4.1 Dataset

Two public single-speaker French (named LM) and British English (named CA) CS
datasets contain 238 sentences (repeated twice) and 97 sentences,
respectively. The French dataset contains 8 visemes, 8 hand shapes and 5 hand
positions to encode 14 vowels and 20 consonants. More details can be referred
to [5] and [11].

Based on CA, which was published by the author in 2019, we build a new five-
speaker British English dataset (the original data contain video and audio).
With forced alignment, the acoustic signal synchronized with the video is
automatically marked. Then, we post-check them manually. The expanded datasets
were recorded by 4 different professional CS interpreters (named EM, KA, LD
and VK, respectively). The specific data is shown in Table 1. The new British
English dataset contains 11 visemes [27], 8 hand shapes and 4 hand positions
to encode 17 vowels and 24 consonants. RGB video images of the interpreter’s
upper body are available at 25 fps, and the spatial resolution is 720 × 1280.

###  4.2 Implementation Details

In the following experiments, we randomly select 80% of the dataset as the
training set and 20% of the dataset as the test set. Five groups of data are
randomly generated for cross validation.

####  4.2.1 Hand shape feature extraction

We use our model to extract hand shape features from hand ROI. In stage 1, all
training set (with noisy labels) and the Adam optimizer are used to train the
model for 200 epochs (the English model is trained on the French pre-trained
model for 100 epochs). In stage 2, a small amount of manually annotated data,
the Adam optimizer and cross-entropy loss function are used to train 60 epochs
with a learning rate of 1e-4. Then, we feed the continuous hand ROI into our
model and feed the features obtained from stage 2 into stage 3. In stage 3,
Bi-LSTM (2 layers) and multi-head SANs (3 layers and 16 heads) are used to
extract the continuous hand shape features. The Connectionist Temporal
Classification (CTC) [28] is used as the loss function. Empirically, the
learning rate is 0.001 and the batch size is 1.

Table 1: A summary of the French and British English datasets. Speaker | Speech amount | Language
---|---|---
| (sentence) |
LM | 238x2 | French
CA | 97 | British English
EM | 100 | British English
KA | 100 | British English
LD | 50 | British English
VK | 43 | British English

####  4.2.2 Continuous CS phoneme recognition

We detect the position of lips and hands at first. The facial landmark
detector in the open source library dlib [29] is used to extract the key
points of the outer edge of lips. Then, the lip ROI is obtained based on these
key points. YOLOv5 is used to detect hand positions. We pre-train the model on
a public hand dataset [30]. Next, we mark a small number of hand bounding
boxes on CS datasets to fine-tune the model. Finally, the hand positions and
ROI can be obtained simultaneously. After that, we extract the features of the
three streams, respectively. We employ a 2D-CNN which contains two
convolutional layers (8 filters, a kernel size of 7×7 pixels, a down-sampling
factor of 3) to extract lip features from lip ROI, and also employ an ANN to
extract hand position features from coordinate values. The method of obtaining
static hand shape features is described in 4.2.1. We concatenate the features
of the three streams. Then, the S3 module which has the same structure and
setting as the continuous hand shape feature extraction experiment is used to
recognize continuous CS phoneme.

####  4.2.3 Evaluation metrics

In the hand shape recognition based on a single image, the classification
accuracy ($acc=\frac{n_{correct}}{n_{total}}$) is used as the evaluation
metrics, where $n_{correct}$ and $n_{total}$ represent the number of correct
samples and the total number of samples, respectively. In continuous hand
shape recognition and CS phoneme recognition, phone error rate
($T_{e}=\frac{n_{i}+n_{d}+n_{s}}{N}$), and phone correct rate
($T_{c}=1-T_{e}$) are used as the evaluation metrics, where $n_{i}$, $n_{d}$
and $n_{s}$ represent the number of insert, delete and replace errors,
respectively. $N$ represents the input sequence length.

###  4.3 Result and Discussion

We report the results of CS hand shape recognition and phoneme recognition
quantitatively and qualitatively.

####  4.3.1 Hand shape recognition

To verify the influence of noisy data on hand shape recognition performance,
we compare ResNet18 directly trained with noisy data (100% of the training
set) with ResNet18 trained with manually annotated data (10% of the training
set). As shown in Table 2, ResNet18 trained with manually annotated data
achieves over 30% improvement on both French and British English CS datasets.
Then, we compare our model which only contains S1 and S2 with ResNet18 trained
with manually annotated data. In both our model and ResNet18, we use the same
amount of manually annotated data. It can be seen from Table 2 that our model
performs better because the self-supervised contrastive learning module makes
full use of noisy data for feature representation. Finally, in order to verify
the performance of our model in multi-lingual and multi-speaker scenarios, we
mix French and British English CS datasets for training. We take out 80% of
each dataset as the mixed training set. Obviously, better results are achieved
due to the increase in the size and diversity of the dataset. The results show
that our model is not affected by multiple speakers. We notice that there is a
slight drop in EM’s result, which may be caused by the small amount of EM’s
data.

Table 2: Comparison results (acc%) using ResNet18 based on supervised learning, and our model. We use noisy data (n) and manually annotated data (a) to train ResNet18. Ours-S3 represents our model without S3 and Ours-S3(multi) represents our model in multi-lingual and multi-speaker scenarios. Method | LM | CA | EM | KA | LD | VK
---|---|---|---|---|---|---
ResNet18(n) | 61.33 | 53.92 | 54.58 | 60.14 | 47.68 | 48.86
ResNet18(a) | 94.32 | 89.17 | 83.58 | 91.49 | 86.61 | 86.55
Ours-S3 | 96.39 | 95.05 | 91.59 | 96.45 | 92.56 | 95.18
Ours-S3(multi) | 96.50 | 95.73 | 91.54 | 96.52 | 92.89 | 96.28
Figure 2: Feature visualization of hand shapes, where (a) is the Grad-CAM++
[31] visual graph. The images of each group in (b)$\sim$(e) are original
image, Grad-CAM [32] visual graph and Guided Grad-CAM [32] visual graph,
respectively.

In Figure 2, to verify the effectiveness of the extracted hand shape features,
we qualitatively visualize the feature map of hands in complex scenes. It can
be seen from (a) that the hand shape is distorted during transition, but our
model still captures the key regions. In (b), the index finger and the middle
finger are overlapped, but the features of the index and middle finger are
both well learned. At the beginning and end of a sentence, the hand is
gradually raised or put down. Therefore, some hands are only partially exposed
in this process, as shown in (c). However, even (c) only shows two fingers,
the hand shape can be correctly recognized. (d) is a twisted hand shape 8. It
can be seen that even though the hand shape is similar to hand shape 4, the
features are still well learned. And (e) is a hand shape 5 blurred by motion.
Although the hand shape is very fuzzy, our model can still reconstruct the
hand shape.

The recognition results of continuous hand shape in each dataset can be seen
in Table 3. Each time the hand shape sequence in a sentence is identified.
Compared with SOTA (i.e., CNN-HMM) [3], a significant improvement of 16.42%
correctness is achieved in the French dataset. In the British English dataset,
our model also achieves high accuracy. Ablation study (Ours-S2 and Ours-SANs)
shows that all modules in our model are important. In addition, more accurate
results can be obtained with manually annotated data. Compared with the model
trained with noisy data (i.e., ReNet18(n)+S3), our model achieves much higher
hand shape recognition correctness.

Table 3: Comparison of continuous hand shape recognition results in $T_{c}$ (%). CNN-HMM [3] is the SOTA in French dataset. ResNet18(n)+S3 represents the model which combines ResNet18(n) and S3. Ours-S2 represents our model without S2. Ours-SANs represents our model without SANs. Ours represents our model. Method | LM | CA | EM | KA | LD | VK
---|---|---|---|---|---|---
CNN-HMM [3] | 68.50 | — | — | — | — | —
ResNet18(n)+S3 | 83.99 | 78.16 | 76.15 | 76.08 | 74.42 | 67.52
Ours-S2 | 84.81 | 82.27 | 79.13 | 83.57 | 75.86 | 75.44
Ours-SANs | 84.88 | 84.75 | 83.81 | 89.40 | 80.47 | 76.12
Ours | 84.92 | 85.25 | 85.19 | 89.52 | 82.22 | 78.94

####  4.3.2 Phoneme recognition

Finally, we consider CS phoneme recognition, which uses the lip, hand
position, and hand shape as the input, and the output is the phoneme class (33
and 41 phonemes in French and British English, respectively). The comparison
under different models (see Table 4) shows that our model achieves the best
results on both French and British English datasets. Compared with SOTA [9],
8.75% and 10.09% improvements are achieved in the two datasets, respectively.
This verifies that our model can still perform well in the multi-lingual
scenario. There are significant differences in performance for each dataset.
This may be due to the limited size of LD and VK datasets. But the exciting
finding is that the model with SANs outperforms other models in Table 4. The
above results show that our model can reduce the influence of noisy data on CS
phoneme recognition.

To verify the performance of our model in the multi-speaker scenario, we mix
data from five speakers in the British English dataset. 80% data of each
speaker is used as the training set. It can be observed that all the results
are greatly improved because the increasing diversity of dataset is beneficial
to self-supervised contrastive learning.

Table 4: Comparison of phoneme recognition results in $T_{e}$ (%). 3D-CNN+TDS [9] is the SOTA. Ours(multi) represents our model in the multi-speaker scenario. Method | LM | CA | EM | KA | LD | VK
---|---|---|---|---|---|---
3D-CNN+TDS [9] | 29.12 | 36.25 | — | — | — | —
ResNet18(n)+S3 | 23.27 | 30.48 | 27.36 | 26.20 | 46.79 | 46.60
Ours-SANs | 21.8 | 27.21 | 25.41 | 25.96 | 43.89 | 45.42
Ours | 20.37 | 26.16 | 23.39 | 24.88 | 38.42 | 42.02
Ours(multi) | — | 19.93 | 17.64 | 14.59 | 20.77 | 26.38

##  5 Conclusions

In this work, we propose an attention self-supervised contrastive learning
based three-stage hand shape feature extraction model. Our model solves the
problem caused by excessive noisy annotation in CS hand shape feature
extraction. Experimental results show superior performance to SOTA methods. As
for future work, we will explore the effective sample pair selection methods
for self-supervised contrastive learning in CS video.

##  6 Acknowledgements

This work was supported by the GuangDong Basic and Applied Basic Research
Foundation (No. 2020A1515110376) and the National Natural Science Foundation
of China (grant No. 61977049). The authors would like to thank the
professional CS speakers from Cued Speech UK for the British English CS
dataset recording.

## References

  * [1] R. O. Cornett, “Cued speech,” _American annals of the deaf_ , vol. 112, no. 1, pp. 3–13, 1967.
  * [2] L. Liu, G. Feng, and D. Beautemps, “Inner lips feature extraction based on clnf with hybrid dynamic template for cued speech,” _EURASIP Journal on Image and Video Processing_ , vol. 2017, no. 1, pp. 1–15, 2017.
  * [3] L. Liu, T. Hueber, G. Feng, and D. Beautemps, “Visual recognition of continuous cued speech using a tandem cnn-hmm approach.” _in Proc. Interspeech_ , pp. 2643–2647, 2018.
  * [4] G. Jocher, A. Stoken, J. Borovec, NanoCode012, A. Chaurasia, TaoXie, L. Changyu, A. V, Laughing, tkianai, yxNONG, A. Hogan, lorenzomammana, AlexWang1900, J. Hajek, L. Diaconu, Marc, Y. Kwon, oleg, wanghaoyang0106, Y. Defretin, A. Lohia, ml5ah, B. Milanko, B. Fineran, D. Khromov, D. Yiwei, Doug, Durgesh, and F. Ingham, “ultralytics/yolov5: v5.0 - YOLOv5-P6 1280 models, AWS, Supervise.ly and YouTube integrations,” Apr. 2021. [Online]. Available: https://doi.org/10.5281/zenodo.4679653
  * [5] L. Liu, G. Feng, and D. Beautemps, “Automatic temporal segmentation of hand movements for hand positions recognition in french cued speech,” _in Proc. ICASSP_ , pp. 3061–3065, 2018.
  * [6] L. Liu, G. Feng, D. Beautemps, and X.-P. Zhang, “A novel resynchronization procedure for hand-lips fusion applied to continuous french cued speech recognition,” _in Proc. EUSIPCO_ , pp. 1–5, 2019.
  * [7] S. Tranter, K. Yu, G. Everinann, and P. C. Woodland, “Generating and evaluating segmentations for automatic speech recognition of conversational telephone speech,” _in Proc. ICASSP_ , pp. 753–756, 2004.
  * [8] R. Rastgoo, K. Kiani, and S. Escalera, “Hand pose aware multimodal isolated sign language recognition,” _Multimedia Tools and Applications_ , vol. 80, no. 1, pp. 127–163, 2021.
  * [9] K. Papadimitriou and G. Potamianos, “A fully convolutional sequence learning approach for cued speech recognition from videos,” _in Proc. EUSIPCO_ , pp. 326–330, 2021.
  * [10] L. Liu, G. Feng, D. Beautemps, and X.-P. Zhang, “Re-synchronization using the hand preceding model for multi-modal fusion in automatic continuous cued speech recognition,” _IEEE Transactions on Multimedia_ , vol. 23, pp. 292–305, 2020.
  * [11] L. Liu, J. Li, G. Feng, and X. S. Zhang, “Automatic detection of the temporal segmentation of hand movements in british english cued speech.” _in Proc. Interspeech_ , pp. 2285–2289, 2019.
  * [12] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” _in Proc. ICML_ , pp. 1597–1607, 2020.
  * [13] A. Graves, A. Mohamed, and G. E. Hinton, “Speech recognition with deep recurrent neural networks,” _in Proc. ICASSP_ , pp. 6645–6649, 2013.
  * [14] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _arXiv preprint arXiv:1706.03762_ , 2017.
  * [15] T. Burger, A. Caplier, and S. Mancini, “Cued speech hand gestures recognition tool,” _in Proc. EUSIPCO_ , pp. 1–4, 2005.
  * [16] N. Aboutabit, D. Beautemps, and L. Besacier, “Hand and lip desynchronization analysis in french cued speech: Automatic temporal segmentation of hand flow,” _in Proc. ICASSP_ , pp. 633–636, 2006.
  * [17] S. Mitra and T. Acharya, “Gesture recognition: A survey,” _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ , vol. 37, no. 3, pp. 311–324, 2007.
  * [18] M. Oudah, A. Al-Naji, and J. Chahl, “Hand gesture recognition based on computer vision: a review of techniques,” _Journal of Imaging_ , vol. 6, no. 8, p. 73, 2020.
  * [19] M. Perimal, S. Basah, M. Safar, and H. Yazid, “Hand-gesture recognition-algorithm based on finger counting,” _Journal of Telecommunication, Electronic and Computer Engineering_ , vol. 10, no. 1-13, pp. 19–24, 2018.
  * [20] C.-M. Pun, H.-M. Zhu, and W. Feng, “Real-time hand gesture recognition using motion tracking,” _International Journal of Computational Intelligence Systems_ , vol. 4, no. 2, pp. 277–286, 2011.
  * [21] Q. Chen, N. D. Georganas, and E. M. Petriu, “Real-time vision-based hand gesture recognition using haar-like features,” _in Proc. IMTC_ , pp. 1–6, 2007.
  * [22] V. S. Kulkarni and S. Lokhande, “Appearance based recognition of american sign language using gesture segmentation,” _International Journal on Computer Science and Engineering_ , vol. 2, no. 03, pp. 560–565, 2010.
  * [23] X. S. Nguyen, L. Brun, O. Lézoray, and S. Bougleux, “A neural network based on spd manifold learning for skeleton-based hand gesture recognition,” _in Proc. CVPR_ , pp. 12 036–12 045, 2019.
  * [24] D. Konstantinidis, K. Dimitropoulos, and P. Daras, “Sign language recognition based on hand and body skeletal data,” _in Proc. 3DTV-CON_ , pp. 1–4, 2018\.
  * [25] M. Abavisani, H. R. V. Joze, and V. M. Patel, “Improving the performance of unimodal dynamic hand-gesture recognition with multimodal training,” _in Proc. CVPR_ , pp. 1165–1174, 2019.
  * [26] S. Narasimhaswamy, Z. Wei, Y. Wang, J. Zhang, and M. Hoai, “Contextual attention for hand detection in the wild,” _in Proc. CVPR_ , pp. 9567–9576, 2019.
  * [27] D. Jachimski, A. Czyzewski, and T. Ciszewski, “A comparative study of english viseme recognition methods and algorithms,” _Multimedia Tools and Applications_ , vol. 77, no. 13, pp. 16 495–16 532, 2018.
  * [28] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” _in Proc. ICML_ , pp. 369–376, 2006.
  * [29] D. E. King, “Dlib-ml: A machine learning toolkit,” _The Journal of Machine Learning Research_ , vol. 10, pp. 1755–1758, 2009.
  * [30] A. Mittal, A. Zisserman, and P. H. Torr, “Hand detection using multiple proposals.” _in Proc. BMVC_ , pp. 1–11, 2011.
  * [31] A. Chattopadhay, A. Sarkar, P. Howlader, and V. N. Balasubramanian, “Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks,” _in Proc. WACV_ , pp. 839–847, 2018.
  * [32] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” _in Proc. ICCV_ , pp. 618–626, 2017.


arxiv.tex

# Workload Behavior Driven Memory Subsystem Design for Hyperscale

Suyash Mahar∗ Hao Wang† Wei Shu§ Abhishek Dhanotia‡
∗UC San Diego †NVIDIA §Tenstorrent ‡Meta Inc

###### Abstract

Hyperscalars run services across a large fleet of servers, serving billions of
users worldwide. However, these services exhibit different behaviors compared
to commonly available benchmark suites, leading to server architectures that
are suboptimal for cloud workloads. As datacenters emerge as the primary
server processor market, optimizing server processors for cloud workloads by
better understanding their behavior is an area of interest. To address this,
we present MemProf, a memory profiler that profiles the three major reasons
for stalls in cloud workloads: code-fetch, memory bandwidth, and memory
latency. We use MemProf to understand the behavior of cloud workloads at Meta
and propose and evaluate micro-architectural and memory system design
improvements that help cloud workloads’ performance.

MemProf’s code analysis shows that cloud workloads at Meta execute the same
code across CPU cores. Using this, we propose shared micro-architectural
structures–a shared L2 I-TLB and a shared L2 cache. Next, to help with memory
bandwidth stalls, using workloads’ memory bandwidth distribution, we find that
only a few pages contribute to most of the system bandwidth. We use this
finding to evaluate a new high-bandwidth, small-capacity memory tier and show
that it performs 1.46$\times$ better than the current baseline configuration.
Finally, we look into ways to improve memory latency for cloud workloads.
Profiling using MemProf reveals that L2 hardware prefetchers, which are
commonly used to reduce memory latency, have very low coverage and consume a
significant amount of memory bandwidth. To help improve future hardware
prefetcher performance, we built an efficient memory tracing tool to collect
and validate production memory access traces. Our memory tracing tool adds
significantly less overhead than DynamoRIO, enabling tracing production
workloads.

##  1 Introduction

In recent years, the market share of the cloud as a portion of the total
server market has constantly been increasing. Cloud deployments now account
for >50% of the total server processor market and are expected to grow even
further [38, 29]. However, CPU benchmarks available today do not accurately
represent microarchitectural and memory system behavior of real cloud
workloads, leaving the processors largely unoptimized for their unique
characteristics.

Cloud workloads show different behavior than the CPU benchmarks available
today across several metrics. For one, the fleet-wide111Server fleet refers to
an ensemble of servers across datacenters. IPC (instructions per cycle) for
major hyperscalers is significantly lower than that of the benchmark suites
[36, 20]. With millions of servers running these workloads, even small
improvements in IPC from understanding the behavior of these workloads can
result in significant cost savings and efficiency improvements across the
fleet.

To improve the performance of cloud workloads, in this paper, we study and
characterize nine microservices serving live production traffic. These
microservices represent a diverse range of workloads at a hyperscalar’s
datacenters and run on a significant portion of the fleet.

Figure 1: Breakdown of where the CPU pipeline slots spend their time across
nine production workloads using top-down analysis [41].

We find that the memory system is the primary reason for CPU pipeline stalls
across most of these cloud workloads, as shown in Figure 1 using the top-down
analysis [41]. These memory system-related stalls can have three sources, (1)
stall on an instruction fetch (code-fetch bound), (2) stall on the memory
system with a high request pending queue occupancy (memory bandwidth bound),
or (3) stall on the memory system with a low request pending queue occupancy
(memory latency bound).

To further dive into these three sources of stalls in cloud workloads, we
created a new tool, MemProf, that enables detailed profiling of workloads’
micro-architectural and memory system behavior. We use observations from
MemProf to propose and evaluate new micro-architectural changes and system
architecture for cloud server processors.

The first source of stalls, code fetch-related stalls are a significant
problem in cloud workloads [3, 36]. Using MemProf, we observe that workloads
at Meta run similar code and share page table mappings across cores. Building
on this observation, we propose sharing the L2 Cache and L2 I-TLB among a
small cluster of cores (e.g., four) that are physically close. Shared µ-arch
structures for instructions help reduce code-fetch stalls by enabling the
cores running similar code to pool their caches for larger apparent cache
size, even though they have the same per-core cache size.

Memory bandwidth is another significant contributor to memory system stalls.
Over the years, DDR memory capacity and bandwidth have been scaling at
significantly different rates [27]. This increasing memory capacity and
bandwidth disparity, along with growing core counts in modern server
processors has resulted in limited per-core memory bandwidth, requiring
processors to populate more memory channels, thus resulting in higher server
TCO (total cost of ownership). We use MemProf to study the memory bandwidth
distributions of production workloads and find that only a small percentage of
pages contribute to most of the memory bandwidth utilization across the
workloads. We use this observation to propose and evaluate new bandwidth-
tiered memory systems as opposed to latency-tiered systems common today. In
this new system, a small, high-bandwidth memory tier serves the bandwidth
needs of the workload. In contrast, a low-bandwidth, large-capacity tier
provides additional memory capacity, lowering servers’ TCO. We find that by
splitting the memory capacity in 30:70% between the high-BW and low-BW tiers,
we can achieve 1.46$\times$ better throughput and 13% better throughput/cost
than the baseline DDR-only configuration.

Finally, we look into ways to improve future memory latency research for cloud
workloads. Using MemProf, we observe that although L2 hardware prefetchers
improve memory latency by prefetching cachelines, they often have very low
coverage in real-world cloud workloads, resulting in minor IPC improvements.
Further, we observe that L2 hardware prefetcher’s inefficiencies result in
high memory bandwidth consumption, exacerbating the memory bandwidth problem.
A major hurdle in optimizing these hardware prefetchers is access to the cloud
workloads to study their memory behavior. To enable broader access to cloud
workloads’ behavior, we built an efficient memory tracing tool capable of
tracing live production workloads and collect several memory access traces of
live production workloads. Our tracing tool has significantly less runtime
overhead compared to state-of-the-art, DynamoRIO. We plan to make the traces
available in the future to help hardware prefetcher research.

This paper is the first to study how cloud workloads interact with the memory
system and to use these observations to propose shared µ-arch structures and
memory bandwidth tiering. While previous works from Google [3, 20] and Meta
[36, 35] have investigated the µ-arch behavior of hyperscale workloads, they
have not performed detailed studies of the code behavior, memory bandwidth
distribution, or hardware prefetcher efficiency. In summary, this paper makes
the following contributions:

  * •

Detailed study and characterization of cloud workloads. We perform a detailed
study of code, memory bandwidth, and memory latency trends at Meta.

  * •

New profiling methodology and tool. We present MemProf, a profiling tool for
cloud workloads that helps understand their interaction with the memory
system.

  * •

Code sharing across cores. We show that CPU cores run very similar code across
all the workloads, enabling performance improvements from shared µ-arch
resources.

  * •

Memory bandwidth distribution. We study the memory bandwidth distribution,
find that very few pages contribute to a large amount of bandwidth and propose
new tiered memory solutions for cloud workloads.

  * •

Hardware prefetcher efficiency. L2 hardware prefetchers incur significant
memory bandwidth overhead with small performance improvements. To address
this, we will make production workload memory traces available to the
community for future hardware prefetcher research.

In the next section, we look into the three reasons for memory system stalls:
code fetch, memory bandwidth, and memory latency. We follow with the
description of MemProf and use it to make observations and proposals in each
area.

##  2 Code, Memory BW, and Latency Challenges in Cloud Datacenters

Existing benchmarks that are generally used to design processors do not
accurately represent the workloads at hyperscalars. As highlighted by several
previous works [3, 5, 20, 36, 35], cloud workloads exhibit fundamentally
different IPC, cache miss rates, and other metrics compared to standard
benchmarks. These factors make the SPEC and other commercially available
benchmarks a bad proxy for studying the performance of server processors in a
datacenter.

Figure 2: Gen-over-gen IPC change for the 9 workloads. Missing Gen1 data
corresponds to services that do not run on Gen1. Years correspond to processor
release year.

Besides standard benchmarks’ little resemblance to cloud workloads, we
motivate the need to better understand cloud workloads by comparing the IPC
change across server generations at Meta. We chart the IPC change over the
years across nine representative cloud workloads and four server generations
running the same software versions. Figure 2 shows that the IPC improvement is
minimal or even negative across generations, necessitating the need for a
deeper understanding of cloud workloads.

To understand the reasons for this inhibited performance growth of server
processors running cloud workloads, we will look into each of the three
primary reasons for memory system stalls and understand how they affect cloud
workloads’ performance.

###  2.1 Increasing Code Footprint of Datacenter Applications

Figure 3: Binary Size trend for Web1 and Web2. Web2 binary size increased by
5.6$\times$ over the last four years.

Cloud workloads are often frontend bound due to their large code footprint,
resulting in high instruction cache miss rates [3]. The code footprint can be
hundreds of MiBs and has been increasing over the years. Figure 3 shows the
increase in the binary size for Web1 and Web2, the two major web services at
Meta over the years. Web2 shows an exponential increase in binary size, a
5.6$\times$ increase over four years.

In contrast, the I-cache sizes have stayed relatively constant (32KB I$ per
core) across several server generations and have only recently started
increasing on some server architectures. Likewise, instruction TLB lookups are
optimized for the latency-sensitive instruction fetch pipeline, making it
harder for them to scale with cloud workloads’ rapidly growing code
footprints. Previous work on cloud workload has observed a similar code-
footprint trend. Kanev et al. [20] show that the code footprint at Google is
growing at a rate of 27% per year. This imbalance between the processor’s
micro-architectural resources and the growing code footprint have resulted in
frontend to be the leading source of bottleneck for many workloads across
several server generations and resulted in stagnant IPC over time.

###  2.2 Memory Bandwidth Scaling Challenge

Figure 4:  Memory bandwidth utilization (%) CDF across three generations
(1-min avg); BW bound and sensitive regions based on DDR memory bandwidth vs.
latency characteristics.

Memory bandwidth is another major concern for cloud workloads, resulting from
poor DRAM bandwidth scaling and increasing CPU core count. To understand cloud
workloads’ memory bandwidth utilization trend, we show the fleet-wide memory
bandwidth consumption increase over the three generations of servers in Figure
4. With the most recent generation (Gen 5), the 1-minute average memory
bandwidth utilization shows that the majority of the fleet is either bandwidth
sensitive or bandwidth bound.

While mirco-benchmarks can generally drive memory bandwidth utilization to
>80%, we observe that production workloads rarely exceed 60% memory bandwidth
utilization, as any further increase results in an exponential increase in
memory latency [32]. Thus, we classify workloads with higher than 60%
bandwidth utilization as memory bandwidth bound (shaded red in Figure 4).
Likewise, workloads with average memory bandwidth utilization between 40%-60%
can have high transient memory bandwidth utilization and are thus classified
as memory BW sensitive. While the fleet-wide data is 1-minute average,
transient peaks can cause these workloads to become significantly bandwidth
bound during shorter time intervals, causing tail latency spikes and limited
overall system CPU utilization.

Besides growing memory bandwidth utilization across the fleet, increasing
disparity between DIMM’s bandwidth and capacity [27] makes alleviating memory
bandwidth problems of cloud workloads harder as hyperscalars need to provision
a large number of memory channels and DIMMs per server for sufficient memory
bandwidth.

The trend of increasing memory bandwidth utilization and the poor bandwidth
scaling of DDR-based memories call for more efficient memory provisioning
strategies in cloud datacenters to reduce server costs and power consumption
[13].

###  2.3 High Memory Latency Leading to CPU Underutilization

Figure 5:  Relative memory latency vs. max CPU utilization for microservices;
red markers indicate latency SLO constraints limiting CPU utilization; marker
size represents service size; latency relative to peak load.

Finally, many cloud workloads are memory latency sensitive as they must meet
service level objectives (SLOs) for request latencies [14]. These workloads
run at the maximum CPU utilization possible without violating SLO guarantees,
often wasting CPU resources. Memory latency limits CPU utilization as an
increase in memory latency lowers the system’s IPC. Lower IPC for workloads
results in cloud workloads spending more cycles doing “on-cpu” work, and thus
results in longer query response times.

To study this trend across the fleet, Figure 5 shows a scatter plot with the
max CPU utilization vs. memory latency of workload relative to latency at peak
load. Peak load is the max throughput a workload can sustain without violating
any SLOs. We make three important observations from the fleet-wide data: (a)
Memory latency increases significantly with increasing CPU utilization. That
is, a large portion of the fleet is operating at a relatively high memory
latency (>0.5). (b) Workloads marked in red have 20-50% stranded CPU cores
because of the latency SLO constraints. These workloads must leave CPU cores
idle to avoid violating SLO guarantees, resulting in wasted resources and
inefficiency at the datacenter scale. (c) And differences in memory access
patterns (e.g., bursty memory load) result in workloads experiencing varying
memory latencies at similar CPU utilization levels.

Hardware prefetching is one of the key techniques used in modern server
processors to hide memory latency. L2 hardware prefetchers, however, have a
significant memory bandwidth overhead. Our analysis in Section 6 reveals that
workloads show a substantial increase in total memory bandwidth consumption
with L2 prefetchers enabled while providing limited performance gains,
signaling that the prefetchers have low efficiency.

Next, we’ll introduce MemProf, a profiling tool to characterize cloud
workloads and explore performance improvement opportunities in cloud server
processors.

##  3 MemProf

So far, our focus has been on the sources and the consequences of memory
system stalls. In order to address these issues, we present MemProf, a code
and memory profiler, to profile and understand the memory subsystem behavior
of cloud workloads. MemProf is an automated tool that measures several
hardware counters, samples events using Intel PEBS [10], and generates reports
on code sharing, memory bandwidth distribution, and memory latency behavior.

###  3.1 Design

Figure 6: MemProf overview. (a) MemProf.Code, (b) MemProf.MemBW, (c)
MemProf.MemLat profile the code, memory BW, and memory latency behavior,
respectively.

Figure 6 shows the overview of MemProf. MemProf has three components that
measure the three aspects of memory system-related stalls. (1) MemProf.Code
measures a workload’s code behavior to understand code sharing across cores
and code bandwidth distribution. (2) MemProf.MemBW generates the memory
bandwidth distribution to help understand how hot memory pages in a system
are. And, (3) MemProf.MemLat generates data to understand the effectiveness of
common techniques in reducing memory latency.

###  3.2 Methodology

Next, we will look at the methodology used by MemProf and understand its
working.

MemProf.Code: For profiling the code behavior of workloads, MemProf samples
per-core L1 I-TLB miss events (FRONTEND_RETIRED.ITLB_MISS) using Intel PEBS.
Larger granularity sampling using I-TLB miss events enables MemProf to cover
wider address space regions. Further, I-TLB-based miss sampling and dense
binary layout of hot code using link-time optimization [30] enable low
frequency, low-overhead code footprint estimation.

With the I-TLB miss event samples, MemProf computes the hit distribution
across the address space of the workload and uses it to generate the code
heatmap, code bandwidth distribution, and compute the correlation between the
miss events of two cores (Figure 6a). For all these calculations, MemProf uses
the virtual address of the samples.

MemProf.Code’s results help understand the code footprint and code sharing
across cores in cloud workloads.

MemProf.MemBW: To understand the memory bandwidth requirements of a workload,
MemProf samples the precise LLC demand load miss event
(MEM_LOAD_RETIRED.L3_MISS) and generates the memory bandwidth distribution
profile for a workload (Figure 6b). As multiple virtual pages can map to the
same physical page, MemProf translates sampled virtual addresses to
corresponding physical addresses using the workload’s page table before
computing the memory bandwidth distribution. MemProf supports sampling over a
configurable interval to study memory bandwidth trends over time and
translates the samples’ address at the end of the sampling interval. In cases
where the page table entries for an address range might not exist at the end
of a measurement interval, MemProf keeps around the last copy of the page
table and uses it to translate any addresses missing in the current copy.

MemProf.MemLat: To measure how effective hardware prefetchers are in reducing
memory latency, MemProf performs two sets of measurements. To compute their
accuracy and coverage, MemProf measures the cache and prefetcher-related
hardware counters and uses the following expressions. (CL = cachelines, pref.
= prefetched)

$\displaystyle\textit{Accuracy}=1-\frac{(\textit{Unused pref. CLs
evicted})}{(\textit{Total pref. CLs})}$ $\displaystyle\textit{Coverage}={}$
$\displaystyle\frac{(\textit{Total pref. CLs})-(\textit{Unused pref. CLs
evicted})}{(\textit{Total CLs brought in})-(\textit{Unused pref. CLs
evicted})}$

To measure the memory bandwidth overhead of enabling L2 hardware prefetchers,
MemProf measures the total system bandwidth consumption and IPC with L2
prefetchers enabled and disabled across all cores. Since these counters are
only available for L2 caches, MemProf only generates accuracy and coverage
data for L2 prefetchers.

Collecting accuracy, coverage, and memory bandwidth overhead of L2 hardware
prefetchers gives us insights into prefetcher behavior of different workloads.

###  3.3 Production Servers and Workloads

Figure 7: Distribution of workloads across the fleet based on % frontend and backend stalls. Marker size is scaled by the workload’s server count. Table 1: Gen 4 system configuration. Parameter | Value
---|---
Cores/Threads | 26/52
L1 I-/D-Cache | 32 KiB/core each, private
L2 Cache | 1 MiB/core, unified, private
| 2 MiB/4 MiB pages fully assoc., 8 entries/thread;
I-L1 TLB | 4 KiB pages 8-way, 128 entries/core
LLC | 1.375 MiB/core

We run containerized production workloads on bare metal machines (that is, no
hypervisor or virtual machines) to characterize microservices in Section 3,
and for studies in Section 4 and 5. These workloads process live production
traffic, represent a large part of the fleet, and exhibit a wide range of
µ-architectural behaviors. We run each of these workloads near the peak load
operating point.

Further, all experiments were run on the fourth generation of servers, with
the hardware configuration listed in Table 1.

We use nine representative workloads running live production traffic to study
the frontend and backend behavior of the fleet. Figure 7 shows the percentage
of CPU pipeline slots across the microservices that are frontend and backend
bound. Across the fleet, we see that workloads show extremely diverse µ-arch
behavior, ranging from highly frontend bound to highly backend bound.

The nine labeled microservices in the scatter plot account for a significant
portion of the fleet and exhibit diverse µ-arch behavior. We study these nine
workloads to perform a representative characterization of cloud workloads at
Meta.

These microservices are Web1, Web2, Ads1, Ads2, Ads3, Cache1, Cache2, Feed,
and Reader. To achieve the maximum query rate for Cache1, hosts use 2/3 of the
cores for running the workload. The rest of the 1/3 cores are used for network
processing and handling NIC IRQs. Most microservices spawn multiple threads,
where only one thread is assigned to a logical core. In contrast, Web2 spawns
multiple processes and assigns up to one process per logical core.

##  4 Code Behavior of Cloud Workloads

Figure 8: % code fetch bound cycles (I-cache and I-TLB). Figure 9: Code bw
distribution for the nine production workloads using precise L1 I-TLB miss
events. X-axis represents code pages sorted by their hotness and the Y-axis
shows the relative memory bandwidth contribution of the ‘X’ MiB of hottest
pages.

As we showed in Section 2, CPU frontend pipeline stalls from instruction cache
and TLB misses are a significant fraction of the memory system stalls across
the nine workloads. To mitigate the code-fetch overhead, in this section, we
look at the code access behavior of cloud workloads at the full system level
and propose µ-arch optimizations.

To understand the impact of a large code footprint, Figure 8 shows the
percentage of CPU cycles where the CPU core was stalled while waiting for an
I-TLB miss or an I-cache miss. Web1, Web2, and Cache2 spend over 20% of their
cycles stalling on instruction fetch. This code fetch overhead is a direct
result of the rapid footprint growth of cloud workload binaries due to their
high development velocities, as shown in Figure 3 for Web1 and Web2 workloads.
Although we use huge pages and code layout optimization techniques [9, 30] to
reduce I-TLB and I-cache misses, we still see fairly high frontend stalls.
This is partly because application developers do not control huge page usage
and code layout for many dynamically linked libraries; and because the code
footprint is too large and still growing.

To address the code-fetch overhead, our analysis with MemProf reveals that
cloud applications execute highly similar code across different CPU cores, in
both virtual and physical address spaces, for multi-threaded and multi-process
applications alike.

###  4.1 Working Set Size for Code

Figure 10: L1, L2, and L3 cache code miss rate.

Next, we will look at the code bandwidth and footprint of the nine workloads
to understand why they are significantly code-fetch bound.

Using MemProf.Code’s L1 I-TLB miss event measurements; we generate the code
bandwidth of the workloads, as shown in Figure 9. We find that all workloads
have very large code footprints, with one notable example being the Web1
workload, which has a code footprint of 125 MiB.

We further investigate the effects of the large code footprint by measuring
the L1-L3 cache code miss rate. Figure 10 shows the code MPKI for the three
levels of caches. For the nine workloads, we see that the large footprint of
cloud workloads results in a high code miss rate across all three levels, with
Web1 and Web2 even fetching a significant amount of code lines from memory.

Using Figure 9 and Figure 10, we make three important observations: (a)
despite the latest processors adopting bigger last-level caches (LLC), certain
workloads’ huge but sparsely accessed code footprint (see Figure 10) causes
diminishing returns in LLC code MPKI improvements. (b) Second, extremely large
L1 code MPKI values suggest that the active code footprint greatly exceeds the
L1 code sizes. (c) Finally, workloads like Cache2 can show abnormally high L1
cache MPKI and code-fetch bound cycles despite a relatively small code
footprint. Investigating further, we found that Cache2 suffers from a high
branch misprediction rate, resulting in increased code accesses and, thus,
being significantly more code-fetch bound.

It is interesting to note that workloads show a ~10$\times$ reduction in code
MPKI from L2 to L3, which is much higher than the L1-to-L2 reduction. While
considering capacity, L2 is often 32$\times$ of L1, while per-core L3 is only
about 1.35$\times$ of L2. This suggests that code benefits from shared L3 and
effectively uses much more capacity than its per-core quota would suggest.

###  4.2 Code-Sharing Behavior for Cloud Workloads

(a) L1 I-TLB miss event distribution for the address space of Web1. Each row
is a 2 MiB slice of the address space. Contiguous rows represent a contiguous
part of the address space.

(b) Hot functions across cores for Web1 stack sampling.

Figure 11: Code sharing across cores using MemProf.

So far, we have looked into the code-fetch overhead of cloud workloads and the
resulting performance loss; next, we will present our observation that cloud
workloads running across cores share code cachelines, allowing shared cache
structures to improve workload’s performance.

To understand if the cores are accessing identical address locations, we used
MemProf to sample L1 I-TLB miss events for different cores running the same
workload. Figure 11a shows the I-TLB miss heatmap for two cores running the
Web1 workload for the first 1.28 GiB of the address space. Across the address
space, both cores show very similar code access patterns, suggesting that the
cores are accessing common code pages. We test our hypothesis by computing the
Pearson correlation coefficient for the two cores; the Web1 workload shows a
correlation value of 0.9997. A Pearson correlation value close to 1 suggests
that the L1 I-TLB miss events for both cores are highly correlated, and thus,
the cores have very similar code access patterns.

Table 2: Core 0 & 1 correlation for L1 I-TLB miss events. Workload | Corr. Value |  | Workload | Corr. Value
---|---|---|---|---
Web1 | 0.9997 |  | Web2 | 0.9881
Ads1 | 0.9977 |  | Ads2 | 0.9921
Ads3 | 0.9833 |  | Feed | 0.9977
Cache1 b/w workload cores |  0.9947 |  |  Cache1 b/w NIC cores |  0.9168
Cache1 b/w workload & NIC |  0.0010 |  |  Cache2 |  0.9978
Reader | 0.9887 |  |  |

Table 2 shows a similar trend for the other workloads. All nine workloads show
a high correlation between cores for L1 I-TLB miss events.

For multithreaded workloads, accessing the same pages implies that the CPU
cores access common mappings across threads. For the Web2 microservice that
spawns multiple processes, we use the page table to verify that code page
mappings have the same translation across processes. For Web2, we observe
common code mappings across processes as all child processes spawn from the
same parent.

To confirm that cores execute the same code, we collect stack samples across
cores and check if they execute the same functions. Figure 11b shows the
average, p90, and p10 weight of the top 100 functions across cores for the
Web1 workload. The L1 I-TLB miss correlation and cores executing identical
functions across them confirm that CPU cores execute identical code, leading
to our first key observation.  Key observation 1: Cores run similar code
across them.

Using this observation about cloud workloads, we will propose two new µ-arch
improvements and evaluate their effectiveness using the Web1 workload. We
chose the Web1 workload as it is significantly frontend bound (Figure 7) and
runs across a large portion of the fleet.

###  4.3 Reducing I-Cache Misses Using a Shared L2 I-Cache

Figure 12: Shared cache design with unified L2 cache shared among four cores.
(a) Private 1 MiB/core L2. (b) Shared, 1 MiB/core shared L2 (total 4 MiB)
cache.

Since CPU cores share code; with a shared L2 cache, the cores will see an
increased apparent cache size for shared cachelines. To take advantage of
this, we propose using a shared L2 architecture (Figure 12) where a cluster of
four cores shares a unified L2 cache. To keep the hardware cost similar, we
propose that the per-core L2 cache size stay similar. Thus, a processor
resembling the Gen4 platform (Table 1) should share 4 MiB of L2 cache among
four cores (1 MiB/core). In a heterogeneous workload environment, the
scheduler can assign multiple threads/processes from a single workload to a
core cluster, enabling heterogeneous workloads to take advantage of the shared
L2 cache.

####  4.3.1 Evaluation Methodology

To evaluate the performance improvements from a shared L2 cache, we use
Intel’s L2 CAT [12] and CDP [11] with Web1 while serving live production
traffic.

While simulations can allow us to model the shared cache design closely,
simulations suffer from several limitations. For example, full-system or
system-call emulations have significant performance overhead which changes the
application’s runtime behavior. On the other hand, while trace-based
simulations have lower overhead, they cannot accurately capture the timing
characteristics, inter-thread dependencies, or RPCs, making them impractical
for cloud workloads.

####  4.3.2 Results

Table 3: Evaluated system configuration for cache scaling. Parameter | Value
---|---
I-L1 Cache | 32 KiB/core, private
L2 Cache | 2 MiB/core, unified, private
LLC | 1.875 MiB/core
Figure 13: IPC scaling and projection along with L2 code MPKI change for Web1
with increasing L2 code cache allocation. Web1 shows an 8.6% experimental and
9.1% projected IPC improvement.

Figure 13 shows the measured and projected IPC along with the L2 code MPKI for
the Web1 workload serving live traffic. We use a machine with the cache
configuration listed in Table 3 and scale the L2 code partition from 128 KiB
to 1.5 MiB while keeping the L2 data partition constant at 0.5 MiB (50% of
Gen4 L2$, based on our study of code-data split in L2 cache). To project
performance beyond 1.5 MiB code partition, we extrapolate the experimental
data using linear projection in logspace. We chose this projection to model
the cache behavior where the performance improves linearly with an exponential
increase in cache size [1].

The Web1 workload shows a real experimental performance improvement of 8.6%
when the cache size increases from 0.5 MiB (50% of private 1 MiB L2 cache) to
1.5 MiB (50% of shared 3 MiB L2 cache), corresponding to a 3$\times{}$
increase in apparent I-cache size. For a 4$\times{}$ increase in I-cache size,
we rely on projected IPC values, which estimate a 9.1% increase.

Thus, based on code sharing across cores and the performance increase from a
larger code cache for Web1, one of the most frontend bound workloads, we
observe that cloud workloads benefit from a larger shared L2 cache.

###  4.4 Reducing I-TLB Misses with a Shared L2 I-TLB

Figure 14: Shared L2 I-TLB design with unified L2 cache shared among 4 cores.
(a) Private 1 MiB/core unified L2. (b) Shared, 1 MiB/core Unified L2 (total 4
MiB) cache.

Figure 15: TLB entry design: (a) private TLB (b) shared TLB. Figure 16: Page
table lookup/update for a shared L2 I-TLB.

Similar to a shared L2 cache, we propose a shared L2 I-TLB to exploit common
page mappings for code across cores for cloud workloads. Figure 14 shows a
shared L2 I-TLB architecture where a cluster of four cores shares the L2
I-TLB, but each core still has a private L2 D-TLB. A shared L2 I-TLB allows
the cores running identical code to share the TLB entries, making a larger
number of TLB entries available to the workload.

Supporting shared I-TLB for multithreaded programs requires no additional
changes to the TLB entries. Each TLB entry includes an ASID (Figure 15a) to
track which process the entry belongs to. Since each CPU core is part of the
same address space in multithreaded programs, the shared unmodified TLB
entries can support multiple cores.

For multiple processes across cores, we propose the shared I-TLB entries to
include multiple ASIDs (Figure 15b). For our design, we propose 4 processes to
share a single L2 I-TLB; thus, each TLB entry only needs 4 ASIDs, one for each
core.

Figure 16 shows the mechanism for a TLB lookup and to install a new entry if
it does not already exist. When a translation request comes into the TLB, if
the virtual address (VFN) and the entry’s application-specific ID (ASID)
match, the corresponding physical address (PFN) is returned. If the entry for
the VFN does not exist, or the entry’s ASIDs do not match the request, the TLB
performs a page table walk and adds a new entry or updates the existing page
table entry.

Finally, as our shared TLB design is similar to shared TLBs in SMT, we expect
a comparable security model.

####  4.4.1 Evaluation Methodology

Next, we will validate whether cloud workloads benefit from larger I-TLB sizes
by doubling the L1 I-TLB entry count. We cannot do a scaling study similar to
Section 4.3 because the cache partitioning feature is not available for TLBs.
Further, as previously stated, µ-arch simulators cannot capture the behavior
of a production environment, making detailed architecture simulation
impractical.

####  4.4.2 Results

Figure 17 shows the change in L1 I-TLB MPKI when the number of entries is
doubled, showing that workloads benefit from an increased I-TLB size. To
increase the count of I-TLB entries, we measure per-thread L1 I-TLB MPKI with
simultaneous multithreading (SMT) on (1$\times$ entries) and off (2$\times$
entries).

Thus, shared code mappings and an increase in performance with more L1 I-TLB
entries show that cloud workloads benefit from a larger, shared L2 I-TLB.
While larger I-TLB results in a lower MPKI, shared TLB can still suffer from
trashing over smaller time intervals if the threads sharing the TLB are in
different application phases.

Figure 17: I-TLB MPKI with 1$\times$ and 2$\times$ L1 I-TLB entries for four
cloud workloads.

##  5 Understanding and Improving Memory Bandwidth

The challenges of memory bandwidth scaling and limited per-DIMM bandwidth have
resulted in cloud workloads becoming more memory bandwidth-bound (Section
2.2). Since current server processors only support a single memory tier,
hyperscalars must populate additional DRAM channels to satisfy the workload’s
memory bandwidth requirements. This results in higher power consumption and
wasted memory capacity at the datacenter scale.

Recent memory industry trends have resulted in new, higher bandwidth memory
technologies like Intel MCR [34], HBM, and HB-DIMMs which can be directly
connected to CPUs to enable high bandwidths. These technologies offer the
potential to explore memory bandwidth tiering as a solution to tackle
bandwidth and capacity scaling challenges. It is important to note here that
memory-bandwidth contrasts with the memory-latency tiering commonly used
today, such as DRAM-Optane tiers.

In this section, we will study the memory bandwidth characteristics of cloud
workloads and look at the potential of high bandwidth memory technologies in
improving cloud workloads’ performance and server TCO at Meta.

###  5.1 Memory Bandwidth Distribution

Figure 18: Memory bandwidth distribution of the workloads over three different
measurement intervals.

Using MemProf, we measured the memory bandwidth distribution over 30, 60, and
120 seconds. Figure 18 shows the amount of memory contributing to different
bandwidth percentiles. Across the nine workloads, we observe that over 30
seconds, the active memory footprint is less than 25% of the total memory
capacity, while the 90%-tile bandwidth is contributed by less than 10% of the
memory capacity. Moreover, a similar memory bandwidth profile for different
measurement intervals suggests that the memory bandwidth distribution remains
relatively constant over time, supporting the possibility of memory bandwidth
tiering.

MemProf, using its LLC demand load miss sampling, enables measuring the
relative hit rate of a memory page. This is unlike current active page
tracking techniques, e.g., Kernel Idle Page Tracking (IPT) [21], which track
only active pages with no visibility into how hot an active page is. Since the
system does not know the “hotness” of a page, these existing techniques cannot
measure the memory bandwidth distribution of a workload.

Using MemProf’s memory bandwidth distribution experiments, we make our second
key observation of cloud workloads’ memory behavior:  Key observation 2: Only
a few pages contribute to most of the memory bandwidth over varying time
intervals. With these insights on memory bandwidth behavior, we will
investigate opportunities to improve system performance without substantially
increasing cost and power overheads.

###  5.2 Memory Bandwidth Tiering

Memory bandwidth tiering enables servers with memory tiers that efficiently
serve memory traffic to the high- and low-bandwidth memory pages. To explore
opportunities for memory bandwidth tiering, we will look at memory
technologies that support considerably higher memory bandwidth than
conventional DDR-DRAM, and study their power requirements and cost overheads.

Figure 19: Mem BW tiering architectures for cloud workloads.

####  5.2.1 Opportunities

Emerging memory technologies like High Bandwidth DIMMs (HB-DIMMs) [28] and
High bandwidth memory (HBM) [37] enable memory tiers with higher than DDR
bandwidth. A higher bandwidth tier in the memory hierarchy enables workloads
to meet their bandwidth needs from a small capacity memory while using DDR or
CXL-based low-bandwidth, large-capacity tier to meet their memory capacity
needs. HB-DIMMs (not to be confused with HBM) provide up to 2$\times$ the
bandwidth of conventional DDR5 memory but are expected to consume up to
2$\times$ the power and cost. On-package HBM [17] shows a similar trend
(higher bandwidth at the expense of higher power and cost [25]).

Figure 19 shows the possible memory bandwidth tiering architecture for cloud
workloads based on our observations of the nine representative workloads.
While emerging memories support significantly higher memory bandwidth, this
comes at increased power consumption and higher cost, making a single high
bandwidth tier impractical.

####  5.2.2 Evaluation

To evaluate the possibility of memory bandwidth tiering in datacenters, we
will expand memory bandwidth using an HB-DIMM tier, while using a CXL-based
memory tier for memory capacity (Figure 19). Please note that while we
evaluate a conservative scenario with a higher memory latency far tier (CXL-
based), possible memory bandwidth tiers can differ only in memory bandwidth
while having similar memory latencies. For example, a high-bandwidth tier uses
HB-DIMMs, while a low-bandwidth tier uses DDR memories.

Table 4: Capacity and theoretical peak memory bandwidth for the three memory bandwidth tiering configurations. Config. | Near Memory | Far Memory
---|---|---
Baseline | 100%, 100 GB/s | -
Ideal | 100%, 200 GB/s (HB-DIMM like) | -
Tiered | 37.5%, 200 GB/s (HB-DIMM like) | 62.5%, 100 GB/s (CXL-like)
Table 5: Measured relative throughput, mem BW, and relative throughput/cost for the three configurations (Table 4). |  | Measured |  | Relative
---|---|---|---|---
| Relative | BW (GiB/s) |  Cost (relative) | Tput.
Config | Tput. |  Near | Far |  Near | Far | Total | /cost
Baseline | 1 | 67.81 | - | 1.0 | - | 1.0 | 1
Ideal | 1.55 | 108.41 | - | 2.0 | - | 2.0 | 0.73
|  |  |  | 0.75 | 0.625 |  |
Tiered |  1.47 |  84.60 |  19.22 | (2$\times$37.5%) | (1$\times$62.5%) |  1.375 |  1.13

We use the three configurations listed in Table 4 for our evaluation. The
baseline configuration resembles current servers deployed in the datacenters
at a hyperscalar. In contrast, the Ideal configuration shows the maximum
possible performance improvement using a high-bandwidth-only tier. Finally,
the Tiered configuration allocates about 1/3 of the capacity using the high-
bandwidth tier that resembles HB-DIMMs, while rest of the capacity is
allocated on far memory that resembles CXL memory. The tiered configuration
uses Maruf et al.’s [27] page placement technique.

We use a dual-socket server with DDR-5 DIMMs and configurable channel counts
to allocate the memory bandwidth. Only one socket has its CPU cores enabled,
with the memory connected to this socket referred to as Near Memory. Memory
connected to the socket with all its CPU cores disabled is referred to as Far
Memory. HB-DIMMs are expected to have DDR-5 like latencies, so we model HB-
DIMMs using DDR-5 based near memory. Likewise, CXL memories are expected to
have latency and bandwidth characteristics similar to NUMA links between the
sockets [27] (Intel UPI). Thus, Far Memory represents a CXL memory tier.

####  5.2.3 Results

Figure 20: Relative throughput of the Reader workload for the three memory
configurations (Table 4).

Figure 20 shows the relative throughput of the Reader microservice serving
live production traffic for the three configurations. At the steady state
($t>100~{}\mathrm{s}$), the Tiered memory configuration achieves 1.46$\times$
better throughput than the Baseline and within 6.32% of the Ideal
configuration. Out of the three configurations, the Tiered configuration takes
the longest to warm up from the page migration overhead between near and far
memories, which the Baseline and Ideal configurations do not have. Due to the
limited and geo-restricted availability of the evaluation server machine, we
only evaluate the Reader microservice, which is the most backend-bound
workload out of the nine representative workloads (Figure 7).

Table 5 lists the three configurations’ measured memory bandwidth and relative
throughput/cost As HB-DIMMs are not commercially available, for our TCO
analysis, we use an early assessment of HB-DIMM’s cost as 2$\times$ that of
DDR-5 DIMMs. The tiered memory configuration provides a 13% improvement in
performance/cost over the Baseline, and a 54.8% improvement over the Ideal
configuration, making it the most cost-efficient memory hierarchy, confirming
our memory bandwidth observations using MemProf.

##  6 Understanding Memory Latency

Figure 21: IPC and mem BW change with L2 prefetchers enabled.

Memory latency is another significant concern for cloud workloads [3], with
server processors often relying on hardware prefetchers to alleviate the
performance penalty. With cloud microservices operating under strict request
latency SLO constraints, workloads must limit CPU utilization to meet the SLO
guarantees, resulting in wasted resources, as discussed in Section 2.2.

A common solution to hide memory latency, the hardware prefetchers, work by
prefetching soon-to-be-used cachelines, thus reducing the overall memory
latency. While prefetchers hide some memory latency, they also significantly
increase memory bandwidth consumption. Figure 21 shows the increase in IPC and
the corresponding increase in memory bandwidth with L2 prefetchers turned on.
Several workloads show a significant increase in memory bandwidth consumption
(e.g., for Cache1, it increases by 31%), signaling prefetcher inefficiencies.
While we present data for L2 prefetchers, from our experiments, we observe
that L1 prefetchers have much higher efficiencies; thus, we omit them from our
study.

Although modern hardware prefetchers monitor bandwidth utilization and
throttle to reduce inefficiencies [16], we observe that workloads like Ads1
and Reader still suffer from a large memory bandwidth overhead.

Using Figure 21, we make several important observations: (a) while some
workloads show negligible IPC improvement, we keep L2 prefetchers enabled as
the servers have enough memory bandwidth headroom for the listed workloads. In
the past, with limited memory bandwidth, turning L2 prefetchers off has
resulted in performance improvements. (b) Workloads with predictable memory
access patterns (e.g., CPU-based inference) like Ads1 show a significant IPC
improvement, suggesting efficient hardware prefetchers are important in
improving performance.

Next, we will look into reasons for prefetcher inefficiencies and future
opportunities to mitigate them using production memory access traces.

###  6.1 Hardware Prefetcher’s Accuracy and Coverage

Figure 22: L2 hardware prefetcher accuracy and coverage.

Figure 22 shows the accuracy and coverage of L2 hardware prefetchers for the
nine workloads and SPEC 2006 CPU benchmark. Most cloud workloads show very low
coverage (<50%) but are relatively accurate (>75%). This tells us that the L2
hardware prefetcher cannot predict a large fraction of access patterns of the
workloads. Further, it is interesting to note that cloud workloads on average,
have considerably lower prefetcher coverage compared to the SPEC benchmark,
38.1% vs. 67.0%.

While there have been proposals to improve hardware prefetchers [26, 4],
research on hardware prefetchers for cloud workloads is limited by access to
these workloads and the difficulty of running them in a simulator. Studying
their memory access traces is an alternative, but today’s memory tracing tools
have a significant runtime overhead, making them impractical for production
workloads.

###  6.2 Efficient Memory Tracing for Future Architecture Research

Tracing memory accesses in production workloads poses several challenges,
necessitating the development of a new tracing technique. In response, we
present MemProf.Trace, a low-overhead memory access tracing tool. Next, we
look at the challenges in tracing memory accesses in a production environment
followed by the description and evaluation of MemProf.Trace.

####  6.2.1 Challenges of Tracing Memory

The two major challenges are:

Performance overhead. Common memory tracing tools use dynamic instrumentation
to trace accesses but add significant execution overhead to the application.
The effect of workload slowdown is different across the different production
workloads we tried to trace, with common causes being: (a) the kernel
scheduler schedules some other thread on the core, (b) requests timeout,
resulting in retries or failures, and (c) the load balancer throttles machine
load.

Dynamic application phases. Tracing memory access at smaller granularities to
limit overhead poses a challenge with workloads that exhibit diverse and
short-lived application phases. Examples of these phases are: parsing a
request, communicating with other microservices, or querying a database. Thus,
collecting a single small trace (millions of accesses) would not be enough to
get representative behavior of the workload.

####  6.2.2 MemProf.Trace – Low-overhead Memory
Tracing

To overcome these challenges, we built a PIN Tool that periodically attaches
itself to a process, records memory access for a configurable duration (in the
order of microseconds), and detaches itself to let the workload continue. We
collect several such traces from multiple hosts and stitch them together to
create a representative workload trace. Thus allowing low-overhead tracing
where services do not violate their request-level latency guarantees.

Table 6: Measured (Prod.) and simulated (Trace) results. | L1D Hit Ratio | R:W Ratio
---|---|---
| Prod. | Trace | Error | Prod. | Trace | Error
Cache1 | 0.93 | 0.88 | 5.38% | 1.84 | 1.92 | -4.34%
Feed | 0.95 | 0.93 | 2.11% | 2.14 | 2.20 | -2.8%
Web1 | 0.94 | 0.90 | 4.25% | 1.72 | 1.67 | 2.3%
Figure 23: Tracing overhead, MemProf vs DynamoRIO.

####  6.2.3 Overhead and Trace Validation

Performance Overhead. We compare MemProf.Trace against state-of-the-art memory
tracer, DynamoRIO [8], and native execution for gcc and bzip2 from the SPEC
2006 CPU benchmark. Figure 23 shows that compared to MemProf, DynamoRIO has
orders of magnitude higher performance overhead. For the experiment, we
configured MemProf’s sampling rate at 100 ms every 5 s. Due to long execution
times, we limit performance evaluation to two workloads.

Trace Validation. To verify the accuracy of the traces, we used a simple cache
simulator with the same cache architecture as the production servers and
verified the L1D hit rate. Table 6 shows the measured and simulated hit rate
for the L1-D cache and the R:W ratios. All workloads show a small error,
signifying that the traces are accurate for studying the memory behavior of
the workloads.

We will make these traces available to the community in the future to help
advance hardware prefetcher research.

##  7 Related Works

Some previous works have analyzed data center workloads code and data behavior
to understand their characteristics. To the best of our knowledge, none of the
prior works have looked at code sharing across threads and processes, memory
bandwidth distribution of production workloads, and the impact of hardware
prefetchers on memory bandwidth and performance.

Ayers et al. [3] analyzed Google’s web search workload, observing large code
footprints and high MPKI for caches. They proposed a large, low-latency L4
cache. In contrast, we suggest sharing L2 I-Cache and L2 I-TLB among four-core
clusters, improving performance with minimal area and power increase. Kundu et
al. [23] also studied a shared L1 cache design, but using an OLTP benchmark.
Similarly, Alves et al. [2] studied shared L2 cache (data+code) using the NAS
benchmark. HPC benchmarks have significantly different performance
characteristics, so they observe significant performance regression from cache
sharing. Tam et al. [39] use software-based L2 cache partitioning to avoid
contention between cores which resembles Intel’s CAT [12].

Kanev et al. [20] investigated “datacenter tax” overhead at Google,
identifying I-cache as a significant bottleneck due to growing code footprints
in cloud workloads. They observed low memory bandwidth consumption resulting
from overprovisioned machines, contrasting our findings that memory bandwidth
is increasingly concerning. Nevertheless, MemProf insights can help future
hardware prefetcher research, offering possibilities for overall performance
improvements.

SoftSKU [36] examines Meta’s cloud workloads, understanding OS and hardware
behavior using "Soft SKUs" by adjusting core/uncore frequencies, LLC
prioritization, and active core counts. They observe similar IPC, cache, and
TLB MPKIs. Accelerometer [35] studies Meta’s cloud workloads, finding most
cycles spent on non-core application tasks, such as compression and
serialization.

Some earlier works [40, 7] have also looked at µ-arch improvements for cloud
workloads. A few workloads have analyzed the commercial benchmarks using
similar techniques, e.g., PIN Tool [18] and top-down analysis [41].

For memory traces, we ensure that they accurately represent production
behavior which we validate using µ-arch statistics. Ranganathan et al. [33]
collected traces from production using dynamoRIO [8]. Our approach is
complementary to theirs. Payer et al. [31] leverage 64-bit register sizes
while executing 32-bit code to trace memory accesses, implementing a
lightweight memory tracing solution. However, 32-bit code has become
increasingly uncommon in modern computing. Daptrace [24] employs page table
access bit sampling to detect hot objects but does not record individual
memory accesses. HMTT [6] uses a dedicated hardware board for DIMM monitoring
without application changes. However, HMTT requires specialized hardware and
server modifications, often impractical in data centers. In contrast, MemProf
avoids hardware changes, ensures minimal intrusiveness, and achieves low
performance overhead.

Some recent works have looked into the memory prefetcher behavior of SPEC and
cloud workloads. Litz et al. [26] present critical slice prefetching (CRISP)
to prefetch hard-to-predict loads. Jamilan et al. [19] and I-SPY [22] rely on
Intel’s LBR and dynamic execution information to optimize data and instruction
prefetching, respectively. Ayers et al. [4] study SPEC and Google workloads
and present an automated way of classifying memory access patterns for
software-based prefetching. Hashemi et al. [15] use machine learning to study
memory traces. MemProf enables low-overhead, accurate traces complementing
their techniques. These works are orthogonal to ours and can be used together
to optimize different aspects of cloud workloads.

##  8 Conclusion

Memory subsystem bottlenecks are one of the primary reasons for stagnating IPC
on cloud workloads. Datacenters are a significant portion of the server
processor market, but benchmarks used for CPU designs often do not represent
their characteristics. In this work, we present a detailed characterization of
Meta’s cloud workloads using MemProf and use the findings that cores share
code cachelines to propose shared L2 cache and shared L2 I-TLB caches. Next,
we look into the memory bandwidth behavior of live cloud workloads and
evaluate memory bandwidth tiering to achieve throughput and TCO improvements.
Finally, we look into ways to improve memory latency. We collect production
memory traces and verify their accuracy for future memory prefetcher research.

## References

  * [1] Alaa R Alameldeen and David A Wood.  IPC considered harmful for multiprocessor workloads.  IEEE Micro, 26(4):8–17, 2006.
  * [2] Marco AZ Alves, Henrique C Freitas, and Philippe OA Navaux.  Investigation of shared l2 cache on many-core processors.  In 22th International Conference on Architecture of Computing Systems 2009, pages 1–10. VDE, 2009.
  * [3] Grant Ayers, Jung Ho Ahn, Christos Kozyrakis, and Parthasarathy Ranganathan.  Memory hierarchy for web search.  In 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 643–656. IEEE, 2018.
  * [4] Grant Ayers, Heiner Litz, Christos Kozyrakis, and Parthasarathy Ranganathan.  Classifying memory access patterns for prefetching.  In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pages 513–526, 2020.
  * [5] Grant Ayers, Nayana Prasad Nagendra, David I August, Hyoun Kyu Cho, Svilen Kanev, Christos Kozyrakis, Trivikram Krishnamurthy, Heiner Litz, Tipp Moseley, and Parthasarathy Ranganathan.  Asmdb: understanding and mitigating front-end stalls in warehouse-scale computers.  In Proceedings of the 46th International Symposium on Computer Architecture, pages 462–473, 2019.
  * [6] Yungang Bao, Mingyu Chen, Yuan Ruan, Li Liu, Jianping Fan, Qingbo Yuan, Bo Song, and Jianwei Xu.  Hmtt: a platform independent full-system memory trace monitoring system.  In Proceedings of the 2008 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, pages 229–240, 2008\.
  * [7] Luiz André Barroso, Kourosh Gharachorloo, and Edouard Bugnion.  Memory system characterization of commercial workloads.  In Proceedings. 25th Annual International Symposium on Computer Architecture (Cat. No. 98CB36235), pages 3–14. IEEE, 1998.
  * [8] Derek Bruening, Qin Zhao, and Saman Amarasinghe.  Transparent dynamic instrumentation.  In Proceedings of the 8th ACM SIGPLAN/SIGOPS conference on Virtual Execution Environments, pages 133–144, 2012.
  * [9] Dehao Chen, David Xinliang Li, and Tipp Moseley.  Autofdo: Automatic feedback-directed optimization for warehouse-scale applications.  In Proceedings of the 2016 International Symposium on Code Generation and Optimization, pages 12–23, 2016.
  * [10] Intel Corporation.  Precise events.  Intel® VTune™ Profiler User Guide.  https://www.intel.com/content/www/us/en/develop/documentation/vtune-help/top/analyze-performance/custom-analysis/custom-analysis-options/hardware-event-list/precise-events.html.
  * [11] Intel Corporation.  Code and data prioritization (CDP) technology.  In Intel 64 and IA-32 Architectures Software Developer’s Manual, volume 3B, pages 17–54 – 17–67. April 2022.
  * [12] Intel Corporation.  Intel® resource director technology (Intel® RDT) allocation features.  In Intel 64 and IA-32 Architectures Software Developer’s Manual, volume 3B, pages 17–50 – 17–67. April 2022.
  * [13] Miyuru Dayarathna, Yonggang Wen, and Rui Fan.  Data center energy consumption modeling: A survey.  IEEE Communications surveys & tutorials, 18(1):732–794, 2015.
  * [14] Jianru Ding, Ruiqi Cao, Indrajeet Saravanan, Nathaniel Morris, and Christopher Stewart.  Characterizing service level objectives for cloud services: Realities and myths.  In 2019 IEEE International Conference on Autonomic Computing (ICAC), pages 200–206. IEEE, 2019.
  * [15] Milad Hashemi, Kevin Swersky, Jamie Smith, Grant Ayers, Heiner Litz, Jichuan Chang, Christos Kozyrakis, and Parthasarathy Ranganathan.  Learning memory access patterns.  In International Conference on Machine Learning, pages 1919–1928. PMLR, 2018.
  * [16] Wim Heirman, Kristof Du Bois, Yves Vandriessche, Stijn Eyerman, and Ibrahim Hur.  Near-side prefetch throttling: Adaptive prefetching for high-performance many-core processors.  In Proceedings of the 27th International Conference on Parallel Architectures and Compilation Techniques, pages 1–11, 2018.
  * [17] Intel Corporation.  Speed-up memory bandwidth bound workloads with HBM.  2022\.  https://www.intel.com/content/www/us/en/high-performance-computing/hpc-high-bandwidth-memory-video.html.
  * [18] Aamer Jaleel et al.  Memory characterization of workloads using instrumentation-driven simulation–a pin-based memory characterization of the spec cpu2000 and spec cpu2006 benchmark suites.  Intel Corporation, VSSAD, 2007.
  * [19] Saba Jamilan, Tanvir Ahmed Khan, Grant Ayers, Baris Kasikci, and Heiner Litz.  APT-GET: Profile-guided timely software prefetching.  In Proceedings of the Seventeenth European Conference on Computer Systems, pages 747–764, 2022.
  * [20] Svilen Kanev, Juan Pablo Darago, Kim Hazelwood, Parthasarathy Ranganathan, Tipp Moseley, Gu-Yeon Wei, and David Brooks.  Profiling a warehouse-scale computer.  In Proceedings of the 42nd Annual International Symposium on Computer Architecture, pages 158–169, 2015.
  * [21] The Linux kernel user’s and administrator’s guide.  Idle page tracking.  https://docs.kernel.org/admin-guide/mm/idle_page_tracking.html, Feb 2015.
  * [22] Tanvir Ahmed Khan, Akshitha Sriraman, Joseph Devietti, Gilles Pokam, Heiner Litz, and Baris Kasikci.  I-spy: Context-driven conditional instruction prefetching with coalescing.  In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 146–159. IEEE, 2020.
  * [23] Partha Kundu, Murali Annavaram, Trung Diep, and John Shen.  A case for shared instruction cache on chip multiprocessors running oltp.  In Proceedings of the 2003 workshop on MEmory performance: DEaling with Applications, systems and architecture, pages 11–18, 2003.
  * [24] Yunjae Lee, Yoonhee Kim, and Heon Y Yeom.  Lightweight memory tracing for hot data identification.  Cluster Computing, 23:2273–2285, 2020.
  * [25] Shang Li, Dhiraj Reddy, and Bruce Jacob.  A performance & power comparison of modern high-speed dram architectures.  In Proceedings of the International Symposium on Memory Systems, pages 341–353, 2018.
  * [26] Heiner Litz, Grant Ayers, and Parthasarathy Ranganathan.  Crisp: critical slice prefetching.  In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pages 300–313, 2022.
  * [27] Hasan Al Maruf, Hao Wang, Abhishek Dhanotia, Johannes Weiner, Niket Agarwal, Pallab Bhattacharya, Chris Petersen, Mosharaf Chowdhury, Shobhit Kanaujia, and Prakash Chauhan.  Tpp: Transparent page placement for cxl-enabled tiered memory.  arXiv preprint arXiv:2206.02878, 2022.
  * [28] James A McCall, Rajat Agarwal, George Vergis, and Bill Nale.  High bandwidth DIMM, March 30 2021.  US Patent 10,963,404.
  * [29] Rick Merritt.  The whales changing the industry.  EE Times, 2019.  https://www.eetasia.com/the-whales-changing-the-industry/.
  * [30] Maksim Panchenko, Rafael Auler, Bill Nell, and Guilherme Ottoni.  Bolt: a practical binary optimizer for data centers and beyond.  In 2019 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), pages 2–14. IEEE, 2019.
  * [31] Mathias Payer, Enrico Kravina, and Thomas R Gross.  Lightweight memory tracing.  In 2013 $\\{$USENIX$\\}$ Annual Technical Conference ($\\{$USENIX$\\}$$\\{$ATC$\\}$ 13), pages 115–126, 2013.
  * [32] Milan Radulovic, Darko Zivanovic, Daniel Ruiz, Bronis R de Supinski, Sally A McKee, Petar Radojković, and Eduard Ayguadé.  Another trip to the wall: How much will stacked dram benefit hpc?  In Proceedings of the 2015 International Symposium on Memory Systems, pages 31–36, 2015.
  * [33] Parthasarathy Ranganathan and Victor Lee.  Advancing systems research with open-source google workload traces.  https://cloud.google.com/blog/topics/systems/workload-traces-for-google-warehouse-scale-computers.
  * [34] Anton Shilov.  SK Hynix, Intel develop MCR DIMM: DDR5-8000+ for high-capacity modules, 2022.  https://www.tomshardware.com/news/sk-hynix-develops-mcr-dimm.
  * [35] Akshitha Sriraman and Abhishek Dhanotia.  Accelerometer: Understanding acceleration opportunities for data center overheads at hyperscale.  In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pages 733–750, 2020.
  * [36] Akshitha Sriraman, Abhishek Dhanotia, and Thomas F Wenisch.  Softsku: Optimizing server architectures for microservice diversity@ scale.  In Proceedings of the 46th International Symposium on Computer Architecture, pages 513–526, 2019.
  * [37] JEDEC Standard.  High bandwidth memory (HBM) DRAM.  Jesd235, 16, 2013.
  * [38] Manoj Sukumaran.  Data center server market tracker – 2Q21 analysis.  Technical report, Omdia, June 2021.  https://omdia.tech.informa.com/OM018242/Data-Center-Server-Market-Tracker--2Q21-Analysis.
  * [39] David Tam, Reza Azimi, Livio Soares, and Michael Stumm.  Managing shared l2 caches on multicore systems in software.  In Workshop on the Interaction between Operating Systems and Computer Architecture, pages 26–33, 2007.
  * [40] Pedro Trancoso, J-L Larriba-Pey, Zheng Zhang, and Josep Torrellas.  The memory performance of dss commercial workloads in shared-memory multiprocessors.  In Proceedings Third International Symposium on High-Performance Computer Architecture, pages 250–260. IEEE, 1997.
  * [41] Ahmad Yasin.  A top-down method for performance analysis and counters architecture.  In 2014 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 35–44. IEEE, 2014.