countblanklinestrue[t]#1

# Optimizing Inference Performance of Transformers on CPUs

Dave Dice  Oracle LabsBurlington, MAUSA<EMAIL_ADDRESS>and  Alex Kogan
Oracle LabsBurlington, MAUSA<EMAIL_ADDRESS>
###### Abstract.

The Transformer architecture revolutionized the field of natural language
processing (NLP). Transformers-based models (e.g., BERT) power many important
Web services, such as search, translation, question-answering, etc. While
enormous research attention is paid to the training of those models,
relatively little efforts are made to improve their inference performance.
This paper comes to address this gap by presenting an empirical analysis of
scalability and performance of inferencing a Transformer-based model on CPUs.
Focusing on the highly popular BERT model, we identify key components of the
Transformer architecture where the bulk of the computation happens, and
propose three optimizations to speed them up. The optimizations are evaluated
using the inference benchmark from HuggingFace, and are shown to achieve the
speedup of up to x2.37. The considered optimizations do not require any
changes to the implementation of the models nor affect their accuracy.

##  1\. Introduction

The introduction of the Transfomer architecture for deep neural networks
(DNNs) by Vaswani et al. (VSP17, ) has literally transformed the field of NLP.
It happened just a few years ago (in 2017, to be exact), and since then the
field has exploded with an enormous wave of Transfomer-based models achieving
state-of-the-art, and often super-human, performance on many NLP tasks, which
just recently have been considered unrealistically difficult to solve. BERT
(DCL19, ), RoBERTa (LOG19, ), ALBERT (LCG20, ), Transformer-XL (DYY19, ) are
only very few examples in the vast sea of published models (XWD20, ). As of
today, Transfomer-based models, and BERT in particular, power many important
Web services, such as search (Nay19, ; NYZ20, ), translation and text
classification (LK20, ).

The big premise of the Transfomer-based models is that they can be pre-trained
on huge amounts of unlabeled data (such as all of Wikipedia or a book corpus),
and later fine-tuned to a specific task (e.g., question-answering) using just
a small amount of labeled, domain-specific data. To achieve high accuracy,
those models feature millions (and, at times, billions) of parameters, and
require long and expensive training. As a result, numerous efforts have been
made to optimize the training performance of those models (LCG20, ; YRH20, ;
LOG19, ; GHL19, ). At the same time, and despite the vast deployment of those
models in practice, far less attention is paid to inference performance.
Furthermore, among the efforts that do target inference performance of
Transformer-based models, many consider GPU or smartphone-based deployments
(FYZ20, ; WBC19, ; WWC20, ; Zhu19, ), even though in many practical settings
the inference is done on small CPU-based systems (NYZ19, ; LK20, ).

This paper comes to address this gap by presenting an empirical analysis of
scalability and performance of inferencing Transfomer-based models on CPUs. We
identify the key component of the Transformer architecture where the bulk of
the computation happens, namely, the matrix multiplication (matmul)
operations, and propose three optimizations to speed them up.

The first optimization is based on the observation that the performance of the
matmul operation is heavily impacted not only by the shape (dimensions) of the
source matrices and the available computing resources (the number of worker
threads), but also by whether (at least) one of those matrices is provided in
a transposed form. We propose a lightweight method to adaptively choose the
appropriate form of source matrices for the inference, which results in
substantial performance improvement of the latter. The second optimization
stems from the observation that an invocation of matmul operations in deep
learning (DL) frameworks incurs a significant sequential overhead, leading to
the poor scalability of those operations. We analyze the source of the
overhead, and demonstrate how the scalability of matmul operations (and the
overall inference performance) can be improved by reducing (some portion of)
that overhead. Finally, the third optimization builds on the realization that
while performant matmul operations are typically implemented by partitioning
matrices into sub-matrices and carrying out the actual computation using
highly optimized inner kernels (GG08, ), the partitioning itself might be
suboptimal and not fully utilize parameters of the underlying hardware (such
as cache capacity). We show how choosing different parameters for matrix
partitioning results in faster matmul operations. We evaluate the efficacy of
our optimizations using the industry-grade inference benchmark from
HuggingFace (WDS20, ).

We note that prior work shows many factors impacting the inference performance
of DNN models (WWW19, ), including the choice of a DL framework, a math
library, a thread pool library, availability of certain hardware features,
such as the support for SIMD (single instruction multiple data), etc. To make
our analysis feasible, we make several conscious choices when setting up our
experimentation environment, focusing on the inference performance of BERT
implemented in the widely used Pytorch framework (pytorch, ) built with the
state-of-the-art oneDNN math library (onednn, ) (previously known as MKL-DNN)
and run on an Intel Skylake processor-powered system (which supports AVX512
SIMD instructions). While the chosen setup is significant, we validate the
generality of many of our findings in other variations of our setup such as
with other Transformer-based models and math libraries. We also note that
despite the focus of our work being on NLP and Transformer-based models in
particular, we believe our findings extend to any model in which matmul
operations consume a significant portion of the inference time.

The rest of the paper is organized as follows. We provide the relevant
background on Transformers and BERT in Section 2. The related work is
discussed in Section 3. We describe our evaluation setup in Section 4 and
provide the analysis of the inference performance of BERT on CPUs in Section
5. Based on this analysis, we describe three optimizations for inference
performance in Section 6. Finally, we conclude in Section 7 with a discussion
of the results and some of the future directions.

##  2\. Background: Transfomer and BERT

The Transformer architecture (VSP17, ) is composed of two stacks of identical
layers; those stacks are called encoder and decoder. For the purpose of this
paper, we will focus on the encoder stack only, which is used exclusively in
many actual Transformer-based models, including BERT. In fact, we will note
upfront that BERT’s model architecture is almost identical to the Transformer
encoder, only tweaking the number of layers, the activation function, etc.
(DCL19, ). Also, we note that BERT itself has multiple configurations that
differ in the various model hyper-parameters (e.g., the “base” configuration
for BERT has $12$ layers while the “large” one has $24$). Unless specified
otherwise, when we say BERT in this paper, we refer to to its “base”
configuration (DCL19, ).

Each encoder layer has two sublayers, the first being a _multi-head self-
attention mechanism_ and the second being a _position-wise fully-connected
feed-forward network_. A _residual connection_ is employed around each of the
sub-layers, followed by _layer normalization_.

The attention mechanism is at the heart of the Transformer architecture. For
the purpose of this paper we will focus on the actual computations performed
by this mechanism; the explanation of the intuition behind those computations
can be found in many excellent sources (Ala18, ; harvard-nlp, ), including in
the original paper (VSP17, ). Specifically, the attention mechanism takes as
an input three matrices Q, K and V and computes the output matrix:

(1) $Attn(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d}_{k}})V$

where $d_{k}$ is the attention input dimension ($64$ for the BERT model). As
mentioned above, each self-attention sublayer includes multiple heads ($12$
for the BERT model). The computed function of this sublayer is given by the
following expressions:

(2) $\displaystyle MultiHead(Q,K,V)=Concat(head_{1},...,head_{h})W^{O}$
$\displaystyle\textnormal{where}~{}head_{i}=Attn(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V})$

where $W^{O}$, $W_{i}^{Q}$, $W_{i}^{K}$ and $W_{i}^{V}$ are parameter
matrices. Overall, the computation of the multi-head self-attention requires 4
matrix multiplications to create input token projections (the Q, K and V
matrices) and the projection of the concatenated output of all the multiple
heads. (We note that when Transformer is implemented in Pytorch, each of those
multiplications are performed during the computation of the corresponding
Linear modules.) In addition, two batched matrix multiplications are required
to calculate the $Attn$ function in Equation 1. Furthermore, the self-
attention sublayer includes the invocation of softmax and layer normalization
operations.

As for the fully-connected feed-forward sublayer, it consists of two linear
transformations with an activation function in between:

(3) $\displaystyle FFN(x)=Act(xW_{1}+b_{1})W_{2}+b_{2}$

where $W_{1}$, $b_{1}$, $W_{2}$ and $b_{2}$ are weight and bias matrices,
respectively (which are model parameters, one set for each layer) and $Act$ is
an activation function, such as _gelu_ (HG16, ). While the inputs and outputs
of the feed-forward sublayer have the same dimensions as the rest of the model
($768$, in case of BERT), the inner-layer has a larger dimensionality ($3072$
for BERT). It is easy to see that the computation of the feed-forward sublayer
requires two matrix multiplication operations (carried by two Linear modules
in Pytorch), as well as an activation function and a layer normalization
operation.

(a) Seq. length 8

(b) Seq. length 64

(c) Seq. length 384

Figure 1. Performance breakdown for BERT by sub-layers and their components.

##  3\. Related Work

There is a relatively small body of work we are aware of on optimizing
inference performance of NLP models on CPUs. Ning et al. (NYZ20, ) describe
their effort on accelerating BERT with ONNX Runtime, an inference engine
compatible with PyTorch and TensorFlow. The idea is to fuse multiple
operations in the computation graph (e.g., matrix multiplication, layer
normalization and gelu) to reduce the amount of overhead (e.g., memory
copying) in invoking each elementary computation individually. They also
experiment with reducing the number of layers in BERT sacrificing (some)
accuracy for higher performance. In general, we note that the operation fusion
is a known technique for optimizing inference performance, and is orthogonal
to the optimizations described in this paper. Also, our techniques aim for
performing the given inference computations faster, but without any change to
the accuracy.

Wu et al. (NYZ19, ) describe another effort to optimize inference of BERT in
Apache MXNet using the GluonNLP toolkit. They report on speedups achieved by
using the MKL math library in MXNet as well as from quantizing the model for
better performance with lower precision. We note that we use MKL as one of the
baseline configurations in our analysis, while model quantization is, once
again, orthogonal to the ideas discussed in this paper and may result in
reduced accuracy.

There is an enormous effort on refining an/or replacing the attention
mechanism with a more efficient alternative that requires less computation
and/or allows scaling for longer sentences, e.g., (CTB20, ; ZGD20, ; WLM20, ;
BPC20, ). While most of those efforts are primarily concerned with speeding up
training, they help inference directly or indirectly as well. Notably, one of
the goals behind the _knowledge distillation_ effort (SDC19, ; SCG19, ; WWD20,
), i.e., training a smaller model (student) to achieve a similar accuracy as a
larger one (teacher), is reducing the inference latency. Indeed, Le and
Kaehler describe how they employ distillation with quantization to speedup
their deployment of BERT on CPUs (LK20, ). We believe the optimizations
described in this paper apply to most of such work. In particular, we show the
speedups achieved by our optimization for inferencing DistilBert (SDC19, ), a
popular model that uses knowledge distillation, are similar to those of BERT.

In a broader context, Liu at et. (LWY19, ) describe an approach called NeoCPU
for optimizing CNN inference on CPUs. In addition to the common optimizations
of operation fusion and inference simplification, NeoCPU manipulates the data
layout flowing through the model to minimize the overhead of transforming the
data between various individual operations. Fang et al. (FYZ20, ) present
TurboTransformers, a GPU-based serving system for Transformer models. They
describe a number of optimizations targeting GPUs, such as memory management
and batch reduction. Optimizing inference of NLP models on GPUs has been also
the motivation behind the work by Wang et. al (WWC20, ). At the same time, Wu
et al. (WBC19, ) describe opportunities and design challenges in enabling
machine learning inference on smartphones and other edge platforms.

##  4\. Evaluation Environment

In this section, we describe the hardware and software setup for our
experiments. We ran the experiments on an Intel-based system featuring two
Intel Xeon Platinum 8167M processors with 26 hyperthreaded cores each, and
runs an Oracle Linux 7.8 OS. To avoid any non-uniform memory access effects,
we use the numactl utility to restrict all our experiments to executing on and
allocating memory from one socket only.

On the software side, we use Pytorch v1.6, a popular DL framework. We compile
Pytorch in the default configuration, which means that it employs MKL as its
default math library, but also includes support for oneDNN. While MKL is a
closed-source library, oneDNN is “an open-source cross-platform performance
library of basic building blocks for deep learning applications” (onednn, ),
and, unless stated otherwise, we use the latter in our experiments.

To invoke oneDNN bindings, one needs to convert a given model as well as the
input tensors into the so-called “mkldnn” data format, which dictates how data
is laid out in memory (onednn-memory-format, ). We note that oneDNN bindings,
however, are available for only a handful of DL operations, such as Linear and
Convolution modules. In practice, this means that for each such supported
operation, Pytorch would seamlessly convert input tensors into the mkldnn
format, apply the corresponding operation and convert the output tensors back
into the default (“dense”) format so they could be sent to other operations
for which oneDNN bindings are not provided. Such conversions do not come for
free, however, and involve memory layout translations and memory copying. To
avoid this overhead, we extend the integration of oneDNN with Pytorch, adding
the missing bindings for various operations invoked by a typical Transformer
model, such as the layer normalization, softmax and gelu activation functions.
The extension comprises of a few hundred lines of C++ and Python code. As we
show in Section 6.1, the resulting setup performs on-par with or better than
the default Pytorch configuration (which employs MKL).

We use the popular Transformers Python package (v3.0.2) from HuggingFace,
which provides a state-of-the-art implementation of numerous Transformer-based
NLP models implemented in Pytorch (and Tensorflow) (WDS20, ). In addition,
Transformers includes an easy-to-use inference benchmark, which we utilize
heavily for our experiments. Furthermore, we utilize mcbench (mcbench, ), the
open-sourced suite of microbenchmarks, which includes, among other things,
microbenchmarks for evaluating the performance of matrix multiplication
operations when invoked directly through the C++ API of the corresponding math
libraries.

##  5\. Inference Performance Analysis

(a) Seq. length 8

(b) Seq. length 64

(c) Seq. length 384

Figure 2. Performance breakdown for BERT by modules.

We instrument the BERT model implemented in Transformers (WDS20, ) and collect
timing information for various sub-layers (multi-head attention, feed-forward)
and modules (Linear, Softmax, etc.) composing the model while executing the
Transformers inference benchmark. We experiment with various input sequence
lengths and vary the number of threads from 1 to 16 (by setting the
OMP_NUM_THREADS environment variable). We note that although our experimental
machine has more than 16 cores, we deliberately decided to focus on smaller
setups, as practical inference deployments typically include a small number of
cores (NYZ19, ; LK20, ).

Figure 1 presents the inference latency as well as the breakdown of runtime
spent in two main sub-layers, attention and feed-forward (along with the small
portion of time not associated with any of the sub-layers, which consists
mostly of input embedding and pooling of the resulting tensor; this time is
denoted as a red box titled “Other”). In addition, we break the time in the
two sub-layers into major components. For the attention sub-layer, this is the
time spent in self-attention (“self”), the linear projection (“dense”), the
layer normalization (“layernorm”) and the rest (“other”). For the feed-forward
sub-layer, this is the time spent in two linear projections (“dense1” and
“dense2”), the layer normalization (“layernorm”) and the rest (“other”), e.g.,
the gelu activation function.

Overall, we see that the feed-forward sub-layer typically consumes more time
than the attention sub-layer. Linear projections, which ultimately translate
into matmul operations, are responsible for that. We do not break down the
attention sub-layer for better readability, yet the data shows that linear
projections for the Q, K and V matrices consume the large share ($50$–$75\%$)
of its time as well. Along with that, the attention sub-layer does include
other modules, e.g., softmax, tensor transpose and reshaping operations, etc.,
which explains why the time share of the attention sub-layer grows as we
increase the number of threads. Specifically, as we show in Section 6.2,
matmul operations, being carried out by carefully optimized math libraries
(oneDNN, in this case), scale almost linearly with the number of threads. At
the same time, other operations (including layer norm, softmax and tensor
reshaping) do not scale. As such, their relative portion grows while the
portion of time spent in matmul operations shrinks. The portion of non-
scalable operation is larger in the attention sub-layer, hence its weight
grows with the number of threads. This growth is more tamed when the input
sequence is large (cf. Figure 1 (c)) since the matmul operations are invoked
with larger matrices, thus consuming a larger portion of time w.r.t. all other
operations.

When examining the total runtime (the black curve in Figure 1), we note that
the overall scalability of the inference latency is relatively low, and
depends on the input sequence length. In particular, for sequences of 8
tokens, we achieve the speedup of only x3.3 when running with 16 threads
versus 1 thread; the speedup goes up to x9.7 for sequences of 384 tokens.
Better scalability with longer sequences is, once again, related to the
scalability of matmul operations in math libraries and the fact that larger
sequences result in heavier matmul operations (with larger matrices) —
reducing the time spent in matmul operations when the number of threads
increases has a larger effect on the overall scalability of the inference
latency.

In Figure 2, we present a different way to breakdown the inference runtime, by
the time spent in various models. While most module names are self-
explanatory, we note that “bmm” stands for batched matrix multiplication, the
operation at the heart of the multi-head attention mechanism (there are two of
those operations per each attention layer in BERT); ”other” stands for the
time spent in computations not included in the specific modules, such as the
time spent on transposing and reshaping tensors, input embedding, pooling of
the resulting tensor, etc.

Not surprisingly, the vast majority of the inference time is spent in the
Linear module, which in practice means matmul operations (the Linear module
applies a linear transformation to the incoming data, calculating a product of
the input matrix with the stored weight matrix). This concurs the results in
Figure 1. When summing up both “linear” and “bmm” runtime portions, the matmul
operations consume between $66.2\%$ and $91.5\%$ of the total runtime. Note
that this portion decreases as we increase the number of threads. As explained
above, this is because matmul operations are executed by a math library
(oneDNN), and are highly optimized and scale almost linearly with the number
of threads.

(a) #threads=1 (oneDNN)

(b) #threads=2 (oneDNN)

(c) #threads=4 (oneDNN)

(d) #threads=16 (oneDNN)

(e) #threads=16 (MKL)

(f) #threads=16 (OpenBLAS)

Figure 3. Matmul operation performance as a ratio between the time to multiply
two non-transposed matrices and the time to multiply a non-transposed matrix
by a transposed one.

##  6\. Optimizing Inference Performance

###  6.1. Adaptive Linear Module

The results in Section 5 show that the key to improving inference performance
on CPU lies in reducing the time spent in matmul operations.

For the context, the API for a matmul operation allows invoking that operation
on two source matrices A and B (producing the destination matrix C=AB) s.t.
each of the source matrices can be provided in the transposed form (onednn-
blas, ). At the same time, the Pytorch Linear module stores the weight matrix
in a transposed form, which means that, during inference, the input matrix (A)
is always non-transposed, while the weight matrix (B) is always transposed. We
believe the Pytorch designers made this choice (to transpose the weight
matrix) to achieve better performance for the backward pass during training
(pytorch-linear, ), a concern which is not relevant for inference111We note
that in Tensorflow, the weight matrix is _always_ given in the normal form to
the matmul operation..

Our experiments with the matmul microbenchmark from mcbench reveal an
interesting observation. Figure 3 demonstrates the ratio between the time to
compute matmul when both source matrices are non-transposed to the time to
compute matmul when (only) the second source matrix is transposed. In other
words, ratio ¿ 1 (ratio ¡ 1) corresponds to cases in which the former (latter,
respectively) method is faster. The shape of the second matrix (B) is
represented by the name of the corresponding data series, while the shape of
the first matrix (A) is given by the sequence length x first dimension of B.
Note that the chosen three shapes are not incidental, and they correspond to
the shapes of weight matrices used in Linear modules of the BERT model.

More concretely, Figure 3 (a)–(d) compare the performance of the matmul
operation in oneDNN across different numbers of threads. We see that for
shorter sequences, multiplying the non-transposed matrices is almost always
faster, and often results in substantial speedups. For longer sequences, the
picture is less clear – one way of applying a mutmul operation is faster than
the other for one shape but worse for another. In general, the faster way of
applying a matmul operation depends on the shape of the source matrices and
the number of threads used. We also confirmed that this observation is not
unique to oneDNN, and is reproducible, at least to some extent, with other
math libraries. Figure 3 (e) and (f) show the results obtained with MKL and
OpenBLAS libraries, respectively (for the latter, we used the benchmark
included in the library sources; for brevity, we include only the result for
$16$ threads).

One may wonder about the reason for this performance difference. In oneDNN,
much like in any math library for high-performance matmul calculation (GG08,
), the matmul operation is coded in assembly, and each of the matmul variants
(e.g., one with both source matrices in the normal form vs. one in which the
second matrix is transposed) results in a different code path, which generates
different memory access patterns. Based on the profiling information produced
by perf, we observe that given a certain configuration (i.e., the same source
matrix shapes and the number of threads), both variants have a similar number
of L1 data cache misses, but the faster variant has a lower number of L3 cache
accesses. This suggests that one reason for performance difference might be
the better utilization of L2 cache by one variant over the other.

#threads | sequence length |
---|---|---
8 | 64 | 384
onednn base |  onednn normal |  onednn almo |  onednn base |  onednn normal |  onednn almo |  onednn base |  onednn normal |  onednn almo |  mkl base |  mkl script
1 | 115 | 82 | 79 $\pm$ 1 (x1.46) | 216 | 193 | 162 (x1.33) | 884 $\pm$ 31 | 1009 | 807 (x1.10) | 967 $\pm$ 15 | 861
2 | 83 | 50 | 50 $\pm$ 1 (x1.66) | 133 | 105 | 97 $\pm$ 1 (x1.37) | 471 $\pm$ 29 | 512 | 413 (x1.14) | 522 | 453
4 | 51 | 34 | 34 (x1.50) | 76 | 64 | 60 (x1.27) | 259 $\pm$ 27 | 285 | 220 (x1.18) | 302 $\pm$ 6 | 245
8 | 35 | 27 | 27 (x1.30) | 49 | 45 | 42 (x1.17) | 135 | 148 | 128 (x1.05) | 193 | 138 $\pm$ 2
16 | 28 | 24 | 24 (x1.17) | 36 | 34 | 32 (x1.12) | 96 $\pm$ 14 | 97 | 82 (x1.17) | 142 $\pm$ 2 | 83 $\pm$ 1

Table 1. BERT-base inference latency (ms), for various sequence lengths. The
numbers in () show the speedup of onednn-almo over onednn-base. The numbers
after the $\pm$ sign specify the standard deviation when it is larger than 1%
of the mean.

Given the results in Figure 3, we propose the following optimization for the
Linear module. Each Linear module is augmented with a transposeFlags array,
specifying whether to use a transposed version of the weights matrix for the
forward pass (inference). Entry $i$ of the array corresponds to the sequence
length of $2^{i}$; the array has $10$ entries corresponding to the maximal
length of $512$ tokens. When creating a Linear module with the given weights
shape $[in,out]$, we generate random matrices with the shape $[2^{i},in]$, for
each $0\leq i<10$, and measure the time to perform a matmul operation when the
weight matrix is transposed or not. Based on the result, we set the
corresponding entry transposeFlags[i]. During the inference time, given the
input of shape $[length,in]$, we calculate $s=\lfloor\log(length)\rfloor$, and
based on the flag in transposeFlags[s], perform the matmul operation with
either weight matrix transposed or not.

To avoid the overhead of transposing the weight matrix during inference, we
keep both variants of the weight matrix (transposed and non-transposed one).
This doubles the memory footprint of the Linear module. While it might not be
a concern on some CPU deployments, there are several ways to mitigate this
drawback. First, some shapes always prefer one form over the other, for all
thread counts (e.g., the shape 3072-768 in Figure 3 (a)–(d)). For this case,
we can keep only the relevant variant of the weight matrix. Second, the length
of the input can be known prior to the deployment of an inference server,
e.g., in a farm of inference servers, certain servers can be configured to
handle input of a certain sequence length. Once again, in this case we can
keep only the relevant variant of the weight matrix. Finally, if the input
range is dynamic, one can store one variant of the weight matrix and transpose
on-demand. The selection of the stored variant can be also dynamic and tuned
based on the actual input lengths seen during the runtime. All those
mitigation ideas are left for the future work.

We note that transposeFlags can be shared among Linear modules of the same
shape. We use a key-value map (dictionary) to store transposeFlags arrays
where the $[in,out]$ tuple of corresponding Linear modules serves as a key.
Thus, when initializing the transposeFlags array, we query the dictionary
first, and if such a shape has been already profiled, we reuse the resulting
array, skipping the profiling phase for that Linear module. For the BERT
model, this optimization allows us to reduce the number of profiling phases
from 73 (6 Linear modules per each of the 12 self-attention layers plus one
for the input embedding) to 3 (one per each different shape). We emphasize
that the profiling is run only once, during the initialization of the model
(and its corresponding Linear modules), and is not invoked during inference.

Table 6.1 compares the performance of the HuggingFace inference benchmark run
on top of several Pytorch variants as described below. Each experiment is run
five times in the same configuration, and mean results are presented. We also
present the standard deviation for the cases where it was relatively high. The
variants we evaluate are mkl-base, which is the Pytorch version installed from
pip (and uses MKL); mkl-script, which is the base version run in the
torchscript mode (which creates “a serializable and optimizable models from
PyTorch code” (pytorch-jit, ) and therefore is a recommended mode for
inference (pytorch-jit2, )); onednn-base, which is the Pytorch version built
from sources and uses oneDNN; onednn-normal, which is the onednn-base version
in which the weight matrix is stored in a normal (non-transposed) shape; and
onednn-almo, which is the onednn-base version with the adaptive Linear module
optimization. We note that the first two variants are included for reference
only, to demonstrate that they perform mostly on-par with (and, at times,
worse than) onednn-base. Thus, we include them for one case only, for brevity.
We also note that the torchscript mode has a smaller impact when applied to
oneDNN-based variants, shaving about $7$-$9$ ms from the reported latency in
each case. The qualitative comparison between oneDNN-based variants does not
change, however (although, quantitatively, the torchscript mode leads to even
larger speedups for the adaptive optimization). Therefore, we do not include
the torchscript mode results for those variants.

The improvements in the inference latency achieved by the adaptive Linear
module optimization, as can be seen in Table 6.1, correlate strongly with the
results in Figure 3. Specifically, higher speedups are achieved on shorter
sequences and fewer number of threads, which are exactly the settings where
the ratios depicted in Figure 3 are the highest. We also note the need for
adaptivity – while onednn-normal performs well on shorter sequences, its
performance suffers on longer ones, which is exactly the settings in which,
according to Figure 3, multiplying the transposed weight matrix is faster. The
adaptive variant selects the correct shape in each case, performing on-par or
better than the other two oneDNN-based variants.

We note that our findings extend to other Transformer-based models. For
instance, Tables 6.1 and 6.1 present the results for RoBERTa and DistilBERT,
respectively, in their “base” configurations, while Tables 6.1 presents the
results for BERT-large, the larger version of the BERT model (DCL19, ). For
brevity, we include only the inference latencies for onednn-base and onednn-
almo variants. While the results for RoBERTa and DistilBERT largely follow
those for BERT-base, the results for BERT-large are slightly different. This
is because BERT-large uses a different number of hidden units (1024 vs. 768 in
other models we have considered), and thus operates with matrices of different
dimensions.

#threads | sequence length
---|---
8 | 64 | 384
onednn base |  onednn almo |  onednn base |  onednn almo |  onednn base |  onednn almo
1 | 116 | 79 (x1.47) | 216 | 162 (x1.33) | 860 | 809 (x1.06)
2 | 83 | 50 (x1.66) | 133 | 97 (x1.37) | 472 | 416 (x1.13)
4 | 50 | 35 (x1.43) | 76 | 60 (x1.27) | 250 | 221 (x1.13)
8 | 35 | 28 (x1.25) | 49 | 41 (x1.20) | 135 | 128 (x1.05)
16 | 29 | 24 (x1.21) | 36 | 32 (x1.12) | 112 | 82 (x1.37)

Table 2. RoBERTa inference latency (ms).

#threads | sequence length
---|---
8 | 64 | 384
onednn base |  onednn almo |  onednn base |  onednn almo |  onednn base |  onednn almo
1 | 58 | 38 (x1.53) | 108 | 81 (x1.33) | 437 | 402 (x1.09)
2 | 41 | 24 (x1.71) | 67 | 49 (x1.37) | 226 | 210 (x1.08)
4 | 26 | 17 (x1.53) | 39 | 30 (x1.30) | 120 | 111 (x1.08)
8 | 18 | 14 (x1.29) | 25 | 21 (x1.19) | 69 | 65 (x1.06)
16 | 14 | 12 (x1.17) | 18 | 16 (x1.12) | 45 | 41 (x1.10)

Table 3. DistilBERT inference latency (ms).

#threads | sequence length
---|---
8 | 64 | 384
onednn base |  onednn almo |  onednn base |  onednn almo |  onednn base |  onednn almo
1 | 460 | 284 (x1.62) | 811 | 786 (x1.03) | 3134 | 3063 (x1.02)
2 | 379 | 163 (x2.33) | 558 | 475 (x1.17) | 1717 | 1726 (x0.99)
4 | 209 | 100 (x2.09) | 300 | 299 (x1.00) | 908 | 908 (x1.00)
8 | 124 | 68 (x1.82) | 171 | 203 (x0.84) | 506 | 501 (x1.01)
16 | 82 | 56 (x1.46) | 109 | 120 (x0.91) | 293 | 318 (x0.92)

Table 4. BERT-large inference latency (ms).

For BERT-large and short sentences, onednn-almo achieves even more impressive
gains over onednn-base compared to BERT-base, reaching the speedup of x2.33.
For longer sentences, however, onednn-almo lags behind or performs on-par with
onednn-base. We identify the reason behind the performance regression as
following: The baseline performance of matmul operations established during
the profiling phase of the adaptive linear module optimization differs from
the actual performance when the inference is executed. In other words, during
profiling, we establish that using the normal form of the weight matrix is
faster than the transposed one, yet when we run inference, using weights in
the normal form ends up being slower! As we expand in Section 6.3, we
hypothesize that this happens due to the poor fitting of matrix partitioning
parameters in the math library (oneDNN) to hardware constraints, such as the
L2 cache capacity.

(g) oneDNN matmul performance

(h) Break down of Pytorch Linear module performance

Figure 4. Matmul performance when invoked directly though oneDNN API and
through the Pytorch Linear module.

###  6.2. Reducing Sequential Overhead

The results in Table 6.1 (as well as in Figures 1 and 2) underline poor
scalability of inference latency, especially for shorter sequences. This is
despite the fact that most of the inference time is spent in matmul operations
(c.f. Figure 2), and those operations exhibit nearly linear scalability. The
latter is demonstrated by the results from the mcbench microbenchmark shown in
Figure 4 (a), in which we measure the time to multiply two matrices of the
shape [8,768] and [768,768] as we vary the number of threads. (These shapes
correspond to the matmul operation invoked by linear projections in the
attention sublayer of the BERT model when the inference is performed on an
input sequence of 8 tokens).

To shed more light on where the matmul operation cycles are spent during
inference, we augmented Pytorch and oneDNN with timestamps. We ran the
following simple code that employed the Linear module only (rather than a
full-fledge model) and thus allowed to focus on the performance of matmul
operations:

⬇

1import torch

2from torch.utils import mkldnn as mkldnn_utils

3net = torch.nn.Linear(768, 768)

4net = mkldnn_utils.to_mkldnn(net)

5seq = torch.rand(8, 768).to_mkldnn()

6for i in range(0, 10000): net(seq)

Note that the forward path through the Linear module above invokes a matmul
operation on two matrices of the same shape as the ones used for the
experiment in Figure 4 (a).

With the collected profiling information, we break down the phases through
which the invocation of the forward pass of the Linear module goes, separating
the time spent in the Python interpreter, Pytorch dispatcher (that directs the
call to the oneDNN implementation of the Linear model), oneDNN dispatcher
(that selects the appropriate low-level matmul function), and finally, the
matmul (aka general matrix multiply, or GEMM) function itself.

The results are presented in the first (left) set of bars in Figure 4 (b).
They show that, indeed, the time in the GEMM function scales with the number
of threads. At the same time, the duration of the rest of the computation
phases does not change as the number of threads increases, implying that
matmul operations in Pytorch incur significant sequential overhead. This
overhead becomes even more substantial when the matmul operation is applied on
smaller matrices and/or with a large number of threads. This, according to the
Amdahl’s law, explains the poor overall scalability of the matmul operation.

We focus on the oneDNN dispatcher as a target for reducing the sequential
overhead. The dispatcher validates the input parameters, identifies the
capabilities of the underlying architecture (e.g., the type of supported AVX
instructions, if any), the number of available threads, etc. Based on this
information, it iterates over the list of available GEMM implementations and
selects the first that is compatible with the given set of requirements for
the matmul operation. While this process is necessary for a correct behavior
of oneDNN with arbitrary input matrices (or, in general, input parameters), we
observe that during inference, which constrains the set of possible inputs to
a few specific shapes, only one particular GEMM function is called, at least
when the number of threads is larger than one222The oneDNN dispatcher may
choose a different function when it detects that only a single thread is
available.. Thus, when more than one thread is used, we implement an
optimization where the dispatching process is reduced to call that function
directly, skipping the validation logic described above.

The result of this optimization is shown in the second (right) set of bars in
Figure 4 (b). We note that the time spent in the oneDNN dispatcher is reduced
substantially, leading to increasing the speedup of x4.42 at 16 threads
compared to a single thread (up from x3.4 without the optimization). Overall,
the speedup is still subpar to the one achieved with mcbench microbenchmark
(cf. Figure 4 (a)) because the rest of the sequential overhead remains.

#threads | sequence length
---|---
8 | 64 | 384
onednn base |  onednn almo+sor |  onednn base |  onednn almo+sor |  onednn base |  onednn almo+sor
1 | 115 | 78 (x1.47) | 216 | 165 (x1.31) | 884 | 806 (x1.10)
2 | 83 | 48 (x1.73) | 133 | 94 (x1.41) | 471 | 412 (x1.14)
4 | 51 | 32 (x1.59) | 76 | 56 (x1.36) | 259 | 215 (x1.20)
8 | 35 | 25 (x1.40) | 49 | 39 (x1.26) | 135 | 123 (x1.10)
16 | 28 | 22 (x1.27) | 36 | 30 (x1.20) | 96 | 78 (x1.23)

Table 5. BERT-base inference latency (ms).

#threads | sequence length |  |  |
---|---|---|---|---
8 | 64 | 384
onednn base |  onednn almo |  onednn almo+sor |  onednn base |  onednn almo |  onednn almo+sor |  onednn base |  onednn almo |  onednn almo+sor
1 | 460 | 288 (x1.60) | 287 (x1.60) | 811 | 579 (x1.40) | 573 (x1.42) | 3134 | 3003 (x1.04) | 2954 $\pm$ 46 (x1.06)
2 | 379 | 165 (x2.30) | 160 (x2.37) | 558 | 317 (x1.76) | 311 (x1.79) | 1717 $\pm$ 22 | 1631 $\pm$ 39 (x1.05) | 1606 $\pm$ 40 (x1.07)
4 | 209 | 101 (x2.07) | 96 $\pm$ 1 (x2.18) | 300 | 191 (x1.57) | 186 $\pm$ 2 (x1.61) | 908 | 868 $\pm$ 15 (x1.05) | 881 (x1.03)
8 | 124 | 68 (x1.82) | 63 (x1.97) | 171 | 120 (x1.42) | 114 (x1.50) | 506 | 480 $\pm$ 7 (x1.05) | 467 $\pm$ 19 (x1.08)
16 | 82 $\pm$ 1 | 55 (x1.49) | 50 (x1.64) | 109 | 87 $\pm$ 1 (x1.25) | 82 (x1.33) | 293 $\pm$ 3 | 285 $\pm$ 12 (x1.03) | 275 $\pm$ 8 (x1.07)

Table 6. BERT-large inference latency (ms) with the modified matrix
partitioning. The numbers in () show the speedup over onednn-base. The numbers
after the $\pm$ sign specify the standard deviation when it is larger than 1%
of the mean.

The effect of reducing the sequential overhead in matmul on the inference
performance is shown in Table 6.2. Here we present the comparison between
onednn-base and onednn-almo+sor, where the latter is the onednn-almo version
with the sequential overhead reduction optimization described in this section
applied. The new optimization shaves another 3–12% from the inference time,
with larger gains recorded at larger thread counts and/or smaller sequence
lengths. This is expected since those are the settings where the sequential
overhead has the most relative impact on the duration of matmul operations.

###  6.3. Modifying Matrix Partitioning

High-performance math libraries, including oneDNN, perform matrix
multiplication by partitioning the arguments into sub-matrices of certain
shape, which are then given to assembly-coded inner-kernels (GG08, ). This
design aims to amortize the cost of moving data across adjacent memory layers,
all while taking advantage of carefully engineered inner-kernels. Hence, on a
high level, the matrix multiplication operation can be expressed as the
following triple-nested loop:

⬇

1for (p = 0; p < sizeK; p+=BK)

2 for (i = 0; i < sizeM; i+=BM)

3 for (j= 0; j < sizeN; j+=BN)

4 $C_{ij}$+=$A_{ip}$$B_{pj}$

Various considerations take place when deciding on how to partition the
matrices (i.e., set BK, BM and BN above), including the size of the caches and
TLB (translation-look-aside buffers), the shape and layout of source matrices,
etc. (GG08, ). Yet, while some of those parameters are clearly hardware
dependent, the oneDNN implementation uses a set of constants to control the
partitioning333e.g., see sgemm_nocopy_driver() in https://github.com/oneapi-
src/oneDNN/blob/63c8b5ce84b0be266d1edad0420390f2e131cb29/src/cpu/x64/gemm/f32/jit_avx512_common_gemm_f32.cpp#L1808-L1816.
We strongly believe that the regressions reported for BERT-large in Section
6.1 are the results of excessively conservative fitting of those parameters to
the actual hardware.

As evidence, we reduce one of the parameters (BK, from $384$ to $64$444We note
that while we tried a few other settings for matrix partitioning, a
comprehensive sensitivity analysis of partitioning parameters is a part of the
future work.) so that, effectively, the matrix multiplication is carried by a
larger number of iterations of the outermost loop, where each inner-kernel is
activated on smaller sub-matrices that are more likely to fit into cache and,
in general, reduce the amount of cache misses. This has a highly positive
effect on the inference performance, as demonstrated in Table 6.2 with the
results of the BERT-Large model, which shows significant gains for onednn-almo
over onednn-base across most sequence lengths and thread counts. Performance
counters (reported by perf) show that with the patched version of oneDNN, the
number of last-level cache (LLC) misses is reduced. Also, while both (patched
and non-patched) versions report a similar number of instructions, the patched
version uses significantly less cycles, yielding a higher IPC (instruction per
cycle) ratio.

#threads | sequence length
---|---
8 | 64 | 384
onednn base |  onednn almo+sor |  onednn base |  onednn almo+sor |  onednn base |  onednn almo+sor
1 | 115 | 80 (x1.44) | 216 | 150 (x1.44) | 884 | 815 (x1.08)
2 | 83 | 48 (x1.73) | 133 | 84 (x1.58) | 471 | 421 (x1.12)
4 | 51 | 32 (x1.59) | 76 | 53 (x1.43) | 259 | 223 (x1.16)
8 | 35 | 24 (x1.46) | 49 | 37 (x1.32) | 135 | 128 (x1.05)
16 | 28 | 22 (x1.27) | 36 | 30 (x1.20) | 96 | 81 (x1.19)

Table 7. BERT-base inference latency (ms) with the modified matrix
partitioning.

The results for BERT-base with the modified matrix partitioning are given in
Table 6.3. They show that reducing the size of matrix partitions has a
favorable effect on this model as well.

##  7\. Discussion

In this paper we present the analysis of the inference performance for BERT,
one of the most prominent models for NLP based on the Transfomer architecture,
on CPU-based systems. The analysis demonstrates clearly that the way to
speeding up inference lies through the optimization of the matmul operation.
Based on this observation, we investigate three optimizations for speeding up
matmul operations, which collectively lead to the inference speedup of up to
x2.37 for Transfomer-based models over established baselines. The
optimizations do not require any changes to the implementation of those
models, and they do not affect their accuracy. We further note that while the
focus of our work has been the Transfomer architecture, our results are
applicable to any machine learning model in which matmul operations consume a
significant portion of the inference time.

Our work underscores the importance of the operation fusion as a technique for
optimizing computation during inference (NYZ20, ; LWY19, ). Such fusion would
reduce the amount of sequential overhead in invoking individual operations
(cf. Figure 4 (b)) and, in general, bring the scalability of high-level
operations, such as the Linear module computation, closer to their low-level
counterpart (cf. Figure 4 (a)). Furthermore, our work demonstrates that tuning
matrix partitioning can lead to substantial matmul speedups. An adaptive
approach similar to the one discussed in Section 6.1, but applied to the
matrix partitioning parameters, might be warranted.

Another related future direction is scaling primitive operations beyond
matmul. While matmul is responsible for the lion’s share of the inference
time, the portion of other operations grows as the number of threads
increases. For instance, for short sequences, the share of the time spent in
the layer normalization operation grows from $2.9\%$ for 1 thread to $12\%$
for 16 threads (cf. Figure 2). Parallelizing those operations and fusing them
with matmul should provide further improvement to the inference performance.

## References

  * [1] Jay Alammar.  The Illustrated Transformer.  http://jalammar.github.io/illustrated-transformer/.  Accessed: 01-07-21.
  * [2] Iz Beltagy, Matthew E. Peters, and Arman Cohan.  Longformer: The long-document transformer.  CoRR, abs/2004.05150, 2020.
  * [3] Qingqing Cao, Harsh Trivedi, Aruna Balasubramanian, and Niranjan Balasubramanian.  Deformer: Decomposing pre-trained transformers for faster question answering.  In Proc. of Conference of the Association for Computational Linguistics (ACL), pages 4487–4497, 2020.
  * [4] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context.  In Proc. of Conference of the Association for Computational Linguistics (ACL), pages 2978–2988, 2019.
  * [5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In Proc. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (NAACL-HLT), pages 4171–4186, 2019.
  * [6] Jiarui Fang, Yang Yu, Chengduo Zhao, and Jie Zhou.  Turbotransformers: An efficient GPU serving system for transformer models.  CoRR, abs/2010.05680, 2020.
  * [7] Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tie-Yan Liu.  Efficient training of BERT by progressively stacking.  In Proceedings of International Conference on Machine Learning (ICML), volume 97, pages 2337–2346, 2019.
  * [8] Kazushige Goto and Robert A. van de Geijn.  Anatomy of high-performance matrix multiplication.  ACM Trans. Math. Softw., 34(3), 2008.
  * [9] Dan Hendrycks and Kevin Gimpel.  Gaussian Error Linear Units (GELUs).  CoRR, abs/1606.08415, 2016.
  * [10] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  ALBERT: A lite BERT for self-supervised learning of language representations.  In Proc. of International Conference on Learning Representations, (ICLR), 2020.
  * [11] Quoc N. Le and Kip Kaehler.  How We Scaled Bert To Serve 1+ Billion Daily Requests on CPUs.  https://robloxtechblog.com/how-we-scaled-bert-to-serve-1-billion-daily-requests-on-cpus-d99be090db26.  Published: 05-27-20, Accessed: 01-06-21.
  * [12] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized BERT pretraining approach.  CoRR, abs/1907.11692, 2019.
  * [13] Yizhi Liu, Yao Wang, Ruofei Yu, Mu Li, Vin Sharma, and Yida Wang.  Optimizing CNN model inference on cpus.  In Proc. of USENIX Annual Technical Conference (ATC), pages 1025–1040, 2019.
  * [14] Pandu Nayak.  Understanding searches better than ever before.  https://blog.google/products/search/search-language-understanding-bert/.  Published: 10-25-19, Accessed: 01-06-21.
  * [15] Emma Ning, Nathan Yan, Jeffrey Zhu, and Jason Li.  Microsoft open sources breakthrough optimizations for transformer inference on gpu and cpu.  https://cloudblogs.microsoft.com/opensource/2020/01/21/microsoft-onnx-open-source-optimizations-transformer-inference-gpu-cpu/.  Published: 01-20-20, Accessed: 01-06-21.
  * [16] Harvard NLP.  The Annotated Transformer.  https://nlp.seas.harvard.edu/2018/04/03/attention.html.  Accessed: 01-07-21.
  * [17] oneAPI.  BLAS functions.  https://oneapi-src.github.io/oneDNN/group__dnnl__api__blas.html.  Accessed: 01-07-21.
  * [18] oneAPI.  oneAPI Deep Neural Network Library (oneDNN).  https://github.com/oneapi-src/oneDNN.  Accessed: 01-07-21.
  * [19] oneAPI.  Understanding Memory Formats.  https://oneapi-src.github.io/oneDNN/understanding_memory_formats.html.  Accessed: 01-07-21.
  * [20] Pytorch.  https://github.com/pytorch/pytorch.  Accessed: 01-07-21.
  * [21] Pytorch.  Efficient forward pass in nn.Linear.  https://github.com/pytorch/pytorch/issues/2159.  Accessed: 01-07-21.
  * [22] Pytorch.  Torchscript.  https://pytorch.org/docs/stable/jit.html.  Accessed: 01-07-21.
  * [23] Pytorch.  Torchscript for Deployment.  https://pytorch.org/tutorials/recipes/torchscript_inference.html.  Accessed: 01-07-21.
  * [24] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter.  CoRR, abs/1910.01108, 2019.
  * [25] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu.  Patient knowledge distillation for BERT model compression.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proc. Conference on Empirical Methods in Natural Language Processing, pages 4322–4331, 2019.
  * [26] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Proc. of Conference on Neural Information Processing Systems (NIPS), pages 5998–6008, 2017.
  * [27] Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma.  Linformer: Self-attention with linear complexity.  CoRR, abs/2006.04768, 2020.
  * [28] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou.  MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers.  In Proc. of Conference on Neural Information Processing Systems (NIPS), 2020.
  * [29] Y. Wang, Q. Wang, and X. Chu.  Energy-efficient inference service of transformer-based deep learning models on gpus.  In IEEE Conferences on Green Computing and Communications (GreenCom), pages 323–331, 2020.
  * [30] Yu Emma Wang.  Mille Crepe Bench: multi-layer performance analysis for deep learning frameworks.  https://github.com/Emma926/mcbench.  Accessed: 12-29-20.
  * [31] Yu Emma Wang, Carole-Jean Wu, Xiaodong Wang, Kim M. Hazelwood, and David Brooks.  Exploiting parallelism opportunities with deep learning frameworks.  CoRR, abs/1908.04705, 2019.
  * [32] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.  Transformers: State-of-the-art natural language processing.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, October 2020\.
  * [33] Carole-Jean Wu, David Brooks, Kevin Chen, Douglas Chen, Sy Choudhury, Marat Dukhan, Kim M. Hazelwood, Eldad Isaac, Yangqing Jia, Bill Jia, Tommer Leyvand, Hao Lu, Yang Lu, Lin Qiao, Brandon Reagen, Joe Spisak, Fei Sun, Andrew Tulloch, Peter Vajda, Xiaodong Wang, Yanghan Wang, Bram Wasti, Yiming Wu, Ran Xian, Sungjoo Yoo, and Peizhao Zhang.  Machine Learning at Facebook: Understanding Inference at the Edge.  In IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 331–344, 2019.
  * [34] Shufan Wu, Tao Lv, Pengxin Yuan, Patric Zhao, Jason Ye, and Haibin Lin.  Optimization for BERT Inference Performance on CPU.  https://medium.com/apache-mxnet/optimization-for-bert-inference-performance-on-cpu-3bb2413d376c.  Published: 09-12-19, Accessed: 01-06-21.
  * [35] Patrick Xia, Shijie Wu, and Benjamin Van Durme.  Which *BERT? A survey organizing contextualized encoders.  In Proc. of Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7516–7533, 2020.
  * [36] Yang You, Jing Li, Sashank J. Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh.  Large batch optimization for deep learning: Training BERT in 76 minutes.  In Proc. of International Conference on Learning Representations (ICLR), 2020.
  * [37] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed.  Big bird: Transformers for longer sequences.  In Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2020.
  * [38] Jeffrey Zhu.  Bing delivers its largest improvement in search experience using Azure GPUs.  https://azure.microsoft.com/en-us/blog/bing-delivers-its-largest-improvement-in-search-experience-using-azure-gpus/.  Published: 11-18-19, Accessed: 01-06-21.


Excluding C (subsolar), N (subsolar), S, Sc, Mn; (19) Excluding P, S, Sc, V,
Mn, C (subsolar); (20) Excluding P, S, C (subsolar), using favoured strat
values, using optical Si abundance; (21) Excluding P, S, C (subsolar), using
favoured strat values, using UV Si abundance; (22) Excluding P, S, Sc, V, Mn,
C (subsolar); (23) Excluding P, S, C (subsolar), N (subsolar); (24) Excluding
Sc, V, Mn, Sr; (25) Excluding very high Na point; (26) Excluding Mn; (27)
Excluding N (subsolar). Using model I; (28) Excluding P, S. Errors reported to
be typically 0.1 - 0.2 dex; (29) Excluding P, S

Table 3:  Results from Bayesian model. Errors given are 1 sigma. N/A indicates that a parameter was not invoked. A description of the parameters is given in Section 2 in the main text. Superscripted indices have the same meaning as in Table LABEL:tab:abundances System |  $[\textrm{Fe}/\textrm{H}]_{\textrm{index}}$ |  $t$/Myrs |  log($d_{\textrm{formation}}$/AU) |  $z_{\textrm{formation}}$/AU |  $f_{c}$ |  log($f_{\textrm{pol}}$) |  log($t_{\textrm{event}}$/Yrs) |  $P$/GPa |  $f_{\textrm{O}_{2}}$ ($\Delta$IW)
---|---|---|---|---|---|---|---|---|---
G166-58a |  $274_{-69}^{+462}$ |  $0.69_{-0.68}^{+8.48}$ |  N/A |  N/A |  N/A |  $-7.92_{-0.04}^{+0.04}$ |  $6.20_{-2.02}^{+1.08}$ |  N/A |  N/A
G241-6b,1 |  $320_{-72}^{+164}$ |  $6.45_{-4.41}^{+9.19}$ |  $0.24_{-0.22}^{+0.16}$ |  N/A |  N/A |  $-5.52_{-0.07}^{+0.06}$ |  $7.24_{-0.56}^{+0.41}$ |  N/A |  N/A
G29-38c,2 |  $338_{-124}^{+126}$ |  $0.02_{-0.02}^{+2.29}$ |  $-1.27_{-0.39}^{+0.31}$ |  $0.14_{-0.01}^{+0.01}$ |  N/A |  $-4.94_{-0.02}^{+0.02}$ |  $4.59_{-2.56}^{+2.20}$ |  N/A |  N/A
GALEX1931+0117d,3 |  $137_{-98}^{+462}$ |  $0.00_{-0.00}^{+1.15}$ |  $0.55_{-0.03}^{+0.03}$ |  N/A |  $0.16_{-0.03}^{+0.04}$ |  $-3.46_{-0.03}^{+0.03}$ |  $3.94_{-2.49}^{+2.54}$ |  $45.0_{-13.3}^{+9.8}$ |  $-2.6_{-0.2}^{+0.3}$
GALEX1931+0117e |  $459_{-167}^{+250}$ |  $0.00_{-0.00}^{+1.77}$ |  N/A |  N/A |  N/A |  $-3.74_{-0.10}^{+0.10}$ |  $3.86_{-2.56}^{+2.82}$ |  N/A |  N/A
GALEX1931+0117f,3 |  $561_{-0}^{+0}$ |  $0.00_{-0.00}^{+0.02}$ |  $0.43_{-0.06}^{+0.04}$ |  $0.07_{-0.04}^{+0.05}$ |  $0.31_{-0.03}^{+0.03}$ |  $-3.58_{-0.04}^{+0.03}$ |  $3.07_{-1.36}^{+1.52}$ |  $51.6_{-7.2}^{+5.0}$ |  $-2.7_{-0.1}^{+0.2}$
GALEXJ2339g,4 |  $413_{-159}^{+178}$ |  $1.94_{-1.02}^{+2.84}$ |  N/A |  $0.07_{-0.05}^{+0.05}$ |  N/A |  $-5.42_{-0.03}^{+0.03}$ |  $5.12_{-3.44}^{+2.19}$ |  N/A |  N/A
GD362h,5 |  $568_{-177}^{+180}$ |  $0.23_{-0.15}^{+0.27}$ |  $-0.81_{-0.23}^{+0.13}$ |  $0.12_{-0.03}^{+0.02}$ |  N/A |  $-5.18_{-0.03}^{+0.02}$ |  $4.49_{-2.93}^{+2.24}$ |  N/A |  N/A
GD378g,6 |  $481_{-311}^{+311}$ |  $2.29_{-1.54}^{+14.25}$ |  N/A |  N/A |  N/A |  $-6.08_{-0.11}^{+0.12}$ |  $6.11_{-4.07}^{+1.46}$ |  N/A |  N/A
GD40b,7 |  $369_{-187}^{+262}$ |  $1.90_{-1.08}^{+13.58}$ |  $-0.57_{-0.07}^{+0.36}$ |  $0.09_{-0.05}^{+0.04}$ |  N/A |  $-5.38_{-0.06}^{+0.06}$ |  $6.50_{-4.39}^{+1.10}$ |  N/A |  N/A
GD424i,8 |  $381_{-95}^{+141}$ |  $5.21_{-4.61}^{+19.86}$ |  $-0.60_{-0.01}^{+0.02}$ |  N/A |  N/A |  $-4.34_{-0.02}^{+0.02}$ |  $7.13_{-0.96}^{+0.57}$ |  N/A |  N/A
GD56a |  $563_{-305}^{+263}$ |  $0.01_{-0.01}^{+3.17}$ |  N/A |  N/A |  N/A |  $-5.06_{-0.09}^{+0.09}$ |  $4.39_{-2.84}^{+2.51}$ |  N/A |  N/A
GD61j,9 |  $450_{-197}^{+260}$ |  $3.09_{-2.76}^{+10.41}$ |  $0.30_{-0.20}^{+0.12}$ |  N/A |  $0.03_{-0.01}^{+0.01}$ |  $-5.83_{-0.03}^{+0.02}$ |  $6.91_{-3.55}^{+0.63}$ |  $40.5_{-18.5}^{+11.7}$ |  $-2.5_{-0.3}^{+0.4}$
HE0106-3253a |  $515_{-302}^{+282}$ |  $0.00_{-0.00}^{+1.41}$ |  $-1.47_{-0.34}^{+0.41}$ |  $0.09_{-0.01}^{+0.01}$ |  $0.60_{-0.06}^{+0.06}$ |  $-4.56_{-0.04}^{+0.04}$ |  $4.09_{-2.54}^{+2.45}$ |  $25.6_{-15.5}^{+18.7}$ |  $-1.9_{-0.6}^{+0.6}$
HS2253+8023k,10 |  $597_{-287}^{+216}$ |  $0.26_{-0.20}^{+9.81}$ |  $-0.44_{-0.16}^{+0.64}$ |  N/A |  N/A |  $-5.19_{-0.06}^{+0.07}$ |  $6.04_{-3.83}^{+1.35}$ |  N/A |  N/A
LHS2534l,11 |  $926_{-15}^{+22}$ |  $4.71_{-0.85}^{+8.56}$ |  $0.55_{-0.02}^{+0.02}$ |  N/A |  $0.52_{-0.12}^{+0.14}$ |  $-8.38_{-0.03}^{+0.03}$ |  $3.62_{-2.30}^{+3.66}$ |  $35.7_{-16.5}^{+14.8}$ |  $-1.3_{-0.2}^{+0.2}$
NLTT43806m |  $817_{-272}^{+110}$ |  $0.78_{-0.75}^{+10.53}$ |  $-0.16_{-0.30}^{+0.35}$ |  $0.10_{-0.06}^{+0.04}$ |  $0.03_{-0.01}^{+0.01}$ |  $-6.56_{-0.05}^{+0.05}$ |  $6.34_{-2.52}^{+1.07}$ |  $45.6_{-15.0}^{+9.7}$ |  $-2.7_{-0.2}^{+0.4}$
PG0843+516d,12 |  $417_{-210}^{+280}$ |  $0.01_{-0.01}^{+1.81}$ |  $-0.07_{-0.29}^{+0.35}$ |  N/A |  $0.64_{-0.09}^{+0.08}$ |  $-4.25_{-0.08}^{+0.09}$ |  $4.13_{-2.55}^{+2.50}$ |  $29.3_{-17.1}^{+21.1}$ |  $-2.5_{-0.3}^{+0.6}$
PG0843+516a |  $463_{-295}^{+313}$ |  $0.00_{-0.00}^{+1.50}$ |  $-0.01_{-0.38}^{+0.36}$ |  N/A |  $0.75_{-0.10}^{+0.08}$ |  $-3.77_{-0.12}^{+0.12}$ |  $3.97_{-2.40}^{+2.64}$ |  $25.8_{-15.0}^{+18.7}$ |  $-1.9_{-0.7}^{+0.6}$
PG1015+161d,13 |  $377_{-232}^{+350}$ |  $0.00_{-0.00}^{+1.12}$ |  $-1.30_{-0.44}^{+0.42}$ |  $0.11_{-0.02}^{+0.02}$ |  $0.34_{-0.12}^{+0.13}$ |  $-5.00_{-0.09}^{+0.10}$ |  $3.79_{-2.31}^{+2.68}$ |  $17.8_{-11.5}^{+19.0}$ |  $-1.7_{-0.7}^{+0.4}$
PG1015+161a |  $499_{-316}^{+293}$ |  $0.00_{-0.00}^{+1.27}$ |  $-1.09_{-0.55}^{+0.56}$ |  $0.11_{-0.03}^{+0.02}$ |  $0.59_{-0.13}^{+0.11}$ |  $-4.72_{-0.12}^{+0.12}$ |  $3.98_{-2.51}^{+2.57}$ |  $31.9_{-18.8}^{+17.4}$ |  $-2.0_{-0.6}^{+0.6}$
PG1225-079n,14 |  $494_{-250}^{+260}$ |  $1.49_{-1.00}^{+2.71}$ |  $-0.87_{-0.32}^{+0.21}$ |  $0.12_{-0.04}^{+0.02}$ |  N/A |  $-6.93_{-0.07}^{+0.07}$ |  $5.74_{-3.51}^{+1.56}$ |  N/A |  N/A
SDSSJ0512-0505o,15 |  $499_{-319}^{+305}$ |  $3.65_{-1.35}^{+1.60}$ |  $-1.04_{-0.55}^{+0.33}$ |  $0.13_{-0.02}^{+0.01}$ |  $0.54_{-0.21}^{+0.17}$ |  $-7.38_{-0.06}^{+0.06}$ |  $3.20_{-2.04}^{+2.53}$ |  $25.5_{-16.6}^{+20.0}$ |  $-1.8_{-0.7}^{+0.5}$
SDSSJ0738+1835p,16 |  $375_{-219}^{+185}$ |  $0.49_{-0.07}^{+0.07}$ |  $0.10_{-0.41}^{+0.27}$ |  N/A |  $0.44_{-0.07}^{+0.07}$ |  $-3.79_{-0.07}^{+0.07}$ |  $2.39_{-1.51}^{+1.78}$ |  $32.3_{-19.6}^{+19.2}$ |  $-2.5_{-0.3}^{+0.6}$
SDSSJ0823+0546o,17 |  $438_{-297}^{+341}$ |  $17.66_{-3.54}^{+3.41}$ |  $0.03_{-0.38}^{+0.34}$ |  N/A |  $0.96_{-0.04}^{+0.02}$ |  $-7.23_{-0.06}^{+0.06}$ |  $3.25_{-2.05}^{+2.34}$ |  $29.0_{-18.9}^{+19.8}$ |  $-2.1_{-0.6}^{+0.6}$
SDSSJ0845+2257q,18 |  $586_{-350}^{+250}$ |  $0.47_{-0.43}^{+12.05}$ |  $-0.21_{-0.29}^{+0.36}$ |  N/A |  $0.31_{-0.09}^{+0.10}$ |  $-3.76_{-0.09}^{+0.08}$ |  $6.23_{-3.45}^{+1.17}$ |  $28.4_{-18.2}^{+19.8}$ |  $-2.1_{-0.6}^{+0.7}$
SDSSJ1043+0855r,19 |  $342_{-178}^{+245}$ |  $0.00_{-0.00}^{+1.92}$ |  $-0.22_{-0.38}^{+0.51}$ |  N/A |  N/A |  $-4.60_{-0.09}^{+0.10}$ |  $4.03_{-2.59}^{+2.64}$ |  N/A |  N/A
SDSSJ1228+1040d,20 |  $421_{-245}^{+284}$ |  $0.01_{-0.01}^{+1.82}$ |  $-0.69_{-0.30}^{+0.42}$ |  $0.12_{-0.05}^{+0.02}$ |  N/A |  $-4.28_{-0.07}^{+0.06}$ |  $4.11_{-2.59}^{+2.55}$ |  N/A |  N/A
SDSSJ1228+1040d,21 |  $486_{-270}^{+291}$ |  $0.00_{-0.00}^{+1.77}$ |  $-0.90_{-0.52}^{+0.25}$ |  $0.12_{-0.04}^{+0.02}$ |  N/A |  $-4.38_{-0.09}^{+0.08}$ |  $3.97_{-2.68}^{+2.66}$ |  N/A |  N/A
SDSSJ1242+5226s,22 |  $374_{-81}^{+111}$ |  $4.93_{-0.39}^{+0.86}$ |  $0.05_{-0.55}^{+0.34}$ |  N/A |  N/A |  $-4.32_{-0.04}^{+0.05}$ |  $4.51_{-3.09}^{+1.60}$ |  N/A |  N/A
SDSSJ2047-1259t,23 |  $523_{-367}^{+297}$ |  $0.62_{-0.12}^{+0.15}$ |  $-0.01_{-0.41}^{+0.34}$ |  N/A |  N/A |  $-4.67_{-0.06}^{+0.06}$ |  $2.83_{-1.95}^{+2.18}$ |  N/A |  N/A
WD0122-227u |  $467_{-299}^{+311}$ |  $7.84_{-4.78}^{+6.14}$ |  $0.03_{-0.40}^{+0.37}$ |  N/A |  $0.57_{-0.20}^{+0.19}$ |  $-8.23_{-0.15}^{+0.15}$ |  $3.96_{-2.58}^{+2.92}$ |  $29.6_{-18.7}^{+19.1}$ |  $-2.0_{-0.6}^{+0.6}$
WD0446-255u,24 |  $814_{-339}^{+113}$ |  $1.09_{-0.75}^{+1.42}$ |  $-0.58_{-0.11}^{+0.13}$ |  $0.13_{-0.02}^{+0.01}$ |  $0.05_{-0.02}^{+0.02}$ |  $-5.62_{-0.04}^{+0.05}$ |  $4.54_{-3.05}^{+2.42}$ |  $36.9_{-22.5}^{+15.5}$ |  $-2.2_{-0.5}^{+0.6}$
WD0449-259u,25 |  $581_{-387}^{+267}$ |  $7.11_{-4.28}^{+5.58}$ |  $-1.03_{-0.61}^{+0.94}$ |  $0.10_{-0.04}^{+0.03}$ |  $0.65_{-0.23}^{+0.18}$ |  $-7.47_{-0.15}^{+0.14}$ |  $3.99_{-2.53}^{+2.70}$ |  $19.9_{-13.1}^{+19.3}$ |  $-1.4_{-0.7}^{+0.3}$
WD1145+017v,3 |  $640_{-327}^{+214}$ |  $0.54_{-0.39}^{+8.70}$ |  $0.13_{-0.44}^{+1.31}$ |  N/A |  N/A |  $-4.87_{-0.12}^{+0.32}$ |  $5.90_{-3.78}^{+1.50}$ |  N/A |  N/A
WD1145+288a |  $502_{-297}^{+285}$ |  $0.01_{-0.01}^{+1.47}$ |  $-0.96_{-0.58}^{+1.21}$ |  $0.12_{-0.04}^{+0.02}$ |  $0.48_{-0.15}^{+0.14}$ |  $-5.30_{-0.12}^{+0.14}$ |  $4.16_{-2.57}^{+2.41}$ |  $31.7_{-19.9}^{+17.5}$ |  $-2.0_{-0.6}^{+0.6}$
WD1232+563a,26 |  $559_{-0}^{+0}$ |  $13.26_{-9.00}^{+24.86}$ |  N/A |  N/A |  N/A |  $-5.10_{-0.03}^{+0.02}$ |  $7.49_{-0.44}^{+0.34}$ |  N/A |  N/A
WD1350-162u,25 |  $161_{-114}^{+440}$ |  $6.84_{-1.13}^{+1.25}$ |  $0.00_{-0.33}^{+0.31}$ |  N/A |  $0.42_{-0.08}^{+0.08}$ |  $-6.04_{-0.06}^{+0.06}$ |  $3.11_{-2.02}^{+2.28}$ |  $22.7_{-14.9}^{+22.7}$ |  $-2.1_{-0.6}^{+0.7}$
WD1425+540w,27 |  $470_{-275}^{+295}$ |  $1.34_{-0.82}^{+4.38}$ |  N/A |  N/A |  N/A |  $-6.59_{-0.08}^{+0.10}$ |  $5.48_{-3.73}^{+1.90}$ |  N/A |  N/A
WD1536+520x,28 |  $445_{-232}^{+294}$ |  $0.19_{-0.14}^{+8.80}$ |  $0.10_{-0.40}^{+0.29}$ |  N/A |  N/A |  $-3.26_{-0.07}^{+0.08}$ |  $5.79_{-3.78}^{+1.54}$ |  N/A |  N/A
WD1551+175a,26 |  $589_{-221}^{+171}$ |  $4.19_{-3.44}^{+20.66}$ |  $-0.66_{-0.01}^{+0.01}$ |  N/A |  N/A |  $-5.38_{-0.04}^{+0.03}$ |  $7.08_{-4.24}^{+0.64}$ |  N/A |  N/A
WD2105-820u |  $210_{-159}^{+486}$ |  $0.00_{-0.00}^{+0.08}$ |  $0.07_{-0.35}^{+0.29}$ |  N/A |  $0.84_{-0.26}^{+0.11}$ |  $-5.96_{-0.13}^{+0.14}$ |  $1.53_{-0.98}^{+3.79}$ |  $18.8_{-12.6}^{+21.4}$ |  $-1.7_{-0.7}^{+0.4}$
WD2115-560u |  $472_{-181}^{+217}$ |  $0.00_{-0.00}^{+0.30}$ |  $0.20_{-0.47}^{+1.10}$ |  N/A |  N/A |  $-5.84_{-0.04}^{+0.04}$ |  $3.20_{-1.74}^{+2.73}$ |  N/A |  N/A
WD2157-574u |  $364_{-199}^{+275}$ |  $0.01_{-0.00}^{+2.44}$ |  N/A |  N/A |  N/A |  $-6.55_{-0.04}^{+0.04}$ |  $3.09_{-2.18}^{+3.70}$ |  N/A |  N/A
WD2207+121a,26 |  $673_{-438}^{+214}$ |  $5.42_{-4.17}^{+22.74}$ |  $0.12_{-0.43}^{+0.28}$ |  N/A |  N/A |  $-5.22_{-0.06}^{+0.06}$ |  $7.13_{-4.58}^{+0.62}$ |  N/A |  N/A
WD2216-657u |  $388_{-272}^{+345}$ |  $7.18_{-0.97}^{+0.97}$ |  $-0.18_{-0.21}^{+0.28}$ |  $0.07_{-0.05}^{+0.06}$ |  N/A |  $-7.25_{-0.05}^{+0.04}$ |  $3.18_{-2.09}^{+2.34}$ |  N/A |  N/A
WDJ1814-7354y,29 |  $382_{-147}^{+199}$ |  $0.02_{-0.02}^{+3.22}$ |  N/A |  N/A |  N/A |  $-5.94_{-0.05}^{+0.04}$ |  $4.76_{-2.43}^{+2.15}$ |  N/A |  N/A
Table 4:  Results from Bayesian model. N/A indicates that a parameter was not invoked, or (for the category column) that another data set was used for that system. Category abbreviations are as follows: HPM = High Pressure Mantle-rich, LPC = Low Pressure Core-rich, PD = Pressure degenerate with oxygen fugacity, PU = Pressure unconstrained, NED = No Evidence of Differentiation, U = Unphysical. * Previously explained as crust-rich. † May be subject to a degeneracy which causes high pressure fragments to appear to be low pressure; see section 4.5.3. Superscripted indices have the same meaning as in Table LABEL:tab:abundances System |  Good Fit? |  Primitive? |  Core Rich? |  Mantle Rich? |  Volatile Rich? |  Volatile Depleted? |  Moderate Volatile Depleted? |  Temperature |  Inferred Parent Core Number Fraction |  Differentiation Sigma |  Category
---|---|---|---|---|---|---|---|---|---|---|---
G166-58a |  Y |  Y |  N |  N |  Y |  N |  N |  N/A |  N/A |  N/A |  NED
G241-6b,1 |  Y |  Y |  N |  N |  Y |  N |  N |  223 |  N/A |  N/A |  NED
G29-38c,2 |  Y |  Y |  N |  N |  N |  N |  Y |  1952 |  N/A |  N/A |  NED
GALEX1931+0117d,3 |  N |  N |  Y |  N |  Y |  N |  N |  130 |  0.01 |  2.7 |  N/A
GALEX1931+0117e |  N |  N |  Y |  N |  Y |  N |  N |  150 |  0.14 |  4.9 |  NED
GALEX1931+0117f,3 |  N |  Y |  N |  N |  Y |  N |  N |  N/A |  N/A |  N/A |  N/A
GALEXJ2339g,4 |  Y |  Y |  N |  N |  Y |  N |  N |  N/A |  N/A |  N/A |  NED
GD362h,5 |  Y |  Y |  N |  N |  N |  N |  Y |  1516 |  N/A |  N/A |  NED
GD378g,6 |  Y |  Y |  N |  N |  Y |  N |  N |  N/A |  N/A |  N/A |  NED
GD40b,7 |  Y |  Y |  N |  N |  N |  Y |  N |  1197 |  N/A |  N/A |  NED
GD424i,8 |  N |  Y |  N |  N |  N |  Y |  N |  1268 |  N/A |  N/A |  NED
GD56a |  Y |  Y |  N |  N |  Y |  N |  N |  N/A |  N/A |  N/A |  NED
GD61j,9 |  Y |  N |  N |  Y |  Y |  N |  N |  198 |  0.16 |  1.3 |  HPM
HE0106-3253a |  Y |  N |  Y |  N |  N |  N |  Y |  2170 |  0.04 |  9.2 |  PD
HS2253+8023k,10 |  Y |  Y |  N |  N |  Y |  N |  N |  912 |  N/A |  N/A |  NED
LHS2534l,11 |  Y |  N |  Y |  N |  Y |  N |  N |  130 |  0.0 |  4.3 |  U
NLTT43806m |  Y |  N |  N |  Y |  Y |  N |  N |  514 |  0.17 |  2.8 |  HPM*
PG0843+516d,12 |  N |  N |  Y |  N |  Y |  N |  N |  429 |  0.17 |  4.6 |  N/A
PG0843+516a |  Y |  N |  Y |  N |  Y |  N |  N |  373 |  0.13 |  5.6 |  PD
PG1015+161d,13 |  N |  N |  Y |  N |  N |  N |  Y |  1976 |  0.07 |  1.4 |  N/A
PG1015+161a |  Y |  N |  Y |  N |  N |  N |  Y |  1761 |  0.1 |  3.3 |  PD
PG1225-079n,14 |  Y |  Y |  N |  N |  N |  N |  Y |  1570 |  N/A |  N/A |  NED
SDSSJ0512-0505o,15 |  Y |  N |  Y |  N |  N |  N |  Y |  1714 |  0.1 |  2.1 |  PD
SDSSJ0738+1835p,16 |  Y |  N |  Y |  N |  Y |  N |  N |  299 |  0.15 |  2.9 |  PU
SDSSJ0823+0546o,17 |  Y |  N |  Y |  N |  Y |  N |  N |  349 |  0.13 |  10.1 |  PU
SDSSJ0845+2257q,18 |  Y |  N |  Y |  N |  Y |  N |  N |  565 |  0.14 |  1.6 |  PU
SDSSJ1043+0855r,19 |  N |  Y |  N |  N |  Y |  N |  N |  587 |  N/A |  N/A |  NED
SDSSJ1228+1040d,20 |  Y |  Y |  N |  N |  N |  N |  Y |  1419 |  N/A |  N/A |  N/A
SDSSJ1228+1040d,21 |  Y |  Y |  N |  N |  N |  N |  Y |  1588 |  N/A |  N/A |  NED
SDSSJ1242+5226s,22 |  Y |  Y |  N |  N |  Y |  N |  N |  330 |  N/A |  N/A |  NED
SDSSJ2047-1259t,23 |  Y |  Y |  N |  N |  Y |  N |  N |  376 |  N/A |  N/A |  NED
WD0122-227u |  Y |  N |  Y |  N |  Y |  N |  N |  349 |  0.1 |  2.2 |  PU
WD0446-255u,24 |  Y |  N |  N |  Y |  N |  Y |  N |  1217 |  0.15 |  N/A |  HPM
WD0449-259u,25 |  Y |  N |  Y |  N |  N |  N |  Y |  1711 |  0.1 |  N/A |  LPC
WD1145+017v,3 |  Y |  Y |  N |  N |  Y |  N |  N |  279 |  N/A |  N/A |  NED
WD1145+288a |  Y |  N |  Y |  N |  N |  N |  Y |  1642 |  0.16 |  1.5 |  PU
WD1232+563a,26 |  N |  Y |  N |  N |  Y |  N |  N |  N/A |  N/A |  N/A |  NED
WD1350-162u,25 |  Y |  N |  Y |  N |  Y |  N |  N |  364 |  0.09 |  N/A |  LPC†
WD1425+540w,27 |  Y |  Y |  N |  N |  Y |  N |  N |  N/A |  N/A |  N/A |  NED
WD1536+520x,28 |  Y |  Y |  N |  N |  Y |  N |  N |  300 |  N/A |  N/A |  NED
WD1551+175a,26 |  Y |  Y |  N |  N |  N |  Y |  N |  1395 |  N/A |  N/A |  NED
WD2105-820u |  Y |  N |  Y |  N |  Y |  N |  N |  318 |  0.09 |  N/A |  LPC
WD2115-560u |  Y |  Y |  N |  N |  Y |  N |  N |  242 |  N/A |  N/A |  NED
WD2157-574u |  Y |  Y |  N |  N |  Y |  N |  N |  N/A |  N/A |  N/A |  NED
WD2207+121a,26 |  Y |  Y |  N |  N |  Y |  N |  N |  287 |  N/A |  N/A |  NED
WD2216-657u |  N |  Y |  N |  N |  Y |  N |  N |  540 |  N/A |  N/A |  NED
WDJ1814-7354y,29 |  Y |  Y |  N |  N |  Y |  N |  N |  N/A |  N/A |  N/A |  NED
Table 5:  Summary of coefficients used in parametrisations of partitioning behaviour. For coefficients sourced from Fischer et al. (2015), those marked with † are results quoted from epsilon modelling. *S partitioning was implemented following the procedure described by Boujibar et al. (2014), but was not included when running the model on white dwarf systems. Element |  Valence |  a |  b $/\textrm{K}$ |  c $/\textrm{K}\;\textrm{GPa}^{-1}$ |  d |  Source |  Equation Used
---|---|---|---|---|---|---|---
Fe |  2 |  0 |  0 |  0 |  0 |  N/A |  (4)
Mn |  2 |  -0.02 |  -5600 |  38 |  0.036 |  Corgne et al. (2008) |  (2)
Ni |  2 |  0.46 |  2700 |  -61 |  0 |  Fischer et al. (2015) |  (3)
Cr |  2 |  -0.3 |  -2200 |  -5 |  0 |  Fischer et al. (2015) † |  (3)
Ga |  3 |  3.5 |  -4800 |  -126 |  -0.97 |  Corgne et al. (2008) |  (2)
Si |  4 |  0.549 |  -12324 |  0 |  0 |  Siebert et al. (2013) |  (2)
Nb |  5 |  4.09 |  -15500 |  -166 |  -0.75 |  Corgne et al. (2008) |  (2)
Ta |  5 |  7.74 |  -20000 |  -264 |  -1.69 |  Corgne et al. (2008) |  (2)
Cu |  1 |  0.3 |  2300 |  -37 |  0.14 |  Corgne et al. (2008) |  (2)
Zn |  2 |  -1.11 |  600 |  -23 |  -0.21 |  Corgne et al. (2008) |  (2)
V |  3 |  -1.5 |  -2300 |  9 |  0 |  Fischer et al. (2015) † |  (3)
Co |  2 |  0.36 |  1500 |  -33 |  0 |  Fischer et al. (2015) |  (3)
P |  5 |  0.64 |  -1593 |  -74.95 |  0 |  Wade & Wood (2005) |  (2)
Tl |  1 |  -0.118 |  -783 |  0 |  0 |  Wood et al. (2008b) |  (2)
W |  4.52 |  3.2 |  -1605 |  -115 |  -0.85 |  Cottrell et al. (2009) |  (2)
O |  -2 |  0.986 |  -3275 |  0 |  0 |  Siebert et al. (2013) |  (2)
C |  4 |  -1 |  4842 |  31 |  0 |  Blanchard et al. (2019) |  (3)
S |  -2 |  0 |  405 |  136 |  0 |  Boujibar et al. (2014) |  N/A *
Table 6:  Tabulated extract of Figure 15, showing predicted elemental abundances in a white dwarf atmosphere as a function of the core fraction of a pollutant fragment and the core–mantle differentiation pressure in its parent. Hx represents the dominant element in the white dwarf atmosphere, either H or He. Mg/Fe and Ca/Fe act as proxies for the fragment core fraction. When this quantity is known, Cr, Ni and/or Si can be used to infer pressure. Note that this table assumes steady-state accretion and heavy pollution (meaning that the total pollution of the elements we model varies between -3 and -4 log units relative to Hx) - see caption of Figure 15 and the main text for further details. Fragment Core Fraction |  Pressure /GPa |  log(Mg/Fe) |  log(Ca/Fe) |  log(Cr/Hx) |  log(Ni/Hx) |  log(Si/Hx)
---|---|---|---|---|---|---
0 |  0 |  0.99 |  -0.06 |  -7.15 |  -9.19 |  -5.05
0 |  10 |  1.0 |  -0.05 |  -7.2 |  -8.67 |  -5.06
0 |  20 |  1.01 |  -0.04 |  -7.23 |  -8.39 |  -5.06
0 |  30 |  1.03 |  -0.03 |  -7.26 |  -8.16 |  -5.07
0 |  40 |  1.06 |  0.0 |  -7.3 |  -7.99 |  -5.09
0 |  50 |  1.1 |  0.04 |  -7.36 |  -7.85 |  -5.12
0 |  60 |  1.13 |  0.07 |  -7.42 |  -7.73 |  -5.14
0.2 |  0 |  0.06 |  -0.99 |  -7.24 |  -6.74 |  -5.27
0.2 |  10 |  0.07 |  -0.98 |  -7.22 |  -6.74 |  -5.26
0.2 |  20 |  0.08 |  -0.97 |  -7.22 |  -6.75 |  -5.26
0.2 |  30 |  0.1 |  -0.96 |  -7.21 |  -6.77 |  -5.25
0.2 |  40 |  0.13 |  -0.93 |  -7.21 |  -6.79 |  -5.24
0.2 |  50 |  0.17 |  -0.89 |  -7.22 |  -6.81 |  -5.22
0.2 |  60 |  0.2 |  -0.85 |  -7.23 |  -6.84 |  -5.22
0.4 |  0 |  -0.33 |  -1.39 |  -7.31 |  -6.53 |  -5.49
0.4 |  10 |  -0.32 |  -1.38 |  -7.24 |  -6.54 |  -5.47
0.4 |  20 |  -0.31 |  -1.37 |  -7.21 |  -6.54 |  -5.45
0.4 |  30 |  -0.29 |  -1.35 |  -7.19 |  -6.56 |  -5.41
0.4 |  40 |  -0.27 |  -1.32 |  -7.16 |  -6.58 |  -5.37
0.4 |  50 |  -0.23 |  -1.28 |  -7.14 |  -6.6 |  -5.31
0.4 |  60 |  -0.19 |  -1.25 |  -7.12 |  -6.62 |  -5.28
0.6 |  0 |  -0.67 |  -1.73 |  -7.37 |  -6.43 |  -5.74
0.6 |  10 |  -0.66 |  -1.72 |  -7.25 |  -6.44 |  -5.7
0.6 |  20 |  -0.65 |  -1.71 |  -7.21 |  -6.44 |  -5.66
0.6 |  30 |  -0.63 |  -1.69 |  -7.17 |  -6.45 |  -5.58
0.6 |  40 |  -0.61 |  -1.66 |  -7.13 |  -6.47 |  -5.49
0.6 |  50 |  -0.57 |  -1.62 |  -7.09 |  -6.49 |  -5.4
0.6 |  60 |  -0.53 |  -1.59 |  -7.06 |  -6.51 |  -5.33
0.8 |  0 |  -1.09 |  -2.15 |  -7.41 |  -6.37 |  -6.1
0.8 |  10 |  -1.08 |  -2.14 |  -7.25 |  -6.37 |  -6.02
0.8 |  20 |  -1.07 |  -2.13 |  -7.2 |  -6.38 |  -5.91
0.8 |  30 |  -1.05 |  -2.11 |  -7.16 |  -6.39 |  -5.77
0.8 |  40 |  -1.03 |  -2.08 |  -7.11 |  -6.4 |  -5.62
0.8 |  50 |  -0.99 |  -2.04 |  -7.05 |  -6.42 |  -5.47
0.8 |  60 |  -0.95 |  -2.01 |  -7.01 |  -6.44 |  -5.38
0.99 |  0 |  -2.48 |  -3.54 |  -7.45 |  -6.33 |  -7.37
0.99 |  10 |  -2.47 |  -3.53 |  -7.26 |  -6.34 |  -6.67
0.99 |  20 |  -2.46 |  -3.52 |  -7.2 |  -6.34 |  -6.3
0.99 |  30 |  -2.44 |  -3.5 |  -7.15 |  -6.35 |  -5.99
0.99 |  40 |  -2.42 |  -3.47 |  -7.09 |  -6.36 |  -5.75
0.99 |  50 |  -2.38 |  -3.43 |  -7.03 |  -6.38 |  -5.54
0.99 |  60 |  -2.34 |  -3.4 |  -6.98 |  -6.39 |  -5.42

##  Appendix B Comments on individual systems

###  B.1 GD61

The depletion in Ni and Fe illustrated in Figure 9 suggests that the pollution
is rich in mantle-like material. The red line shows how the median fit would
change if the pressure was fixed to 0 GPa while using the same posterior
distribution for the other parameters. While the difference due to pressure is
very much a secondary effect, the fit is noticeably improved at higher
pressure. This is driven by Si partitioning and the indirect change in mantle
Fe content. Our median model hits the artificial $D_{\textrm{Si}}$ cap, which
implies that our pressure constraint could be stronger if we allowed Si to
partition more strongly into the core.

The median pressure is 40 GPa, which (given the composition inferred by our
model) corresponds to a planet of mass 0.6 $\textrm{M}_{\oplus}$. We
illustrate the posterior distribution of pressure in Figure 11, which exhibits
a peak between 40 and 60 GPa - i.e., roughly Earth-like pressure, and hence
Earth-like mass.

Our results are generally in agreement with previous work by Farihi et al.
(2013). We find that GD61 is in a steady-state phase of accretion, which is
consistent with Farihi et al. (2013) and the presence of an infra-red excess.
Farihi et al. (2013) additionally found that the pollution contained excess O,
implying the accretion of water. We follow a similar analysis, computing the
amount of O in the parent body’s mantle which would be left over if all other
components were fully oxidised. Assuming that all remaining O is contained in
$\textrm{H}_{2}\textrm{O}$, and that the parent body’s mass is indeed 0.61M⊕
as implied by our model, we find that there is enough excess O (34% of the
mantle’s O budget, by number) to account for significant amounts of water. The
implied total mass of water in the parent body is $5.1^{+2.1}_{-2.9}\times
10^{23}\;\textrm{kg}$, where the error comes from propagating our error on
pressure. Farihi et al. (2013) found that GD61 has accreted (at most) $5\times
10^{19}\;\textrm{kg}$ of water (based on the trace H content in the atmosphere
of GD61). This discrepancy implies that either our inferred parent body
differentiation pressure (and hence mass) is an overestimate, or that GD61 has
so far accreted only a very small amount of the parent body’s water budget
(roughly 0.01%).

###  B.2 WD0446-255

High pressure and low oxygen fugacity is favoured. Compared to the median fit
(with pressure = $37^{+15}_{-22}\;\textrm{GPa}$), the log-likelihood is
decreased by 0.66 when fixing pressure to 0. All elements, apart from Mg, fit
better at high pressure. Cr and Fe show the greatest change with pressure, but
as with GD61, the Ni abundance is not strongly affected despite the fragment
being mantle-rich. We discuss this behaviour in the Appendix.

The depletion in siderophile elements pushes the model to a mantle rich
fragment with 5.5% core. Compared to GD61 it has a low Mg/Fe ratio, only
slightly above solar, as shown in Figure 6. However, our model finds that the
Mg abundance is anomalously low: the model is able to match the abundance of
all 9 other elements modelled to within 1 sigma, but predicts an Mg abundance
roughly 2 sigma above that observed. This may be down to random chance: the
probability that at least one of the 10 data points is observed at least 2
sigma away from its true value is 37%. The sensitivity to pressure is
therefore greater than suggested by Figure 6. To best match the low Mg, the
model favours a build-up phase solution. This is consistent with the results
of Swan et al. (2019). The high refractory element abundances (and slight O
depletion) suggest incomplete condensation.

###  B.3 NLTT43806

Our model finds a good fit to the data by invoking the accretion of mantle-
rich material. Again, high pressure is preferred. However, the pollutant has
previously been explained as being composed (at least partially) of crust-rich
material due to the high Al and Ca abundances (Zuckerman et al., 2011). Our
model does not reproduce these Al or Ca observations because we don’t include
crustal material. We infer that our fit could be further improved by including
crustal material, as was found by Harrison et al. (2018).

###  B.4 WD0449-259

The model finds 2 degenerate solutions for this system. The first solution is
a highly core-rich fragment, with extreme heating, accreting in build
up/steady state. This roughly follows the heating effects arrow from the 80%
core contour to the data point in the middle panel of Figure 6. The second
solution invokes a less core-rich fragment, with decreased heating, accreting
in the declining phase. This is similar to the first solution, but starting at
a lower core fraction, and following part of both the heating and sinking
arrows to the data point.

Importantly, both solutions favour low pressure (and high oxygen fugacity) in
order to make Ni as siderophilic as possible. Given that this is a core-rich
fragment (as suggested by the super-solar Fe/Mg and Ni/Mg ratios), moving as
much Ni into the core as possible helps to match the high observed Ni/Fe.

Swan et al. (2019) reported an anomalously high Na abundance for this system
which our model cannot reproduce. The cause is unknown. We therefore leave
open the question of how to explain these anomalous lines and consider only
the other observed (or constrained) elements.

###  B.5 WD1350-162

The median model is moderately core-rich (41%), with low pressure favoured by
the low Si abundance. The high observed Mg value relative to other elements is
explained by accretion in the declining phase. This is consistent with the
lack of observed infra-red excess. However, the model seems to also admit a
(less favoured) high pressure solution (with additional heating and a slightly
higher fragment core fraction). We discuss this important degeneracy in
section 4.5.3.

As with WD0449-259, an anomalously high Na abundance was reported for this
system by Swan et al. (2019) which we exclude from our modelling.

###  B.6 WD2105-820

This system has a strong requirement for differentiation (>4$\sigma$) due to
its low Ca/Fe ratio, which suggests a core-rich fragment (84%). Our model
explains the comparatively high Mg abundances as being caused by accretion in
the declining phase. The preference for low pressure is driven by an upper
bound on Si: core-like fragments become more Si rich as pressure increases.

###  B.7 PG0843+516

The model is able to fit this data by the accretion of a core rich fragment
(73%), driven by the high Fe relative to Ca, Mg and Si. We find a degeneracy
between pressure and oxygen fugacity, which is caused by Si. Increasing
pressure makes Si more siderophilic, but increasing oxygen fugacity makes Si
less siderophilic. It’s possible to increase (or decrease) both in such a way
as to keep the final Si abundance within the errors. Including a Cr or Ni data
point may help break the degeneracy.

As noted by Xu et al. (2019), there is a known discrepancy between metal
abundances based on UV and optical data. We ran our model on two data sets for
this system, taken from Gänsicke et al. (2012) and Xu et al. (2019). Gänsicke
et al. (2012) used UV data from HST’s COS instrument, and we use their
abundances reported for Al, Ni, Fe, Cr, Mg, Si and O. The abundances of Xu et
al. (2019) are based on optical data from HIRES at Keck, from which we use Ca,
Fe, Mg and Si. The Mg abundances are consistent to within the reported errors,
but the Fe and Si abundances are both significantly higher in Xu et al.
(2019). Our results are based on the version presented in Xu et al. (2019),
because we were unable to simultaneously fit the extreme Cr and Ni abundances
in Gänsicke et al. (2012).

It is possible that the material in the WD’s photosphere may have been
completely recycled in between the UV and optical observations. The sinking
timescales for this WD are very short - of the observed elements, the longest
sinking timescale belongs to Si (25 days). This implies that the material in
the WD’s photosphere may have been completely recycled in between the UV and
optical observations. However, as a DA white dwarf with
$\textrm{T}_{\textrm{eff}}\gtrsim$15,000-18,000 K, our interpolated sinking
timescales for this white dwarf should be treated with caution (see section
4.4).

###  B.8 HE0106-3253 and PG1015+161

These systems are similar to PG0843+516: the model infers a core-rich fragment
with the only pressure-sensitive element being Si. This leads to a
pressure/oxygen fugacity degeneracy. Along with PG0843+516, they appear
towards the bottom left of the bottom panel in Figure 6, in the region where
Si is reasonably sensitive to pressure. This suggests that these objects would
potentially yield tight pressure constraints, if another element were present
to break the pressure/oxygen fugacity degeneracy.

PG1015+161 has multiple data sets, from Gänsicke et al. (2012) and Xu et al.
(2019). As with PG0843+516, we use the set from Xu et al. (2019) since the
model is unable to fit the data from Gänsicke et al. (2012) (in particular,
the low Si/Mg ratio is difficult to reconcile with the other detected
abundances).

###  B.9 SDSSJ0512-0505

This system has roughly solar Fe/Mg and Ca/Mg ratios, but a depletion in Cr,
which the model can explain as the accretion of a primarily core-rich
fragment. To match the low Cr/Fe ratio, the model reduces the partition
coefficient of Cr. It can do this by either lowering pressure or increasing
oxygen fugacity, leading to a degeneracy.

###  B.10 SDSSJ0823+0546

Our model finds that SDSSJ0823+0546 has accreted a core-rich (96%) fragment.
The only pressure-sensitive element detected is Ni. As discussed in section
2.4, Ni is typically poor for constraining the pressure of core-rich
fragments, and we correspondingly find that the model is unable to constrain
pressure. A Si detection would strongly constrain pressure: our model predicts
that the Si abundance would vary by more than an order of magnitude across the
range of pressures we consider.

###  B.11 SDSSJ0738+1835

Cr, Ni and Si are all present, but the model can’t constrain pressure because
it finds multiple high-likelihood regions of parameter space, leading to a
bimodal pressure distribution. All of these solutions invoke accretion of a
moderately core-rich fragment (44%) in the declining phase. However, there are
3 possible combinations of pressure, oxygen fugacity and stellar metallicity.
Two of these solutions invoke high pressure, while the other invokes low
pressure.

###  B.12 SDSSJ0845+2257

Pressure could not be constrained for this system due to a combination of both
the large error bars and the inference that the fragment is only slightly
core-rich (33%), reducing the effect of changing pressure.

###  B.13 WD0122-227 and WD1145+288

WD0122-227 and WD1145+288 are polluted by core-rich fragments, but in each
case the only potential for information about pressure comes from an upper
bound on Si. Since these upper bounds are not strong, pressure is
unconstrained.

###  B.14 LHS2534

The model was able to fit the data, but the result was deemed unphysical. The
median model corresponds to a parent body composed of mostly O (79%), with
very low Fe content (6%) - consequently it has an unrealistically small core.
This behaviour is a consequence of moving to an unusually high
metallicity/high formation distance part of parameter space. Hollands et al.
(2021) concluded that this system was polluted by crustal material. We don’t
consider crustal material in our model, which may explain why we were unable
to find a satisfactory fit to the data. We also note that this object is
magnetic, which complicates spectral analysis (Hollands et al., 2021).

###  B.15 GALEX1931+0117

We adopted data from Gänsicke et al. (2012) for this system, from which we
found no evidence of differentiation. However, data were also available from
Vennes et al. (2011) and Melis et al. (2011). The fit to the data from Vennes
et al. (2011) was deemed to be unphysical, with a parent body composed of
mostly O (89%) and low Fe (3%), similar to LHS2534. This was likely driven by
the relative enrichment in Si and O compared to the other data sets. The data
set from Melis et al. (2011) has a low Ca/Mg ratio which the model was unable
to fit given that the short sinking timescales effectively restrict it to
steady state accretion, and that the O abundance constrains the effect of
incomplete condensation. Melis et al. (2011) suggested that the abundances may
be explained by accretion of a parent body which has been stripped of its
crust (and most of its mantle) by the AGB stellar wind. Similarly, Harrison et
al. (2018) found that their fit to the data was improved if the parent body
had a large crust which was removed by collisions. Our inability to fit this
dataset may be because we don’t include selective depletion of crustal
material. We also note that our sinking timescales for hot DAs such as this
may not be accurate (see 4.4).

##  Appendix C Predicted core/mantle composition

In this section, we show predictions for the core and mantle composition of an
Earth-like body which forms under different pressure/oxygen fugacity
conditions. Figure LABEL:fig:stacked_composition_plot illustrates the change
in number abundance of various elements in the core and mantle as a function
of pressure and oxygen fugacity. In each panel, the vertical thickness of each
shaded segment indicates the abundance (by number) of the corresponding
element. The top panels show both the core (hatched area) and mantle
(unhatched area) compositions. Our model predicts that the core number
fraction increases with increasing pressure (as Si and O partition more
strongly into the core) and decreases with increasing oxygen fugacity (as
metallic Fe is oxidised and incorporated into the mantle).

The behaviour discussed in section 2.4 is visible in Figure
LABEL:fig:stacked_composition_plot. In the left-hand middle panel, the dark
blue strip shows the mantle abundance of Ni, which increases significantly
with increasing pressure (as Ni becomes less siderophilic). In contrast, the
core abundance of Ni (shown in the lower left-hand panel) changes very little.
The lower left-hand panel also shows a dramatic increase in core Si (red) as
pressure increases, while the mantle abundance remains roughly constant. As
pressure increases, the increase in core Cr (pink), as well as the noticeable
decrease in mantle Cr, are both visible.

##  Appendix D Pressure Sensitivity of Different Elements

Pressure sensitivity varies by element, and also depends on the overall
fragment composition. We wish to calculate how the log bulk number abundance
of an element, $M$, in the fragment, $X_{b}$, varies with pressure.

We firstly calculate $X_{b}$. For an element differentiating with partition
coefficient $D$, the normalised core number abundance in the parent, $N_{c}$
is

$N_{c}=\frac{N_{b}D}{Dw+(1-w)},$ (24)

where $N_{b}$ is the normalised bulk number abundance of element $M$ in the
parent and w is the parent core number fraction. Similarly, the normalised
mantle number abundance in the parent, $N_{m}$ is

$N_{m}=\frac{N_{b}}{Dw+(1-w)}$ (25)

The fragment’s composition is determined by combining the parent core and
mantle material in arbitrary proportions. Taking $f$ to be the fraction of
core-like material in the fragment, we find that

$X_{b}=\frac{fDN_{b}+(1-f)N_{b}}{Dw+(1-w)}$ (26)

As the pressure changes, $D$ and $w$ also change. Differentiating equation
LABEL:eq:X_b with respect to $P$ gives

$\frac{dX_{b}}{dP}=\frac{N_{b}(f-w)\frac{dD}{dP}-N_{b}(f(D-1)+1)(D-1)\frac{dw}{dP}}{(1+(D-1)w)^{2}},$
(27)

in which we note that $P$ could be replaced by any variable which affects $D$
and/or $w$. Note that if the parent core fraction does not vary with $P$, the
right hand side vanishes when $f=w$. In other words, we recover the intuitive
result that if the fragment samples the parent’s core and mantle in the same
proportion as the parent itself, the abundance of $M$ is unaffected. In our
model, the parent core fraction generally increases with pressure (i.e.,
$\frac{dw}{dP}>0$, see Figure LABEL:fig:stacked_composition_plot), which means
that even when $f=w$ the abundance of $M$ varies with pressure.

Elemental observations are typically expressed as log number abundances (base
10), so we now find $\frac{d\log X_{b}}{dP}$. Using the identity

$\frac{d\log(X_{b})}{dP}=\frac{1}{X_{b}\ln(10)}\frac{dX_{b}}{dP}$ (28)

we find

$\frac{d\log(X_{b})}{dP}=\frac{(f-w)\frac{dD}{dP}-(f(D-1)+1)(D-1)\frac{dw}{dP}}{\ln(10)(f(D-1)+1)(w(D-1)+1}$
(29)

The greater the magnitude of this quantity, the easier it is to resolve the
core–mantle differentiation pressure in the parent body. Observational errors
are typically on the order of 0.1 log units, so as a rough order of magnitude
estimate we require $|\frac{d\log(X_{b})}{dP}|>0.01\textrm{GPa}^{-1}$ in order
to resolve the pressure to within 10 GPa. This threshold could be lower if
multiple elements are detected.

Figure LABEL:fig:dlogX_dP_v_fcf shows how pressure sensitivity changes with
fragment core fraction for different elements, assuming a bulk Earth parent
composition.

Despite Ni having a partition coefficient which spans many orders of magnitude
over the pressures we investigate, it only becomes sensitive to pressure at
very low fragment core fractions ($\approx$0.01). This is because the value of
$f$ at which $\frac{d\log(X_{b})}{dP}=0$ is reduced by virtue of
$D_{\textrm{Ni}}$ being large and decreasing with pressure. Because Ni is
highly siderophilic, its abundance in the core is not sensitive to pressure
and even a small amount of core-like material overwhelms the mantle’s Ni
contribution. This is why the Ni abundance of certain systems shows little
sensitivity to pressure, despite being polluted by apparently mantle-rich
material.

In the context of Figure 6, $\frac{d(\log X_{b})}{dP}$ is the quantity which
sets how spread apart the pressure contours are at a given fragment core
fraction. Since we plot ratios of 2 elements, the contours will additionally
be spread apart further (along the relevant axis direction) if the elements
have sensitivities with opposite signs (and vice versa). Pressure sensitivity
is a function of pressure, which means that along a line of constant fragment
core fraction, the pressure contours are not necessarily evenly spaced.