# ProbSAINT: Probabilistic Tabular Regression for Used Car Pricing

Kiran Madhusudhanan<EMAIL_ADDRESS>0000-0001-6356-8646
Information Systems and Machine Learning Lab
VWFS Data Analytics Research Center
University Of HildesheimHildesheimGermany ,  Gunnar Behrens  Data Analytics &
AI Engineering
Volkswagen Financial Services AGBraunschweigGermany ,  Maximilian Stubbemann
<EMAIL_ADDRESS>0000-0003-1579-1151 Information Systems and Machine
Learning Lab
VWFS Data Analytics Research Center
University Of HildesheimHildesheimGermany and  Lars Schmidt-Thieme  schmidt-
<EMAIL_ADDRESS>0000-0001-5729-6023 Information Systems and
Machine Learning Lab
VWFS Data Analytics Research Center
University Of HildesheimHildesheimGermany

###### Abstract.

Used car pricing is a critical aspect of the automotive industry, influenced
by many economic factors and market dynamics. With the recent surge in online
marketplaces and increased demand for used cars, accurate pricing would
benefit both buyers and sellers by ensuring fair transactions. However, the
transition towards automated pricing algorithms using machine learning
necessitates the comprehension of model uncertainties, specifically the
ability to flag predictions that the model is unsure about. Although recent
literature proposes the use of boosting algorithms or nearest neighbor-based
approaches for swift and precise price predictions, encapsulating model
uncertainties with such algorithms presents a complex challenge. We introduce
ProbSAINT, a model that offers a principled approach for uncertainty
quantification of its price predictions, along with accurate point predictions
that are comparable to state-of-the-art boosting techniques. Furthermore,
acknowledging that the business prefers pricing used cars based on the number
of days the vehicle was listed for sale, we show how ProbSAINT can be used as
a dynamic forecasting model for predicting price probabilities for different
expected offer duration. Our experiments further indicate that ProbSAINT is
especially accurate on instances where it is highly certain. This proves the
applicability of its probabilistic predictions in real-world scenarios where
trustworthiness is crucial.

Used Car Price, Attention, Deep Learning, Tabular Regression

††conference: ; 2024;

##  1\. Introduction

The Used car market is a vast and complex ecosystem with millions of vehicles
traded each year. For instance, the German used car market was valued at 113.2
Billion USD in 2021 and is expected to reach a value of 171.03 Billion USD by
2027, with a compound annual growth rate (CAGR) of 7.12% during the forecasted
period (Intelligence, 2022). Consequently, there is a need for a scalable
automated pricing system to accommodate this demand. This increase in volume
is accompanied by considerable volatility in the used car market. For example,
during the COVID-19 pandemic, supply-chain challenges and shortage of chips
and other electronic components, hindered the supply of both new and used
cars, resulting in inflated prices and higher profit margins for these
vehicles. Thus, complex machine learning techniques capable of swiftly
adapting to rapidly changing market conditions are necessary for pricing.
Furthermore, to ensure the trustworthiness of model predictions in an
automated pricing system, quantification of uncertainties is crucial.

Used car pricing is a complex task involving features that describe vehicle
characteristics such as vehicle-model, brand, and registration year, as well
as usage attributes like the damage, offer duration, and mileage. Moreover,
temporal features such as initial listing date and the selling date are also
crucial factors. Previous machine learning methods (Venkatasubbu and Ganesh,
2019; Samruddhi and Kumar, 2020) for used-car pricing rely on simple linear or
neighborhood-based models. Consequently, they struggle to manage the complex
mixture of numerical, categorical and temporal features mentioned above.
Recent studies utilize more complex Gradient Boosted Decision Tree models
(GBDTs) (Jawed et al., 2023; Cui et al., 2022) for used-car pricing. However,
they fail to compare their results with state-of-the-art deep learning
solutions or restrict experiments to smaller datasets. In contrast to the
specific area of used car pricing, recent advances in tabular learning have
moved in the direction of complex deep attention-based neural networks
(Borisov et al., 2022). In this work, we transfer and adapt these state-of-
the-art models to the specific use case of used car pricing, resulting in
pricing models which are able to handle complex feature sets while also
providing enhanced prediction accuracies. Although numerous works exists on
improving prediction accuracies on tabular dataset, uncertainty quantification
has received less attention. This lack of study limits their applicability in
industrial settings such as used car pricing, where trustworthiness is
crucial. To address this issue, previous approaches have measured the
uncertainty of model predictions as the standard deviation of values produced
by an ensemble of models (Malinin et al., 2021; Lakshminarayanan et al.,
2017). However, these techniques offer uncertainty in a post-hoc manner across
multiple models rather than training a single model to learn uncertainty from
a distributional loss.

This emphasizes the necessity for state-of-the-art attention-based tabular
model SAINT (Somepalli et al., 2021) to be enhanced to provide uncertainty
quantification. Building on this premise, we introduce our novel approach,
ProbSAINT. While SAINT was initially designed for tabular classification and
does not inherently offer uncertainty quantification, we extend its
capabilities to the probabilistic tabular regression setting. The proposed
ProbSAINT model incorporates a probabilistic loss, specifically the log-
likelihood loss, allowing the model to learn the distribution of the response
variable instead of focusing solely on individual point predictions. This
approach provides a principled method for modeling uncertainty without the use
of post-hoc ensembles as mentioned before. Furthermore, to our knowledge, the
evaluation of an attention-based tabular model such as SAINT (Somepalli et
al., 2021) on large real-world regression datasets is lacking in the
literature. We close this gap in the literature by demonstrating that both
ProbSAINT and the underlying SAINT are proficient in tackling real-world
regression tasks, such as used car pricing with a dataset comprising 2 million
data points.

In the context of used-car pricing, the _offer duration_ feature, indicating
how long a specific car has been on the market at the time of sale prediction,
holds significant importance. However, this information is typically
unavailable when predicting the initial sales price for a used car to be
listed on a selling platform. Often, the _expected offer duration_ represent a
strategic decision by businesses, factoring in margins and the volume of
similar used cars available. In other words, understanding the price
elasticity of a given vehicle across multiple offer durations for optimal
pricing is of interest. To address this, we propose the use of ProbSAINT for
conditional-inference procedure named _Probabilistic Dynamic Forecasting_ to
evaluate the price elasticity for multiple expected offer duration in the
future. In section 7, we illustrate how ProbSAINT learns distinct price
elasticities for multiple vehicles.

By offering more precise predictions alongside principled uncertainty values,
our approach enhances the reliability of machine-learning-based pricing
methods in the eyes of human experts. In doing so, it significantly reduces
the requirement for ensuring trustworthy predictions through the
implementation of extensive hand-crafted rules, which are frequently employed
in real-world business contexts. Thus, ProbSAINT can facilitate a move towards
fully automated used car pricing. Our specific contributions are as follows.

  * •

We propose ProbSAINT, to offer probabilistic outputs, thereby providing
confidence values for used car pricing.

  * •

We demonstrate the advantages of ProbSAINT empirically and qualitatively by
utilizing appropriate distributional metrics.

  * •

We experimentally show that the underlying SAINT architecture serves as a
valuable deep learning alternative for large-scale used car price predicion in
comparison to the state-of-the-art boosting methods.

  * •

We introduce and illustrate the utilization of ProbSAINT as a _probabilistic
dynamic forecasting_ model with the ability to grasp market dynamics.

##  2\. Related Work

Machine learning techniques have been previously employed for various price
prediction settings, For example, real estate price prediction (Tchuente and
Nyawa, 2022), house price estimation (Wang and Wu, 2018), inventory demand
estimation (Bajari et al., 2015) are few use-cases where machine learning
solutions have already seen an impact. On the topic of used car price
prediction Venkatasubbu and Ganesh (2019) suggest the use of Lasso regression
model for careful feature selection. The elimination of non-informative
features was shown to improve the performance for used car price prediction
within the Indian market. In another study, Samruddhi and Kumar (2020) propose
a nearest neighbor approach for pricing used cars. However, the study ignore
the temporal aspect of the problem and perform matching with the entire
training dataset. Despite these efforts to apply machine learning for used car
pricing, these methods primarily employ simple linear models. Recent studies
utilize more complex Gradient Boosted Decision Tree models (GBDTs) (Jawed et
al., 2023; Cui et al., 2022) for used-car pricing. However, they fail to
compare their results with state-of-the-art deep learning solutions or
restrict experiments to smaller datasets.

Advancement in the field of tabular regression is yet to be explored for the
topic of used car pricing. Typically, gradient boosted tree algorithms like
XGBoost (Chen and Guestrin, 2016), LightGBM (Ke et al., 2017) or CatBoost
(Prokhorenkova et al., 2018) are used for tabular regression tasks due to
their superior performance. Shwartz-Ziv and Armon (2022) compare the
performance of several deep learning models with XGBoost method and conclude
that efficient deep learning models for tabular data is still an active
research area. Recent studies on using deep learning solutions for tabular
data benefits from the use of attention mechanisms. Somepalli et al. (2021)
introduce SAINT, a Transformer model (Vaswani et al., 2017), that utilizes
self attention and inter-sample attention that rivals the performance of
boosting tree algorithms for classification tasks. Although, SAINT was
introduced for tabular classification tasks, Borisov et al. (2022),
experiments on the effectiveness of using SAINT for tabular regression on a
single dataset with 20 thousand samples. Here, the SAINT model, even though
being the best deep learning model, was unable to outperform its gradient
boosted counterparts. Evaluation on a single dataset does not provide
qualitative understanding of the effectiveness of SAINT. A different study
(Grinsztajn et al., 2022) highlights the subpar performance of deep learning
methods for tabular data. It concludes that, despite being the top-performing
deep learning model, SAINT still falls short when compared to Gradient Boosted
Decision Trees (GBDTs). In this study, we extend the application of the SAINT
model for tabular regression tasks and assess its performance on a substantial
real-world pricing dataset. Furthermore, as the uncertainty quantification of
tabular data models is a relatively less explored area, we modify the SAINT
framework to generate probabilistic outputs. This adaptation aims to increase
the model’s reliability in an industrial environment

Uncertainty quantification is a crucial aspect for any decision-making
process. Previous works on the topic of price prediction ignores this aspect
of the problem (Jawed et al., 2023; Cui et al., 2022; Samruddhi and Kumar,
2020). Even though boosting methods are highly competent within the tabular
data regime, they are restricted to a scalar value for regression. In order to
predict a Gaussian distribution output, the model requires at least two
degrees of freedom, for example, the mean and standard deviation of the
predictions. Hence, uncertainty quantification with boosted trees is a
challenge. CatBoost (Prokhorenkova et al., 2018) and other boosting methods
attempt to circumvent this challenge by providing uncertainty over an ensemble
of models (Malinin et al., 2021; Sprangers et al., 2021). Another popular
approach, Monte Carlo (MC) dropout (Gal and Ghahramani, 2016) uses the dropout
(Srivastava et al., 2014) regularization term to quantify uncertainty. While
several works have used MC dropout (Alarab et al., 2021) for uncertainty
quantification, it has been shown to represent incorrect posterior for deeper
neural networks (Foong et al., 2020). Above mentioned methods are not a
principled method to model uncertainty, as the model is not trained for
uncertainty quantification bur rather use ad-hoc post-processing to quantify
uncertainty. More principled techniques to model uncertainty have been
researched for deep learning models. Normalizing flow models (Papamakarios et
al., 2021) are an emerging field of research to model exact posterior
distributions from given data with the help of a series of bijective
transformations. Even though normalizing flows are widely adopted within the
computer vision domain (Abdelhamed et al., 2019) and time series domain (Rasul
et al., 2020), extrapolation to tasks with a scalar value output is less
explored. Generalized linear models (GLMs) (Nelder and Wedderburn, 1972) offer
a principled approach to modelling uncertainty. These can be easily
incorporated to any model by learning the distribution parameters of the
target. Therefore, we adapt the SAINT model to a Generalized Linear model that
easily learns the distribution parameters and provides probabilistic outputs.

##  3\. Problem Definition

The problem of used car pricing involves learning from historic prices to
predict the final _sales price_ for an unseen used car. In addition, the task
of _probabilistic used car pricing_ involves calculating an expected price and
a measure of uncertainty for the prediction, or more general, to predict the
distribution of likely prices.

The abstract problem can be formalized as follows. Assume we have given a
dataset $\mathcal{D}\subseteq\mathbb{R}^{D}\times\mathbb{R}$ of
$|\mathcal{D}|=N$ many samples drawn from an unknown distribution $p$. The
task is to find a _model_

$\hat{p}\colon\mathbb{R}^{D}\to\text{Dist}(\mathbb{R})$

which predicts for predictors $x\in\mathbb{R}^{D}$ a distribution
$\hat{p}(y\mid x)$ for the target $y$. Models should minimize the expected
Negative Log Likelihood (NLL):

$\displaystyle\ell(\hat{p};p):=-\mathbb{E}_{(x,y)\sim p}\log(\hat{p}(y\mid
x))$

Models often choose a specific shape of target distribution in advance, e.g.,
a normal distribution. To predict for every $x$ a distribution then simplifies
to predict the distribution’s parameters, i.e., its mean and variance:

$\displaystyle\hat{m}$ $\displaystyle\colon\mathbb{R}^{D}\to\mathbb{R}^{2}$
$\displaystyle\hat{p}(y\mid x)$
$\displaystyle:=p_{\text{normal}}(y\mid\mu:=\hat{m}(x)_{1},\sigma^{2}:=\hat{m}(x)_{2})$

and the Negative Log Likelihood loss (NLL) in
$\hat{m}(x)=(\mu_{x},\sigma_{x}^{2})$ then is

(1) $\ell(\hat{m};p)=\mathbb{E}_{(x,y)\sim
p}\left[\quad\frac{1}{2}\big{(}\text{log}(\text{max}(\sigma_{x}^{2},\epsilon))+\frac{(y-\mu_{x})^{2}}{\text{max}(\sigma_{x}^{2},\epsilon)}\big{)}\right]$

Here, $\epsilon>0$ is chosen as a small value to prevent divisions by $0$ and
to provide numeric stability.

##  4\. Methodology: ProbSAINT

Figure 1. Architecture of proposed ProbSAINT model. Every block contains layer
normalization following each attention and feed-forward layer.

The recent success of deep learning can often be attributed to the expressive
power of the Transformer architecture (Vaswani et al., 2017). However, the
direct application of the Transformer architecture on tabular data has not
witnessed significant performance improvements similar to other domains
(Grinsztajn et al., 2022), which has led to attention-based tabular models
that fundamentally modify the base Transformer architecture. For example,
SAINT (Somepalli et al., 2021) adds a specific embedding technique for
numerical features and introduces inter-sample attention, which leads to
significant performance gains in tabular classification tasks. For more
details, we refer the reader to Somepalli et al. (2021). For our novel model
ProbSAINT, we modify the SAINT model to perform the task of tabular regression
with uncertainty quantification. In detail, the individual components of our
model are as follows.

###  4.1. Encoding

As previously stated, the task of pricing used cars requires a model that can
effectively handle both continuous and categorical variables. Typically, deep
learning solutions embed categorical variables into a $d$-dimensional space,
leaving numerical features unprocessed (Huang et al., 2020). As a result,
categorical variables often dominate the embedding space. The SAINT model,
addresses this by using a separate, fully-connected network with ReLU
activation for each continuous feature. This projects the single-dimensional
numerical feature into the same $d$-dimensional vector space where the
categorical features are embedded, allowing a richer feature representation.
More specifically, our encoder $E$ maps an input vector $x\in\mathbb{R}^{D}$
to a sequence of length $D$ of $d$-dimensional vectors, i.e.,

$\displaystyle E\colon\mathbb{R}^{D}$ $\displaystyle\to(\mathbb{R}^{d})^{D}$
$\displaystyle x$ $\displaystyle\mapsto(E_{1}(x_{1}),\dots,E_{D}(x_{D})),$

where $E_{i}:\mathbb{R}\to\mathbb{R}^{d}$ is the encoder of the $i$-th feature
of $\mathcal{D}$.

###  4.2. Self Attention and Inter-Sample Attention

SAINT relies on the alternating use of self-attention and inter-sample
attention. While self-attention is a concept borrowed from the Transformer
architecture (Vaswani et al., 2017), inter-sample attention was first
introduced in SAINT. Unlike the original attention mechanism, which applies
attention across different features, inter-sample attention enables the
computation of attention across different data points within a given batch.
The ProbSAINT model along with the inter-sample attention is illustrated in
Figure 1. The authors of the SAINT model suggest that the inter-sample
attention enables the model to borrow missing or noisy features from other
similar data points, which is beneficial in the tabular data setting.
Therefore, the SAINT model is composed of a stack of $L$ identical blocks,
each of which alternates between self-attention and inter-sample attention.

In detail, let $z^{(1)},\dots,z^{(m)}$ be a batch of $m$ data points, where a
data point $z^{(i)}$ is a sequence of $D$ many $d$-dimensional vectors. We
represent such datapoints via a matrix $Z^{(i)}\in\mathbb{R}^{D\times d}$
where $Z^{(i)}_{k,l}$ is the $l$-th feature of the $k$-th element of the
sequence. We represent the batch $(Z^{(1)},\dots,Z^{(m)})$ via a tensor
$Z\in\mathbb{R}^{m\times D\times d}$, where $Z_{i,k,l}\coloneqq
Z^{(i)}_{k,l}$. The _self-attention_ layer applies the normal attention
mechanism on each datapoint over the features or columns, which can be seen as
a function

$\text{{self-attention}}_{D,d}\colon\mathbb{R}^{D\times
d}\to\mathbb{R}^{D\times d}.$

A self-attention layer applies this to all datapoints of a batch individually
and can be seen as a function

$\text{{Self-Attention}}\colon\mathbb{R}^{m\times D\times
d}\to\mathbb{R}^{m\times D\times d}.$

Then, the _inter-sample_ layer is applied over the datapoints or rows in a
batch, which works as follows: We concat all sequences so that we have only
one vector for each datapoint which can be interpreted as a map

$\texttt{concat}\colon\mathbb{R}^{m\times D\times d}\to\mathbb{R}^{1\times
m\times(Dd)}$

on which the normal attention mechanism is applied, for which the output is
then split back to a sequence via

$\texttt{split}\colon\mathbb{R}^{m\times(Dd)}\to\mathbb{R}^{m\times D\times
d}.$

The full inter-sample attention is then given via

$\displaystyle\texttt{InterSample-Attention}\colon\mathbb{R}^{m\times D\times
d}\to\mathbb{R}^{m\times D\times d}$ $\displaystyle
Z\mapsto\texttt{split}(\texttt{self-attention}_{m,Dd}(\texttt{concat}(Z))).$

###  4.3. Distributional Output

Our complete model is composed of an encoder and $L$ blocks of attention
layers as illustrated in Figure 1. Furthermore, we incorporate a final 2-layer
MLP with ReLU activation at the hidden layer and no activation at the output,
to predict means and standard deviations. This approach differs from SAINT,
where the 2-layer MLP only produces a single output for regression target. The
model is trained on the objective defined in Equation 1.

Table 1. Example used-car features and corresponding data types. The “sales
price” is our target feature.

Feature | Type | Feature | Type
---|---|---|---
Odometer | Numeric | Available Date | Date
Condition | Categorical | Engine Type | Categorical
Age | Integer | Engine Size | Numeric
Offer Duration | Integer | Engine Power | Numeric
Make | Categorical | Fuel Type | Categorical
Initial Registration | Date | Transmission | Categorical
Model | Categorical | Euro Emissions Std | Categorical
Model Variant | Categorical | Sales Date | Date
Brand | Categorical | Sales Price | Numeric

##  5\. Experimental Setting

###  5.1. Dataset

Given the absence of large-scale benchmark datasets for used car pricing, we
train and evaluate the ProbSAINT model on externally acquired and standardized
data extracted from real-world Business-to-Customer (B2C) online websites such
as autoscout24, mobile.de, among others. The dataset that we derive consists
of 65 features, which include 55 categorical values, 7 numerical, and 3 date
features. Spanning a period from July 1, 2018, to August 20, 2022, the data
comprises approximately 2 million records. Table 1 provides a summary of the
key features of the B2C dataset. We train the model on these 65 features with
the aim of predicting the final sales price, also conditioned on the offer
duration for a specific used car.

###  5.2. Data Preprocessing

We convert categorical features into integer encodings using sklearn
(Pedregosa et al., 2011). This encoding treats any new categories that may
appear in the testing range as unknown. In our scenario, this could involve
the introduction of a new vehicle or a different variant of an existing
vehicle. We also preprocess the dataset by replacing any missing values with a
placeholder value. For the ‘Date’ features in the dataset, we generate integer
features such as day of the month, month of the year, and year. For the
months, we create embeddings as follows.

$\displaystyle\text{SinMonth}(m)$
$\displaystyle=\texttt{Sin}\big{(}\frac{2\pi}{12}m\big{)},$
$\displaystyle\text{CosMonth}(m)$
$\displaystyle=\texttt{Cos}\big{(}\frac{2\pi}{12}m\big{)}.$

These periodic embeddings reflect the seasonal nature of the different months.
Furthermore, we also incorporate the absolute time information by determining
the number of days from a specified date to capture any trend information in
the data.

###  5.3. Training and Evaluation

In our experiments, we adopt a time-wise split to divide the dataset into
training, validation, and testing data. For example, if ‘2022-03-20’ is set as
the beginning of the testing period, all records up to ‘2022-02-20’ form the
training data. The validation data is composed of the last month’s data,
spanning from ‘2022-02-20’ to ‘2022-03-20’, while the testing data includes
three months of records, from ‘2022-03-20’ to ‘2022-06-20’. This time-wise
split reflects real-world scenarios where the testing data lies in the future.
Consequently, we create two versions of the B2C data: ‘B2C-March’, which
starts the testing period on ‘2022-03-20’, and ‘B2C-June’, which starts the
testing period on ‘2022-05-20’. Although retraining on the validation data
could be a beneficial strategy, for the current research, we limit the model,
to train only on the training split. A previous study (Jawed et al., 2023) on
a similar dataset reported minimal to no performance enhancement after
retraining on the validation data.

For the task of probabilistic prediction, all the neural network models, and
the NGBoost model, were trained using Negative Log Likelihood (NLL) as
outlined in Equation 1. The CatBoost model, which offers uncertainty
quantification, was trained using the ‘RMSEWithUncertainty’ loss function.
Regardless of the loss function used during training, we employ NLL as a
metric to assess the quality of probabilistic predictions. In addition, for
the evaluation of non-probabilistic point prediction tasks, we utilize Mean
Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE), which are
defined as follows:

$\displaystyle\text{MAE}(\hat{m},p)$ $\displaystyle=\mathbb{E}_{(x,y)\sim
p}|y-\mu_{x}|,$ $\displaystyle\text{MAPE}(\hat{m},p)$
$\displaystyle=\mathbb{E}_{(x,y)\sim
p}\frac{|y-\mu_{x}|}{\text{max}(\epsilon,|y|)}.$

##  6\. Probabilistic Price Prediction

Our results section begins with a comparison of ProbSAINT with other methods
that offer probabilistic interpretations for tabular regression. We evaluate
the model using the Negative Log Likelihood (NLL) as a probabilistic metric.
Additionally, we report the deviation of the mean prediction ($\mu_{x}$) from
the true selling price ($y$) using the Mean Absolute Percentage Error (MAPE).

The main focus of our research is to determine if the ProbSAINT model, which
is trained on a distribution loss, can outperform the state-of-the-art
boosting alternatives in quantifying uncertainty. In the following section, we
first present the chosen baselines for our experiments, followed by an
evaluation of the results in Table 2.

#### Baselines.

CatBoost, the system currently in use, serves as a primary baseline for our
research. For uncertainty quantification, CatBoost provides an out-of-the-box
uncertainty quantification measure when trained using the
”RMSEWithUncertainty” loss function, a model we refer to as ”CatBoostUn”. We
describe below the core ideas for our chosen baselines.

  1. (1)

MC-Dropout (Gal and Ghahramani, 2016) (ICML’16): MC-Dropout uses dropout
during the testing phase as an approximate Bayesian inference in deep Gaussian
processes. Here, we use the best performing SAINT model as the backbone for
this method.

  2. (2)

NGBoost (Duan et al., 2020) (ICML’20): NGBoost is a modular framework that
learns using the ”Natural Gradient” descent as an alternative to stochastic
gradient descent. NGBoost considers the parameters of a conditional
distribution as targets for a multiparameter boosting algorithm.

  3. (3)

CatBoostUn (Malinin et al., 2021) (ICLR’21): The well known CatBoost library
allows uncertainty quantification of its predicted values when trained with
the loss function ”RMSEWithUncertainty”. The model predicts the uncertainty
from an ensemble-of-trees. The study claims that ensembles of gradient
boasting models were able to successfully detect anomalous inputs.

  4. (4)

ProbMLP: In order to contrast the importance of the SAINT architecture
backbone in ProbSAINT, we use a deep Multi-Layer Perceptron (MLP) model as the
base architecture.

Table 2. Performance comparison of the proposed ProbSAINT model with relevant
baselines on probabilistic metric of NLL and point prediction MAPE metric.
(lower is better).

| NLL | MAPE
---|---|---
| B2C-March | B2C-June | B2C-March | B2C-June
MC-Dropout | 22.145 |  $\,\pm\,$1.128 | 11.099 |  $\,\pm\,$0.256 | 0.058 | $\,\pm\,$0.000 | 0.056 | $\,\pm\,$0.000
NGBoost | 8.462 |  $\,\pm\,$0.058 | 8.228 |  $\,\pm\,$0.017 | 0.059 |  $\,\pm\,$0.001 | 0.055 |  $\,\pm\,$0.001
CatBoost | 8.491 |  $\,\pm\,$0.003 | 8.217 | $\,\pm\,$0.001 | 0.063 |  $\,\pm\,$0.000 | 0.057 |  $\,\pm\,$0.000
ProbMLP | 8.355 | $\,\pm\,$0.076 | 8.314 |  $\,\pm\,$0.027 | 0.065 |  $\,\pm\,$0.004 | 0.065 |  $\,\pm\,$0.003
ProbSAINT | 8.192 |  $\,\pm\,$0.026 | 8.144 |  $\,\pm\,$0.022 | 0.056 |  $\,\pm\,$0.001 | 0.053 |  $\,\pm\,$0.001

#### Analysis.

Table 2 presents a comparison of the proposed ProbSAINT with other methods
used for uncertainty quantification. As observed from Table 2, despite
extensive modifications to these GBDT methods, the NLL loss for these methods
is higher than that of the proposed ProbSAINT model on both dataset splits.
Interestingly, the high-quality probabilistic output from ProbSAINT does not
compromise point prediction accuracy, as evidenced by the MAPE values for the
corresponding models. The second best model in terms of the MAPE value is the
MC-Dropout method that uses a SAINT backbone. The fact that the MC-Dropout
model performs second best on the MAPE values is not a surprise here, as the
underlying SAINT model was trained and tuned for the point prediction task.
However, for the uncertainty quantification task the MC-Dropout method, that
uses the dropout at testing phase, fails to capture uncertainty as shown by
the NLL error rates. To summarize, while GBDT models are reliable for quick
and accurate point predictions, ProbSAINT emerges as a far superior
alternative for probabilistic forecasting.

Figure 2. Qualitative comparison of ProbSAINT with that of the second best
CatBoostUn baseline. The x-axis denotes the confidence score and y-axis
indicates the corresponding MAPE error.

###  6.1. Probabilistic Prediction Quality at Multiple Confidence Levels

To further evaluate the benefits of employing ProbSAINT for uncertainty
quantification, we graphically represent the confidence and the corresponding
error for ProbSAINT and compare it with that of the CatBoostUn method, as
depicted in Figure 2. In this context, we compute the confidence scores via

$\displaystyle C=1-\frac{\sigma_{x}}{\mu_{x}},$

where $\mu_{x},\sigma_{x}$ indicates the mean and standard deviation of the
prediction, respectively. As depicted in Fig. 2, CatBoostUn tends to make
overconfident predictions, resulting in a larger area under the curve and
outliers compared to ProbSAINT. More specifically, one can observe that the
CatBoost error increases for highly confident predictions. Furthermore, even
in the area of 0.93 to 0.95 confidence, the MAPE difference between ProbSaint
and CatBoost increases. Thus, the ability of ProbSAINT to learn the standard
deviation and mean using the NLL loss provides it with a clear advantage over
the baseline.

(a) ProbSAINT

(b) CatBoostUn

Figure 3. Qualitative analysis of ProbSAINT and CatBoostUn baseline. X-axis
denotes the indices of different vehicles order in the decreasing order of
their true selling price and Y-axis denotes the price.

###  6.2. Individual Price Predictions

We furthermore compare the standard deviation and mean from ProbSAINT and
CatBoostUn by plotting the difference to the true value in Figure 3. Here, the
interesting observation is, that ProbSAINT slightly overpredict prices while
CatBoost slightly underprices. However, for both methods the true value is
always within the standard deviations, which indicates that both methods
produce reasonable distributions.

###  6.3. Point Price Prediction

The probabilistic prediction results clearly show the advanatage of ProbSAINT.
We now investigate wether the underlying model of ProbSAINT, namely SAINT,
serves as a potent model for tabular regression on larger datasets. To address
this auxiliary research question, we base our study on a previous survey paper
(Borisov et al., 2022). This study compares various deep learning techniques
with GBDTs and concludes that GBDTs outperform deep learning techniques for
tabular tasks. However, the authors’ evaluation of their methods is based on a
considerably smaller dataset, comprising nearly 20 thousand records for
tabular regression, in contrast to the used-car pricing dataset, which
contains 2 million rows. Consequently, we strive to answer this significantly
relevant research question using results from Table 3.

#### Baselines.

As mention before, we consider GBDT methods as a strong baseline for our
analysis as it has been shown to outperform other models in recent studies
(Borisov et al., 2022; Grinsztajn et al., 2022). We choose additional
baselines considering their superior performance on the previous study
(Borisov et al., 2022). These are as follows.

  1. (1)

KNN : K-Nearest Neighbor (KNN), identifies most similar data samples from the
training data corresponding to the testing sample and averages the prediction
across $K$ many samples.

  2. (2)

MLP: Multi-Layer Perceptrons (MLP) are deep learning solutions that employ
multiple layers of fully connected layers with ReLU non-linearity in between
each layer.

  3. (3)

LightGBM (Ke et al., 2017) (NeurIPS’17): A powerful gradient boosting
technique that speeds up the conventional Gradient Boosted Decision Tree’s
(GBDT), by ignoring instances with smaller gradients from the calculation of
information gain and reduces the number of features by bundling mutually
exclusive features.

  4. (4)

XGBoost (Chen and Guestrin, 2016) (KDD’16): Another powerful scalable and
effective gradient boosting technique widely used to achieve state-of-the-art
performance in many machine learning challenges.

  5. (5)

CatBoost (Prokhorenkova et al., 2018) (NeurIPS’18): Gradient boosting
technique that uses the target statistics to encode high fidelity categorical
features into numerical features. This allows CatBoost models to be trained
with almost no preprocessing steps for efficient handling and with minimum
information loss.

  6. (6)

DeepFM (Guo et al., 2017) (IJCAI’17): A factorization-machine based neural
network initially proposed for Click-Through-Rate (CTR) prediction. The model
was shown to be competitive in (Borisov et al., 2022).

We also experimented with additional baselines like the TabTransformer (Huang
et al., 2020) and the FT-Transformer (Gorishniy et al., 2021) for the used car
pricing task. These models encode each categorical variable with an attention
layer, leading to large memory requirements to process the 2 million records
with 56 categorical features in the used-car dataset. Hence, these model
results could not be reported in the final evaluations. Additionally, recent
work (Jawed et al., 2023) on used-car pricing compares these models on a much
smaller dataset size and report almost double the error as that of the GBDT
models for point prediction.

#### Analysis.

From the Table 3, we note that for larger datasets, such as the used-car
pricing dataset, the SAINT model demonstrates performance on par with the GBDT
models. The SAINT model outperforms in the ‘B2C-March’ split and exhibits
comparable accuracy in the ‘B2C-June’ split for both MAPE and MAE metrics.
Additionally, the results from Table 2 indicate that for ‘B2C-June’ the SAINT
model is capable of similar performance as the GBDT methods in Table 3. It is
also worth noting that the standard deviation values of the deep learning
models are larger compared to the GBDT models, suggesting a higher sensitivity
to random sources like parameter initialization. However, when compared to
(Borisov et al., 2022) and (Jawed et al., 2023), where deep learning models
face challenges with the regression task compared to GBDT models, we observe a
clear indication that deep learning models could close the performance gap as
the size of the dataset increases. However, the superiority of SAINT is not as
clear as the one of ProbSAINT in the probabilistic setting. This indicates
that our novel probabilistic training is a key factor for successful
predictions.

Table 3. Performance comparison of various tabular regression models on used-
car pricing datasets. The models are evaluated on MAPE and MAE metrics. (lower
is better).

| MAPE | MAE
---|---|---
| B2C-March | B2C-June | B2C-March | B2C-June
KNN | 0.175 |  $\,\pm\,$0.002 | 0.173 |  $\,\pm\,$0.002 | 6346.18$\,\pm\,$ | $102.68$ | 6398.92$\,\pm\,$ | $146.49$
MLP | 0.064 |  $\,\pm\,$0.001 | 0.067 |  $\,\pm\,$0.001 | 2018.22$\,\pm\,$ | $30.60$ | 2133.89$\,\pm\,$ | $30.66$
LightGBM | 0.058 |  $\,\pm\,$0.000 | 0.053 |  $\,\pm\,$0.000 | 1846.80$\,\pm\,$ | $5.76$ | 1732.12$\,\pm\,$ | $0.00$
XGBoost | 0.058 |  $\,\pm\,$0.000 | 0.054 |  $\,\pm\,$0.000 | 1847.37$\,\pm\,$ | $0.00$ | 1737.77$\,\pm\,$ | $0.00$
CatBoost | 0.058 |  $\,\pm\,$0.000 | 0.053 |  $\,\pm\,$0.000 | 1860.81$\,\pm\,$ | $1.64$ | 1713.94$\,\pm\,$ | $7.89$
DeepFM | 0.058 |  $\,\pm\,$0.000 | 0.065 |  $\,\pm\,$0.002 | 1865.43$\,\pm\,$ | $8.90$ | 2006.16$\,\pm\,$ | $10.3$
SAINT | 0.056 |  $\,\pm\,$0.001 | 0.055 |  $\,\pm\,$0.002 | 1834.51$\,\pm\,$ | $41.19$ | 1782.04$\,\pm\,$ | $68.07$

Algorithm 1 Dynamical Forecasting for a specific vehicle with ProBSAINT

1:Trained ProbSAINT, Vehicle v with feature vector $x$

2:Outputs = []

3:$i$ = coordinate of $x$ which displays the offer days

4:$O$ = [15, 45, 75, 105, 150]

5:for $o$ in $O$ do

6: $x_{i}$ = $o$

7: $\mu$, $\sigma$ = ProbSAINT$(x)$

8: Outputs.append([$\mu,\sigma$])

9:end for

##  7\. Probabilistic Dynamic Forecasting

The process of selling a used car involves a balancing act between the profit
margin on the sale of a used-car and the volume of used-cars sold. In certain
situations, it may be advantageous to sell the used-car as quickly as
possible. Conversely, in other scenarios, it might be more beneficial to set a
higher price for the used-car and allow it to remain listed on the platform
for a longer duration.

The data considered in Section 5 includes the feature _offer durations_ which
indicates, the duration a specific car has been on the market at the time of
sale. While this feature is beneficial during training and testing, it does
not help in predicting the initial sales price for a new car entering the
market. Here, a reasonable procedure is to identify a maximum of days
$d_{\max}$ that the car should be on the market, allow the model to predict
the sales price with the offer durations being set to $0$ and with the offer
durations being set to $d_{max}$ and choose a sales price between the two
predicted values. This raises the question how ProbSAINT behaves with respect
to varying this feature. Therefore, in this section, we investigate how the
predicted prices vary for different models and different offer durations.

In this scenario, the task of pricing used cars can be viewed as a temporal
forecasting task, where the objective is to predict the future price of an
unknown used-car over various offer durations. However, unlike typical
forecasting tasks, we do not have time-varying information for each used-car.
Instead, we have a data point indicating the price at which the used-car was
sold. Our used-car dataset comprises 2 million such data points, representing
used-cars sold at various _offer durations_. The question then is: Can a model
trained on these 2 million records learn the general market dynamics of how
the selling price fluctuates across different offer durations for different
used-cars?

To answer this, we use a ProbSAINT model trained on B2C-March split, and
choose 3 different used-cars for which all features except _offer durations_
are fixed. Then, we let ProbSAINT predict the output distribution parameters
$\mu$ and $\sigma$ for different amount of offer durations, ranging from 15 to
150. This procedure of generating dynamical forecasts with ProbSAINT is shown
in Algorithm 1. Figure 4 presents three distinct used cars, denoted as
UsedCar-1, UsedCar-2, and UsedCar-3, each sold at different _Offer durations_.
For instance, UsedCar-1 (represented in red) was sold after 62 days on the
platform, while UsedCar-2 (in blue) and UsedCar-3 (in green) were sold after
48 and 98 days, respectively, as indicated by the plus symbol. The y-axis is
normalized by the predicted price at 15 _Offer duration_ to capture the
underlying trend.

#### Analysis

Our results provide evidence that each model learns distinct market dynamics
when it remains on the platform for an extended period. While UsedCar-1 and
UsedCar-3’s price predictions do not show a clear trend with respect to
increasing offer durations, ProbSAINT predicts a decreasing price trend for
UsedCar-2 with an increasing offer duration. A possible outcome from the
prediction is that the UsedCar-2 should be priced by risking a lower selling
price due to the decreasing trend. It is further notable, that for all three
models the price predicted after $45$ offer durations is higher than after
$15$. Thus, ProbSAINT indicates that keeping vehicles on the market for a
short amount of time can be beneficial in contrast to immediate sales. We are
of the opinion that understanding of these market dynamics could be
advantageous in determining the best price for used cars.

Figure 4. Probabilistic prediction of price with respect to change in offer
durations. The X-axis denotes the expected offer durations for a used-car, and
the Y-axis denotes the normalized price prediction from the ProbSAINT for
varying offer durations. The ’+’ data point denotes the ”true selling price”
for the ”true offer durations”, and the ’o’ denotes predictions. Comparing
three different model instances, the ProbSAINT learns varying market dynamics
for the different models. Table 4. Hyperparameter configurations of various
models used in the study.

Model | Hyperparameters | Type | Values
---|---|---|---
SAINT ProbSAINT MC-Dropout | Dim | Categorical | [32, 64, 128, 256]
Depth | Categorical | [1, 2, 3, 6, 12]
Heads | Categorical | [2, 4, 8]
Dropout | Categorical | [0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8]
MLP | Dim | Ordinal | (10-100)
Num. layers | Ordinal | (2, 5)
Learning Rate | Continous | (0.0005, 0.001)
NGBoost | Learning Rate | Continous | (001, 0.3)
Base | Categorical | [DecisionTree, GradBoost]
Mini Batch Frac | Continous | (0.6, 1.0)
CatBoost CatBoostUn | Learning Rate | Continous | (0,01, 0.3)
Max Depth | Ordinal | (2, 12)
L2 Leaf Reg. | Continous | (0.5, 30)
DeepFM | Droupout | Continous | (0, 0.9)
XGBoost | Max Depth | Ordinal | (2, 12)
Alpha | Continous | (1e-8, 1.0)
Lambda | Continous | (1e-8, 1.0)
Eta | Continous | (0.01, 0.3)
LightGBM | Learning Rate | Continous | (0,01, 0.3)
Lambda L1 | Continous | (1e-8, 10)
Lambda L2 | Continous | (1e-8, 10)
Num. Leaves | Ordinal | (2, 4096)
KNN | Neighbors | Ordinal | (3, 42)

##  8\. Hyperparameters

For hyperparameter tuning each model, we use Optuna (Akiba et al., 2019) with
the default Tree-structured Parzen Estimator (TPE) algorithm. The TPE
algorithm is a Sequential Model Based Optimization strategy (SMBO) that
suggests the next best trial parameters conditioned on the evaluated trials
from a pre-defined hyperparameter space. This facilitates effective
hyperparameter search within an allocated budget. For our experiments, we set
the budget as 20 trials for the Gradient Boosted Decision Tree (GBDT) models
and 10 trials for the neural network models. The trials were defined
considering the fact that the neural network models require longer training
times in comparison to GBDT models. The model specific hyperparameters are
listed in Table 4.

##  9\. Deployed System

Our currently deployed system utilizes a CatBoost model similar, but not
exactly as described in section 6 that predicts the ‘sales price’ based on the
65 features mentioned in section 5.2. The CatBoost pricing model, deployed on
AWS, is live for the German market and was subsequently expanded to include
the Italian, Spanish, and Czech markets. In 2023, the CatBoost solution
predicted almost 2M pricing requests, with a human expert acceptance rate of
50-60%.

##  10\. Conclusion and Future Work

We presented ProbSAINT, a novel deep neural network for probabilistic
regression on tabular data. By adapting the well-established SAINT approach
and incorporating a 2-layer MLP head, ProbSAINT is able to predict
distributions instead of point predictions. Thus, ProbSAINT is able to provide
confidence values for its predictions. This is especially required in real-
world applications where trustworthiness is crucial, such as predicting for
used car prices. Our experiments on real-world industry datasets show that
ProbSAINT outperforms boosting-based state-of-the-art baselines for
probabilistic regression tasks. Further experiments indicate that ProbSAINTs
prediction are particularly accurate when it has a high confidence, validating
that ProbSAINT computes reasonable target distributions. Finally, we
demonstrate that ProbSAINT can be used to compute prices for multiple offer
durations, allowing it to include background information like the maximal
desired time in the market.

In this study, we trained ProbSAINT using an end-to-end approach, without the
use of auxiliary or self-supervised pre-training tasks. In future work, we
plan to develop useful pre-training tasks to leverage larger volume of
additional data. However, determining the optimal pre-training strategy for
probabilistic industrial applications such as used car pricing is an open
problem. ProbSAINT’s ability to provide reliable point- and probabilistic
outputs paves the way for future work on adapting established deep learning
techniques like transfer learning and self-supervised representation learning.

## References

  * (1)
  * Abdelhamed et al. (2019) Abdelrahman Abdelhamed, Marcus A. Brubaker, and Michael S. Brown. 2019.  Noise Flow: Noise Modeling With Conditional Normalizing Flows. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_.
  * Akiba et al. (2019) Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019.  Optuna: A Next-generation Hyperparameter Optimization Framework. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_.
  * Alarab et al. (2021) Ismail Alarab, Simant Prakoonwit, and Mohamed Ikbal Nacer. 2021\.  Illustrative discussion of mc-dropout in general dataset: uncertainty estimation in bitcoin.  _Neural Processing Letters_ 53 (2021), 1001–1011.
  * Bajari et al. (2015) Patrick Bajari, Denis Nekipelov, Stephen P Ryan, and Miaoyu Yang. 2015.  Machine learning methods for demand estimation.  _American Economic Review_ 105, 5 (2015), 481–485.
  * Borisov et al. (2022) Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. 2022.  Deep neural networks and tabular data: A survey.  _IEEE Transactions on Neural Networks and Learning Systems_ (2022).
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  XGBoost: A Scalable Tree Boosting System.  _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (2016).  https://api.semanticscholar.org/CorpusID:4650265
  * Cui et al. (2022) Baoyang Cui, Zhonglin Ye, Haixing Zhao, Zhuome Renqing, Lei Meng, and Yanlin Yang. 2022\.  Used Car Price Prediction Based on the Iterative Framework of XGBoost+LightGBM.  _Electronics_ 11, 18 (2022).  https://doi.org/10.3390/electronics11182932
  * Duan et al. (2020) Tony Duan, Avati Anand, Daisy Yi Ding, Khanh K Thai, Sanjay Basu, Andrew Ng, and Alejandro Schuler. 2020.  Ngboost: Natural gradient boosting for probabilistic prediction. In _International conference on machine learning_. PMLR, 2690–2700.
  * Foong et al. (2020) Andrew Foong, David Burt, Yingzhen Li, and Richard Turner. 2020\.  On the expressiveness of approximate inference in bayesian neural networks.  _Advances in Neural Information Processing Systems_ 33 (2020), 15897–15908.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In _international conference on machine learning_. PMLR, 1050–1059.
  * Gorishniy et al. (2021) Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. 2021.  Revisiting deep learning models for tabular data.  _Advances in Neural Information Processing Systems_ 34 (2021), 18932–18943.
  * Grinsztajn et al. (2022) Léo Grinsztajn, Edouard Oyallon, and Gaël Varoquaux. 2022.  Why do tree-based models still outperform deep learning on typical tabular data?. In _Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022_.  http://papers.nips.cc/paper_files/paper/2022/hash/0378c7692da36807bdec87ab043cdadc-Abstract-Datasets_and_Benchmarks.html
  * Guo et al. (2017) Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017.  DeepFM: A Factorization-Machine based Neural Network for CTR Prediction.  _IJCAI_ abs/1703.04247 (2017).  https://api.semanticscholar.org/CorpusID:970388
  * Huang et al. (2020) Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. 2020\.  Tabtransformer: Tabular data modeling using contextual embeddings.  _arXiv preprint arXiv:2012.06678_ (2020).
  * Intelligence (2022) Mordor Intelligence. 2022\.  Used Cars Market in Germany Size and Share Analysis.  https://www.mordorintelligence.com/industry-reports/germany-used-car-market.  Accessed: 2024-05-01.
  * Jawed et al. (2023) Shayan Jawed, Jan Stening, and Lars Schmidt-Thieme. 2023\.  Pricing Used Vehicles at Volkswagen Financial Services AG. In _2023 IEEE International Conference on Big Data (BigData)_. IEEE Computer Society, 1736–1743.
  * Ke et al. (2017) Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017\.  LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In _Neural Information Processing Systems_.  https://api.semanticscholar.org/CorpusID:3815895
  * Lakshminarayanan et al. (2017) Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017.  Simple and scalable predictive uncertainty estimation using deep ensembles.  _Advances in neural information processing systems_ 30 (2017).
  * Malinin et al. (2021) Andrey Malinin, Liudmila Prokhorenkova, and Aleksei Ustimenko. 2021\.  Uncertainty in gradient boosting via ensembles.  _ICLR_ (2021).
  * Nelder and Wedderburn (1972) John Ashworth Nelder and Robert WM Wedderburn. 1972.  Generalized linear models.  _Journal of the Royal Statistical Society Series A: Statistics in Society_ 135, 3 (1972), 370–384.
  * Papamakarios et al. (2021) George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. 2021\.  Normalizing flows for probabilistic modeling and inference.  _The Journal of Machine Learning Research_ 22, 1 (2021), 2617–2680.
  * Pedregosa et al. (2011) Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011\.  Scikit-learn: Machine learning in Python.  _Journal of machine learning research_ 12, Oct (2011), 2825–2830.
  * Prokhorenkova et al. (2018) Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. 2018.  CatBoost: unbiased boosting with categorical features.  _Advances in neural information processing systems_ 31 (2018).
  * Rasul et al. (2020) Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs Bergmann, and Roland Vollgraf. 2020.  Multivariate probabilistic time series forecasting via conditioned normalizing flows.  _arXiv preprint arXiv:2002.06103_ (2020).
  * Samruddhi and Kumar (2020) K Samruddhi and R Ashok Kumar. 2020.  Used car price prediction using k-nearest neighbor based model.  _Int. J. Innov. Res. Appl. Sci. Eng.(IJIRASE)_ 4 (2020), 629–632.
  * Shwartz-Ziv and Armon (2022) Ravid Shwartz-Ziv and Amitai Armon. 2022.  Tabular data: Deep learning is not all you need.  _Information Fusion_ 81 (2022), 84–90.
  * Somepalli et al. (2021) Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. 2021\.  Saint: Improved neural networks for tabular data via row attention and contrastive pre-training.  _arXiv preprint arXiv:2106.01342_ (2021).
  * Sprangers et al. (2021) Olivier Sprangers, Sebastian Schelter, and Maarten de Rijke. 2021.  Probabilistic gradient boosting machines for large-scale probabilistic regression. In _Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining_. 1510–1520.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _J. Mach. Learn. Res._ 15 (2014), 1929–1958.  https://api.semanticscholar.org/CorpusID:6844431
  * Tchuente and Nyawa (2022) Dieudonné Tchuente and Serge Nyawa. 2022.  Real estate price estimation in French cities using geocoding and machine learning.  _Annals of Operations Research_ (2022), 1–38.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Venkatasubbu and Ganesh (2019) Pattabiraman Venkatasubbu and Mukkesh Ganesh. 2019.  Used cars price prediction using supervised learning techniques.  _Int. J. Eng. Adv. Technol.(IJEAT)_ 9, 1S3 (2019).
  * Wang and Wu (2018) Changchun Wang and Hui Wu. 2018.  A new machine learning approach to house price estimation.  _New Trends in Mathematical Sciences_ 6, 4 (2018).


# Full Phase Space Resonant Anomaly Detection

Erik Buhmann  Institute for Experimental Physics, Universität Hamburg, Luruper
Chaussee 149, 22761 Hamburg, Germany  Cedric Ewen<EMAIL_ADDRESS>hamburg.de Institute for Experimental Physics, Universität Hamburg, Luruper
Chaussee 149, 22761 Hamburg, Germany  Gregor Kasieczka  Institute for
Experimental Physics, Universität Hamburg, Luruper Chaussee 149, 22761
Hamburg, Germany  Vinicius Mikuni<EMAIL_ADDRESS>National Energy Research
Scientific Computing Center, Berkeley Lab, Berkeley, CA 94720, USA  Benjamin
Nachman  Physics Division, Lawrence Berkeley National Laboratory, Berkeley, CA
94720, USA  Berkeley Institute for Data Science, University of California,
Berkeley, CA 94720, USA  David Shih  New High Energy Theory Center, Rutgers
University
Piscataway, New Jersey 08854-8019, USA

###### Abstract

Physics beyond the Standard Model that is resonant in one or more dimensions
has been a longstanding focus of countless searches at colliders and beyond.
Recently, many new strategies for resonant anomaly detection have been
developed, where sideband information can be used in conjunction with modern
machine learning, in order to generate synthetic datasets representing the
Standard Model background. Until now, this approach was only able to
accommodate a relatively small number of dimensions, limiting the breadth of
the search sensitivity. Using recent innovations in point cloud generative
models, we show that this strategy can also be applied to the full phase
space, using all relevant particles for the anomaly detection. As a proof of
principle, we show that the signal from the R&D dataset from the LHC Olympics
is findable with this method, opening up the door to future studies that
explore the interplay between depth and breadth in the representation of the
data for anomaly detection.

##  I Introduction

In the pursuit of physics beyond the Standard Model (BSM), it is apparent that
a parallel path is required to the traditional signal-specific paradigm. Many
physics models have too many parameters to scan efficiently and it may be that
new phenomena are not described by known physics models. While most BSM
searches posit a specific model or models in the design and execution of an
analysis, anomaly detection is an alternative strategy that tries to be
agnostic to any particular signal model.

Modern machine learning has accelerated the development and deployment of
anomaly detection methods to particle physics [1, 2, 3]. One of the biggest
challenges with any anomaly detection method is the need to balance signal
sensitivity with background specificity, i.e. it is necessary to be able
estimate the background in any signal-sensitive region. This is one reason
that resonant anomaly detection has attracted significant attention [4, 5, 6,
7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]: sideband regions can
naturally be used to estimate backgrounds.

While a number of anomaly detection methods have been applied to resonances,
the most sensitive approaches use weakly-supervised learning techniques that
are tailored for the resonant case [9, 1, 11]. Of these, nearly all use only a
handful of features to avoid sculpting the resonant feature and/or to cope
with previous computational limitations. Feature selection implicitly
introduces a prior over theoretical models, as only certain signals would be
observable with the chosen features. While general symmetry considerations can
provide the basis for a model agnostic feature reduction, the number of
relevant features required to maintain broad sensitivity is still large [22,
21].

We propose a strategy to perform weakly supervised, resonant anomaly detection
using a maximal feature set — all of the relevant particles in an event. Our
approach is based on the Cathode method introduced in [10]: first, conditional
generative models are trained on sideband regions and interpolated into the
signal region in order to create a synthetic sample of events following the
background distribution; second, a classifier is trained to distinguish
between background and data, and the classifier output serves as the anomaly
score. In [10], the method was demonstrated using an ordinary normalizing flow
trained on a small set of high level features (masses and $n$-subjettinesses).
Here, we extend Cathode using recent innovations in point cloud generative
models in order to generate the full set of low-level features – the four
momenta of all of the jet constituents.

We will compare two different state-of-the-art approaches to particle cloud
generation: a score-based diffusion model [23, 24] parametrized by transformer
layers [25] for permutation equivariance [26, 27]; and a continuous
normalizing flow (CNF) [28] trained with flow-matching [29] using EPiC layers
[30] for permutation equivariance.

We will see that both generative models have comparable performance on the
anomaly detection task. They are able to use all of the particles inside jets
to identify the anomaly in the R&D dataset of the LHC Olympics data challenge
[1, 31]. Currently, our approach is limited in the sense that the sensitivity
to the anomaly is only present when the BSM cross section is quite large.
However, we expect this sensitivity to improve with future innovations with
the generative modeling and/or the classifier. We expect that searching over
so many dimensions enhances the breadth of sensitivity at the cost of depth of
sensitivity to particular physics models. Future studies will explore the
details of the breadth/depth tradeoff.

This paper is organized as follows. Section II briefly introduces the dataset.
Our machine learning methods are described in Sec. III. The numerical results
are presented in Sec. IV and the paper ends with conclusions and outlook in
Sec. V.

##  II Dataset

To illustrate our approach, we use the R&D dataset from the LHC Olympics data
challenge [31, 1] which is briefly described in the following. The background
is modeled as inclusive dijets and the signal is resonant boson production
$A\to B(\to qq^{\prime})C(\to qq^{\prime})$ with masses $m_{A},m_{B},m_{C}$ =
3.5, 0.5, 0.1 TeV, respectively. The events are generated with Pythia8 [32]
and Delphes3.4.1 [33, 34, 35] provides a detector simulation. Jets are defined
using the anti-$k_{T}$ jet clustering algorithm [36] as implemented in FastJet
[37] with $R=1$. The focus is on events defined by two high energy jets
capturing the decay products of the $B$ and $C$ particles. The pair of jets
has a large dijet mass (corresponding to $m_{A}$) and non-trivial substructure
(corresponding to the two-prong decay). Consequently, the leading jet is
required to have $p_{T}>1.2$ TeV.

Each event is represented by up to 700 particles with a three-momentum per
particle; after selection of the two most energetic jets, the maximum particle
multiplicity observed is reduced to 279 per jet. For many previous studies,
this high-dimensional information was compressed into a small set of features,
such as the masses of the leading jets and the $n$-subjettiness observables
$\tau_{1}$ and $\tau_{2}$ [38, 39] of the two jets. Instead, we consider all
of the particles within the leading two jets.

##  III Methods

We introduce two approaches for generating the dijet system. Both approaches
are based on the same structure: one model generates overall jet properties
and a second model generates the jet constituents conditioned on the overall
features. For the first model, each jet is represented by its transverse
momentum $\mathrm{p_{T}}$, pseudo-rapidity $\eta$, azimuthal angle $\phi$, jet
mass $m$, and constituent particle multiplicity $N$. Jet constituents
(particles) are generated in the relative set of coordinates defined by
$(\mathrm{p_{T}}_{\text{rel}},\Delta\eta,\Delta\phi)$, where
$\mathrm{p_{T}}_{\text{rel}}=\mathrm{p_{T}}_{\text{part.}}/\mathrm{p_{T}}_{\text{jet}},\Delta\eta=\eta_{\text{part.}}-\eta_{\text{jet}}$
and $\Delta\phi=\phi_{\text{part.}}-\phi_{\text{jet}}$.

Similarly to previous studies, we define the signal region of interest for
events with $3300~{}\text{GeV}<m_{jj}<3700~{}\text{GeV}$. The events used for
the training in the sidebands are restricted between the interval
$2300~{}\text{GeV}<m_{jj}<3300~{}\text{GeV}$ and
$3700~{}\text{GeV}<m_{jj}<5000~{}\text{GeV}$. The interval range for the
sideband region is defined such that enough training data is available for the
generative model while avoiding boundary effects at the extremes of the dijet
mass distribution. Moreover, 60% of the events in the sideband regions are
used for training and testing, while 40% are used for the validation and
further refinement of the generative model. In order to get the $m_{jj}$
values in the SR, we perform a kernel density estimate (KDE) fit to the
training data. The KDE was implemented with a gaussian kernel and a bandwidth
of $0.001$ using the Scikit-learn [40] library.

###  III.1 Diffusion Generative Models

Our first generative approach is based on the diffusion models introduced in
Ref. [26]. The neural network architecture used for both the dijet system
generation and particle generation are based on a combination of DeepSets [41]
with attention layers [42] such that permutation equivariance for the network
outputs is guaranteed both at the jet- and particle-generation level. Both
diffusion models incorporate time information by feeding random Fourier
features [43] through two fully connected layers. This conditional information
is then combined with values of $m_{jj}$ after a log-transformation. In the
case of the particle generation model, the additional jet kinematic
information is passed through a fully connected layer before being
concatenated with the other conditional inputs. A diagram showing the
interplay between the generative models is shown in Fig. 1. Hyperparameter
choices are listed in Tab. 1.

To generate events, we use the trained diffusion models with the DDIM [44]
sampler with 512 time-intervals and additional second-order correction [45] to
improve precision. A simple pre-processing is applied to all inputs to the
generative models to ensure all distributions, for particle and jet
kinematics, are transformed to have zero mean and unit variance. The training
is carried out using the Perlmutter supercomputer interfaced with the Horovod
package [46] for distributed training. 16 NVIDIA A100 GPUs are used
simultaneously during training, while a single GPU is used for evaluation. All
models are trained for up to 300 epochs with a cosine learning rate schedule
[47] with initial learning rate of $1.6\times 10^{-3}$. If the loss function
does not decrease for 30 consecutive epochs, evaluated in a separate testing
set representing 10% to the sample size, the training is stopped. The
implementation of all models is carried out using Keras backend [48] with
TensorFlow [49].

###  III.2 Flow Matching Generative Models

The second approach is based on the optimal transport flow matching training
objective for training continuous normalizing flows (CNFs)[28] introduced in
Ref. [29]. For the jet feature model, the architecture is a fully connected
network. The particle feature model is based on the EPiC-FM model introduced
in Ref. [50] and combines the flow matching objective with fast and linearly
scalable equivariant point cloud (EPiC) layers [30]. Before feeding the point
cloud into the EPiC layers, the data is projected into a higher dimensional
representation. Because EPiC layers require additional global features as
inputs, these are calculated with an average and summation pooling. Both
models incorporate an embedded time vector and interpolation into the signal
region is possible due to the conditioning of the jet feature model on
$m_{jj}$. The conditioning for the particle feature model consists of the
embedded time and the jet features $\mathrm{p_{T}}$, $\eta$, $\phi$, and $m$.
In both models the conditioning is added to all multilayer perceptrons (MLPs).
A visualization of the model architecture can be found in Fig. 2, and for an
overview of all hyperparameter choices we refer to Tab. 1.

In the generation process, the second order midpoint ordinary differential
equation solver with 250 steps (500 function evaluations) from the Torchdyn
[51] library is used. As for the diffusion models, a simple pre-processing
ensures all distributions to have zero mean and unit variance. The models are
implemented with PyTorch [52] using the PyTorch Lightning [53] framework.

###  III.3 Classifier

To obtain the final anomaly score, we employ classifiers trained to
distinguish the generated, synthetic background samples from the data - this
is the weakly-supervised anomaly detection approach proposed in Ref. [10]
following the CWoLa framework of Refs. [54, 4, 5]. In this work, we employ a
transformer-based classifier architecture with nearly the same structure as
the transformer used in the diffusion model. It first takes the dijet system
consisting of the jet kinematic information and uses DeepSets combined with
attention layers to map the jets to a common latent embedding where the
clustered particles are also represented after applying a similar set of
DeepSets layers with added attention. The attention layers used to process the
particle information consider in the attention matrix only particles belonging
to the same jet, with same set of functions reused between the two jets to
ensure the entire classifier is permutation invariant with respect to the
order of the input jets and the order of the input particles belonging to each
jet. In the same latent embedding space, the dijet system and particles are
then concatenated in the particle dimension, resulting in a hybrid graph with
$N_{1}+N_{2}+2$ “particles”, represented by the two jets containing $N_{1}$
and $N_{2}$ particles, respectively. A fully connected layer is then added
after the concatenation, operating only at the feature level, followed by a
dropout operation [55] and LeakyRelu activation function. An average pooling
operation is then used in the particle dimension to combine the information of
all the objects. A second fully connected layer with dropout and LeakyRelu
activation are used before the output layer with a fully connected layer of
single output and sigmoid activation function. The classifier shares the same
learning rate schedule and early stopping criteria as the diffusion models. A
visual representation of the classifier architecture is shown in Fig. 3.

Figure 1: Network architecture of the diffusion generative model. Dijet
kinematic information is first generated with a dedicated diffusion model. For
each jet created, particles are generated through a second diffusion process
conditioned on the jet kinematic information. A conditional embedding is used
to process the conditional inputs used for both generative models consisting
of the invariant mass of the dijet system ($m_{jj}$), the time information,
and the jet kinematic information in the case of the particle generation
model. Figure 2: Network architecture of the flow matching generative models.
The upper architecture first generates the kinematic dijet information, the
lower architecture generates the particles of each jet based on the jet
kinematic information. The jet feature model is conditioned on time $t$ and
dijet mass $m_{jj}$, while the particle feature model is explicitly
conditioned on the jet features $\mathrm{p_{T}}$, $\eta$, $\phi$, and $m$. $N$
is used to determine the number of particles per jet. Figure 3: Network
architecture of the classifier model. Dijet kinematic information is first
processed with a transformer model. For each jet, particles are processed
through a second transformer model shared between the two jets. The outputs of
the transformer model are then combined through a concatenation operation.

##  IV Results

We evaluate the performance of the generative model first by comparing
distributions of generated quantities with the ones present in the simulation
of sideband events. Histograms of the jet and particle kinematic information
are shown in Fig. 4. We observe a good agreement between generated and
simulated quantities, often to better than 10% accuracy.

Figure 4: Comparison between jet (first five histograms) and particle (last
three histograms) kinematic distributions for samples in the dijet mass
sideband. Distributions generated by the diffusion model are shown in blue
while expected distributions from the pseudo-data are shown as filled
histograms.

We further evaluate the quality of the generative models using a binary
classifier trained to distinguish between generated and reference samples, as
proposed in [56]. Using the same classifier architecture described in Sec.
III, trained on generated vs. reference events in the sideband region, we
obtain an area under the curve (AUC) metric of $0.54$ and $0.53$ for the
diffusion and flow matching models respectively, values very close to the best
attainable result of $0.50$ representing the scenario where the classifier is
unable to tell the difference between the samples.

The next step is to generate background observations in the signal region.
Values of $m_{jj}$ are first generated by sampling values from a kernel
density estimator trained to learn the probability density of the $m_{jj}$
spectra based on the values of the observations coming from the sideband
regions. The values sampled in the signal region are then used as conditional
inputs to the generative models. A total of 200k background events are
generated in the signal region, roughly a factor of two more than the amount
of true background events in the data within the signal region.111In [10] an
even larger amount of oversampling, i.e. a larger number of synthetic
background events, was found to be helpful in improving the performance of the
weakly-supervised classifier. It would be interesting to study that here as
well, but we save this for future work. A comparison of the simulated and
generated background distributions in the signal region is shown in Fig. 5. We
observe a good agreement between simulated and generated background events
with only noticeable differences observed at the extremes of the relative
$\eta$ and $\phi$ distributions. In the absence of signal, the classifier
trained to distinguish between generated vs. reference samples yield an AUC of
$0.48$ and $0.42$ for the diffusion and flow matching models respectively.

Figure 5: Comparison between jet (first five histograms) and particle (last
three histograms) kinematic distributions for samples in the dijet mass signal
region. Distributions generated by the diffusion model are shown in blue while
expected background distributions from the pseudo-data are shown as filled
histograms.

As a last step, we derive anomaly scores by training the classifier in the
signal region to distinguish the neural network-generated background events
from the data observed in the signal region. During the training, 200k
generated samples are trained against the data consisting of 100k background
events and different amount of injected signal. From this dataset, 80% is used
for training and 20% for validation. The training is stopped when the loss in
the validation set does not decrease for 20 consecutive epochs. For each
different signal injection value we train ten classifiers with different
random initialization.

The evaluation of the classifier is performed using an independent dataset
[31, 57] consisting of 100k signal and background events, used to study
different performance metrics. From the classifier outputs, we calculate the
significance improvement characteristic curve (SIC) defined as the signal
efficiency divided by the square root of the background efficiency versus the
signal efficiency. The SIC represents a multiplicative scaling by which the
significance of a signal would increase corresponding to a threshold cut given
by a particular signal efficiency.

In Fig. 6, we show an example of the median of the receiver operating
characteristic (ROC) and SIC curves, obtained for a signal region containing
2000 signal and 100k background events. The uncertainty band is derived from
the 68% confidence bands of 10 independent classifier trainings. The values
are compared with an idealized anomaly detector [10] created by training the
classifier with “generated” background events simulated with the same physics
simulator used to generate the background in the signal region. A total of
200k background events are used to train the classifier taken from an
independent background sample [57]. The idealized classifier training follows
the same training strategy discussed before. The evaluation is always
performed on a large set of signal events and independent background sample
[57], to mitigate the statistical fluctuations from the signal injection. Both
SIC and AUC values are always calculated using the signal and background
efficiencies.

Figure 6: Significance improvement characteristic curve (SIC) (left) and
receiver operating characteristic (ROC) (right) obtained by a classifier
trained on a sample containing 2000 signal and 100k background events. SIC and
ROC values are left empty if the number of background events is below 10 after
applying a given threshold with the classifier output to avoid large
statistical fluctuations.

Both generative models show excellent performance at identifying the signal
component, showing a maximum value of the SIC curve near 40, limited by the
amount of background events available in the signal region. We also
investigate the maximum SIC value and AUC obtained by training the classifier
with different amounts of signal injection. The results are shown in Fig. 7,
compared with the idealized anomaly detector results and with a retrained
version of the original Cathode method [10] in Fig. 7.

Figure 7: Median of the Maximum value of the significance improvement
characteristic (SIC) curve (left) and area under the curve of the receiver
operating characteristic (AUC) (right) as a function of the number of injected
signal events. Uncertainties are derived from the 68% confidence bands
calculated over ten independent classifier trainings with random
initialization.

At lower amounts of signal injection, low-dimensional Cathode outperforms even
the idealized anomaly detector due to the high dimensionality of the inputs to
the latter vs. the hand-picked high-level features input to the former. Above
1k signal injected events, the idealized anomaly detector matches the maximum
performance obtained by the low-dimensional Cathode and continues to improve
as the amount of signal increases. Similar conclusions are made for the
diffusion and flow matching generative models, which are able to reach a
maximum SIC value as high as 50 for large amounts of signal injection. At
lower amounts of signal injection, the AUC of the ROC curve obtained by the
models is compatible with 50%, showing that the background prediction in the
signal region is precise.

##  V Conclusions and Outlook

Anomaly detection provides a new set of tools for exploring particle physics
data in search of BSM physics. Machine learning is empowering these approaches
to make the most of our complex datasets. We have explored how state-of-the-
art generative neural networks can be deployed in a resonant anomaly detection
mode to use all of the available information in hadronic final states. On the
LHC Olympics dataset, we find that this method is able to automatically
identify the presence of anomalies, even in the high- and variable-dimensional
feature space so long as there is enough signal. This performance comes at the
tradeoff of specificity. A simpler model trained on a small set of features
known to be useful for the signal performs better at low signal injection and
worse at relatively high signal injection. This crossover point currently
happens around 5-6$\sigma$ in injected signal, but future innovations may be
able to bring this down. It will also be exciting to explore the breadth of
signals that this new approach can find. Quantifying the discovery volume will
be a complex challenge that is already difficult for existing methods and ever
more acute with the full phase space.

## Code Availability

The code used to produce all results presented in this paper are available at
https://github.com/ViniciusMikuni/LHCO_diffusion and https://github.com/uhh-
pd-ml/LHCO_EPiC-FM.

## Acknowledgments

The authors would like to thank Manuel Sommerhalder for providing reference
results for low-dimensional Cathode; Tobias Quadfasel for assistance in
preparing some figures; and Darius Faroughy, Julie Pagès and Manuel Szewc for
early discussions. VM and BN are supported by the U.S. Department of Energy
(DOE), Office of Science under contract DE-AC02-05CH11231. DS is supported by
DOE grant DOE-SC0010008. This research used resources of the National Energy
Research Scientific Computing Center, a DOE Office of Science User Facility
supported by the Office of Science of the U.S. Department of Energy under
Contract No. DE-AC02-05CH11231 using NERSC award HEP-ERCAP0021099. EB is
funded by a scholarship of the Friedrich Naumann Foundation for Freedom and by
the German Federal Ministry of Science and Research (BMBF) via Verbundprojekts
05H2018 - R&D COMPUTING (Pilotmaßnahme ErUM-Data) Innovative Digitale
Technologien für die Erforschung von Universum und Materie. EB, CE, and GK
acknowledge support by the Deutsche Forschungsgemeinschaft under Germany’s
Excellence Strategy – EXC 2121 Quantum Universe – 390833306 and via the KISS
consortium (05D23GU4) funded by the German Federal Ministry of Education and
Research BMBF in the ErUM-Data action plan. This research was supported in
part through the Maxwell computational resources operated at Deutsches
Elektronen-Synchrotron DESY, Hamburg, Germany.

## References

  * Kasieczka  _et al._ [2021a] G. Kasieczka _et al._ , (2021a), arXiv:2101.08320 [hep-ph] .
  * Aarrestad  _et al._ [2021] T. Aarrestad _et al._ , (2021), arXiv:2105.14027 [hep-ph] .
  * Karagiorgi  _et al._ [2021] G. Karagiorgi, G. Kasieczka, S. Kravitz, B. Nachman, and D. Shih, (2021), arXiv:2112.03769 [hep-ph] .
  * Collins  _et al._ [2018] J. H. Collins, K. Howe, and B. Nachman, Phys. Rev. Lett. 121, 241803 (2018), arXiv:1805.02664 [hep-ph] .
  * Collins  _et al._ [2019] J. H. Collins, K. Howe, and B. Nachman, Phys. Rev. D99, 014038 (2019), arXiv:1902.02634 [hep-ph] .
  * Nachman and Shih [2020] B. Nachman and D. Shih, Phys. Rev. D 101, 075042 (2020), arXiv:2001.04990 [hep-ph] .
  * Andreassen  _et al._ [2020] A. Andreassen, B. Nachman, and D. Shih, Phys. Rev. D 101, 095004 (2020), arXiv:2001.05001 [hep-ph] .
  * Stein  _et al._ [2020] G. Stein, U. Seljak, and B. Dai, in  _34th Conference on Neural Information Processing Systems_ (2020) arXiv:2012.11638 [cs.LG] .
  * Amram and Suarez [2020] O. Amram and C. M. Suarez,  (2020), 10.1007/JHEP01(2021)153, arXiv:2002.12376 [hep-ph] .
  * Hallin  _et al._ [2021] A. Hallin, J. Isaacson, G. Kasieczka, C. Krause, B. Nachman, T. Quadfasel, M. Schlaffer, D. Shih, and M. Sommerhalder, (2021), arXiv:2109.00546 [hep-ph] .
  * Collins  _et al._ [2021] J. H. Collins, P. Martín-Ramiro, B. Nachman, and D. Shih, (2021), arXiv:2104.02092 [hep-ph] .
  * Benkendorfer  _et al._ [2020] K. Benkendorfer, L. L. Pottier, and B. Nachman, (2020), arXiv:2009.02205 [hep-ph] .
  * Kasieczka  _et al._ [2021b] G. Kasieczka, B. Nachman, and D. Shih (2021) arXiv:2107.02821 [stat.ML] .
  * Hallin  _et al._ [2023] A. Hallin, G. Kasieczka, T. Quadfasel, D. Shih, and M. Sommerhalder, Phys. Rev. D 107, 114012 (2023), arXiv:2210.14924 [hep-ph] .
  * Chen  _et al._ [2023] M. F. Chen, B. Nachman, and F. Sala, JHEP 07, 188 (2023), arXiv:2212.10579 [hep-ph] .
  * Kamenik and Szewc [2023] J. F. Kamenik and M. Szewc, Phys. Lett. B 840, 137836 (2023), arXiv:2210.02226 [hep-ph] .
  * Sengupta  _et al._ [2023] D. Sengupta, S. Klein, J. A. Raine, and T. Golling, (2023), arXiv:2305.04646 [hep-ph] .
  * Raine  _et al._ [2023] J. A. Raine, S. Klein, D. Sengupta, and T. Golling, Front. Big Data 6, 899345 (2023), arXiv:2203.09470 [hep-ph] .
  * Golling  _et al._ [2023] T. Golling, G. Kasieczka, C. Krause, R. Mastandrea, B. Nachman, J. A. Raine, D. Sengupta, D. Shih, and M. Sommerhalder, (2023), arXiv:2307.11157 [hep-ph] .
  * Bickendorf  _et al._ [2023] G. Bickendorf, M. Drees, G. Kasieczka, C. Krause, and D. Shih, “Combining Resonant and Tail-based Anomaly Detection,” (2023), arXiv:2309.12918 [hep-ph] .
  * Finke  _et al._ [2023] T. Finke, M. Hein, G. Kasieczka, M. Krämer, A. Mück, P. Prangchaikul, T. Quadfasel, D. Shih, and M. Sommerhalder, “Back To The Roots: Tree-Based Algorithms for Weakly Supervised Anomaly Detection,” (2023), arXiv:2309.13111 [hep-ph] .
  * Dillon  _et al._ [2022] B. M. Dillon, R. Mastandrea, and B. Nachman, Phys. Rev. D 106, 056005 (2022), arXiv:2205.10380 [hep-ph] .
  * Vincent [2011] P. Vincent, Neural Computation 23, 1661 (2011), https://direct.mit.edu/neco/article-pdf/23/7/1661/851298/neco_a_00142.pdf .
  * Song  _et al._ [2020] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, arXiv e-prints , arXiv:2011.13456 (2020), arXiv:2011.13456 [cs.LG] .
  * Vaswani  _et al._ [2023] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” (2023), arXiv:1706.03762 [cs.CL] .
  * Mikuni  _et al._ [2023] V. Mikuni, B. Nachman, and M. Pettee, (2023), arXiv:2304.01266 [hep-ph] .
  * Mikuni and Nachman [2023] V. Mikuni and B. Nachman, (2023), arXiv:2306.03933 [hep-ph] .
  * Chen  _et al._ [2018] T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud, CoRR abs/1806.07366 (2018), 1806.07366 .
  * Lipman  _et al._ [2023] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” (2023), arXiv:2210.02747 [cs.LG] .
  * Buhmann  _et al._ [2023a] E. Buhmann, G. Kasieczka, and J. Thaler, (2023a), arXiv:2301.08128 [hep-ph] .
  * Kasieczka  _et al._ [2019] G. Kasieczka, B. Nachman, and D. Shih, “Official Datasets for LHC Olympics 2020 Anomaly Detection Challenge (Version v6) [Data set].” (2019), https://doi.org/10.5281/zenodo.4536624.
  * Sjöstrand  _et al._ [2008] T. Sjöstrand, S. Mrenna, and P. Skands, Computer Physics Communications 178, 852–867 (2008).
  * de Favereau  _et al._ [2014] J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lemaître, A. Mertens, and M. Selvaggi, Journal of High Energy Physics 2014 (2014), 10.1007/jhep02(2014)057.
  * Mertens [2015] A. Mertens, Journal of Physics: Conference Series 608, 012045 (2015).
  * Selvaggi [2014] M. Selvaggi, Journal of Physics: Conference Series 523, 012033 (2014).
  * Cacciari  _et al._ [2008] M. Cacciari, G. P. Salam, and G. Soyez, Journal of High Energy Physics 2008, 063 (2008).
  * Cacciari  _et al._ [2012] M. Cacciari, G. P. Salam, and G. Soyez, The European Physical Journal C 72 (2012), 10.1140/epjc/s10052-012-1896-2.
  * Thaler and Van Tilburg [2012] J. Thaler and K. Van Tilburg, Journal of High Energy Physics 2012 (2012), 10.1007/jhep02(2012)093.
  * Thaler and Van Tilburg [2011] J. Thaler and K. Van Tilburg, Journal of High Energy Physics 2011 (2011), 10.1007/jhep03(2011)015.
  * Pedregosa  _et al._ [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, Journal of Machine Learning Research 12, 2825 (2011).
  * Zaheer  _et al._ [2017] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Póczos, R. Salakhutdinov, and A. J. Smola, CoRR abs/1703.06114 (2017), 1703.06114 .
  * Vaswani  _et al._ [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, CoRR abs/1706.03762 (2017), 1706.03762 .
  * Tancik  _et al._ [2020] M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. Barron, and R. Ng, Advances in Neural Information Processing Systems 33, 7537 (2020).
  * Song  _et al._ [2020] J. Song, C. Meng, and S. Ermon, CoRR abs/2010.02502 (2020), 2010.02502 .
  * Karras  _et al._ [2022] T. Karras, M. Aittala, T. Aila, and S. Laine, Advances in Neural Information Processing Systems 35, 26565 (2022).
  * Sergeev and Balso [2018] A. Sergeev and M. D. Balso, arXiv preprint arXiv:1802.05799 (2018).
  * Loshchilov and Hutter [2016] I. Loshchilov and F. Hutter, CoRR abs/1608.03983 (2016), 1608.03983 .
  * Chollet [2017] F. Chollet, “Keras,” https://github.com/fchollet/keras (2017).
  * Abadi  _et al._ [2016] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, _et al._ , in  _OSDI_ , Vol. 16 (2016) pp. 265–283.
  * Buhmann  _et al._ [2023b] E. Buhmann, C. Ewen, D. A. Faroughy, T. Golling, G. Kasieczka, M. Leigh, G. Quétant, J. A. Raine, D. Sengupta, and D. Shih, (2023b), arXiv:2310.00049 [hep-ph] .
  * [51] M. Poli, S. Massaroli, A. Yamashita, H. Asama, J. Park, and S. Ermon, “TorchDyn: Implicit Models and Neural Numerical Methods in PyTorch,” .
  * Paszke  _et al._ [2019] A. Paszke _et al._ , Advances in Neural Information Processing Systems 32 , 8024 (2019), Advances in Neural Information Processing Systems 32 pp. 8024–8035.
  * Falcon and The PyTorch Lightning team [2019] W. Falcon and The PyTorch Lightning team, “PyTorch Lightning,”  (2019).
  * Metodiev  _et al._ [2017] E. M. Metodiev, B. Nachman, and J. Thaler, JHEP 10, 174 (2017), arXiv:1708.02949 [hep-ph] .
  * Srivastava  _et al._ [2014] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, Journal of Machine Learning Research 15, 1929 (2014).
  * Krause and Shih [2023] C. Krause and D. Shih, Phys. Rev. D 107, 113003 (2023), arXiv:2106.05285 [physics.ins-det] .
  * Shih [2021] D. Shih, “Additional qcd background events for lhco2020 r&d (signal region only),” https://zenodo.org/record/5759086 (2021).
  * Leigh  _et al._ [2023] M. Leigh, D. Sengupta, G. Quétant, J. A. Raine, K. Zoch, and T. Golling, (2023), arXiv:2303.05376 [hep-ph] .
  * Xu  _et al._ [2015] B. Xu, N. Wang, T. Chen, and M. Li, CoRR abs/1505.00853 (2015), 1505.00853 .
  * Clevert  _et al._ [2016] D.-A. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and accurate deep network learning by exponential linear units (elus),” (2016), arXiv:1511.07289 [cs.LG] .
  * Kingma and Ba [2014] D. Kingma and J. Ba, (2014), arXiv:1412.6980 [cs] .
  * Loshchilov and Hutter [2019] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” (2019), arXiv:1711.05101 [cs.LG] .

Table 1: Hyperparameter choices for all models. The conditioning variables
refer to time $t$, jet variables $p_{T}$, $\eta$, $\phi$, $m$, particle
multiplicity $N$, dijet mass $m_{jj}$.

Hyperparameter | Jet-Diffusion | Particle-Diffusion | Jet-FM | Particle-FM | Classifier
---|---|---|---|---|---
Explicit Conditioning |  $t$, $m_{jj}$ |  $t$, $p_{T}$, $\eta$, $\phi$, $m$, $N$, $m_{jj}$ |  $t$, $m_{jj}$ |  $t$, $p_{T}$, $\eta$, $\phi$, $m$, | /
Time Embedding | Fourier [43] | Fourier [43] | Sin/Cos | Cosine [58] | /
Activation function | LeakyReLU($0.01$) [59] | LeakyReLU($0.01$) [59] | ELU [60] | LeakyReLU($0.01$) [59] | LeakyReLU($0.01$) [59]
Batch size | 128 | 128 | 128 | 1024 | 128
Optimizer | Adam [61] | Adam [61] | AdamW [62] | AdamW [62] | Adam [61]
Initial learning rate | $1.6\times 10{-3}$ | $1.6\times 10^{-3}$ | $10^{-3}$ | $10^{-3}$ | $10^{-3}$
Weight decay | / | / | $5\cdot 10^{-5}$ | $5\cdot 10^{-5}$ | /
Learning rate scheduling | Cosine annealing [47] | Cosine annealing [47] | Constant | Cosine with warm-up | /
Warm-up epochs | / | / | / | 500 | /
Training epochs | 300 | 300 | 10000 | 5000 | 300
Early stopping patience | 50 | 50 | 100 | / | 50
Number of GPUs | 16 | 16 | 1 | 1 | 16
Model weights | $\sim 1.3M$ | $\sim 1.4M$ | $\sim 380k$ | $\sim 2M$ | 438k