11institutetext: Institute for Fundamental Science and Department of Physics,
University of Oregon, Eugene, OR 97403, USA22institutetext: Physics Division,
Lawrence Berkeley National Laboratory, Berkeley, CA 94720, USA33institutetext:
Department of Physics, University of California, Berkeley, CA 94720,
USA44institutetext: Berkeley Institute for Data Science, University of
California, Berkeley, CA 94720, USA

# Non-resonant Anomaly Detection
with Background Extrapolation

Kehang Bai, 2,3 Radha Mastandrea 3,4 and Benjamin Nachman<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Complete anomaly detection strategies that are both signal sensitive and
compatible with background estimation have largely focused on resonant
signals. Non-resonant new physics scenarios are relatively under-explored and
may arise from off-shell effects or final states with significant missing
energy. In this paper, we extend a class of weakly supervised anomaly
detection strategies developed for resonant physics to the non-resonant case.
Machine learning models are trained to reweight, generate, or morph the
background, extrapolated from a control region. A classifier is then trained
in a signal region to distinguish the estimated background from the data. The
new methods are demonstrated using a semi-visible jet signature as a benchmark
signal model, and are shown to automatically identify the anomalous events
without specifying the signal ahead of time.

##  1 Introduction

Despite the impressive predictability of the Standard Model (SM), it is a
well-known fact that it does not account for all known phenomena and is
plagued with a number of aesthetic issues. As such, the search for new,
fundamental interactions is a central goal of particle physics across
virtually every experiment. However, given that there is an uncountable number
of new physics models, we simply do not have the bandwidth to test all
possibilities, and it is certain that there are alternative hypotheses that we
have not yet considered.

For this reason, a new paradigm has emerged to complement traditional, model-
specific approaches. The new anomaly detection protocols seek to explore data
with as little bias as possible in order to be broadly sensitive to many
scenarios. Such protocols have been significantly advanced by modern machine
learning Kasieczka:2021xcg ; Aarrestad:2021oeb ; Karagiorgi:2021ngt ;
Feickert:2021ajf  Collins:2018epr ; DAgnolo:2018cun ; Collins:2019jip ;
DAgnolo:2019vbw ; Farina:2018fyg ; Heimel:2018mkt ; Roy:2019jae ;
Cerri:2018anq ; Blance:2019ibf ; Hajer:2018kqm ; DeSimone:2018efk ;
Mullin:2019mmh ; 1809.02977 ; Dillon:2019cqt ; Andreassen:2020nkr ;
Nachman:2020lpy ; Aguilar-Saavedra:2017rzt ; Romao:2019dvs ; Romao:2020ojy ;
knapp2020adversarially ; collaboration2020dijet ; 1797846 ; 1800445 ;
Amram:2020ykb ; Cheng:2020dal ; Khosa:2020qrz ; Thaprasop:2020mzp ;
Alexander:2020mbx ; aguilarsaavedra2020mass ; 1815227 ; pol2020anomaly ;
Mikuni:2020qds ; vanBeekveld:2020txa ; Park:2020pak ; Faroughy:2020gas ;
Stein:2020rou ; Chakravarti:2021svb ; Batson:2021agz ; Blance:2021gcs ;
Bortolato:2021zic ; Collins:2021nxn ; Dillon:2021nxw ; Finke:2021sdf ;
Shih:2021kbt ; Atkinson:2021nlt ; Kahn:2021drv ; Dorigo:2021iyy ;
Caron:2021wmq ; Govorkova:2021hqu ; Kasieczka:2021tew ; Volkovich:2021txe ;
Govorkova:2021utb ; Hallin:2021wme ; Ostdiek:2021bem ; Fraser:2021lxm ;
Jawahar:2021vyu ; Herrero-Garcia:2021goa ; Aguilar-Saavedra:2021utu ;
Tombs:2021wae ; Lester:2021aks ; Mikuni:2021nwn ; Chekanov:2021pus ;
dAgnolo:2021aun ; Canelli:2021aps ; Ngairangbam:2021yma ; Bradshaw:2022qev ;
Aguilar-Saavedra:2022ejy ; Buss:2022lxw ; Alvi:2022fkk ; Dillon:2022tmm ;
Birman:2022xzu ; Raine:2022hht ; Letizia:2022xbe ; Fanelli:2022xwl ;
Finke:2022lsu ; Verheyen:2022tov ; Dillon:2022mkq ; Caron:2022wrw ;
Park:2022zov ; Kamenik:2022qxs ; Hallin:2022eoq ; Kasieczka:2022naq ;
Araz:2022zxk ; Mastandrea:2022vas ; Schuhmacher:2023pro ; Roche:2023int ;
Golling:2023juz ; Sengupta:2023xqy ; Mikuni:2023tok ; Golling:2023yjq ;
Vaslin:2023lig ; ATLAS:2023azi ; Chekanov:2023uot ; CMSECAL:2023fvz ;
Bickendorf:2023nej ; Finke:2023ltw ; Buhmann:2023acn ; Freytsis:2023cjr . One
powerful anomaly detection strategy (“weak supervision”) is to isolate a
region of phase space and compare data to a background-only reference. Signal
model-agnostic evidence for new physics emerges when the data is not
statistically consistent with the reference.

The core task of weakly supervised anomaly detection strategies is to
construct the reference sample of background-only events. Existing strategies
follow one of two approaches: (1) the reference sample is from (background-
only) simulation and (2) the reference sample is learned from sidebands
Kasieczka:2021tew . The advantage of the simulation approach is that it puts
few assumptions on the form of the signal. In particular, this approach can
accommodate signals that are non-resonant, either due to the new physics being
very massive or consisting of non-reconstructable (e.g. invisible) particles.
The challenge with a simulation-based reference is that the simulation
accuracy limits the anomaly detection sensitivity. In contrast, sideband
methods posit that the signal, if it exists, is localized in at least one
known dimension $m$. Focusing on a particular interval in $m$, a signal-
sensitive region of phase space is then constructed using other features $x$,
and regions in $m$ away from the posited resonance are used to estimate the
background $p_{\text{background}}(x|m)$. The benefit of sideband methods is
that the reference is estimated directly from data, while the challenge is
that not all signals are resonant.

We propose an approach to merge the advantages of the simulation-based and
sideband-based cases to form a strategy that learns the reference from data
without requiring a resonance. This method extends and modifies existing
resonance approaches Andreassen:2020nkr ; Nachman:2020lpy ; Hallin:2021wme ;
Golling:2022nkl ; Hallin:2022eoq ; 1815227 ; Sengupta:2023xqy  to allow for
extrapolation in $m$, as opposed to interpolation. Instead of the signal
region being defined by an interval in a one-dimensional $m$, we now have a
multidimensional $m$ for which the signal region is defined by thresholds111A
non-rectangular signal region is also possible, but makes the accounting more
difficult. $m_{0}>c_{0},m_{1}>c_{1},...,m_{n}>c_{n}$ for
$m,c\in\mathbb{R}^{n}$. The conditional reference $p_{\text{background}}(x|m)$
is learned (directly or indirectly) from the complement of the signal region
and then extrapolated to the signal region. If the machine learning models are
smooth and the background does not qualitatively change in the signal region,
there is reason to believe that the extrapolation can be accurate222We leave
additional studies to impose minimalism in the extrapolation to future work
via e.g. monotonicity Kitouni:2021fkh .. A new feature of the non-resonant
case is that we need to additionally estimate $p_{\text{background}}(m)$. This
is also needed in the resonant case, but it is less important (since $p(m)$ is
relatively constant in the signal region) and thus approximate / simplified
methods have been considered so far. We propose to estimate
$p_{\text{background}}(m)$ using a likelihood-ratio method Andreassen:2020nkr
and then combine this estimate with three proposals for estimating
$p_{\text{background}}(m)$ based on reweighting Andreassen:2020nkr ,
generative models Hallin:2021wme , and template morphing Golling:2022nkl .

The growing anomaly detection for fundamental interactions literature includes
a number of proposals that can accommodate non-resonant new physics. In the
weakly supervised case, simulation-based background estimates do not require
resonant new physics DAgnolo:2018cun ; DAgnolo:2019vbw ; dAgnolo:2021aun ;
Letizia:2022xbe . Nearly all data-based weakly supervised methods include non-
resonant $x$ features Andreassen:2020nkr ; Nachman:2020lpy ; Hallin:2021wme ;
Golling:2022nkl ; Hallin:2022eoq ; 1815227 ; Sengupta:2023xqy ;
Bickendorf:2023nej . In special cases, symmetries can be used to define the
reference directly in data without requiring $m$ to be resonant Tombs:2021wae
; Lester:2021aks ; Birman:2022xzu ; Finke:2022lsu . Extrapolation of
generative models has been considered for background estimation, but combined
with classical model-specific search strategies Lin:2019htn . Unsupervised
methods like those based on autoencoders (e.g. Ref. Hajer:2018kqm ;
Farina:2018fyg ; Heimel:2018mkt  and many others) do not require resonances
for achieving signal sensitivity in general. However, except in special cases,
they often lack background estimation strategy Mikuni:2021nwn , or any
guarantees of optimality. Our approach is unique because it combines signal
sensitivity with background estimation333The reference does not need to be
used directly to estimate the background after making a cut on the anomaly
score. One could use any classical non-resonant background estimation strategy
like the ABCD method, possibly also with machine learning Kasieczka:2020pil .
and is asymptotically optimal Nachman:2020lpy  in the limit of large datasets
and effective machine learning.

This paper is organized as follows. In Sec. 2, we introduce the methodology we
follow for our tail-based non-resonant anomaly detection procedure, providing
details on the background extrapolation task. In Sec. 3, we show a realistic
physics example, considering jets emerging from a complex dark sector as our
non-resonant signal. We conclude in Sec. 4.

##  2 Methodology

###  2.1 Locating overdensities

The goal of a weakly-supervised anomaly search is to identify regions of phase
space that are more represented by data (forming overdensities) than they are
by the reference dataset. The search is performed in a signal region (SR) and
the reference is estimated using information from a neighboring region.

In the resonant case, the SR is defined by an interval around a posited new
particle mass $m\in[m_{0}-\delta,m_{0}+\delta]$. The neighboring region
(called sideband or SB) is the region (or a subset of the region) outside of
this interval. If the signal exists and is at mass $m_{0}$, we expect that
most of the signal is in the SR with little contamination in the SB. The
reference background template is estimated using information from the SB.
Since we expect the phase space to be smoothly varying with $m$ and since
$\delta$ is expected to be small relative to regions over which large changes
occur in $p_{\text{background}}(x|m)$, we expect the background to be well-
estimated by interpolating from the SB (see further details in Sec. 2.2).
Given samples from the background dataset444There are variations on this where
the density is used directly without samples Nachman:2020lpy , but the overall
idea is the same., overdensities are identified with the Classification
WithOut LAbels procedure (CWoLa) Metodiev:2017vrx ; Collins:2018epr ;
Collins:2019jip . Events in data are all assigned the label “signal" and all
events from the background template are labeled “background". An optimal
classifier trained to distinguish these datasets will implicitly learn a
function monotonically related to the signal / background likelihood ratio.
Thresholding the classifier will then isolate an anomaly-like region of phase
space.

In the non-resonant case, the setup is slightly different. Instead of having
SB that surround the SR, the neighboring region is only on one side and is now
called a control region or CR. However, we allow for $m$ to be
multidimensional – the more dimensions available to constrain
$p_{\text{background}}(x|m)$, the better accuracy we might expect for
reconstruction in the SR555We validated this with simplified Gaussian
samples.. Resonances are in fact nearly a special case of this setup, since
one could define e.g. $m^{\prime}=1/|m-m_{0}|$. As in the resonant case, we
start by constructing background templates in the CR, and we expect most of
the signal to be in the SR so that the CR can be used to estimate
$p_{\text{background}}$ without bias. A schematic diagram of the non-resonant
setup is presented in Fig. 1. The challenge now is to extrapolate into the SR
instead of interpolating as in the resonant case. Given the background
template, the actual anomaly detection proceeds using the same CWoLa procedure
described above. The main aspect that distinguishes different approaches is
how the background templates are generated.

Figure 1: A schematic of the setup for non-resonant anomaly detection. The
signal region (SR) is defined by a one-sided cut on a context variable $m>c$.
While the figure shows a one-dimensional context variable, in practice the
context can be multidimensional. A number of other features $x$ are used for
the CWoLa anomaly detection classifier. It is also possible to include $m$ in
the classifier.

###  2.2 Constructing background templates

To construct samples of background-only events in the SR, we take the
following steps:

  1. 1.

Estimate $p_{\mathrm{data}}(x|m)$ in the CR. We first learn the distribution
of the chosen features for background-only events, conditioned on the context
variables, in the CR. It is assumed that the conditioning on the context will
allow the learned distribution to be extrapolated into the SR. For the
interpolation case (i.e resonant AD), the analogous first step would be to
learn $p_{\mathrm{data}}(x|m)$ in the SB. Since we assume that the signal is
mostly absent from the CR, $p_{\text{data}}\approx p_{\text{background}}$.

  2. 2.

Estimate $p_{\mathrm{background}}(m)$ in the SR. In order to sample from
$p_{\mathrm{background}}(x|m)$ in the SR, we need to be able to generate SR-
like context $m$. In the interpolation case, this might be done by performing
a parametric fit to a histogram of $m$ in the SB and interpolating to the SR.
This could also work in the extrapolation case, but (1) the shape may be non-
trivial in the tails of $m$ and (2) this approach does not scale well to more
than one dimension.

To circumvent these challenges, we estimate $p_{\mathrm{background}}(m)$ in
the SR by reweighting context in the CR, $p_{\mathrm{simulation}}(m)$, to
match data and then extrapolating this function into the SR. A binary
classifier parameterized in $m$ Cranmer:2015bka ; Baldi:2016fzo  is trained to
distinguish simulated context from data context in the CR. The classifier
output is then interpreted as a likelihood ratio $w(m)$ for estimating the
background Andreassen:2020nkr . As in step (1), we assume that conditioning
the network on the context will enable the learned weights to extrapolate into
the SR.

  3. 3.

Sample $p_{\mathrm{background}}(x|m)$ in the SR. We first estimate
$p_{\mathrm{background}}(x|m)$ from the learned model in the CR, drawing
context from $p_{\mathrm{simulation}}(m)$, in the CR. The resulting events are
then weighted by $w(m)$ to arrive at our estimate of
$p_{\text{background}}(x,m)$ in the SR, following Eq. 1.

$\begin{split}p_{\rm background}(x,m)&=p_{\rm background}(x|m)p_{\rm
background}(m)\\\ &=p_{\rm background}(x|m)p_{\rm simulation}(m)\frac{p_{\rm
background}(m)}{p_{\rm simulation}(m)}\\\ &=p_{\rm background}(x|m)p_{\rm
simulation}(m)w(m)\end{split}$ (1)

We consider three approaches for approximating $p_{\mathrm{data}}(x|m)$ in the
CR, based on three techniques previously developed for the resonance case:
Simulation-Assisted Likelihood-Free Anomaly Detection (SALAD)
Andreassen:2020nkr , Classifying Anomalies through Outer Density Estimation
(CATHODE) Hallin:2021wme , and Flow-Enhanced Transportation for Anomaly
detection (FETA) Golling:2022nkl . Our proposed methods are:

  1. 1.

Reweight (SALAD-inspired). The parameterized, binary classifier used to
approximate $p_{\text{background}}(m)$ is extended to include also $x$ so that
the likelihood ratio in both $x$ and $m$ is estimated at the same time.

  2. 2.

Generate (CATHODE-inspired). A generative neural network (normalizing flow
pmlr-v37-rezende15 ) is trained to model data in the CR, conditioned on the
context $m$. Given values of $m$, one can sample features $x$ from the
normalizing flow.

  3. 3.

Morph (FETA-inspired). A normalizing flow-based model is trained to map
samples in the CR from MC to data, conditioned on the context $m$. Like SALAD,
FETA starts from a simulation, but instead of reweighting, it morphs the
features directly Golling:2023mqx ; Bright-Thonney:2023sqf .

Note that $w(m)$ is estimated with SALAD in all three cases, so our new
approaches are SALAD-{SALAD, CATHODE, FETA} hybrid methods.

We evaluate the performance of the Reweight, Generate, and Morph-constructed
background samples against two benchmarks: (1) a fully supervised classifier
trained to discriminate pure background from pure signal, and (2) an
“idealized" classifier trained to discriminate pure background from true
background $+$ signal. The idealized classifier represents the actual best
performance to which our background generation methods should asymptote.

###  2.3 Network architectures and training specifications

All networks are implemented in PyTorch NEURIPS2019_9015  and optimized with
Adam kingma2017adam . For the Generate and Morph methods, we construct
normalizing flow networks with the nflows package nflows . All models are
trained with a train-validation split of 2/3-1/3 and are evaluated at the
epoch of lowest validation loss. For the Reweighting, Generate, and Morph
methods, architectures and hyperparameters were optimized through manual
tuning such that in the case of zero signal injection, a binary classifier
could not distinguish reconstructed background in the CR from data in the CR.
This was necessary in order to keep the SR blinded.

The Reweighting method is implemented with a binary classifier, consisting of
a dense neural network with 3 layers of 100 nodes each. We use a batch size of
512, a learning rate of $10^{-3}$, and train for up to 50 epochs with an early
stopping patience of 5. The Generate method is implemented with 4 layers of
(Masked Affine Autoregressive Transform block of 128 hidden features, Reverse
Permutation). Training is with a batch size of 216, a learning rate of
$10^{-3}$, and is carried out for up to 20 epochs with an early stopping
patience of 5. For the Morph method, the base density flow uses the same
architecture as the Generate method. The transfer flow is similar, but layers
have only 64 hidden features, the batch size is reduced to 128, and the
learning rate to $10^{-4}$.

For all anomaly detection classifier networks, we use a fully-connected
architecture with 3 layers of 64 nodes each. We use a batch size of 512, a
learning rate of $10^{-3}$, and train for up to 50 epochs with an early
stopping patience of 5. Networks are only trained on the non-context features.
Data is additionally standard scaled to the mean and standard deviation of the
train set before network training.

For the dataset, we consider a two-dimensional context $m$ and a five-
dimensional feature space $x$ (to be explained in detail in Sec. 3.1). The
values of all seven variables are minmax-scaled to the range (-2.5, 2.5) based
on the minimum and maximum of the simulation (pure background) dataset before
any network training is carried out. This was found to give more reliable flow
training. When the number of signal events is small, the exact events that are
injected into the “data" is important. Therefore, we rerun the procedures with
10 random signal injections and report the median and a 68-percentile spread.
For each signal injection, we retrain the models with a different random
initialization 10 times and average over these models. We found (as in
previous studies) that this ensembling helps with sensitivity and stability.

##  3 Application with dark QCD jets

This section illustrates the use of the above background extrapolation method
in the case of finding Beyond the Standard Model (BSM) physics at the LHC. The
physics model we are interested in is jets that contain both stable and
unstable hadrons from a strongly interacting dark sector, or dark QCD. This
type of signal comes from e.g. Hidden Valley models Strassler:2006im ;
Carloni:2010tw ; Carloni:2011kk ; Knapen:2021eip . In particular, we assume
there exist some new massive quarks charged under dark QCD that are promptly
produced and go through QCD-like showering and hadronization to form “dark
hadrons”. It is possible that a fraction of these dark hadrons decay back to
the SM hadrons, and the rest remain invisible. This results in the type of
dark QCD signature called semi-visible jets Cohen_2015 ; Cohen_2017 ;
Beauchesne_2018 ; Bernreuther_2021 ; CMS2022darkQCDsearch ;
atlascollaboration2023search . Semi-visible jets are challenging to
distinguish from SM QCD jets, but the unique signature from certain dark
hadron mass scales and missing energy may be used for identification.

We assume that the dark quarks are produced through a mediator such as a
massive gauge boson $Z^{\prime}$. When the mediator is produced on-shell, we
can reconstruct the invariant mass of the mediator from the semi-visible jets.
However, when the invisible fraction of the jet is large, there will be a
large amount of missing energy, resulting in a low invariant mass resolution
that is no longer resonant. The invisible fraction of dark hadron decays is
defined using the parameter, $r_{\rm inv}$, the ratio of the number of stable
dark hadrons over number of total dark hadrons.

For this study, we choose a 4 TeV $Z^{\prime}$ decay into a pair of dark
quarks. There are 3 flavours of dark quarks, all with the same mass of
$m_{q_{\rm D}}=250$ GeV. There are two types of dark hadrons, the dark pion
and dark rho, whose masses are set to be the same as the confinement scale of
the dark sector, $m_{\pi_{\rm D}}=m_{\rho_{\rm D}}=\Lambda_{\text{D}}=500$
GeV. Note that this choice of $\Lambda_{\text{D}}$ might result in a small
number of dark mesons in the shower, and since the dark mesons are relatively
heavy, the jet substructure comes from the dark meson decays and is less
dependent on the hadronization process in the dark sector. Future work is
needed to study the effects of different mass parameters and hadronization
uncertainties Cohen_2020 ; Cohen_2023 .

###  3.1 Dataset specifications

Signal events are generated using the Hidden Valley module from Pythia 8.310
bierlich2022comprehensive . Background (“data" and “simulation") events are
generated using MadGraph5 aMC@NLO Alwall_2014  and showered using Pythia
8.310. All events use Delphes deFavereau:2013fsa  for detector simulation with
the CMS card, and jets are clustered using the anti-$k_{t}$ Cacciari:2008gp
algorithm with $R=1.0$ using FastJet Cacciari:2011ma . For the background
samples, the data and simulation events differ in the choices of
renormalization and factorization scales (1 vs. 2, respectively) as well as
the tuning (14 vs. 25). A minimum of 200 GeV transverse momentum is required
for the leading jet in the event. In Fig. 2, we show the $m_{\rm jj}$
distribution for a few selections of $r_{\rm inv}$ values. To ensure the non-
resonance of the signal, we set $r_{\rm inv}=1/3$ in this study.

Figure 2: A histogram of $m_{jj}$, corresponding to the reconstructed mass of
the $Z^{\prime}$ particle, for a selection of choices within the Hidden Valley
model with parameter $r_{\rm inv}$. In this study, we set $r_{\rm inv}=1/3$,
at which point the $Z^{\prime}$ signal is no longer clearly resonant.

We use the scalar sum of jet transverse momenta ($H_{\rm T}$) and missing
transverse momentum (MET) as the context variables to define a two-dimensional
SR666These were chosen since they are nearly independent. This is not strictly
necessary for the method to work, but we found that it improves the
extrapolation quality and the same features can be used for a classical matrix
method / ABCD background estimate.. For a five-dimensional feature space, we
use the dijet invariant mass $m_{\rm jj}$, and the two-pronged and three-
pronged N-subjetiness Thaler:2011gf ; Thaler:2010tr  for both the leading and
subleading jets. Distributions of context and feature variables for signal and
background is shown in Fig. 3. The SR is defined by the cuts $H_{\rm T}>600$
GeV, and MET $>75$ GeV, which ensures the signal-over-background ratio is much
lower in the CR than that of the SR777In addition to the selection of
features, the definition of CR and SR will introduce some model dependence. We
expect that these regions will work well for a wide range of physics model
parameters, especially when the $Z^{\prime}$ mass is higher.. We summarize the
number of generated events in Tab. 1.

(a) Two-dimensional context variables

(b) Five-dimensional feature variables

Figure 3:  Histograms of the observables used in the Hidden Valley non-resonant anomaly detection task, for background (“data "and “simulation") and signal events).  Purpose | Dataset | Number CR events | Number SR events
---|---|---|---
Training for generative models | Simulation (bkg.) | 9,983k | 126k
Data (bkg.) | 9,247k | 72k
Data (sig.) | 14k | 50k
Generated samples | Reweight |  |
Generate | N/A | 126k
Morph |  |
Evaluation | Ideal AD samples | N/A | 68k
Fully supervised set | N/A | 13k bkg., 30k sig.
Test set | N/A | 150k bkg., 20k sig.
Table 1: Breakdown of events generated for the Hidden Valley model. We found
that large training sets in the CR were necessary for good performances of the
Generate and Morph (the flow-based) methods, while the Reweight method worked
well even with a smaller set. We did not explore oversampling in the SR, which
may improve performance Hallin:2021wme .

###  3.2 Results

####  3.2.1 Closure tests

As a first test of extrapolation, we check that we are able to generate
realistic background samples in both the CR and SR under zero-signal
injection. In Fig. 4, we show the five feature distributions as constructed by
the Reweight, Generate, and Morph methods in relation to truth, the zero-
signal data. Feature reconstruction is excellent in the CR, and the ratios of
the marginals to the truth are consistent with unity across virtually all of
the domains of the observables. Reconstruction is good overall in the SR,
although the reconstructed distributions (particularly for the Morph method)
deviate visibly.

(a) Control region distributions.

(b) Signal region distributions.

Figure 4:  Feature distributions as constructed by the Reweight, Generate, and
Morph methods in the absence of signal. Errorbars in the ratio plots reflect
Poisson statistics. Note that the values of the context and features have all
been minmax-scaled to the range (-2.5, 2.5).

In Fig. 5(a), we show a more rigorous test by training a binary classifier to
discriminate between the constructed samples and background-only data. We show
the spread of ROC AUCs (receiver operating characteristic area-under-curves)
across $10$ binary classifier runs; for a classifier trained on two identical
datasets, the ROC AUC is expected to be 0.5 for a perfect classifier, in the
limit of infinite statistics and training time. Again, we see that there is
excellent reconstruction in the CR and slightly worse reconstruction in the
SR. In Fig. 5(b), we show the classifier significance improvement
characteristic (SIC) curves for the discrimination task in the SR. The SIC,
defined as the true positive rate / $\sqrt{\rm{{false\ positive\ rate}}}$,
gauges the factor by which a signal’s significance would improve by making a
cut on the classifier score. We expect SIC $\gg 1$ for a well-performing
classifier, while in the zero-signal case, the classifier would perform
poorly. For all of the methods, the significance found in the zero-signal case
is negligible, less than the idealized classifier – the fact that the SIC
exceeds 1 for the idealized classifier indicates that we are operating in the
regime of finite statistics and training time.

(a) ROC AUC spread in the CR and SR. The horizontal line is at 0.5, marking
true random.

(b) Significance improvement characteristic in the SR.

Figure 5:  Results for a classifier tasked with discriminating extrapolated
background samples from true background. The “Ideal" corresponds to an
idealized classifier, which has been trained to discriminate two statistically
identical sets of background, and thus represents a realistic random
classifier. Errorbars and errorbands show a 68-percentile spread across 10
independent classifier runs.

As stated in Sec. 2.3, architectures and hyperparameters for the extrapolation
networks were selected so as to make the spread of ROC AUCs in the CR
consistent with random. There is a (seemingly unavoidable) systematic error
associated with the extrapolation, such that Reweight, Morph, and Generate
methods show a drop in similarity to true background in the SR. We believe
that most of this drop can be attributed to the more general challenge of
extrapolation for neural networks.

####  3.2.2 Signal detection tests

In this section, we carry out an anomaly detection test for the dark QCD
physics model. The test is done by generating a signal-background
discriminator using a weekly-supervised binary classifier following the CWoLa
procedure, as explained in Sec. 2. The classifier is trained to discriminate
the background reference against data that consists of SR background with a
known amount of signal injection. For the simulation, we scan over seven
signal injections of signal ($S$) over background ($B$) of
$S/B\in\\{0,2.5,5,7.5,12,16,20\\}\times 10^{-3}$ (corresponding to
significances $S/\sqrt{B}\in\\{0,0.66,1.34,2.01,3.22,4.29,5.37$},
respectively). In this section, we provide a number of metrics derived from
this binary classifier.

We focus on metrics derived from the classifier SIC curves for the
discrimination task in the SR. In Fig. 6(a), we show the maximum of the SIC
across all signal efficiencies, corresponding to the largest possible
significance improvement we could achieve. In Fig. 6(b), we show the SIC
evaluated at a fixed rejection (given by 1 / false positive rate ) of
$10^{3}$. The results are encouraging: the Reweight method appears to give
comparable performance with an idealized classifier on both metrics and at
almost every signal injection; both methods could potentially “discover" a
signal originally at $\sim 1\sigma$ by elevating it to the $5\sigma$
threshold. The Generate and Morph methods perform less optimally, but there is
good evidence that the Generate method could elevate a signal originally at
$2\sigma$ to the detection threshold, and the Morph method could elevate a
signal at $3\sigma$ to detection. It would be interesting for future studies
to explore how these findings change when considering different signal and
background models.

(a) Maximum of the Significance Improvement Characteristic.

(b) Significance Improvement Characteristic evaluated at a rejection of
$10^{3}$.

Figure 6:  Metrics for an anomaly detection classifier tasked with
discriminating extrapolated background samples from data in the SR. Errorbands
represent the 68-percentile spread over 10 different signal injections, all of
which have been score-averaged over 10 independent classifier runs.

##  4 Conclusions

In this paper, we have extended background interpolation methods to the
extrapolation case to enable weakly supervised, non-resonant anomaly detection
(AD). Our strategy is based on the construction of a realistic set of
background events through extrapolation from a background enriched control
region (CR) into a signal region (SR). We proposed three extrapolation
approaches based on reweighting, generation, and morphing which parallel the
SALAD, CATHODE, and FETA methods, respectively. Studies in the resonant case
have shown complementarity between these methods Golling:2023yjq  which may be
especially important in the extrapolation case.

The proposed methods are tested on a realistic BSM signal consisting of dark
QCD jets. We found that in the zero-signal case, there was good closure
between the reconstructed samples and the background data, showing that our
background templates were well-modeled and unbiased. The agreement between
predicted background and the true background in CR implies our methods are
optimized for interpolation, and deviation in the SR might come from the
intrinsic difficulty of extrapolation. Finally, we have also shown the
performances of all the methods in a realistic AD task, demonstrating their
abilities to elevate non-significant signal injection to the threshold of
possible detection. While we only showed the performance for a single physics
signal model, we stress that the protocol was optimized on only the
background, and the sensitivity should extend to a broad class of new physics
scenarios.

Our study, while encouraging, brings to light some of the many challenges
associated with non-resonant AD. Future work is necessary in order to develop
more robust non-resonant AD – possible avenues may involve using networks or
data preprocessing techniques that are especially well-suited for
extrapolation. It it also worth testing the learning of $w(m)$ function given
multiple different simulation sets that are qualitatively different than the
data, in order to verify the effectiveness of the Reweight method.

With background extrapolation methods, the range of possible signals that
could be detected using an overdensity analysis would be greatly expanded. Not
only are we able to explore a larger parameter phase space of known resonance
signals that was previously not reconstructable, but we are also able to
access off-shell effects from heavy particles. It is likely that no one method
will achieve the best sensitivity to all scenarios so a set of techniques are
required to achieve broad coverage.

## Data and code availability

The physics datasets and parameter cards used for all simulations have been
made available on Zenodo at https://zenodo.org/records/10154213. The analysis
code is available at https://github.com/hep-lbdl/non-resonant-AD-
extrapolation.

## Acknowledgments

We thank Timothy Cohen, Sascha Diefenbacher, Laura Jeanty, Simon Knapen, and
Christiane Scherb for the useful discussions. In particular, we thank
Christiane Scherb for the help on Hidden Valley simulations, and Laura Jeanty
for providing detailed feedback on the manuscript. KB would like to thank
Javier Montejo Berlingen for providing training and inspirations prior to this
project. KB is supported by the U.S. Department of Energy, Office of Science,
Office of High Energy Physics program under Award Number DE-SC0020244. KB’s
visit to LBNL for this collaboration is supported by the US ATLAS Center
program. BN and RM are supported by the U.S. Department of Energy (DOE),
Office of Science under contract DE-AC02-05CH11231 and Grant No. 63038 from
the John Templeton Foundation. RM is additionally supported by Grant No. DGE
2146752 from the National Science Foundation Graduate Research Fellowship
Program. This research used resources of the National Energy Research
Scientific Computing Center, a DOE Office of Science User Facility supported
by the Office of Science of the U.S. Department of Energy under Contract No.
DE-AC02-05CH11231 using NERSC award HEP-ERCAP0021099.

## References

  * (1) G. Kasieczka et al., The LHC Olympics 2020: A Community Challenge for Anomaly Detection in High Energy Physics, arXiv:2101.08320.
  * (2) T. Aarrestad et al., The Dark Machines Anomaly Score Challenge: Benchmark Data and Model Independent Event Classification for the Large Hadron Collider, arXiv:2105.14027.
  * (3) G. Karagiorgi, G. Kasieczka, S. Kravitz, B. Nachman, and D. Shih, Machine Learning in the Search for New Fundamental Physics, arXiv:2112.03769.
  * (4) M. Feickert and B. Nachman, A Living Review of Machine Learning for Particle Physics, arXiv:2102.02770.
  * (5) J. H. Collins, K. Howe, and B. Nachman, Anomaly Detection for Resonant New Physics with Machine Learning, Phys. Rev. Lett. 121 (2018), no. 24 241803, [arXiv:1805.02664].
  * (6) R. T. D’Agnolo and A. Wulzer, Learning New Physics from a Machine, Phys. Rev. D99 (2019), no. 1 015014, [arXiv:1806.02350].
  * (7) J. H. Collins, K. Howe, and B. Nachman, Extending the search for new resonances with machine learning, Phys. Rev. D99 (2019), no. 1 014038, [arXiv:1902.02634].
  * (8) R. T. D’Agnolo, G. Grosso, M. Pierini, A. Wulzer, and M. Zanetti, Learning Multivariate New Physics, arXiv:1912.12155.
  * (9) M. Farina, Y. Nakai, and D. Shih, Searching for New Physics with Deep Autoencoders, arXiv:1808.08992.
  * (10) T. Heimel, G. Kasieczka, T. Plehn, and J. M. Thompson, QCD or What?, SciPost Phys. 6 (2019), no. 3 030, [arXiv:1808.08979].
  * (11) T. S. Roy and A. H. Vijay, A robust anomaly finder based on autoencoder, arXiv:1903.02032.
  * (12) O. Cerri, T. Q. Nguyen, M. Pierini, M. Spiropulu, and J.-R. Vlimant, Variational Autoencoders for New Physics Mining at the Large Hadron Collider, JHEP 05 (2019) 036, [arXiv:1811.10276].
  * (13) A. Blance, M. Spannowsky, and P. Waite, Adversarially-trained autoencoders for robust unsupervised new physics searches, JHEP 10 (2019) 047, [arXiv:1905.10384].
  * (14) J. Hajer, Y.-Y. Li, T. Liu, and H. Wang, Novelty Detection Meets Collider Physics, arXiv:1807.10261.
  * (15) A. De Simone and T. Jacques, Guiding New Physics Searches with Unsupervised Learning, Eur. Phys. J. C79 (2019), no. 4 289, [arXiv:1807.06038].
  * (16) A. Mullin, H. Pacey, M. Parker, M. White, and S. Williams, Does SUSY have friends? A new approach for LHC event analysis, arXiv:1912.10625.
  * (17) G. M. Alessandro Casa, Nonparametric semisupervised classification for signal detection in high energy physics, arXiv:1809.02977.
  * (18) B. M. Dillon, D. A. Faroughy, and J. F. Kamenik, Uncovering latent jet substructure, Phys. Rev. D100 (2019), no. 5 056002, [arXiv:1904.04200].
  * (19) A. Andreassen, B. Nachman, and D. Shih, Simulation Assisted Likelihood-free Anomaly Detection, Phys. Rev. D 101 (2020), no. 9 095004, [arXiv:2001.05001].
  * (20) B. Nachman and D. Shih, Anomaly Detection with Density Estimation, Phys. Rev. D 101 (2020) 075042, [arXiv:2001.04990].
  * (21) J. A. Aguilar-Saavedra, J. H. Collins, and R. K. Mishra, A generic anti-QCD jet tagger, JHEP 11 (2017) 163, [arXiv:1709.01087].
  * (22) M. Romão Crispim, N. Castro, R. Pedro, and T. Vale, Transferability of Deep Learning Models in Searches for New Physics at Colliders, Phys. Rev. D 101 (2020), no. 3 035042, [arXiv:1912.04220].
  * (23) M. C. Romao, N. Castro, J. Milhano, R. Pedro, and T. Vale, Use of a Generalized Energy Mover’s Distance in the Search for Rare Phenomena at Colliders, arXiv:2004.09360.
  * (24) O. Knapp, G. Dissertori, O. Cerri, T. Q. Nguyen, J.-R. Vlimant, and M. Pierini, Adversarially Learned Anomaly Detection on CMS Open Data: re-discovering the top quark, arXiv:2005.01598.
  * (25) ATLAS Collaboration, Dijet resonance search with weak supervision using 13 TeV pp collisions in the ATLAS detector, arXiv:2005.02983.
  * (26) B. M. Dillon, D. A. Faroughy, J. F. Kamenik, and M. Szewc, Learning the latent structure of collider events, arXiv:2005.12319.
  * (27) M. C. Romao, N. Castro, and R. Pedro, Finding New Physics without learning about it: Anomaly Detection as a tool for Searches at Colliders, arXiv:2006.05432.
  * (28) O. Amram and C. M. Suarez, Tag N’ Train: A Technique to Train Improved Classifiers on Unlabeled Data, arXiv:2002.12376.
  * (29) T. Cheng, J.-F. Arguin, J. Leissner-Martin, J. Pilette, and T. Golling, Variational Autoencoders for Anomalous Jet Tagging, arXiv:2007.01850.
  * (30) C. K. Khosa and V. Sanz, Anomaly Awareness, arXiv:2007.14462.
  * (31) P. Thaprasop, K. Zhou, J. Steinheimer, and C. Herold, Unsupervised Outlier Detection in Heavy-Ion Collisions, arXiv:2007.15830.
  * (32) S. Alexander, S. Gleyzer, H. Parul, P. Reddy, M. W. Toomey, E. Usai, and R. Von Klar, Decoding Dark Matter Substructure without Supervision, arXiv:2008.12731.
  * (33) J. A. Aguilar-Saavedra, F. R. Joaquim, and J. F. Seabra, Mass Unspecific Supervised Tagging (MUST) for boosted jets, arXiv:2008.12792.
  * (34) K. Benkendorfer, L. L. Pottier, and B. Nachman, Simulation-Assisted Decorrelation for Resonant Anomaly Detection, arXiv:2009.02205.
  * (35) Adrian Alan Pol and Victor Berger and Gianluca Cerminara and Cecile Germain and Maurizio Pierini, Anomaly Detection With Conditional Variational Autoencoders, arXiv:2010.05531.
  * (36) V. Mikuni and F. Canelli, Unsupervised clustering for collider physics, arXiv:2010.07106.
  * (37) M. van Beekveld, S. Caron, L. Hendriks, P. Jackson, A. Leinweber, S. Otten, R. Patrick, R. Ruiz de Austri, M. Santoni, and M. White, Combining outlier analysis algorithms to identify new physics at the LHC, arXiv:2010.07940.
  * (38) S. E. Park, D. Rankin, S.-M. Udrescu, M. Yunus, and P. Harris, Quasi Anomalous Knowledge: Searching for new physics with embedded knowledge, arXiv:2011.03550.
  * (39) D. A. Faroughy, Uncovering hidden patterns in collider events with Bayesian probabilistic models, arXiv:2012.08579.
  * (40) G. Stein, U. Seljak, and B. Dai, Unsupervised in-distribution anomaly detection of new physics through conditional density estimation, arXiv:2012.11638.
  * (41) P. Chakravarti, M. Kuusela, J. Lei, and L. Wasserman, Model-Independent Detection of New Physics Signals Using Interpretable Semi-Supervised Classifier Tests, arXiv:2102.07679.
  * (42) J. Batson, C. G. Haaf, Y. Kahn, and D. A. Roberts, Topological Obstructions to Autoencoding, arXiv:2102.08380.
  * (43) A. Blance and M. Spannowsky, Unsupervised Event Classification with Graphs on Classical and Photonic Quantum Computers, arXiv:2103.03897.
  * (44) B. Bortolato, B. M. Dillon, J. F. Kamenik, and A. Smolkovič, Bump Hunting in Latent Space, arXiv:2103.06595.
  * (45) J. H. Collins, P. Martín-Ramiro, B. Nachman, and D. Shih, Comparing Weak- and Unsupervised Methods for Resonant Anomaly Detection, arXiv:2104.02092.
  * (46) B. M. Dillon, T. Plehn, C. Sauer, and P. Sorrenson, Better Latent Spaces for Better Autoencoders, arXiv:2104.08291.
  * (47) T. Finke, M. Krämer, A. Morandini, A. Mück, and I. Oleksiyuk, Autoencoders for unsupervised anomaly detection in high energy physics, arXiv:2104.09051.
  * (48) D. Shih, M. R. Buckley, L. Necib, and J. Tamanas, Via Machinae: Searching for Stellar Streams using Unsupervised Machine Learning, arXiv:2104.12789.
  * (49) O. Atkinson, A. Bhardwaj, C. Englert, V. S. Ngairangbam, and M. Spannowsky, Anomaly detection with Convolutional Graph Neural Networks, arXiv:2105.07988.
  * (50) A. Kahn, J. Gonski, I. Ochoa, D. Williams, and G. Brooijmans, Anomalous Jet Identification via Sequence Modeling, arXiv:2105.09274.
  * (51) T. Dorigo, M. Fumanelli, C. Maccani, M. Mojsovska, G. C. Strong, and B. Scarpa, RanBox: Anomaly Detection in the Copula Space, arXiv:2106.05747.
  * (52) S. Caron, L. Hendriks, and R. Verheyen, Rare and Different: Anomaly Scores from a combination of likelihood and out-of-distribution models to detect new physics at the LHC, arXiv:2106.10164.
  * (53) E. Govorkova, E. Puljak, T. Aarrestad, M. Pierini, K. A. Woźniak, and J. Ngadiuba, LHC physics dataset for unsupervised New Physics detection at 40 MHz, arXiv:2107.02157.
  * (54) G. Kasieczka, B. Nachman, and D. Shih, New Methods and Datasets for Group Anomaly Detection From Fundamental Physics, 7, 2021.  arXiv:2107.02821.
  * (55) S. Volkovich, F. D. V. Halevy, and S. Bressler, The Data-Directed Paradigm for BSM searches, arXiv:2107.11573.
  * (56) E. Govorkova et al., Autoencoders on FPGAs for real-time, unsupervised new physics detection at 40 MHz at the Large Hadron Collider, arXiv:2108.03986.
  * (57) A. Hallin, J. Isaacson, G. Kasieczka, C. Krause, B. Nachman, T. Quadfasel, M. Schlaffer, D. Shih, and M. Sommerhalder, Classifying Anomalies THrough Outer Density Estimation (CATHODE), arXiv:2109.00546.
  * (58) B. Ostdiek, Deep Set Auto Encoders for Anomaly Detection in Particle Physics, arXiv:2109.01695.
  * (59) K. Fraser, S. Homiller, R. K. Mishra, B. Ostdiek, and M. D. Schwartz, Challenges for Unsupervised Anomaly Detection in Particle Physics, arXiv:2110.06948.
  * (60) P. Jawahar, T. Aarrestad, M. Pierini, K. A. Wozniak, J. Ngadiuba, J. Duarte, and S. Tsan, Improving Variational Autoencoders for New Physics Detection at the LHC with Normalizing Flows, arXiv:2110.08508.
  * (61) J. Herrero-Garcia, R. Patrick, and A. Scaffidi, Signal-agnostic dark matter searches in direct detection data with machine learning, arXiv:2110.12248.
  * (62) J. A. Aguilar-Saavedra, Anomaly detection from mass unspecific jet tagging, arXiv:2111.02647.
  * (63) R. Tombs and C. G. Lester, A method to challenge symmetries in data with self-supervised learning, arXiv:2111.05442.
  * (64) C. G. Lester and R. Tombs, Stressed GANs snag desserts, a.k.a Spotting Symmetry Violation with Symmetric Functions, arXiv:2111.00616.
  * (65) V. Mikuni, B. Nachman, and D. Shih, Online-compatible Unsupervised Non-resonant Anomaly Detection, arXiv:2111.06417.
  * (66) S. V. Chekanov and W. Hopkins, Event-based anomaly detection for new physics searches at the LHC using machine learning, arXiv:2111.12119.
  * (67) R. T. d’Agnolo, G. Grosso, M. Pierini, A. Wulzer, and M. Zanetti, Learning New Physics from an Imperfect Machine, arXiv:2111.13633.
  * (68) F. Canelli, A. de Cosa, L. L. Pottier, J. Niedziela, K. Pedro, and M. Pierini, Autoencoders for Semivisible Jet Detection, arXiv:2112.02864.
  * (69) V. S. Ngairangbam, M. Spannowsky, and M. Takeuchi, Anomaly detection in high-energy physics using a quantum autoencoder, arXiv:2112.04958.
  * (70) L. Bradshaw, S. Chang, and B. Ostdiek, Creating Simple, Interpretable Anomaly Detectors for New Physics in Jet Substructure, arXiv:2203.01343.
  * (71) J. A. Aguilar-Saavedra, Taming modeling uncertainties with Mass Unspecific Supervised Tagging, arXiv:2201.11143.
  * (72) T. Buss, B. M. Dillon, T. Finke, M. Krämer, A. Morandini, A. Mück, I. Oleksiyuk, and T. Plehn, What’s Anomalous in LHC Jets?, arXiv:2202.00686.
  * (73) S. Alvi, C. Bauer, and B. Nachman, Quantum Anomaly Detection for Collider Physics, arXiv:2206.08391.
  * (74) B. M. Dillon, R. Mastandrea, and B. Nachman, Self-supervised Anomaly Detection for New Physics, arXiv:2205.10380.
  * (75) M. Birman, B. Nachman, R. Sebbah, G. Sela, O. Turetz, and S. Bressler, Data-directed search for new physics based on symmetries of the SM, Eur. Phys. J. C 82 (2022), no. 6 508, [arXiv:2203.07529].
  * (76) J. A. Raine, S. Klein, D. Sengupta, and T. Golling, CURTAINs for your Sliding Window: Constructing Unobserved Regions by Transforming Adjacent Intervals, arXiv:2203.09470.
  * (77) M. Letizia, G. Losapio, M. Rando, G. Grosso, A. Wulzer, M. Pierini, M. Zanetti, and L. Rosasco, Learning new physics efficiently with nonparametric methods, arXiv:2204.02317.
  * (78) C. Fanelli, J. Giroux, and Z. Papandreou, ”Flux+Mutability”: A Conditional Generative Approach to One-Class Classification and Anomaly Detection, arXiv:2204.08609.
  * (79) T. Finke, M. Krämer, M. Lipp, and A. Mück, Boosting mono-jet searches with model-agnostic machine learning, JHEP 08 (2022) 015, [arXiv:2204.11889].
  * (80) R. Verheyen, Event Generation and Density Estimation with Surjective Normalizing Flows, arXiv:2205.01697.
  * (81) B. M. Dillon, L. Favaro, T. Plehn, P. Sorrenson, and M. Krämer, A Normalized Autoencoder for LHC Triggers, arXiv:2206.14225.
  * (82) S. Caron, R. R. de Austri, and Z. Zhang, Mixture-of-theories Training: Can We Find New Physics and Anomalies Better by Mixing Physical Theories?, arXiv:2207.07631.
  * (83) S. E. Park, P. Harris, and B. Ostdiek, Neural Embedding: Learning the Embedding of the Manifold of Physics Data, arXiv:2208.05484.
  * (84) J. F. Kamenik and M. Szewc, Null Hypothesis Test for Anomaly Detection, arXiv:2210.02226.
  * (85) A. Hallin, G. Kasieczka, T. Quadfasel, D. Shih, and M. Sommerhalder, Resonant anomaly detection without background sculpting, arXiv:2210.14924.
  * (86) G. Kasieczka, R. Mastandrea, V. Mikuni, B. Nachman, M. Pettee, and D. Shih, Anomaly Detection under Coordinate Transformations, arXiv:2209.06225.
  * (87) J. Y. Araz and M. Spannowsky, Quantum-probabilistic Hamiltonian learning for generative modelling & anomaly detection, arXiv:2211.03803.
  * (88) R. Mastandrea and B. Nachman, Efficiently Moving Instead of Reweighting Collider Events with Machine Learning, in 36th Conference on Neural Information Processing Systems, 12, 2022.  arXiv:2212.06155.
  * (89) J. Schuhmacher, L. Boggia, V. Belis, E. Puljak, M. Grossi, M. Pierini, S. Vallecorsa, F. Tacchino, P. Barkoutsos, and I. Tavernelli, Unravelling physics beyond the standard model with classical and quantum anomaly detection, arXiv:2301.10787.
  * (90) S. Roche, Q. Bayer, B. Carlson, W. Ouligian, P. Serhiayenka, J. Stelzer, and T. M. Hong, Nanosecond anomaly detection with decision trees for high energy physics and real-time application to exotic Higgs decays, arXiv:2304.03836.
  * (91) T. Golling et al., The Mass-ive Issue: Anomaly Detection in Jet Physics, in 34th Conference on Neural Information Processing Systems, 3, 2023.  arXiv:2303.14134.
  * (92) D. Sengupta, S. Klein, J. A. Raine, and T. Golling, CURTAINs Flows For Flows: Constructing Unobserved Regions with Maximum Likelihood Estimation, arXiv:2305.04646.
  * (93) V. Mikuni and B. Nachman, High-dimensional and Permutation Invariant Anomaly Detection, arXiv:2306.03933.
  * (94) T. Golling, G. Kasieczka, C. Krause, R. Mastandrea, B. Nachman, J. A. Raine, D. Sengupta, D. Shih, and M. Sommerhalder, The Interplay of Machine Learning–based Resonant Anomaly Detection Methods, arXiv:2307.11157.
  * (95) L. Vaslin, V. Barra, and J. Donini, GAN-AE : An anomaly detection algorithm for New Physics search in LHC data, arXiv:2305.15179.
  * (96) ATLAS Collaboration, Anomaly detection search for new resonances decaying into a Higgs boson and a generic new particle $X$ in hadronic final states using $\sqrt{s}=13$ TeV $pp$ collisions with the ATLAS detector, arXiv:2306.03637.
  * (97) S. V. Chekanov and R. Zhang, Boosting sensitivity to new physics with unsupervised anomaly detection in dijet resonance search, arXiv:2308.02671.
  * (98) CMS ECAL Collaboration, D. Abadjiev et al., Autoencoder-based Anomaly Detection System for Online Data Quality Monitoring of the CMS Electromagnetic Calorimeter, arXiv:2309.10157.
  * (99) G. Bickendorf, M. Drees, G. Kasieczka, C. Krause, and D. Shih, Combining Resonant and Tail-based Anomaly Detection, arXiv:2309.12918.
  * (100) T. Finke, M. Hein, G. Kasieczka, M. Krämer, A. Mück, P. Prangchaikul, T. Quadfasel, D. Shih, and M. Sommerhalder, Back To The Roots: Tree-Based Algorithms for Weakly Supervised Anomaly Detection, arXiv:2309.13111.
  * (101) E. Buhmann, C. Ewen, G. Kasieczka, V. Mikuni, B. Nachman, and D. Shih, Full Phase Space Resonant Anomaly Detection, arXiv:2310.06897.
  * (102) M. Freytsis, M. Perelstein, and Y. C. San, Anomaly Detection in Presence of Irrelevant Features, arXiv:2310.13057.
  * (103) T. Golling, S. Klein, R. Mastandrea, and B. Nachman, Flow-enhanced transportation for anomaly detection, Phys. Rev. D 107 (2023), no. 9 096025, [arXiv:2212.11285].
  * (104) O. Kitouni, N. Nolte, and M. Williams, Robust and Provably Monotonic Networks, in 35th Conference on Neural Information Processing Systems, 11, 2021.  arXiv:2112.00038.
  * (105) J. Lin, W. Bhimji, and B. Nachman, Machine Learning Templates for QCD Factorization in the Search for Physics Beyond the Standard Model, JHEP 05 (2019) 181, [arXiv:1903.02556].
  * (106) G. Kasieczka, B. Nachman, M. D. Schwartz, and D. Shih, ABCDisCo: Automating the ABCD Method with Machine Learning, arXiv:2007.14400.
  * (107) E. M. Metodiev, B. Nachman, and J. Thaler, Classification without labels: Learning from mixed samples in high energy physics, JHEP 10 (2017) 174, [arXiv:1708.02949].
  * (108) K. Cranmer, J. Pavez, and G. Louppe, Approximating Likelihood Ratios with Calibrated Discriminative Classifiers, arXiv:1506.02169.
  * (109) P. Baldi, K. Cranmer, T. Faucett, P. Sadowski, and D. Whiteson, Parameterized neural networks for high-energy physics, Eur. Phys. J. C76 (2016), no. 5 235, [arXiv:1601.07913].
  * (110) D. Rezende and S. Mohamed, Variational inference with normalizing flows, Proceedings of the 32nd International Conference on Machine Learning 37 (07–09 Jul, 2015) 1530–1538.
  * (111) T. Golling, S. Klein, R. Mastandrea, B. Nachman, and J. A. Raine, Flows for Flows: Morphing one Dataset into another with Maximum Likelihood Estimation, arXiv:2309.06472.
  * (112) S. Bright-Thonney, P. Harris, P. McCormack, and S. Rothman, Chained Quantile Morphing with Normalizing Flows, arXiv:2309.15912.
  * (113) A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, Pytorch: An imperative style, high-performance deep learning library, in Advances in Neural Information Processing Systems 32, pp. 8024–8035.  Curran Associates, Inc., 2019.
  * (114) D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, 2017\.
  * (115) C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios, nflows: normalizing flows in PyTorch, nov, 2020.
  * (116) M. J. Strassler and K. M. Zurek, Echoes of a hidden valley at hadron colliders, Phys. Lett. B 651 (2007) 374–379, [hep-ph/0604261].
  * (117) L. Carloni and T. Sjostrand, Visible Effects of Invisible Hidden Valley Radiation, JHEP 09 (2010) 105, [arXiv:1006.2911].
  * (118) L. Carloni, J. Rathsman, and T. Sjostrand, Discerning Secluded Sector gauge structures, JHEP 04 (2011) 091, [arXiv:1102.3795].
  * (119) S. Knapen, J. Shelton, and D. Xu, Perturbative benchmark models for a dark shower search program, Phys. Rev. D 103 (2021), no. 11 115013, [arXiv:2103.01238].
  * (120) T. Cohen, M. Lisanti, and H. K. Lou, Semivisible jets: Dark matter undercover at the lhc, Physical Review Letters 115 (Oct., 2015) [arXiv:1503.00009].
  * (121) T. Cohen, M. Lisanti, H. K. Lou, and S. Mishra-Sharma, Lhc searches for dark sector showers, Journal of High Energy Physics 2017 (Nov., 2017) [arXiv:1707.05326].
  * (122) H. Beauchesne, E. Bertuzzo, G. G. di Cortona, and Z. Tabrizi, Collider phenomenology of hidden valley mediators of spin 0 or 1/2 with semivisible jets, Journal of High Energy Physics 2018 (Aug., 2018) [arXiv:1712.07160].
  * (123) E. Bernreuther, T. Finke, F. Kahlhoefer, M. Krämer, and A. Mück, Casting a graph net to catch dark showers, SciPost Physics 10 (Feb., 2021) [arXiv:2006.08639].
  * (124) CMS Collaboration, Search for resonant production of strongly coupled dark matter in proton-proton collisions at 13 tev, Journal of High Energy Physics 2022 (June, 2022) [arXiv:2112.11125].
  * (125) ATLAS Collaboration, Search for non-resonant production of semi-visible jets using run 2 data in atlas, arXiv:2305.18037.
  * (126) T. Cohen, J. Doss, and M. Freytsis, Jet substructure from dark sector showers, Journal of High Energy Physics 2020 (Sept., 2020) [arXiv:2004.00631].
  * (127) T. Cohen, J. Roloff, and C. Scherb, Dark sector showers in the lund jet plane, Physical Review D 108 (Aug., 2023) [arXiv:2301.07732].
  * (128) C. Bierlich, S. Chakraborty, N. Desai, L. Gellersen, I. Helenius, P. Ilten, L. Lönnblad, S. Mrenna, S. Prestel, C. T. Preuss, T. Sjöstrand, P. Skands, M. Utheim, and R. Verheyen, A comprehensive guide to the physics and usage of pythia 8.3, 2022.
  * (129) J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H.-S. Shao, T. Stelzer, P. Torrielli, and M. Zaro, The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations, Journal of High Energy Physics 2014 (jul, 2014).
  * (130) DELPHES 3 Collaboration, J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lemaitre, A. Mertens, and M. Selvaggi, DELPHES 3, A modular framework for fast simulation of a generic collider experiment, JHEP 02 (2014) 057, [arXiv:1307.6346].
  * (131) M. Cacciari, G. P. Salam, and G. Soyez, The anti-$k_{t}$ jet clustering algorithm, JHEP 04 (2008) 063, [arXiv:0802.1189].
  * (132) M. Cacciari, G. P. Salam, and G. Soyez, FastJet User Manual, Eur. Phys. J. C 72 (2012) 1896, [arXiv:1111.6097].
  * (133) J. Thaler and K. Van Tilburg, Maximizing Boosted Top Identification by Minimizing N-subjettiness, JHEP 02 (2012) 093, [arXiv:1108.2701].
  * (134) J. Thaler and K. Van Tilburg, Identifying Boosted Objects with N-subjettiness, JHEP 03 (2011) 015, [arXiv:1011.2268].


# Offensive language detection on Twitter

Nikhil Chilwant
Matriculation No. : 2577689
<EMAIL_ADDRESS>

Syed Taqi Abbas Rizvi
Matriculation No. : 2577651
<EMAIL_ADDRESS>

Hassan Soliman
Matriculation No. : 2576774
<EMAIL_ADDRESS>

###### Abstract

Detection of offensive language in social media is one of the key challenge
for social media . Researchers have proposed many advanced methods to
accomplish this task. In this report, we try to use the learnings from their
approach and incorporate our ideas to improve upon it. We have successfully
achieved accuracy of 74% in classifying offensive tweets. We also list
upcoming challenges in the abusive content detection in the social media
world.

##  1 Introduction

The given problem is important to avoid the misuse of Twitter. We observed the
given data in _train.tsv_ , _dev.tsv_ and _test.tsv_ and identified the
following characteristics of tweets that make this problem challenging:

  1. (a)

The short length of tweets: It is difficult to predict the sentiment, tone and
context of the document with a short length of the document [1].

  2. (b)

Removal of Twitter handles, images and URLs for privacy reasons in the data
limits the scope for identifying the tweet as offensive. For example, a tweet
that shares a link to an offensive article/image may be classified as
offensive by a human.

  3. (c)

Tweets are not present in a thread form and that limits the understanding of
the context. For example, the tweet ’@USER You are only believing her Because
she is a Women.’ is difficult to identify as offensive confidently without
knowing all the tweets in the thread.

  4. (d)

It is challenging to detect a sarcastic offence. Furthermore, it is difficult
to spot whether the sarcasm was meant comically or offensively. We also want
to avoid classifying a funny tweet as an offensive one.

  5. (e)

No grammar policing, as discussed in the assignment.

  6. (f)

To reduce human error in the data labeling step, every training data row is
usually annotated by multiple persons and usually, final decision for each
label is made based on majority vote. In the case of provided _train.tsv_
data, it is not given whether this practice was followed or not. So removal of
human errors, if present, remains to be one of the biggest challenges. The
difficulty is further increased by lack of a standard definition for an
offensive tweet.

##  2 Literature review

We looked at previous work done in this regard and found the following
techniques and models have been used:

  1. (A)

Cleaning techniques:

    1. (a)

Lower casing of words [2].

  2. (B)

Classification features:

    1. (a)

Character n-gram weighted by TF-IDF: Handles the case of spelling mistakes and
spelling variations like ’kill yrslef’. It has been found to be more effective
than token based n-grams [3, 2].

    2. (b)

Frequency of URL mentions and punctuation comment and token lengths,
capitalization, words that cannot be found in English dictionaries, and the
number of non-alpha numeric characters present in tokens [3].

    3. (c)

Word generalization [3]

    4. (d)

Sentiment analysis using PMI based approach [4], word list approach [3].
Further emoticons, punctuations and elongated words (e.g. ’Ohhhh’) can be used
[4].

    5. (e)

Part of Speech (POS) tags [3, 2]

    6. (f)

Offensiveness level score: Compute a score based on co-occurrences of
offensive words in a range of distance [3].

    7. (g)

Knowledge based approach: Use pre-constructed semantically connected network
of words to detect offensive tweet [3].

    8. (h)

Lexical feature: Use a word list to identify hate/offensive words [4].

  3. (C)

Classifiers: The supervised learning approach has been predominately used and
widely suggested for classification. The Support Vector Machine (SVM)
classifier performs better among classifiers 111Recent papers suggest using
Deep Learning approach but we ruled it out as it is out of scope for this
course.[3].

##  3 Our approach

We used the learnings from the literature review and tried to develop an
approach on top of them. First, we finalized features and text pre-processing
technique. Later, we improved the performance by using over-sampling and then
analyzed our results. This section presents all carried out steps in the same
sequence.

###  3.1 Pre-processing

We did not correct or remove any row from the training data because the
language data is subjective, and we did not want to introduce any biases from
our side. First, we removed repetitive and uninformative words as a part of
the pre-processing step. We also ignored the case of words by converting all
characters to lower-case. The tweet obtained at the end of this step is called
as a ’clean tweet’. However, it should be noted that, in _twitterverse_ ,
upper-cased words are called shouting.

Secondly, we used NLTK’s Wordnet Lemmatizer [5] for lemmatizing and Porter
Stemmer for stemming the tweets obtained after the first step.

###  3.2 Data balancing

A preliminary analysis of the training data showed an imbalance in the class
labels. 69.8% of the training data was labeled as non-offensive and remaining
as offensive. Since is important to train the model for both categories of
tweets equally, we need to do data balancing [6]. As the provided data is
small, we applied random over-sampling on the training data to handle the data
imbalance problem. For this, we used _imbalance-learn_ [7] as it provides a
number of re-sampling techniques commonly used in mitigating between-class
imbalance in datasets.

###  3.3 Features Selection

Feature/model/vector/transformer?
The below table lists all the features that we considered as well as the
decision about it.

Feature |  Selected/Rejected |  Reason
---|---|---
TF-IDF for clean tweets with character n-gram |  Selected |  It provided us with a sparse matrix. Also considering the execution was completed within reasonable time, we included the whole feature set.
TF-IDF for lemmatized tweets with character n-grams |  Rejected |  No improvement in performance. We did reach 74% accuracy with TF-IDF for lemmatized tweet when we used word n-gram with range between 1 to 4 but the character n-gram performed gave same or better performance in every case.
TF-IDF for stemmed tweets with character n-grams |  Rejected |  No improvement in the performance was observed after GridSearchCV.
Text statistic like: number of Hashtags, existence of a URL link, length of a tweet, word count of tweet |  Rejected |  No improvement in the performance was observed after GridSearchCV.
NLTK sentimental analysis |  Rejected |  No improvement in the performance. It affected the performance negatively during GridSearchCV.
Profanity Analysis |  Selected |  We used profanity-check-1.0.3 and that uses combined word list of offensive words to calculate offensiveness score for the text.
Word generalisation |  Rejected |  We could not find a word list or collection of word that caters to tweet classification task.
POS tagging using CMU TweetNLP |  Rejected |  No improvement in the performance was observed after running preliminary experiments.
Offensive level score |  Rejected |  No improvement in the performance. This could be because of the short length of tweets and inability to handle spelling variations.
Knowledge based approach |  Rejected |  Out of scope for this project

###  3.4 Classifier

The literature review unanimously agreed upon supervised learning using SVM
classifier as the best approach, and our experiments also confirmed this.
Among the available classifiers in the SciKit, SGD classifier was found to be
the best performing [8]. Its hyperparameters were determined by using
_GridSearchCV_ (CV: Cross-validation) of SciKit. [9]. We selected weight
parameters for the feature union from section 3.3 using the same
GridSearchCV222Click here for the detailed logs.

##  4 Model parameters and result

This section gives quick overview of the results that we have achieved. We
analysed the results in the next section.

During the phase of hyperparameter estimation for classifier using
GraphSearchCV, the accuracy score reached 76.3%333Click here for the detailed
logs. The corresponding model parameters were found as given in table 1,2:

Table 1: Feature union weights. Feature name | Weight
---|---
Profanity | 0.8
TF-IDF on clean tweet with character n-gram | 1.2
Sentiment | 0.0
| TF-IDF for lemmatized tweets with
---
character n-grams
0.0
| TF-IDF for stemmed tweets with
---
character n-grams
0.0
Table 2: SGD classifier parameters Parameter name | Parameter value
---|---
Loss function | modified huber
Penalty (aka regularization term) | l2
Alpha | 0.001
Random state | 69
Maximum iterations | 100
Stopping criterion | none

The final summary of tweet classification is as below:

Table 3: Result produced by ’metrics.classification_report’ | Precision | Recall | F1-score | Support
---|---|---|---|---
NOT | 0.69 | 0.85 | 0.76 | 500
OFF | 0.80 | 0.63 | 0.70 | 500
Accuracy |  |  | 0.74 | 1000
Macro avg | 0.75 | 0.74 | 0.73 | 1000
Weighted avg | 0.75 | 0.74 | 0.73 | 1000

The confusion matrix is as shown in the figure 1:

Figure 1: Confusion matrix

##  5 Observations and analysis

This section tries to explain shortcomings of the models and highlights where
the model has performed good.

###  5.1 Misclassification of Tweets

A tweet can be misclassified in two ways: an offensive tweet is classified as
non-offensive and a non-offensive tweet is classified as offensive. Here are
some examples of misclassified tweets from the development set which are
interesting and important to understand the workings of our model. We used
ELI5’s [10] TextExplainer functionalty for this purpose.

####  5.1.1 Offensive tweets misclassified as non-offensive

  * •

@USER He is! (collision emoji)
@USER that’s unepic

For the first tweet, after pre processing only two elements are left: _’He’_
and a collision emoji. Similarly, for the second tweet, after pre-processing,
we get _’unepic’_. These misclassifications portray the problem of lack of
context. It is difficult for even a human to classify these tweets due to
their short length and lack of context. Therefore, many recent studies on
tweet classification consider tweets in a thread format.

The screenshorts for the ELI5 analysis gives result shown below. Note that the
ELI5 treats our model like black box and then tries to find how it worked
using word n-grams and so, one sees the full tweet here. The important numbers
are ’mean_KL_divergence’ and ’score’. Since the ’mean_KL_divergence’ is close
to zero, it can be inferred as the ELI5 is succeeding to guess the model. The
intensity of the colour shows weights of terms444To avoid divergence from the
main topic, we have limited explanation of all ELI5 terms. One can read
extensive documentation at
https://eli5.readthedocs.io/en/latest/_notebooks/text-explainer.html for
details about how ELI5 performs this task..

  * •

@USER @USER achichincle lamebotas!

Our tfidf vectors are based on character n-grams ranging from 3-grams to
6-grams with word boundary, but the training dataset is almost exclusively in
English and _’achichincle’_ and _’lamebotas’_ are Mexican words. As a result,
this tweet was misclassified.

  * •

@USER @USER LOVE HER!! She is a BADASS!
@USER @USER Fuuckkkk youuuuu

For the first one, it is a subjective choice to label it as offensive. Without
the relevant context, we can not be sure if it is sarcastic or if it was
labeled as such due to the word _’BADASS’_. All other elements of the tweet
maybe aiming to praise the subject of the tweet and with our profanity checker
not tagging _’BADASS’_ as profanity. It was labeled as not offensive. The next
tweet presents the classical problem of various spellings to write a word. The
Wordnet Lemmatizer did not correctly transform _’Fuuckkkk’_ to _’Fuck’_ ,
hence our profanity checker did not mark it as profanity. Here we also see
that our character n-grams also failed to detect the pattern.

####  5.1.2 Non-offensive tweets misclassified as offensive

  * •

@USER She is thick though? Wtf!!
@USER But honestly the cameras suck ass they’re great if you don’t zoom in
@USER @USER hey y’all take this shit to dms aight

These tweets were labeled as offensive by our model, which is mainly down to
the use of the internet slang or curse words like _’Wtf’_ , _’suck’_ , _’ass’_
and _’shit’_ respectively. For example, the first tweet was marked as profane
by the profanity checker with the confidence of 57%. If we check the same
tweet without the use of _’Wtf’_ ; the confidence goes down to 12%. The lack
of context and the small length of the tweets also contribute to such
misclassification.

  * •

@USER It would be so nice if the Trump supporters could tell us why they
believe he is so great without comparing him to every criminal in the history
of American politics.

@USER @USER Our USA WW2 soldiers were all Antifa anti-fascists too. They
killed them.

These tweets have no profane words in the content. However the context of
these tweets is quite a sensitive one according to our training dataset
resulting in many offensive tweets regarding such topics. So tweets using
words such as ’Antifa’, ’Trump’ or ’fascist’, resulting them to be
misclassified as offensive. It should be also noted that with character
n-grams, we gain the advantage that spelling mistakes are smoothed over,
however, it may find difficulty in differentiating between ’fascists’ and
’anti-fascists’. There is also human judgement involved due to the polar
subjects, a different dataset may label them as non-offensive.

####  5.1.3 Outlook

The above examples explained how our model performs the classification task.
Tweet is a very subjective content, with contextual relevance from outside
world (outside Twitter). As a result, it is difficult to classify them. It is
also heavily subject to our dataset as well. For example, the tweets related
to polarizing topics were labeled as offensive in the training dataset. The
topic itself became offensive in some regards according to our model. This can
be mitigated by a larger and diverse dataset. Curse words are used in such
creative ways, which are difficult to detect their offensive or non offensive
intentions. A profanity checker maybe should accommodate for use of curse
words when used without ill intention. There maybe some advanced features that
could be found which makes it easier to predict the offensive nature of tweets
but the one used in our model have been explained the feature selection
section of our reports.

###  5.2 Challenges faced in abusive content detection on social media

This section tries to provide definition of abusive content and the challenges
faced in detecting it.

We have already listed a few important challenges in the introduction. Due to
subjectivity involved, no final consensus about the definition of abusive
content has been reached yet [2]. For the purpose of this report, however, we
define abusive content as ’a content that is aimed at making derogatory,
humiliating or insulting remarks towards any person/ group based on their
demography, gender, race or any other traits that could affect their
psychological health in any way’. Our definition is based on a clinical basis
and hence tries to reduce subjectivity.

Based on the above definition of abusive content, following are the challenges
for detection of abusive content on the social media using computers:

  1. (a)

Multimedia content:
Social media has multimedia content like images, GIF, videos etc. A classifier
based on single media type i.e. text for the task in this assignment is an
effective and easy to implement. However, as we incorporate multimedia
content, we need to build more sophisticated classifiers.

  2. (b)

Multi-lingual communication:
People are finding new means to communicate on the social media (e.g. using
cryptic code language which only people from certain group can understand) and
a lot of moderators are still catching up to this technologically. With
multiple languages spilling into the internet language, new words are being
adopted from various backgrounds at a higher rate than ever before.

  3. (c)

Concern for privacy:
A social media enterprise has to comply to the regulations imposed by the
governments but they cannot afford to risk customer’s privacy for this task.
However, if one has information about the producer of the content, one could
make more informed guess by looking at the past content. It is an interesting
algorithmic challenge to come up with an algorithm that uses the user
information without giving them a feeling that the _’big brother’_ is
watching.

  4. (d)

Context and Debate:
Classifiers also have to understand to about the context of the content. A lot
of abusive or offensive content may be used for educational purpose, to
protest against it or to provide news on it. A system should be able to
differentiate when abuse is being promoted and otherwise.

  5. (e)

Enormity of Task:
With the rise of social media, huge amount of content is produced everyday.
Many social media organizations are trying to develop automatic classifiers
but human judgement is still needed. With rise of social media usage around
the world, the definition of abusive content may have to become more nuanced
than the current meaning or require regional moderation. Additionally, this
rising volume translates to rising computational requirements.

##  6 Advance Analysis

This section lists our new ideas that can help in improve accuracy of the
tweet classification. We have tried some of them and provided the source code
as well.

###  6.1 Word2Vec

We looked at Word2Vec word embedding with extracting meaningful vectors that
helps not only with decreasing the dimensionality of the data, but also has
shown improvement in accuracy and prediction [11].

Word2Vec tries to obtain vectors that best describe the words and extract
certain relations, for example words like "queen" and "king" would have
similar vectors since they appear in the same window or context.

We tried to apply that to our data and test it against the same model
explained previously, but the model performed worse and it resulted an
accuracy of 51%.

This maybe explained by the training dataset of Word2Vec. It is a huge dataset
compiled from different forms of formal literature like novels, newspapers,
books and reports. However, Internet language has a different lexicon and many
words have different connotation in _twitterverse_ than they would in common
literature.

We could not find any resource similar to Word2Vec for internet language. Here
we should also note the rate of change of the common verbiage of the internet
language is unusually erratic and rapid compared to formal languages. This
makes having a resource like Word2Vec a difficult task.

###  6.2 Advanced Profanity Checker

Our model misclassified the tweet ’@USER @USER Fuuckkkk youuuuu’ as our
profanity checker is dictionary based approach. We propose an improvement in
the profanity checker would pre-trained using a character n-gram approach.
This will help us to detect spelling variations in the profane words.

In the internet language, we see ample of curse words for myriads of uses
including as adjective, adverbs or verb. Any content is not marked as
offensive by the use of profanity but by intention, subject or opinion, an
advance profanity checker should consider this also.

###  6.3 Polarizing Topics or Terms

When we observed results obtained after running the classifier on development
data we observed that whenever words like ’Trump’, ’politics’, ’criminal’
come, the tweet is likely to be classified as offensive as the training data
had offensive tweets containing those words. This increases the chance of
misclassifying a non-offensive tweet as offensive.

Topics like politics are controversial in nature and we need to apply stronger
offensive filters for tweets about them by focusing on other terms that can
provide clue for classification. For example, a tweet like ’Trump is good.’
should not be classified by heavily relying on a single word ’Trump’ but by
considering that this words indicates a controversial/multi-opinion topic and
hence other terms like ’good’ should not be neglected.

We propose finding the polarizing topics in the training data by doing topic
extraction and sentiment analysis on the content. And if the training data has
similar amount of tweets for positive and negative sentiment for that class
label, then it can be classified as polarizing subject. Once a tweet is
identified as about polarizing subject, we downsize weights for words like
’Trump’ and reclassify the tweet.

###  6.4 Representation Learning

As explained above, some words may have a heavy connotation due to the
training data. For example, whenever ’fascists’ was used in a tweet, the model
likely predicted it was an offensive tweet. However, if there is a tweet like
’Fascists are BAD’, it is likely not an offensive tweet.

For this we propose, doing topic extraction and sentiment analysis on the
training data and then using representation learning techniques determine the
leaning of the tweet. For example, if a topic has a heavy imbalance of
offensive tweets with positive sentiment, then a tweet with negative
representation for that subject should be classified as non-offensive. For
example, ’war is always a tragedy’ should be classified as a non-offensive
tweet where as ’war is a great thing’ should be classified as offensive.

## References

  * [1] Google Inc.  Natural Language API Basics | Cloud Natural Language API | Google Cloud, 2018.
  * [2] Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber.  Automated hate speech detection and the problem of offensive language.  In Proceedings of the 11th International AAAI Conference on Web and Social Media, ICWSM ’17, pages 512–515, 2017.
  * [3] Anna Schmidt and Michael Wiegand.  A survey on hate speech detection using natural language processing.  In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1–10, 2017.
  * [4] Saif M Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu.  Nrc-canada: Building the state-of-the-art in sentiment analysis of tweets.  arXiv preprint arXiv:1308.6242, 2013.
  * [5] Edward Loper and Steven Bird.  Nltk: The natural language toolkit.  In In Proceedings of the ACL Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics. Philadelphia: Association for Computational Linguistics, 2002.
  * [6] Google Inc.  Imbalanced Data | Data Preparation and Feature Engineering for Machine Learning | Google Developers, 2019.
  * [7] Guillaume Lemaître, Fernando Nogueira, and Christos K. Aridas.  Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning.  Journal of Machine Learning Research, 18(17):1–5, 2017.
  * [8] Working with text data | scikit-learn 0.21.3 documentation 2019.  https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html.  Accessed: 2019-09-11.
  * [9] Tuning the hyper-parameters of an estimator.  https://scikit-learn.org/stable/modules/grid_search.html#tuning-the-hyper-parameters-of-an-estimator.  Accessed: 2019-09-11.
  * [10] ELI5 Github repository.  https://github.com/TeamHG-Memex/eli5, 2019.  Accessed: 2019-09-11.
  * [11] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781, 2013.