# Generalized Reference Kernel
for One-class Classification††thanks: The work of Jenni Raitoharju was funded
by Academy of Finland (project 324475).

Jenni Raitoharju  Programme for Environmental Information
Finnish Environment Institute, Jyväskylä, Finland  Alexandros Iosifidis
DIGIT, Department of Electrical and Computer Engineering
Aarhus University, Aarhus, Denmark

###### Abstract

In this paper, we formulate a new generalized reference kernel hoping to
improve the original base kernel using a set of reference vectors. Depending
on the selected reference vectors, our formulation shows similarities to
approximate kernels, random mappings, and Non-linear Projection Trick.
Focusing on small-scale one-class classification, our analysis and
experimental results show that the new formulation provides approaches to
regularize, adjust the rank, and incorporate additional information into the
kernel itself, leading to improved one-class classification accuracy.

###### Index Terms:

One-class classification, kernel methods, Support Vector Data Description,
One-class Support Vector Machine

##  I Introduction

One-class classification aims at building a model for a class by using data
from this _target class_ only. During inference, the model may also see
_outliers_ not belonging to the target class and it should be able to
recognize that they do not fit to the model. This kind of methods are suitable
for _anomaly detection_ , where only samples of the normal situation are
available, while any kind of anomaly should be detected [pimentel2014review].

Traditional well-known one-class classification techniques include One-class
Support Vector Machine (OC-SVM) [scholkopf1999support] and Support Vector Data
Description (SVDD) [tax2004support], which are commonly applied as non-linear
models exploiting the _kernel trick_. Various extensions of both methods have
been proposed (e.g., [mygdalis2017geocc, tian2018ramp, sohrab2021multimodal]),
and recently also deep neural network-based variants have been proposed
[ruff2018deep, chong2020simple, gautam2021graph]. Some recent works have shown
that also the traditional methods used on top of deep features may be useful.
In [sohrab2020boosting], both SVDD and OC-SVM were used to improve the
classification accuracy of very small classes on top of deep classification.
In [sohn2021learning], OC-SVM applied on top of features obtained via self-
supervised representation learning was shown to yield better results than end-
to-end trained deep learning approaches.

In this work, we focus on small-scale one-class classification tasks that do
not provide enough training data for deep models. Thus, we focus on
traditional one-classification methods and specifically on the kernel used
with the methods. Previous works have attempted to improve the kernels for
one-class classification by applying multikernel approaches that combine
multiple kernels [gautam2019localized, guo2021multi] or have applied
approximate features to allow distributed implementation
[miao2018distributed]. We take a different approach and formulate a new
_generalized reference kernel_ hoping to improve the original base kernel
using a set of reference vectors. Depending on the selected reference vectors,
our formulation shows similarities to approximate kernels, random mappings,
and Non-linear Projection Trick (NPT). It also reveals ways to regularize,
adjust the rank, and incorporate additional information into the kernel
itself.

The rest of the paper is organized as follows: Section II first briefly
introduces the one-class classification methods, SVDD and OC-SVM, used in the
experiments and then reviews works on kernel-based methods, in particular
different approximate kernel methods, random features, and NPT, which all have
similarities to our proposed kernel formulation. Section III introduces the
proposed generalized reference mapping and kernel formulations. Section IV
provides our experimental results and Section V concludes the paper.

##  II Related Work

###  II-A One-class Classification

In this paper, we focus a one-class classification scenario with a training
set $\mathbf{X}=[\mathbf{x}_{1},\dots,\mathbf{x}_{n}]\in\mathbb{R}^{D\times
N}$ and a test set with
$\hat{\mathbf{X}}=[\hat{\mathbf{x}}_{1},\dots,\hat{\mathbf{x}}_{\hat{N}}]\in\mathbb{R}^{D\times{\hat{N}}}$,
where $D$ is the data dimension and $N,{\hat{N}}$ are the number of training
and testing data samples. All the samples in the training set belong to same
class of interest or a _target class_ , and the goal is to use this data to
build a model that can predict whether an unseen test sample belongs to the
target class or is an _outlier_.

SVDD [tax2004support] aims at enclosing the training data inside a minimum
hypersphere by minimizing the following objective function:

$\displaystyle\min\quad$ $\displaystyle
F(R,\mathbf{c})=R^{2}+C\sum_{i=1}^{N}\xi_{i}$ s.t.
$\displaystyle\|\mathbf{x}_{i}-\mathbf{c}\|_{2}^{2}\leq R^{2}+\xi_{i},$
$\displaystyle\xi_{i}\geq 0,\>\>\forall i\in\\{1,\dots,N\\},$ (1)

where $R$ is the radius of the hypersphere and $\mathbf{c}\in\mathbb{R}^{D}$
is the center of the hypersphere. The hyperparameter $C>0$ and the slack
variables, $\xi_{i}$ are used for controlling the trade-off between the volume
of the hypersphere and the amount of target samples allowed outside the
hypersphere. The Lagrangian dual of (II-A) (for derivation, see
[tax2004support]) can be given as

$\displaystyle\max\quad$ $\displaystyle
L=\sum_{i=1}^{N}\alpha_{i}\mathbf{x}_{i}^{\intercal}\mathbf{x}_{i}-\sum_{i}^{N}\sum_{j}^{N}\alpha_{i}\alpha_{j}\mathbf{x}_{i}^{T}\mathbf{x}_{j},$
s.t. $\displaystyle 0\leq\alpha_{i}\leq C.$ (2)

Solving (II-A) gives a value $\alpha_{i}$ for each training sample
$\mathbf{x}_{i}$. These values define which values are inside the hypersphere
($\alpha_{i}=0$), support vectors on the boundary ($0<\alpha_{i}<C$), and
outliers ($\alpha_{i}=C$). This, in turn, allows to solve $R$ and $\mathbf{c}$
and use these to classify unseen samples.

OC-SVM [scholkopf1999support] aims at separating all the training data from
the origin and maximizes the distance $\rho$ from this hyperplane to the
origin:

$\displaystyle\min\quad$
$\displaystyle\frac{1}{2}||\mathbf{w}||^{2}+\frac{1}{\nu
N}\sum_{i=1}^{N}\xi_{i}-\rho$ s.t.
$\displaystyle\mathbf{w}^{T}\mathbf{x}_{i}\geq\xi_{i}-\rho,\>\>$
$\displaystyle\xi_{i}\geq 0,\>\>\forall i\in\\{1,\dots,N\\},$ (3)

where $\mathbf{w}$ is a weight vector, slack variables $\xi_{i}$ allow some
data points to lie within the margin, and hyper-parameter $\nu$ sets an upper
bound on the fraction of training samples allowed within the margin. The dual
of (II-A) can be given as:

$\min
L=\frac{1}{2}\sum_{i=1}^{N}\alpha_{i}\alpha_{i}\mathbf{x}_{i}^{T}\mathbf{x}_{i},\textrm{s.t.
}0\leq\alpha_{i}\leq\frac{1}{\nu N},\sum_{i=1}^{N}\alpha_{i}=1.$ (4)

Now samples on the hyperplane have $0<\alpha_{i}<\frac{1}{\nu N}$ and they can
be used to solve $\rho.$

###  II-B Kernel-based Methods

Non-linear versions of various pattern recognition techniques can be obtained
by using the _kernel trick_. The kernel trick was introduced already in 1960s
for kernel perceptrons [aizerman1964theoretical] and became popular in 1990s
along with Support Vector Machines [cortes1995support], and has been combined
also with state-of-the-art convolutional neural networks [mairal2016end,
chen2020convolutional] with promising results.

The main idea of kernel methods is to non-linearly map the input data to a
feature space employing a _mapping function_
$\phi(\cdot):\mathbf{x}_{i}\in\mathbb{R}^{\mathcal{D}}\rightarrow\phi(\mathbf{x}_{i})\in\mathcal{F}$
and then apply the original linear method in that space. The feature space
$\mathcal{F}$ usually has the properties of Hilbert spaces
[scholkopf1999input, argyriou2009when] and it is often selected to be higher
or even infinite-dimensional space leading to the classes to be more likely
linearly separable. While this makes it infeasible to operate directly on the
mapped samples, many pattern recognition techniques use only inner products of
the input samples, not the samples by themselves (as examples, see SVDD (II-A)
and OC-SVM (4)). Thus, if the inner products are known, the mapped samples
$\phi(\mathbf{x}_{i})$ and even the function $\phi(\cdot)$ are not needed.
This is the basis of the kernel trick: the inner products of the mapped
samples are obtained using a _kernel function_ :

$\kappa(\mathbf{x}_{i},\mathbf{x}_{j})\triangleq\phi(\mathbf{x}_{i})^{T}\phi(\mathbf{x}_{j})$
(5)

without explicitly using the function $\phi(\cdot)$. An example of the kernel
function is the widely used Radial Basis Function (RBF):

$\kappa(\mathbf{x}_{i},\mathbf{x}_{j})=\exp\left(\frac{-\|\mathbf{x}_{i}-\mathbf{x}_{j}\|_{2}^{2}}{2\sigma^{2}}\right),$
(6)

where $\sigma$ is a hyperparameter. The kernel trick has been used to create
many widely-used non-linear variants of originally linear pattern recognition
techniques, such Kernel Principal Component Analysis (KPCA)
[scholkopf1998kpca] from Principal Component Analysis (PCA), Kernel
Discriminant Analysis (KDA) [baudat2000generalized, mika1999fisher] from
Linear Discriminant Analysis (LDA), and non-linear SVDD [tax2004support].

An inherit drawback of kernel methods for large datasets is the fact that they
require the calculation of the so-called _kernel matrix_
$\mathbf{K}_{\mathbf{X}\mathbf{X}}\in\mathbb{R}^{N\times N}$, having elements
$[\mathbf{K}_{\mathbf{X}\mathbf{X}}]_{i,j}=\kappa(\mathbf{x}_{i},\mathbf{x}_{j})$,
and the exploitation of $\mathbf{K}_{\mathbf{X}\mathbf{X}}$ in order to learn
the model parameters. For example, KPCA and KDA solve a (generalized)
eigendecomposition problem defined on $\mathbf{K}_{\mathbf{X}\mathbf{X}}$ and,
thus, their time complexity is a function of $N^{3}$. For large scale
problems, such a time complexity is prohibitive and, thus, approximate kernel
approaches have been proposed to overcome this issue [mahoney2011randomized,
choromanski2022hybrid, gittens2016revisiting]. They aim at constructing a low-
rank approximation of the kernel matrix $\mathbf{K}_{\mathbf{X}\mathbf{X}}$ to
reduce the computational complexity of applying the main algorithm.

Some of well-known techniques to generate approximate kernels include _random
sampling_ methods (e.g., [drineas2005nystrom, drineas2006fastII,
alaoui2015fast, gittens2016revisiting]), _random projection_ methods (e.g.,
[ailon2009fast]), and _random Fourier features_ (e.g., [rahimi2007random,
avron2017random, liao2020random]). Random sampling or _Nyström methods_ sample
a subset of the columns of $\mathbf{K}_{\mathbf{X}\mathbf{X}}$ to construct
the approximate matrix. An important aspect in this process is the
distribution used to sample the columns. While it would be possible to simply
use a uniform distribution (vanilla Nyström), other distributions have been
shown to be more efficient [drineas2005nystrom]. Random projection methods
multiply the original kernel matrix with a random data-independent projection
matrix, thus selecting linear combinations of the columns of
$\mathbf{K}_{\mathbf{X}\mathbf{X}}$. Random Fourier features take a different
approach. Unlike random sampling and random projection methods that try to
approximate the kernel matrix $\mathbf{K}_{\mathbf{X}\mathbf{X}}$, they try to
approximate the kernel function $\kappa(\mathbf{x}_{i},\mathbf{x}_{j})$ using
a data-independent approach. It was demonstrated in [yang2012comparison,
gittens2016revisiting] that the data-dependent Nyström methods can lead to
better kernel approximations than data-independent random projection methods
or random Fourier features.

To give a more mathematical explanation, random sampling methods approximate
the full kernel matrix $\mathbf{K}_{\mathbf{X}\mathbf{X}}$ by randomly
selecting a subset of $n$ samples,
$\hat{\mathbf{X}}=[\hat{\mathbf{x}}_{1},\dots,\hat{\mathbf{x}}_{m}]\in\mathbb{R}^{D\times
n}$, among the original $N$ samples and then constructing the low-rank matrix
as

$\hat{\mathbf{K}}_{\mathbf{X}\mathbf{X}}=\mathbf{K}_{\mathbf{X}\hat{\mathbf{X}}}\mathbf{K}_{\hat{\mathbf{X}}\hat{\mathbf{X}}}^{+}\mathbf{K}_{\hat{\mathbf{X}}\mathbf{X}},$
(7)

where
$\mathbf{K}_{\mathbf{X}\hat{\mathbf{X}}}=[\kappa(\mathbf{x}_{i},\hat{\mathbf{x}}_{j})]_{ij}\in\mathbb{R}^{N\times
n}$,
$\mathbf{K}_{\hat{\mathbf{X}}\hat{\mathbf{X}}}=[\kappa(\hat{\mathbf{x}}_{i},\hat{\mathbf{x}}_{j})]_{ij}$
$\in\mathbb{R}^{n\times n}$,
$\mathbf{K}_{\mathbf{X}\hat{\mathbf{X}}}=[\kappa(\mathbf{x}_{i},\hat{\mathbf{x}}_{j})]_{ij}\in\mathbb{R}^{n\times
N}$, and $(\cdot)^{+}$ denotes a pseudo inverse. Besides random sampling, the
vector in $\hat{\mathbf{X}}$ can be selected, e.g., via clustering the input
vectors and using the cluster centroids [zhang2010clustered].

As derived in [rahimi2007random, avron2017random], random Fourier features are
based on the knowledge that for shift invariate kernel functions, i.e., kernel
functions of form
$\kappa(\mathbf{x}_{i},\mathbf{x}_{j})=\kappa(\mathbf{x}_{i}-\mathbf{x}_{j})$
having $\kappa(0)=1$, there is a probability distribution $p_{\kappa}(\cdot)$
such that

$\kappa(\mathbf{x}_{i},\mathbf{x}_{j})=\int_{\mathbb{R}^{D}}e^{-2\pi
i\boldsymbol{\eta}^{T}(\mathbf{x}_{i}-\mathbf{x}_{j})}p_{\kappa}(\boldsymbol{\eta})d\boldsymbol{\eta},$
(8)

i.e., the inverse Fourier transform of a kernel function $\kappa(\cdot,\cdot)$
is a probability distribution $p_{\kappa}(\cdot)$. Drawing
$\boldsymbol{\eta}_{1},\dots,\boldsymbol{\eta}_{M}$ from $p_{\kappa}(\cdot)$
and defining

$\phi_{\boldsymbol{\eta}}(\mathbf{x}_{i})=\frac{1}{\sqrt{m}}[e^{-2\pi
i\boldsymbol{\eta}_{1}^{T}\mathbf{x}_{i}},\dots,e^{-2\pi
i\boldsymbol{\eta}_{m}^{T}\mathbf{x}_{i}}]^{*},$ (9)

leads to

$\kappa(\mathbf{x}_{i},\mathbf{x}_{j})=\mathbb{E}_{\boldsymbol{\eta}}[\phi_{\boldsymbol{\eta}}(\mathbf{x}_{i})^{T}\phi_{\boldsymbol{\eta}}(\mathbf{x}_{j})].$
(10)

For the RBF kernel in (6), the corresponding probability distribution
$p_{\kappa}(\cdot)$ is $\mathcal{N}(0,\sigma^{2}\mathbf{I})$
[yang2012comparison] and, thus, random Fourier features can be generated by
sampling vectors $\boldsymbol{\eta}_{1},\dots,\boldsymbol{\eta}_{M}$ from this
distribution and computing $\phi_{\boldsymbol{\eta}}(\mathbf{x}_{i})$
according to (9). The mapped representations
$\phi_{\boldsymbol{\eta}}(\mathbf{x}_{i})$ can be then used as new features
for $\mathbf{x}_{i}$, which can be used with the original linear methods, or
they can be used to construct a low-rank approximation (assuming $M<N$), of
the kernel matrix as
$\hat{\mathbf{K}}_{\mathbf{X}\mathbf{X}}=[\phi_{\boldsymbol{\eta}}(\mathbf{x}_{i})^{T}\phi_{\boldsymbol{\eta}}(\mathbf{x}_{j})]_{ij}$.

The optimal low-rank approximations in terms spectral or Frobenius norms can
be obtained using eigenvalue decomposition [li2015large], but it has
prohibitive cubic complexity. Furthermore, it is not clear what is the best
way to measure the optimality of the transformation. Therefore, most of the
research on approximate kernels focuses on the following aspects: 1) How to
evaluate the optimality of the approximation? [yang2012comparison,
liao2020random, gittens2016revisiting, li2019towards], 2) How to construct
more efficiently more optimal approximations? [musco2017recursive,
li2015large, avron2017faster]. Some works focus on analyzing the performance
of the approximate kernels in a specific application [bach2013sharp,
alaoui2015fast], and some works study the rank of the low-rank approximations
[bach2013sharp, wang2018numerical]. Interestingly, all this research assumes
that the original kernel is optimal or contains all the relevant information
that can be extracted. However, this assumption has not been justified.

While the kernel trick can be used in numerous applications, there are also
methods that cannot be presented in terms of inner products only (e.g.,
Ellipsoidal Support Vector Data Description (ESVDD)). In such cases, the
mapping function $\phi(\cdot)$ would be directly needed to first map the input
vectors to the feature space and then apply the method directly on these new
features. However, for many kernel functions the corresponding feature space
can be infinite dimensional and/or the mapping function can be unknown. The
problem can be approached by mapping the inputs $\mathbf{x}_{i}$ to their
_pre-images_ $\boldsymbol{\phi}_{i}$ [kwok2004preimage, mika1998kernel], which
satisfy
$\boldsymbol{\phi}_{i}^{T}\boldsymbol{\phi}_{j}\approx\kappa(\mathbf{x}_{i},\mathbf{x}_{j})$.
Random Fourier features (9) can be directly used are pre-images as
$\boldsymbol{\phi}_{i}=\hat{\phi}(\mathbf{x}_{i})$. When the computational
complexity of obtaining the pre-images is not a concern, NPT [kwak2013NPT]
provides an alternative solution, while its approximate version can be used
for large-scale problems [iosifidis2016NystromNPT].

NPT maps the data to the _effective subspace_ $\mathcal{P}$ of the feature
space $\mathcal{F}$ by as follows: First, the original kernel matrix
$\mathbf{K}_{\mathbf{X}\mathbf{X}}\in\mathbb{R}^{N\times N}$ is computed and
centered (see Appendix (.4)). Then, the eigendecomposition of the centered
kernel matrix $\mathcal{K}$ is computed as

$\mathcal{K}_{\mathbf{X}\mathbf{X}}=\mathbf{U}\boldsymbol{\Lambda}\mathbf{U}^{-1}=\mathbf{U}\boldsymbol{\Lambda}\mathbf{U}^{T}=\mathbf{U}_{r}\boldsymbol{\Lambda}_{r}\mathbf{U}_{r}^{T},$
(11)

where $\mathbf{U}$ contains as its columns the eigenvectors,
$\boldsymbol{\Lambda}$ is a diagonal matrix with the eigenvalues on its
diagonal, the second equality follows from the symmetry, and in the last step
only the non-zero eigenvalues and the corresponding eigenvalues are kept,
i.e., $\mathbf{U}_{r}\in\mathbb{R}^{N\times r}$ and
$\boldsymbol{\Lambda}_{r}\in\mathbb{R}^{r\times r}$, where $r$ is the rank of
$\mathcal{K}_{\mathbf{X}\mathbf{X}}$. Finally, the matrix containing the pre-
images,
$\boldsymbol{\Phi}=[\boldsymbol{\phi}_{1},\dots,\boldsymbol{\phi}_{N}]\in\mathbb{R}^{r\times
N}$, is computed as

$\mathbf{\Phi}=\boldsymbol{\Lambda}_{r}^{\frac{1}{2}}\mathbf{U}_{r}^{T}.$ (12)

In the test phase, the pre-image or a test vector $\hat{\mathbf{x}}_{j}$ is
computed as follows: First the kernel vector
$\mathbf{k}_{\mathbf{X}\hat{\mathbf{x}}_{j}}=[\kappa(\mathbf{x}_{i},\hat{\mathbf{x}}_{j})]_{i}\in\mathbb{R}^{N\times
1}$ is computed and centered (see Appendix (.5)). Finally, the pre-image of
the test vector is computed as

$\hat{\boldsymbol{\phi}}_{j}=\boldsymbol{\Lambda}_{r}^{-\frac{1}{2}}\mathbf{U}_{r}^{T}\mathit{k}_{\mathbf{X}\hat{\mathbf{x}}_{j}},$
(13)

where $\mathit{k}_{\mathbf{X}\hat{\mathbf{x}}_{j}}$ is the centered kernel
vector. As computing the pre-images via NPT requires the eigendecomposition of
$\mathcal{K}_{\mathbf{X}\mathbf{X}}$, it suffers from the inherent problem of
standard kernel methods related to their computational cost. Therefore, it is
most suitable for smaller problem that cannot be solved using the kernel
trick. An approximate version of NPT [iosifidis2016NystromNPT] uses a Nyström
approximation of the full kernel matrix, making the computational cost of the
eigendecomposition of the kernel matrix lower. Since both NPT and its
approximate version are based on the kernel matrix, they still rely on a
similar assumption of the original kernel matrix being the optimal solution.

##  III Generalized Reference Mapping and Kernel

As discussed above, a lot of research from different perspectives has been
done to construct better kernel approximations in large-scale problems. While
it is known that approximate methods can leads to implicit regularization of
the methods for noisy data [mahoney2011implementing], most of the above
discussed research shares the same underlying assumption that the original
kernel function or kernel matrix is optimal and worth approximating as closely
as possible. While some studies focus on kernel function selection
[liu2014research, ong2005hyperkernels, motai2014kernel], and _hyperkernels_ or
_multikernels_ have been proposed to combine different kernel functions to
create better kernels [ong2005hyperkernels, guo2021multi], the optimality of
the kernel functions has received surprisingly little attention in general. In
this paper, we take a different approach from the works discussed in Section
II-B and, instead of attempting to approximate the original kernel, we
formulate a new kernel definition using a set of selected reference vectors.
We focus on small-scale one-class classification problems, where the
computational complexity is not a concern but rather the lack of training
samples. We hope to use our new kernel as an implicit data augmentation for
such problems.

In our kernel definition, we use the following notations: We consider a
training set
$\mathbf{X}=[\mathbf{x}_{1},\dots,\mathbf{x}_{n}]\in\mathbb{R}^{D\times N}$
and a test set
$\hat{\mathbf{X}}=[\hat{\mathbf{x}}_{1},\dots,\hat{\mathbf{x}}_{\hat{N}}]\in\mathbb{R}^{D\times{\hat{N}}}$.
As we focus on one-class classification, the training set is assumed to
contain samples from a single class only. Both training and test sets are
assumed _standardized_ using the training set mean and standard deviation.
Furthermore, our kernel uses a set of reference vectors
$\mathbf{R}=[\mathbf{r}_{1},\dots,\mathbf{r}_{M}]\in\mathbb{R}^{D\times M}$,
where $M$ can be less, equal, or greater than $N$. We will later return the
selection of $M$ and the reference vectors. Our kernel definition is built on
an original kernel to which we refer to as the _base kernel_. We use a tilde
to denote all the related terms (kernel matrix $\tilde{\mathbf{K}}$, kernel
vector $\tilde{\mathbf{k}}$, and kernel function
$\tilde{\kappa}(\cdot,\cdot)$), whereas the terms referring to proposed
reference kernel do not have tildes. Furthermore, we use subscripts with
kernel matrices and vectors to denote the data over which they have been
computed as follows:

$\mathbf{K}_{\mathbf{X}\hat{\mathbf{X}}}=[\kappa(\mathbf{x}_{i},\hat{\mathbf{x}}_{j})]_{ij}\in\mathbb{R}^{N\times{\hat{N}}},$
(14)

where $\mathbf{x}_{i}\in\mathbf{X}$ and
$\hat{\mathbf{x}}_{j}\in\hat{\mathbf{X}}$, and

$\mathbf{k}_{\mathbf{X}\hat{\mathbf{x}}_{j}}=[\kappa(\mathbf{x}_{i},\hat{\mathbf{x}}_{j})]_{i}\in\mathbb{R}^{N\times
1},$ (15)

where $\mathbf{x}_{i}\in\mathbf{X}$.

To formulate a new kernel using a base kernel function
$\tilde{\kappa}(\cdot,\cdot)$ and a set of reference vectors $\mathbf{R}$, we
first compute the reference base kernel matrix
$\tilde{\mathbf{K}}_{\mathbf{R}\mathbf{R}}$ and center it (see Appendix (.4)).
Then, we calculate the eigendecomposition of the centered base kernel matrix
as $\tilde{\mathcal{K}}_{\mathbf{R}\mathbf{R}}$

$\tilde{\mathcal{K}}_{\mathbf{R}\mathbf{R}}=\mathbf{U}\boldsymbol{\Lambda}\mathbf{U}^{-1}=\mathbf{U}\boldsymbol{\Lambda}\mathbf{U}^{T}=\mathbf{U}_{r}\boldsymbol{\Lambda}_{r}\mathbf{U}_{r}^{T},$
(16)

where $\mathbf{U}$ contains as its columns the eigenvectors and
$\boldsymbol{\Lambda}$ is a diagonal matrix with the eigenvalues on its
diagonal. We keep only the $r$ non-zero eigenvalues $\lambda_{t}$ and the
corresponding eigenvectors $\mathbf{u}_{t}$, where $r$ is the rank of
$\tilde{\mathcal{K}}_{\mathbf{R}\mathbf{R}}$.

We now define our generalized reference mapping function $\phi(\cdot)$:

$\displaystyle[\phi(\mathbf{x}_{i})]_{t}$
$\displaystyle=\sum_{m=1}^{M}\frac{1}{\lambda_{t}}u_{tm}\tilde{\kappa}(\mathbf{x}_{i},\mathbf{r}_{m})=\frac{1}{\lambda_{t}}\mathbf{u}_{t}^{T}\tilde{\mathit{k}}_{\mathbf{R}\mathbf{x}_{i}}$
$\displaystyle\rightarrow\phi(\mathbf{x}_{i})$
$\displaystyle=\boldsymbol{\Lambda}_{r}^{-\frac{1}{2}}\mathbf{U}_{r}^{T}\tilde{\mathit{k}}_{\mathbf{R}\mathbf{x}_{i}},$
(17)

where $\phi(\mathbf{x}_{i})\in\mathbb{R}^{r}$ and
$\tilde{\mathit{k}}_{\mathbf{R}\mathbf{x}_{i}}$ is the centered version (using
Appendix (.5)) of the base kernel vector
$\tilde{\mathbf{k}}_{\mathbf{R}\mathbf{x}_{i}}$. Using the defined mapping
function $\phi(\cdot)$, we can either map all the samples into the
corresponding feature space and apply the linear methods on the new features
or compute the kernel matrices using the corresponding kernel function:

$\displaystyle\kappa(\mathbf{x}_{i},\mathbf{x}_{j})$ $\displaystyle=$
$\displaystyle\phi(\mathbf{x}_{i})^{T}\phi(\mathbf{x}_{j})=\tilde{\mathit{k}}_{\mathbf{R}\mathbf{x}_{i}}^{T}\mathbf{U}_{r}\boldsymbol{\Lambda}_{r}^{-1}\mathbf{U}_{r}^{T}\tilde{\mathit{k}}_{\mathbf{R}\mathbf{x}_{i}}$
(18) $\displaystyle=$
$\displaystyle\tilde{\mathit{k}}_{\mathbf{R}\mathbf{x}_{i}}^{T}\tilde{\mathcal{K}}_{\mathbf{R}\mathbf{R}}^{+}\tilde{\mathit{k}}_{\mathbf{R}\mathbf{x}_{i}},$

where
$\tilde{\mathcal{K}}_{\mathbf{R}\mathbf{R}}^{+}=\mathbf{U}\Lambda^{-1}\mathbf{U}^{T}$
is the pseudo inverse of $\tilde{\mathcal{K}}_{\mathbf{R}\mathbf{R}}$. Using
this kernel function, the kernel matrix $\mathbf{K}_{\mathbf{X}\mathbf{X}}$
becomes

$\mathbf{K}_{\mathbf{X}\mathbf{X}}=\phi(\mathbf{X})^{T}\phi(\mathbf{X})=\tilde{\mathcal{K}}_{\mathbf{R}\mathbf{X}}^{T}\tilde{\mathcal{K}}_{\mathbf{R}\mathbf{R}}^{+}\tilde{\mathcal{K}}_{\mathbf{R}\mathbf{X}}=\tilde{\mathcal{K}}_{\mathbf{X}\mathbf{R}}\tilde{\mathcal{K}}_{\mathbf{R}\mathbf{R}}^{+}\tilde{\mathcal{K}}_{\mathbf{R}\mathbf{X}}.$
(19)

Similarly, the test kernel matrix $\mathbf{K}_{\mathbf{X}\hat{\mathbf{X}}}$
becomes

$\mathbf{K}_{\mathbf{X}\hat{\mathbf{X}}}=\tilde{\mathcal{K}}_{\mathbf{X}\mathbf{R}}\tilde{\mathcal{K}}_{\mathbf{R}\mathbf{R}}^{+}\tilde{\mathcal{K}}_{\mathbf{R}\hat{\mathbf{X}}}.$
(20)

As can be seen, the above definitions for reference mapping and reference
kernel function are not following the typical approach, where only the kernel
function is known, while $\phi(\cdot)$ is possibly both undefined and
infinite-dimensional. Instead, we now have an explicit formula for
$\phi(\cdot)\in\mathbb{R}^{r}$, where $r$ is the rank of
$\tilde{\mathcal{K}}_{\mathbf{R}\mathbf{R}}$, and the formula depends on the
selected reference vectors. Here, it should be noted we see the reference
vectors as hyperparameters of the mapping/kernel function. They can be either
data-dependent or data-independent as discussed below, but both cases are
common also for the common kernel function. For example, the value of $\sigma$
in the RBF kernel function is often scaled according to the training data.
Despite the unusual definition, kernel function in (18) fulfils the definition
of a proper kernel function [bishop2006pattern]: A necessary and sufficient
condition for a function $\kappa(\cdot,\cdot)$ to be a valid kernel is that
the kernel matrix $\mathbf{K}_{\mathbf{X}\mathbf{X}}$ is positive
semidefinite. Our kernel matrix is Gram matrix by definition (19) and, thus,
it is always positive semidefinite.

We call our new definitions _generalized reference mapping and kernel_ ,
because different selection of the reference vectors $\mathbf{R}$ produces
kernels, which are equivalent to different approximate solutions discussed in
Section II-B. In particular, we consider the following seven cases:

  1. 1.

$\mathbf{R}=\mathbf{X}\in\mathbb{R}^{D\times N}$: Here, the target class
training samples are used as reference vectors, and the mapping function
$\phi(\cdot)$ in (III) is equivalent to NPT in (12) and (13). The kernel
function $\kappa(\cdot,\cdot)$ in (18) is almost equal to the base kernel
function with some difference coming from the centering operations.

  2. 2.

$\mathbf{R}=\mathbf{N}\in\mathbb{R}^{D\times N}\sim\mathcal{N}(0,1)$: Here,
$N$ random reference vectors are drawn from the normal distribution. As the
training data is standardized and contains samples from the target class only,
the reference vectors come approximately from the training distribution. This
may have a regularizing effect. With this reference vector selection, the
generalized reference mapping has connections with the mappings used by
randomized single hidden-layer feedforward network one-class classifiers,
e.g., [leng2015one, iosifidis2017one], with two differences, i.e., the
centering of the base kernel vector
$\tilde{\mathit{k}}_{\mathbf{R}\mathbf{x}_{i}}$ and the normalization of the
mapping $\phi(\mathbf{x}_{i})$, which follows from the projection and
normalization with the eigenpairs of
$\tilde{\mathcal{K}}_{\mathbf{R}\mathbf{R}}$ in (III).

  3. 3.

$\mathbf{R}=\mathbf{X}^{*}\in\mathbb{R}^{D\times M},\>M<N$: Here, we pick
randomly a subset of the training samples. This option corresponds to the
vanilla Nyström approximate kernel approach (7). Using this reference vector
selection along with the mapping function in (III) can be seen as an
approximate variant of NPT.

  4. 4.

$\mathbf{R}=\mathbf{N}\in\mathbb{R}^{D\times M}\sim\mathcal{N}(0,1),\>M<N$:
Here, $M<N$ random reference vectors drawn from the normal distribution are
used. This approach is similar to the random projection approximate kernel
approaches discussed in Section II-B. Again, using this version with the
mapping function in (III) allows to extend the approximate kernel method to
the NPT setting.

  5. 5.

$\mathbf{R}=[\mathbf{X},\mathbf{X}_{neg}]\in\mathbb{R}^{D\times(N+T)}$: Here,
the reference vectors are the target class training samples augmented with
some of the non-target class training samples. Note that this variant violates
the basic assumption of one-class classification that no negative samples are
available during training. In this paper, this case is included only to see if
such data augmentation (if possible) would help. In future work, this approach
of exploiting negative samples in SVDD can be compared with another
conceptually similar but methodologically different approach, ‘SVDD with
negative examples’, proposed in [tax2004support] or with class-specific
methods (e.g., [iosifidis2016class, raitoharju2019null]).

  6. 6.

$\mathbf{R}=[\mathbf{X},\mathbf{N}]\in\mathbb{R}^{D\times(N+T)}$: Here, the
reference vectors are the target class training vectors augmented by $T$
random vectors drawn from normal distribution. This approach can be seen as an
implicit data augmentation scheme for non-linear one-class classification.

  7. 7.

$\mathbf{R}=\mathbf{N}\in\mathbb{R}^{D\times(N+T)}\sim\mathcal{N}(0,1)$: Here,
we draw $N+T$ random reference vectors from the normal distribution for direct
comparison against the previous two cases.

##  IV Experiments

###  IV-A Experimental Setup

We experimentally compared the seven cases of our proposed generalized
reference approach, both by using the proposed kernel function in (18) to
create the corresponding kernel matrix to be used in the kernel implementation
of the one-class classification techniques and by using the proposed mapping
function in (III) to generate new features to be used in the linear
implementation of the one-class classification techniques. The kernel approach
was compared against the base kernel and the mapping approach was compared
against NPT using the base kernel function. The base kernel function in all
experiments was the RBF kernel (6) and the one-class classification techniques
used in the experiments were SVDD and OC-SVM. The codes for linear SVDD and
both linear and kernel OC-SVM were obtained from LIBSVM
library111https://www.csie.ntu.edu.tw/ cjlin/libsvm/. The code for kernel SVDD
was obtained from dd-tools222https://www.tudelft.nl/ewi/over-de-
faculteit/afdelingen/intelligent-systems/pattern-recognition-
bioinformatics/pattern-recognition-bioinformatics/data-and-software/dd-tools.
Our own codes were implemented on Matlab R2017b and are publicly
available333https://github.com/JenniRaitoharju/GeneralizedReferenceKernel.

We selected for our experiments six different small-scale datasets from UCI
machine learning repository [dua2019uci] shown in Table I. For each class in
these datasets, we created a different one-class classification task by
considering this class as the target class and all the other classes as
outliers. Thus, we had a total of 14 different tasks as shown in Table I. For
each task, we selected randomly 70% of the samples as our training set and the
remaining 30% as the test set. This was repeated five times and the same
splits were used to test the performance of all methods. For each split, the
experiments were also repeated five times. Thus, each reported result is an
average of 25 different runs. For each run, the hyperparameters were selected
using a random 5-fold cross-validation approach within the training set. Thus,
also for non-random kernel/mappings variants, there may be differences between
the repetitions due to the differences in validation splits that may lead to
different hyperparameter selection.

TABLE I: Dataset properties Dataset | $C$ | $N_{tot}$ | $D$ | Task | Target | $N$ | $T$
---|---|---|---|---|---|---|---
Iris | 3 | 150 | 4 | Iris1 | Setosa | 35 | 35
Iris2 | Versicolor | 35 | 35
Iris3 | Virginica | 35 | 35
Seeds | 3 | 210 | 7 | Seed1 | Kama | 49 | 49
Seed2 | Rosa | 49 | 49
Seed3 | Canadian | 49 | 49
Ionosphere | 2 | 351 | 34 | Ion1 | Bad | 88/89 | 88/89
Ion2 | Good | 157/158 | 88/89
Sonar | 2 | 208 | 60 | Son1 | Rock | 68 | 68
Son2 | Mines | 78 | 68
Qualitative | 2 | 250 | 6 | Bank1 | Bankr. | 74/75 | 74/75
bankruptcy | Bank2 | No bankr. | 100/101 | 74/75
Somerville | 2 | 143 | 6 | Happ1 | Happy | 54 | 47
happiness | Happ2 | Unhappy | 47 | 47
$C$ \- number of classes, $N_{tot}$\- total number of samples,
$D$ \- dimensionality, Task - subtask abbreviation, Target - target class
in the subtask, $N$, $T$ \- as defined in Section III

We experimented with the cases 3-4 listed in Section III by setting
$M=\lfloor\frac{N}{2}\rfloor$ and with cases 5-7 by setting
$T=\min(N,N_{neg})$, i.e., in these cases we doubled the number of reference
vectors, if the number of negative samples allowed it. While the number of
negative samples is not relevant for cases 6-7, we used the same $T$ to allow
direct comparison. The number of training samples, $N$, and the value of $T$
used for cases 5-7 are also shown in Table I. When forming $\mathbf{U}_{r}$
and $\boldsymbol{\Lambda}_{r}$ in (16) we consider values $<10^{-6}$ as zeros.
The hyperparameter values selected by cross-validation included $\sigma$ for
RBF kernel, $C$ for SVDD, and $\nu$ for OC-SVM. Hyperparameter $\sigma$ was
set as $\sqrt{sd_{aver}}$, were $d_{aver}$ is the average squared distance
been the training samples and $s$ is selected from the following options:
$[10^{-1},10^{0},10^{1},10^{2},10^{3}]$. Hyperparameters $C$ and $\nu$ are
selected from $[0.1,0.2,0.3,0.4,0.5,0.6]$. As the performance metric, we use
Geometric Mean (Gmean), because it takes into account both True Positive Rate
(TPR) and True Negative Rate (TNR) as
$\text{Gmean}=\sqrt{\text{TPR}\times\text{TNR}}.$

###  IV-B Experimental Results and Discussion

TABLE II: Average Gmean values (and kernel matrix ranks) for SVDD with the
base kernel function and the seven cases of our proposed generalized reference
kernel function

| $M=N$ | $M=\lfloor N/2\rfloor$ | $M=N+T$
---|---|---|---
Data | Base | Case 1 | Case 2 | Case 3 | Case 4 | Case 5 | Case 6 | Case 7
Iris1 | 80.3$\pm$9.6 (34.4) | 80.3$\pm$9.6 (18.4) |  90.2$\pm$4.3 (23.7) | 86.2$\pm$8.0 (12.4) |  89.9$\pm$4.6 (14.8) |  90.0$\pm$3.1 (34.0) | 88.4$\pm$5.6 (16.5) | 86.2$\pm$8.1 (19.0)
Iris2 |  91.0$\pm$4.9 (35.0) |  91.0$\pm$4.9 (21.8) | 90.9$\pm$5.0 (24.7) | 90.0$\pm$5.4 (13.1) |  91.0$\pm$4.9 (15.8) | 89.5$\pm$4.6 (35.0) |  91.2$\pm$4.9 (32.1) | 90.3$\pm$5.7 (31.1)
Iris3 | 88.0$\pm$4.5 (34.8) | 88.6$\pm$4.7 (23.3) |  90.3$\pm$3.0 (27.3) |  89.0$\pm$3.8 (14.2) |  89.0$\pm$4.0 (14.9) |  89.0$\pm$4.8 (34.8) | 86.8$\pm$8.7 (23.8) | 86.5$\pm$9.4 (28.0)
Seed1 | 86.1$\pm$3.7 (47.1) | 87.2$\pm$3.2 (21.6) |  87.4$\pm$2.9 (28.6) | 86.5$\pm$4.4 (15.6) |  87.2$\pm$3.4 (19.2) |  88.7$\pm$3.3 (49.0) | 87.2$\pm$2.4 (30.7) | 86.9$\pm$4.2 (36.2)
Seed2 |  92.3$\pm$8.5 (48.8) |  92.3$\pm$8.5 (31.4) | 91.6$\pm$8.4 (40.1) |  93.6$\pm$2.9 (19.2) | 92.2$\pm$8.5 (22.4) | 90.5$\pm$2.7 (49.0) | 91.9$\pm$8.4 (47.4) |  92.2$\pm$8.5 (40.3)
Seed3 | 87.0$\pm$10.6 (49.0) | 87.1$\pm$10.6 (32.7) |  92.4$\pm$3.5 (45.8) |  93.0$\pm$3.2 (22.2) | 90.2$\pm$7.2 (22.4) | 88.3$\pm$10.0 (44.7) |  92.7$\pm$3.5 (47.9) | 91.3$\pm$4.4 (45.1)
Ion1 | 29.4$\pm$4.3 (87.6) | 26.5$\pm$3.8 (34.0) |  54.2$\pm$9.4 (87.2) | 22.7$\pm$8.6 (33.0) |  56.4$\pm$10.8 (43.0) | 26.7$\pm$5.7 (77.0) | 26.5$\pm$3.8 (34.0) |  56.1$\pm$8.6 (87.6)
Ion2 |  89.0$\pm$1.5 (157.4) | 83.6$\pm$2.7 (121.8) | 84.0$\pm$3.1 (156.4) | 81.7$\pm$5.4 (66.3) |  84.1$\pm$2.5 (77.4) | 85.0$\pm$1.8 (157.4) | 84.4$\pm$2.9 (157.4) |  86.0$\pm$3.8 (157.4)
Son1 | 53.9$\pm$6.6 (68.0) |  55.6$\pm$5.0 (64.4) | 53.9$\pm$5.4 (65.3) |  56.8$\pm$4.2 (33.0) | 51.8$\pm$5.5 (33.0) |  54.2$\pm$6.1 (67.4) | 53.2$\pm$6.8 (66.6) | 53.7$\pm$7.9 (66.4)
Son2 | 54.5$\pm$9.1 (78.0) | 53.8$\pm$8.2 (64.3) |  54.7$\pm$8.8 (72.9) |  56.0$\pm$8.0 (38.0) | 55.6$\pm$5.9 (38.0) | 53.5$\pm$7.9 (67.4) |  56.8$\pm$4.9 (62.9) | 54.9$\pm$9.2 (72.2)
Bank1 |  95.5$\pm$3.4 (24.0) | 90.3$\pm$4.6 (17.5) | 93.8$\pm$3.3 (23.4) | 87.7$\pm$7.2 (10.0) |  93.0$\pm$3.8 (22.9) |  95.2$\pm$2.3 (24.0) | 94.0$\pm$2.6 (24.0) | 93.3$\pm$2.6 (23.6)
Bank2 | 28.5$\pm$16.2 (67.4) | 15.1$\pm$7.8 (18.9) |  67.4$\pm$9.1 (67.4) | 14.9$\pm$7.7 (15.5) |  62.3$\pm$9.0 (49.0) | 35.6$\pm$14.7 (67.4) | 13.7$\pm$8.8 (27.6) |  68.0$\pm$10.0 (67.4)
Happ1 |  51.6$\pm$8.9 (45.6) |  51.6$\pm$8.9 (8.0) | 50.4$\pm$8.1 (22.5) |  52.2$\pm$8.4 (6.6) | 51.8$\pm$8.9 (12.7) |  53.4$\pm$9.0 (14.0) | 51.6$\pm$8.9 (10.4) | 51.1$\pm$7.5 (22.8)
Happ2 |  43.2$\pm$9.0 (44.6) |  43.2$\pm$9.0 (7.9) | 42.5$\pm$10.0 (11.1) |  44.2$\pm$9.6 (7.0) | 40.9$\pm$10.8 (10.4) | 42.7$\pm$10.1 (11.2) | 42.6$\pm$9.9 (10.4) |  44.0$\pm$9.2 (11.3)
Aver. | 69.3$\pm$7.2 (58.7) | 67.6$\pm$6.5 (34.7) |  74.6$\pm$6.0 (49.7) | 68.2$\pm$6.2 (21.9) |  73.9$\pm$6.4 (28.3) | 70.2$\pm$6.1 (52.3) | 68.6$\pm$5.9 (42.3) |  74.3$\pm$7.1 (50.6)

TABLE III: Average Gmean values (and feature space dimensions) for SVDD with
NPT using the base kernel function and the seven cases of our proposed
generalized reference mapping function

| $M=N$ | $M=\lfloor N/2\rfloor$ | $M=N+T$
---|---|---|---
Data | Base | Case 1 | Case 2 | Case 3 | Case 4 | Case 5 | Case 6 | Case 7
Iris1 | 89.8$\pm$4.1 (14.5) | 89.8$\pm$4.1 (14.5) |  91.3$\pm$3.7 (12.8) |  91.8$\pm$5.5 (9.1) | 88.3$\pm$8.7 (10.0) | 90.8$\pm$3.8 (35.8) | 91.1$\pm$3.8 (17.6) |  91.9$\pm$3.6 (15.2)
Iris2 | 90.7$\pm$4.9 (5.0) | 90.7$\pm$4.9 (5.0) |  90.9$\pm$4.9 (6.2) |  91.3$\pm$4.8 (5.7) | 90.4$\pm$4.7 (4.6) |  91.4$\pm$5.6 (12.2) | 90.9$\pm$4.6 (9.5) | 90.9$\pm$4.6 (10.2)
Iris3 | 89.4$\pm$3.1 (18.9) | 89.4$\pm$3.1 (18.9) |  89.9$\pm$3.5 (25.5) |  90.4$\pm$2.8 (12.8) | 90.0$\pm$3.4 (14.4) | 88.4$\pm$5.3 (37.8) | 89.8$\pm$3.6 (33.1) |  90.2$\pm$3.1 (28.1)
Seed1 | 83.1$\pm$3.9 (15.0) | 83.1$\pm$3.9 (15.0) |  84.3$\pm$4.1 (31.5) | 83.8$\pm$3.4 (11.6) |  83.9$\pm$3.6 (17.4) |  86.4$\pm$2.6 (61.7) | 83.1$\pm$3.3 (41.6) | 83.6$\pm$4.8 (33.8)
Seed2 |  90.9$\pm$4.8 (21.0) |  90.9$\pm$4.8 (21.0) | 90.8$\pm$4.9 (31.0) | 89.7$\pm$4.3 (13.9) |  90.8$\pm$4.7 (16.3) |  93.1$\pm$3.6 (33.1) | 91.1$\pm$5.1 (38.4) | 91.0$\pm$4.7 (66.6)
Seed3 |  93.6$\pm$3.5 (11.7) |  93.6$\pm$3.5 (11.7) | 93.5$\pm$3.4 (10.7) |  93.8$\pm$3.3 (9.7) | 93.6$\pm$3.5 (10.1) |  93.7$\pm$3.4 (32.9) | 93.6$\pm$3.4 (15.0) | 93.6$\pm$3.4 (14.1)
Ion1 | 33.0$\pm$18.5 (34.0) | 33.0$\pm$18.5 (34.0) |  41.2$\pm$13.2 (87.2) | 32.2$\pm$18.2 (33.0) |  44.4$\pm$19.1 (43.0) | 34.0$\pm$10.7 (85.4) | 33.0$\pm$18.5 (34.0) |  46.8$\pm$8.5 (175.4)
Ion2 |  86.4$\pm$1.8 (105.4) |  86.4$\pm$1.8 (105.4) | 86.1$\pm$2.1 (156.4) | 85.1$\pm$2.6 (53.9) |  86.3$\pm$2.1 (77.4) | 85.8$\pm$2.2 (226.8) |  86.5$\pm$1.6 (178.8) | 86.2$\pm$2.0 (245.0)
Son1 | 50.5$\pm$8.6 (65.7) | 50.5$\pm$8.6 (65.7) |  50.9$\pm$7.8 (67.0) |  51.2$\pm$7.3 (33.0) | 50.3$\pm$7.9 (33.0) |  51.5$\pm$6.4 (134.2) | 51.4$\pm$6.1 (134.2) | 50.0$\pm$7.8 (135.0)
Son2 | 56.5$\pm$9.2 (76.6) | 56.5$\pm$9.2 (76.6) |  57.1$\pm$8.7 (77.0) |  55.4$\pm$8.5 (38.0) | 54.3$\pm$7.6 (38.0) |  57.2$\pm$10.1 (135.8) | 55.9$\pm$9.5 (140.1) | 55.0$\pm$9.2 (145.0)
Bank1 | 93.4$\pm$3.3 (17.6) | 93.5$\pm$3.1 (17.6) |  94.6$\pm$2.7 (56.4) | 94.3$\pm$2.9 (13.2) |  95.4$\pm$2.8 (34.6) | 95.3$\pm$2.5 (72.3) | 94.6$\pm$3.0 (53.8) |  96.1$\pm$2.9 (76.4)
Bank2 | 32.9$\pm$27.6 (9.6) | 32.9$\pm$27.6 (9.6) |  60.6$\pm$15.6 (99.4) | 31.0$\pm$27.1 (6.0) |  57.2$\pm$13.9 (49.0) | 38.1$\pm$5.0 (90.4) | 32.9$\pm$27.6 (15.4) |  68.4$\pm$12.7 (174.0)
Happ1 | 39.5$\pm$15.8 (20.9) | 39.5$\pm$15.8 (20.9) |  42.8$\pm$9.4 (47.0) | 39.2$\pm$17.0 (16.0) |  41.8$\pm$16.5 (20.2) |  45.1$\pm$10.4 (62.0) | 41.7$\pm$9.2 (66.4) | 44.1$\pm$10.0 (67.4)
Happ2 |  47.9$\pm$13.3 (9.4) |  47.9$\pm$13.3 (9.4) | 44.7$\pm$11.8 (20.5) | 40.4$\pm$12.4 (11.4) |  41.2$\pm$11.4 (10.2) | 43.0$\pm$11.3 (34.7) |  44.8$\pm$11.6 (33.3) | 40.9$\pm$12.4 (25.0)
Aver. | 69.8$\pm$8.7 (30.4) | 69.8$\pm$8.7 (30.4) |  72.8$\pm$6.8 (52.0) | 69.3$\pm$8.6 (19.1) |  72.0$\pm$7.8 (27.0) | 71.0$\pm$5.9 (75.4) | 70.0$\pm$7.9 (57.9) |  73.5$\pm$6.4 (86.5)

TABLE IV: Average Gmean values (and kernel matrix ranks) for OC-SVM with the
base kernel function and the seven cases of our proposed generalized reference
kernel function

| $M=N$ | $M=\lfloor N/2\rfloor$ | $M=N+T$
---|---|---|---
Data | Base | Case 1 | Case 2 | Case 3 | Case 4 | Case 5 | Case 6 | Case 7
Iris1 |  88.6$\pm$4.4 (34.3) | 62.5$\pm$22.6 (13.8) | 77.3$\pm$7.7 (32.7) | 58.7$\pm$26.0 (9.8) |  72.8$\pm$23.4 (16.0) |  95.1$\pm$5.0 (18.5) | 64.3$\pm$24.4 (23.8) | 70.9$\pm$7.4 (34.7)
Iris2 |  91.1$\pm$4.2 (30.6) | 30.0$\pm$22.5 (25.9) | 76.1$\pm$9.0 (34.0) | 47.9$\pm$17.1 (13.8) |  79.2$\pm$7.6 (16.0) |  89.4$\pm$5.4 (35.0) | 69.0$\pm$6.9 (35.0) | 72.1$\pm$9.1 (35.0)
Iris3 |  90.2$\pm$3.2 (34.8) | 60.3$\pm$15.5 (13.1) | 76.1$\pm$11.7 (34.0) | 43.8$\pm$21.1 (9.0) |  74.4$\pm$10.6 (16.0) |  93.7$\pm$2.0 (19.2) | 69.2$\pm$16.3 (30.6) | 65.3$\pm$22.7 (26.4)
Seed1 |  86.5$\pm$2.1 (45.8) | 49.4$\pm$23.9 (24.2) | 74.2$\pm$7.3 (48.0) | 43.1$\pm$18.4 (12.8) |  75.6$\pm$6.7 (23.0) |  86.7$\pm$5.3 (42.2) | 70.8$\pm$6.8 (49.0) | 74.8$\pm$6.1 (49.0)
Seed2 |  89.6$\pm$4.4 (48.2) | 59.1$\pm$27.8 (36.8) | 82.0$\pm$7.3 (48.0) | 56.1$\pm$22.0 (17.9) |  81.1$\pm$7.5 (23.0) |  88.7$\pm$5.4 (23.2) | 63.5$\pm$16.7 (42.9) | 81.8$\pm$6.9 (49.0)
Seed3 |  92.3$\pm$4.4 (48.9) | 53.9$\pm$28.7 (31.2) | 78.7$\pm$7.0 (48.0) | 36.7$\pm$27.8 (15.4) |  83.2$\pm$7.5 (23.0) |  89.5$\pm$7.7 (39.0) | 72.9$\pm$9.2 (49.0) | 82.3$\pm$6.3 (49.0)
Ion1 | 36.7$\pm$2.6 (87.6) |  51.0$\pm$9.8 (84.1) | 48.5$\pm$14.0 (85.0) |  49.2$\pm$10.0 (42.5) | 47.3$\pm$9.9 (42.2) |  70.6$\pm$4.0 (87.6) | 51.1$\pm$11.4 (79.1) | 52.9$\pm$10.1 (87.6)
Ion2 |  88.9$\pm$2.3 (157.4) | 47.0$\pm$24.9 (34.0) | 76.5$\pm$5.2 (156.4) | 50.9$\pm$16.6 (48.5) |  67.4$\pm$14.8 (77.4) |  88.2$\pm$3.0 (157.4) | 79.3$\pm$2.7 (157.4) | 79.7$\pm$4.3 (157.4)
Son1 |  53.2$\pm$5.7 (68.0) | 47.2$\pm$5.7 (67.0) | 50.2$\pm$6.8 (67.0) | 46.7$\pm$11.4 (33.0) |  50.1$\pm$6.3 (33.0) |  63.9$\pm$4.3 (68.0) | 46.9$\pm$5.8 (67.2) | 53.7$\pm$5.6 (68.0)
Son2 |  55.1$\pm$8.8 (78.0) | 50.8$\pm$6.7 (77.0) | 51.4$\pm$9.0 (77.0) | 48.5$\pm$9.2 (38.0) |  48.9$\pm$9.3 (38.0) |  67.9$\pm$4.8 (76.6) | 51.7$\pm$8.6 (78.0) | 51.7$\pm$6.6 (78.0)
Bank1 |  93.0$\pm$3.4 (23.4) | 62.1$\pm$3.6 (10.2) | 69.9$\pm$9.7 (24.0) | 58.6$\pm$14.2 (18.1) |  69.5$\pm$8.5 (24.0) |  86.0$\pm$4.4 (24.0) | 69.3$\pm$10.0 (24.0) | 70.0$\pm$8.5 (24.0)
Bank2 | 20.7$\pm$12.0 (67.4) | 57.4$\pm$6.8 (57.9) |  80.9$\pm$9.0 (67.4) | 57.3$\pm$14.4 (37.0) |  82.3$\pm$8.2 (49.0) | 78.3$\pm$4.3 (65.8) | 59.4$\pm$21.3 (67.4) |  82.2$\pm$4.8 (67.4)
Happ1 |  54.6$\pm$7.1 (47.0) | 44.9$\pm$10.9 (36.5) | 45.2$\pm$9.2 (36.0) |  48.6$\pm$8.4 (23.3) | 48.5$\pm$8.3 (20.6) |  52.7$\pm$8.6 (45.9) | 47.6$\pm$6.9 (41.2) | 47.5$\pm$7.8 (37.1)
Happ2 | 44.3$\pm$8.0 (44.6) |  47.9$\pm$7.8 (38.2) | 42.9$\pm$9.2 (34.4) | 42.3$\pm$6.0 (17.9) |  43.4$\pm$8.7 (17.4) | 45.5$\pm$9.4 (32.7) |  48.0$\pm$9.6 (26.0) | 42.4$\pm$12.9 (27.0)
Aver. |  70.3$\pm$5.2 (58.3) | 51.7$\pm$15.5 (39.3) | 66.4$\pm$8.7 (56.6) | 49.2$\pm$15.9 (24.1) |  66.0$\pm$9.8 (29.9) |  78.3$\pm$5.3 (52.5) | 61.6$\pm$11.2 (55.0) | 66.2$\pm$8.5 (56.4)

TABLE V: Average Gmean values (and feature space dimensions) for OC-SVM with
NPT using the base kernel function and the seven cases of our proposed
generalized reference mapping function

| $M=N$ | $M=\lfloor N/2\rfloor$ | $M=N+T$
---|---|---|---
Data | Base | Case 1 | Case 2 | Case 3 | Case 4 | Case 5 | Case 6 | Case 7
Iris1 | 62.5$\pm$22.6 (13.8) | 62.5$\pm$22.6 (13.8) |  77.3$\pm$7.7 (32.7) | 58.7$\pm$26.0 (9.8) |  72.8$\pm$23.4 (16.0) |  95.1$\pm$5.0 (19.3) | 64.3$\pm$24.4 (32.6) | 70.9$\pm$7.4 (46.8)
Iris2 | 30.0$\pm$22.5 (25.9) | 30.0$\pm$22.5 (25.9) |  76.1$\pm$9.0 (34.0) | 47.9$\pm$17.1 (13.8) |  79.2$\pm$7.6 (16.0) |  89.4$\pm$5.4 (50.6) | 69.0$\pm$6.9 (67.2) | 72.1$\pm$9.1 (68.5)
Iris3 | 60.3$\pm$15.5 (13.1) | 60.3$\pm$15.5 (13.1) |  76.1$\pm$11.7 (34.0) | 43.8$\pm$21.1 (9.0) |  74.4$\pm$10.6 (16.0) |  93.7$\pm$2.0 (19.2) | 69.2$\pm$16.3 (54.1) | 65.3$\pm$22.7 (49.1)
Seed1 | 49.4$\pm$23.9 (24.2) | 49.4$\pm$23.9 (24.2) |  74.2$\pm$7.3 (48.0) | 43.1$\pm$18.4 (12.8) |  75.6$\pm$6.7 (23.0) |  86.7$\pm$5.3 (52.1) | 70.8$\pm$6.8 (97.0) | 74.8$\pm$6.1 (97.0)
Seed2 | 59.1$\pm$27.8 (36.8) | 59.1$\pm$27.8 (36.8) |  82.0$\pm$7.3(48.0) | 56.1$\pm$22.0 (17.9) |  81.1$\pm$7.5 (23.0) |  88.7$\pm$5.4 (23.2) | 63.5$\pm$16.7 (83.2) | 81.8$\pm$6.9 (97.0)
Seed3 | 53.9$\pm$28.7 (31.2) | 53.9$\pm$28.7 (31.2) |  78.7$\pm$7.0 (48.0) | 36.7$\pm$27.8 (15.4) |  83.2$\pm$7.5 (23.0) |  89.5$\pm$7.7 (49.2) | 72.9$\pm$9.2 (97.0) | 82.3$\pm$6.3 (97.0)
Ion1 | 51.0$\pm$9.8 (84.1) |  51.2$\pm$9.8 (84.1) | 48.5$\pm$14.0 (85.0) |  49.2$\pm$10.0 (42.5) | 47.3$\pm$9.9 (42.2) |  70.6$\pm$4.0 (151.0) | 51.1$\pm$11.4 (140.2) | 52.9 $\pm$10.1 (175.4)
Ion2 | 47.0$\pm$24.9 (34.0) | 47.0$\pm$24.9 (34.0) |  76.5$\pm$5.2 (156.4) | 50.9$\pm$16.6 (48.5) |  67.4$\pm$14.8 (77.4) |  88.2$\pm$3.0 (238.6) | 79.3$\pm$2.7 (244.8) | 79.7$\pm$4.3 (245.0)
Son1 | 47.2$\pm$5.7 (67.0) | 47.2$\pm$5.7 (67.0) |  50.2$\pm$6.8 (67.0) | 46.7$\pm$11.4 (33.0) |  50.1$\pm$6.3 (33.0) |  63.9$\pm$4.3 (132.3) | 46.9$\pm$5.8 (125.0) | 53.7$\pm$5.6 (135.0)
Son2 | 50.8$\pm$6.7 (77.0) | 50.8$\pm$6.7 (77.0) |  51.4$\pm$9.0 (77.0) | 48.5$\pm$9.2 (38.0) |  48.9$\pm$9.3 (38.0) |  67.9$\pm$4.8 (135.9) | 51.7$\pm$8.6 (140.2) | 51.7$\pm$6.6 (145.0)
Bank1 | 62.1$\pm$4.2 (10.8) | 61.6$\pm$3.7 (10.2) |  70.1$\pm$9.5 (73.4) | 59.6$\pm$14.3 (18.1) |  68.4$\pm$9.0 (36.0) |  86.0$\pm$4.4 (75.2) | 69.3$\pm$10.0 (97.4) | 69.8$\pm$7.5 (147.8)
Bank2 | 58.9$\pm$8.3 (60.7) | 58.0$\pm$5.7 (57.9) |  80.9$\pm$9.0 (99.4) | 57.3$\pm$14.5 (37.0) |  82.4$\pm$8.1 (49.0) | 78.5$\pm$6.7 (82.7) | 59.4$\pm$21.3 (140.4) |  82.2$\pm$4.8 (174.0)
Happ1 | 45.3$\pm$11.0 (36.5) | 44.9$\pm$10.9 (36.5) |  45.4$\pm$9.1 (38.8) |  48.6$\pm$8.4 (23.3) | 48.5$\pm$8.3 (20.6) |  52.7$\pm$8.6 (69.5) | 47.6$\pm$6.9 (66.9) | 47.5$\pm$7.8 (65.5)
Happ2 |  48.4$\pm$7.3 (38.9) | 47.9$\pm$7.8 (38.2) |  42.9$\pm$9.2 (34.8) | 42.3$\pm$6.0 (17.9) | 43.4$\pm$8.7 (17.4) | 45.5$\pm$9.4 (40.0) |  48.0$\pm$9.6 (35.5) | 42.4$\pm$12.9 (36.8)
Aver. | 51.9$\pm$15.6 (39.6) | 51.7$\pm$15.4 (39.3) |  66.4$\pm$8.7 (62.6) | 49.2$\pm$15.9 (24.1) |  65.9$\pm$9.8 (30.8) |  78.3$\pm$5.4 (81.4) | 61.6$\pm$11.2 (101.5) | 66.2$\pm$8.4 (112.8)

The experimental results in terms of Gmean are given in Table II-V. Tables II
shows SVDD results using the base kernel matrix along with the seven cases of
our proposed generalized kernel matrix (19) via kernel implementation and
Table III shows the results using NPT along with the seven cases of our
proposed generalized kernel mapping (III) via linear implementation. Tables IV
and Tables V have the corresponding results for OC-SVM. The tables also give
the average kernel matrix ranks or feature space dimensions for the mappings.
In each table, the best performance for each number of reference vectors is
bolded.

First, we see that the results for the base NPT and for the case 1 of the
proposed mapping are indeed almost identical. For the kernel setting, the
difference is larger in particular for OC-SVM. Our additional experiments
verify that the differences indeed follow mainly from the centering operation
and, in most cases, OC-SVM suffers from the centering, while for SVDD
centering slightly improves the results. Also in general, we see that the
conclusions for SVDD and OC-SVM are quite different, which shows that future
analysis should focus on the kernel properties with respect to a specific
algorithm.

For SVDD, the random cases 2, 4, and 7 quite consistently improve the results
compared to the base method or to using training vectors as reference vectors.
In particular, there are few cases (Ion1, Bank2), where the base methods
clearly fail, but the proposed kernel and mapping with random reference
vectors leads to clearly improved results. The random reference vectors indeed
seem to provide implicit regularization and, thus, a more robust performance.
The results are on average slightly better when there are more reference
vectors, but the differences are quite insignificant. Thus, using the proposed
generalized reference kernel as an implicit data augmentation ($M>N$) may
slightly help, but it is more beneficial to replace the training samples with
random data than to augment the training samples with random data (case 6).

Looking at the kernel matrix ranks, we see that the proposed implementation
with the case 2 often leads to lower ranks but better results than the
original method. On the hand, in the tasks where the random reference vectors
lead to significant improvements do not have lower rank. This gives some
indication that the random reference vector variant can adapt the rank to the
task at hand. For the base NPT, ranks are lower than for the base kernel,
whereas for the proposed approach the ranks are more similar between the
kernel and mapping, while for the mapping the variance in ranks is higher.

For OC-SVM, as mentioned already, centering clearly harms the performance.
Thus, the base kernel performs much better than the base NPT, while most of
the proposed generalized reference kernel variants also achieve a lower
performance than the corresponding variant for SVDD. A very interesting
exception for both generalized reference kernel and mapping with OC-SVM is the
case 5 that uses also negative samples as reference vectors. The case 5
results are consistently outperforming all the other cases and also all the
SVDD results. While it should be remembered that comparing this case directly
with the other cases is unfair, because the basic assumption of not having
negative examples is violated, the results still open interesting future work
opportunities either as using the case 5 as a way to exploit some negative
samples within the regular one-class implementation or as an indication that
for OC-SVM it may be more beneficial to pick reference vectors outside the
training data distribution. Furthermore, it is interesting that only OC-SVM
with generalized reference kernel and mapping can exploit the negative data
efficiently, while the same is not true for SVDD with the proposed approaches.
Further analyzing this difference may lead to better understanding how to
select most suitable kernels for different algorithms.

##  V Conclusions

In this paper, we proposed a new kernel formulation based on a base kernel and
reference vectors that can be selected in different ways. Different cases
(i.e., different ways to select the reference vectors) of our _generalized
reference mapping and kernel_ have links to different prior works including
approximate kernels and NPT. Our formulation also provides a way to extend
random sampling and random projection-based approximate kernel methods into an
NPT-like setting. In this paper, we considered the new formulation in the
context of small-scale one-class classification and with RBF kernels, but in
the future it can be used also in different tasks and with different kernels.

Our experimental results show that the new formulation can help to implicitly
regularize and adjust the rank of the kernel matrices. It also allows to
incorporate additional information into the kernel itself, leading to improved
one-class classification accuracy. For SVDD, random reference vectors led to
best classification results and more robust performance indicating that this
approach provides implicit regularization. For OC-SVM, using negative samples
as references vectors led to significant classification performance
improvements. While this approach violates the assumption that one-class
classification cannot use negative examples for training, it can be considered
in the future in the class-specific classification context or in one-class
classification with few negative examples. It may also indicate that for OC-
SVM it would be more beneficial to select random reference vector outside the
training distribution.

As the conclusions for SVDD and OC-SVM are quite different, this suggests that
the new kernel formulation should be further studied with respect to a
specific method. On the other hand, further analyzing the causes for the
observed differences may help to understand how to optimize the reference
vectors for the method and task at hand.

In NPT, the training data needs to be centered in $\mathcal{F}$, i.e.
$\sum_{i=1}^{N}\phi(\mathbf{x}_{i})=0$. If this assumption is not met,
$\mathcal{P}$ is a manifold in $\mathcal{F}$ and not a subspace of
$\mathcal{F}$, since $\mathbf{0}$ does not necessarily belong is to
$\mathcal{P}$ [kwak2013NPT]. In our formulation, $\mathcal{F}$ is spanned by
function $\phi(\cdot)$, meaning that the centering operation is not mandatory.
Nevertheless, in practice we apply KPCA for the reference data with the base
kernel $\tilde{\kappa}(\cdot,\cdot)$ and, therefore, we also opt to center our
reference data in the base kernel space $\tilde{\mathcal{F}}$ by centering the
corresponding kernel function $\tilde{\mathbf{K}}_{\mathbf{R}\mathbf{R}}$. To
align our training and testing data, we also center
$\tilde{\mathbf{K}}_{\mathbf{R}\mathbf{X}}$ and
$\tilde{\mathbf{K}}_{\mathbf{R}\hat{\mathbf{X}}}$ with respect to the center
of the reference data $\mathbf{R}$.

Let us denote by $\tilde{\boldsymbol{\mu}}$ the mean of the $M$ uncentered
reference vectors
$\tilde{\Phi}(\mathbf{R})=[\tilde{\phi}(\mathbf{r}_{1}),\dots,\tilde{\phi}(\mathbf{r}_{M})]\in\tilde{\mathcal{F}}$:

$\tilde{\boldsymbol{\mu}}=\frac{1}{M}\sum_{i=1}^{M}\tilde{\phi}(\mathbf{r}_{i})=\frac{1}{M}\tilde{\Phi}(\mathbf{R})\mathbf{1}_{M},$
(.1)

where $\mathbf{1}_{M}\in\mathbb{R}^{M}$ is a vector of ones. The centered
reference vectors in $\tilde{\mathcal{F}}$ are given by

$\tilde{\Psi}(\mathbf{R})=\tilde{\Phi}(\mathbf{R})-\tilde{\boldsymbol{\mu}}\mathbf{1}_{M}^{T}=\tilde{\Phi}(\mathbf{R})(\mathbf{I}-\frac{1}{M}\mathbf{1}_{M}\mathbf{1}_{M}^{T})=\tilde{\Phi}\mathbf{C}_{M},$
(.2)

where we denote by
$\mathbf{C}_{M}=\mathbf{I}-\frac{1}{M}\mathbf{1}_{M}\mathbf{1}_{M}^{T}$ the
centering matrix for the reference vectors. The training vectors $\mathbf{X}$
(and test vectors $\hat{\mathbf{X}}$ accordingly) can be centered with respect
to $\tilde{\boldsymbol{\mu}}$ in $\tilde{\mathcal{F}}$ as follows:

$\displaystyle\tilde{\Psi}(\mathbf{X})$ $\displaystyle=$
$\displaystyle\tilde{\Phi}(\mathbf{X})-\tilde{\boldsymbol{\mu}}\mathbf{1}_{N}^{T}=\tilde{\Phi}(\mathbf{X})-\frac{1}{M}\tilde{\Phi}(\mathbf{R})\mathbf{1}_{M}\mathbf{1}_{N}^{T}$
(.3) $\displaystyle=$
$\displaystyle\tilde{\Phi}(\mathbf{X})-\tilde{\Phi}(\mathbf{R})\mathbf{C}_{N},$

where $\mathbf{C}_{N}=\frac{1}{M}\mathbf{1}_{M}\mathbf{1}_{N}^{T}$. By using
$\tilde{\Psi}(\mathbf{R})$ and $\tilde{\Psi}(\mathbf{X})$ (or
$\tilde{\Psi}(\hat{\mathbf{X}})$), the centered kernel matrices
$\tilde{\mathcal{K}}_{\mathbf{R}\mathbf{R}}$ and
$\tilde{\mathcal{K}}_{\mathbf{R}\mathbf{X}}$ (or
$\tilde{\mathcal{K}}_{\mathbf{R}\hat{\mathbf{X}}}$) can be calculated as
follows:

$\tilde{\mathcal{K}}_{\mathbf{R}\mathbf{R}}=\tilde{\Psi}(\mathbf{R})^{T}\tilde{\Psi}(\mathbf{R})=\mathbf{C}_{M}\tilde{\mathbf{K}}_{\mathbf{R}\mathbf{R}}\mathbf{C}_{M}$
(.4)

and

$\tilde{\mathcal{K}}_{\mathbf{R}\mathbf{X}}=\tilde{\Psi}(\mathbf{R})^{T}\tilde{\Psi}(\mathbf{X})=\mathbf{C}_{M}\left(\tilde{\mathbf{K}}_{\mathbf{R}\mathbf{R}}-\tilde{\mathbf{K}}_{\mathbf{R}\mathbf{X}}\mathbf{C}_{N}\right),$
(.5)

Note that the centering formulas for NPT can be obtained by setting
$\mathbf{R}=\mathbf{X}$ and by using the kernel at hand as the base kernel in
the formulas. xconference˙101719.bbl

  *[OC-SVM]: One-class Support Vector Machine
  *[SVDD]: Support Vector Data Description
  *[NPT]: Non-linear Projection Trick
  *[RBF]: Radial Basis Function
  *[KPCA]: Kernel Principal Component Analysis
  *[PCA]: Principal Component Analysis
  *[KDA]: Kernel Discriminant Analysis
  *[LDA]: Linear Discriminant Analysis
  *[ESVDD]: Ellipsoidal Support Vector Data Description