11institutetext: Signal Processing Laboratory LTS4, EPFL, Lausanne,
Switzerland22institutetext: Embark Studios, Stockholm, Sweden
22email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

# Knowledge-Distilled Graph Neural Networks for Personalized Epileptic Seizure
Detection ††thanks: This work is supported by the PEDESITE project
2This work was done by the third author when she was a research intern in the
same lab as the other authors at EPFL.

Qinyue Zheng 11 Arun Venkitaraman 11 Simona Petravic 22 Pascal Frossard 11

###### Abstract

Wearable devices for seizure monitoring detection could significantly improve
the quality of life of epileptic patients. However, existing solutions that
mostly rely on full electrode set of electroencephalogram (EEG) measurements
could be inconvenient for every day use. In this paper, we propose a novel
knowledge distillation approach to transfer the knowledge from a sophisticated
seizure detector (called the teacher) trained on data from the full set of
electrodes to learn new detectors (called the student). They are both
providing lightweight implementations and significantly reducing the number of
electrodes needed for recording the EEG. We consider the case where the
teacher and the student seizure detectors are graph neural networks (GNN),
since these architectures actively use the connectivity information. We
consider two cases (a) when a single student is learnt for all the patients
using pre-selected channels; and (b) when personalized students are learnt for
every individual patient, with personalized channel selection using a Gumbel-
softmax approach. Our experiments on the publicly available Temple University
Hospital EEG Seizure Data Corpus (TUSZ) show that both knowledge-distillation
and personalization play significant roles in improving performance of seizure
detection, particularly for patients with scarce EEG data. We observe that
using as few as two channels, we are able to obtain competitive seizure
detection performance. This, in turn, shows the potential of our approach in
more realistic scenario of wearable devices for personalized monitoring of
seizures, even with few recordings.

###### Keywords:

Personalized seizure detection Graph neural networks Knowledge distillation.

##  1 Introduction

Epilepsy is a neurological disorder that is characterized by recurring,
unprovoked seizures caused by surges of electrical activity in the brain and
affects nearly three million people [26]. About one third of the patients do
not respond to treatment by drugs [17]. Hence, real-time seizure monitoring is
crucial for improving the patients’ quality of life, for example, by alerting
caregivers that their assistance is needed once a seizure occurs. A continuous
monitoring of the electroencephalogram (EEG) is useful in identifying and even
predicting seizures in critically ill patients [19], particularly with the use
of deep-learning approaches [27, 21, 12, 1, 23] The monitoring is usually
performed in a hospital environment over the course of several days, which
makes it infeasible to monitor patients long-term in non-ambulatory settings.
Wearable devices could overcome the need of specialised intrusive medical
equipment and hospital environment and enable real-time seizure monitoring on
a daily basis. Existing measurement devices [3] that use EEG head caps with
over 20 wired electrodes are however uncomfortable and difficult to wear over
prolonged intervals and lighter and more discrete wearables are desirable for
patients. Previous studies have attempted to reduce the number of EEG
electrodes needed for seizure detection [8, 28, 9] with promising results.
However, these solutions typically involve training detection systems from
scratch for the new setting, and fail to incorporate the already existing
historical EEG data of the patient recorded with many electrodes. Due to the
nature of the disorder itself, seizure data is sparse in the number of
available seizures and difficult to collect, and it is thus important to
meaningfully use previous data. Further, it is known that the signals from the
different regions of the brain (captured through the EEG electrodes) are not
independent and exhibit strong inter-channel dependencies that could be viewed
as a brain graph or a network. Hence, we ask the question:

How to transfer information gained from a full set of channels/graph to
settings with a reduced number of channels/subgraph while actively using the
connectivity information?

In this paper, we address this question by developing a novel approach for
knowledge distillation (KD) with graph neural networks (GNNs) applied to
seizure detection. Our motivation for the use of GNNs comes from the
observation that they have been used extensively in applications with graph-
structured data, and more recently have shown to result in promising seizure
detection performance [22, 30]. More specifically, we propose a seizure
detection model that consists of three interconnected blocks. Firstly, we have
the knowledge distillation block, whereby we transfer the knowledge from a
pre-trained seizure detection model to obtain a model that is light-weight and
uses only a reduced set of input channels and the corresponding subgraph.
Secondly, a channel selection block, which takes the full multi-channel input
and retains the signal only on a reduced set of channels that are either pre-
selected or learnt in a fully data-driven manner. Lastly, we have the GNN
based seizure detection model that classifies the input in the form of the
multi-channel signal from a reduced set of channels/electrodes and the
corresponding subgraph, into seizure or non-seizure segments.

Our goal is to also investigate the influence of two important aspects in
seizure detection performance with reduced channels: (i) prior knowledge
(through the use of the teacher model), and (ii) personalization/ patient-
specific detection. The specific contributions of our paper are as follows:

  * •

We propose new GNN models for epileptic seizure detection that build on
knowledge distillation to generate models that are both light-weight and work
on subgraphs of reduced nodes/channels. To the best of our knowledge, this is
the first KD approach dedicated to obtaining subgraph GNNs with reduced
channels.

  * •

We propose two different models for seizure detection with reduced channels,
namely one with pre-selected (clinically motivated) channels and one with
data-driven channels obtained from Gumbel softmax channel selection.

  * •

By applying our approach on pre-trained GNN that uses a full electrode set, we
obtain personalized (patient-specific) and global (non patient-specific) GNN
models that are both lightweight (using only $\approx 3\%$ of the parameters
of the teacher) and requires only a reduced subset of electrodes (requiring as
low as only $10\%$ of the original electrodes)

  * •

We demonstrate the results of our approach on the TUH Seizure Corpus, which is
one of the most popular and diverse datasets for epileptic seizures.

  * •

We show empirically that the combination of personalization and KD could
significantly improve seizure detection in cases of very scarce data, and in
cases when the measurements are made from the relatively ’non-informative’
electrodes.

Finally, it could be noted that epilepsy seizure detection is a very active
research problem. In particular, there has been a steady increase in the
number of graph-based approaches, and particularly GNNs applied to the problem
of seizure detection and classification [30, 22, 5]. However, to the best of
our knowledge no prior works exist that tackle the problem of channel
reduction with GNNs and KD, particularly for seizure detection. While KD has
been used in multiple settings related to GNNs [6, 7, 15, 4, 31, 32, 33], it
has not been employed to the task of data-driven subgraph identification,
which is the main objective in this paper.

##  2 Preliminaries

We now briefly review some of the basic concepts from GNNs and KD.

Graph Neural Networks Graph Neural Networks (GNNs) refer to a class of deep
learning models designed for graph-structured data [24]. GNNs learn the
representations of the nodes/channels in a graph and predict the labels or
properties of nodes/edges by actively using the the underlying graph
structure. Due to the graph structure, GNNs naturally provide an aspect of
interpretability or explainability. GNNs have been shown to significantly
outperform the use of CNNs or other non-graph approaches in many applications.
While study and development of GNNs is an active research area, we consider
the specific case of Graph convolutional networks (GCNs) in our work, since
they form one of the simplest and most popular GNNs that directly generalize
the convolution operation from CNNs to a graph setting [16]. A multi-layer GCN
has the layer-wise propagation rule in the hidden layers:

$\centering H^{(l+1)}=\sigma({A}H^{(l)}\Theta^{(l)})\@add@centering$ (1)

where $H^{l}\in\mathbb{R}^{N\times D}$ is the hidden node features at $l$-th
layer; $H^{0}$ denoting the input, $\sigma$ a non-linear activation function
such as ReLU or sigmoid, $A$ the adjacency matrix, and $\Theta{(l)}$ being the
weight matrix in the $l$-th layer that is learnt from the data for a given
task. Put simply, the graph convolution operation takes the weighted sum of
the features of the neighbors of a node and applies a non-linear activation
function to produce the updated features for the node. This operation is
repeated for each layer, allowing the model to learn more complex
representations of the graph structure and node features. The final output of
a GCN is typically obtained by applying a linear layer to the features of the
nodes in the final layer. Finally, depending on whether the task is regression
or classification, the parameters of the GNN are learned by minimizing a loss
function, respectively.

Knowledge Distillation Knowledge distillation (KD) [11] refers to transferring
knowledge from a large/sophisticated pre-trained neural network (known as the
teacher network) to a smaller network (known as the student network). The
student represents a light-weight model derived from the teacher while
enforcing the performance to be similar to that of the teacher. A distillation
loss is used during training to guide the student to replicate the teacher’s
behavior as closely as possible. Different types of knowledge can be
transferred, but the most straightforward one is response-based KD, which
refers to the response of the output layer of the teacher. A widely used
example of this is the class probability called as soft targets defined using
a softmax function as

$p(z_{i},T)={\exp(z_{i}/T)}/{\sum_{j}{}\exp(z_{j}/T)},\vspace{-.1in}$ (2)

where $p_{i}$ is the probability of belonging to class $i$, $z$ is the vector
of logits (outputs of the last layer of the teacher to a given input). The
temperature $T$ controls the contribution of each soft target to the
knowledge. When $T$ is equal to 1, we get the standard softmax function, but
as $T$ increases, the probability distribution is softened. The distillation
loss can be seen as comparing the class probabilities obtained from the
teacher and the student. It enforces the distribution of the outputs produced
by the student to be close to that of the teacher. The Kullback-Leibler (KL)
divergence is therefore often used as the distillation loss function, and
minimizing this loss during training makes the logits of the student get
closer to the logits of the teacher [10]. Let $z_{t}$ and $z_{s}$ denote the
representation produced by the teacher and student models, respectively, for
the same input. Then, the final loss function used to train the student is a
weighted average of the two terms and is defined as

$\displaystyle L_{S}{}={}$
$\displaystyle(1-\delta)L_{D}(p(z_{t},T),p(z_{s},T)){+}\>\delta
L_{CE}(y,p(z_{s},1)),$ (3)

where $L_{D}$ is the distillation loss function, $p(z_{t},T)$ are the teacher
soft targets, $p(z_{s},T)$ are the student soft targets, $L_{CE}$ is the cross
entropy loss function, $y$ are the ground truth labels, and $\alpha$ is the
weighting factor. The parameter $\delta$ represents the relative weight given
to the teacher’s knowledge over the new training data corresponding to the
student training $-$ the higher $\delta$, the lesser the model relies on the
teacher for the training of the student. We shall consider the KD as part of
our approach later in Section 3.

##  3 KD with GNNs for Seizure Detection

###  3.1 Proposed Model

We first propose our approach to design a global seizure detection student GNN
that works on data with reduced nodes/channels and the corresponding subgraph,
obtained using KD from a teacher GNN that operates on the complete node set.
Let $D$ denote the number of nodes/channels in the full measurement. Let $A$
denote the adjacency matrix of the graph describing the connections between
the different channels. The adjacency matrix could be obtained in different
ways like a correlation matrix, functional connectivity, or simply the matrix
that captures the physical proximity of the electrodes on the scalp. In our
paper, we use the latter.

Let $\mathbf{x}\in\mathbb{R}^{D\times T}$ denote the input signal consisting
of the recordings /measurements from all the $D$ channels for $T$ time
samples. Let us consider a GNN with parameters $\theta$ and let
$z_{\theta}(\mathbf{x},A)$ denote the output of the last layer or the logits
learnt by the GNN, where $A\in\mathbb{R}^{D\times D}$ denotes the graph
between the channels. Further, let us use subscripts $t$ and $s$ for the
teacher and student GNNs, respectively: $z_{\theta_{t}}(\cdot,A)$ and
$z_{\theta_{s}}(\cdot,A)$ denote the output layers from the teacher and
student GNNs, respectively. The teacher network is learnt by minimizing the
following the binary cross entropy function $BCE(\cdot,\cdot)$ between the
class label $y$ and the model prediction $f^{t}_{\theta_{t}}(\mathbf{x})$

$\mathcal{L}_{CE}(\theta_{t})=\mathbb{E}_{\mathbf{x}}\left(BCE(y,z_{\theta_{t}}(\mathbf{x},A))\right),$
(4)

with respect to $\theta_{t}$, where $\mathbb{E}$ denotes the expected value
obtained by averaging over all training samples $\mathbf{x}$. We use the BCE
function since we consider here only the seizure versus non-seizure
classification problem. In order to train the student GNN from the pre-trained
teacher, we minimize a regularized BCE cost, where the regularization term is
given by the distillation loss that minimizes the KL divergence between the
soft-output of the teacher and student GNNs:

$\displaystyle\mathcal{L}_{D}(\theta_{t}*,\theta_{s})=\mathbb{E}_{\mathbf{x}}\left(KL(p(z_{\theta_{t}*},T)(\mathbf{x},A),p(z_{\theta_{s}}(\mathbf{x},A),T))\right),$
(5)

where $\theta_{t}*$ denotes the parameters of the pre-trained teacher. Then,
the student network is trained by minimizing the total loss function:

$L_{S}(\theta_{s})\triangleq(1-\delta)\mathcal{L}_{D}(\theta_{t}*,\theta_{s})+\delta\,\mathcal{L}_{BCE}(\theta_{s}).$
(6)

Our formulation so far uses the same input for both the student and teacher,
and hence, the same number of input channels. This is because the KD
formulation assumes that the input to both the student and the teacher are of
the same class, as we discussed in the Preliminaries. However, our ultimate
goal is to transfer knowledge to a student that uses the measurements from
reduced set of nodes/channels $\mathbf{x}^{d}$ with $d<D$, and not
$\mathbf{x}$. In other words, we wish to train a student model that works on a
subgraph $A^{\prime}$ of the original graph $A$. We achieve this by modifying
the graph used by the student deleting the edges from the full graph with
adjacency matrix $A$ as follows:

${A}^{\prime}=W^{\top}\,A\,W,$ (7)

where $W\in\mathbb{R}^{D\times d}$ denotes the selection matrix which is a
permutation of the matrix given by concatenation of a identity matrix of
dimension $d$ with an all zero matrix of size $(D-d)\times d$ $-$ retains only
the subgraph of $d$-size subset of the channels.111 In general, $A^{\prime}$
may not necessarily be a connected graph, unless specifically regularized to
be so. The input $\mathbf{x}_{d}$ is then given by
$\mathbf{x}_{d}=W^{\top}\mathbf{x}\in\mathbb{R}^{d}$, corresponding to the
nodes of the subgraph defined by $W$. This in turn means that we must use
$z_{\theta_{s}}(\mathbf{x}_{d},A^{\prime})$ and not
$z_{\theta_{s}}(\mathbf{x},A)$ in the total loss function in (6). Further, in
order that the hidden nodes corresponding to the deleted channels are not
pooled in the GNN, we multiple the output of each hidden layer of the GNN also
by $W$. This in turn means that in practice the student GNN working on $D$
nodes can be fed with zeroes at the test time on the discarded channels,
corresponding to having only the reduced set of measurement channels as input
for seizure detection. We note that, while the specific application setting
used in this work is that of scalp EEG channels, our proposed approach can be
applied also to other multi-channel settings such as fMRI, where there is
knowledge of connectivity across channels/measurements. The use of GNNs also
makes our approach inherently interpretable in terms of connectivity of the
brain regions.

We consider three different instances of our model in this work: (a) Global
Student GNN with Pre-Selected channel reduction (GS-PS) model, (b) Global
Student GNN with data-driven channel reduction (GS-DD) model, and (c)
Personalized Student with Data-Driven channel reduction (PS-DD) model We
describe them next.

###  3.2 GS-PS Model

We first consider the case when the reduced electrodes are preselected, or
known already. In particular, we chose the four channels of T3, T4, T5, and T6
of the 19 channels from the T-20 montage [14] as the reduced electrode set.
This is motivated by input from neuroscientists that say these temporal
channels can be relatively more indicative channels for seizure in general
[9]. In this case, the $W$ matrix from Eq. (7) corresponds to a diagonal
matrix with ones only at the indices corresponding to T3, T4, T5, and T6. We
also validate the choice of these channels through the following experiment.
We conduct an experiment where a new model with the same architecture as the
teacher (keeping the full electrode channels) is trained to learn relevance
weights $w$ for each electrode: this was simply achieved by applying a
learnable diagonal matrix $M\in\mathbb{R}^{D\times D}$ to the input before the
GNN such that the effective input to the GNN was defined as
$\mathbf{x}_{M}^{\prime}=M\cdot\mathbf{x}\in\mathbb{R}^{D\times D}$. We notice
that the weights assigned to the temporal and some of the occipital electrodes
were the highest, in particular, T2,T3,T4, and T5, were given large weights. A
more practical reason for the choice of temporal channels is the development
of wearable sensors: many state of the art wearable sensors are of the behind
the ear type, corresponding to these four temporal channels [9, 28]. We apply
the proposed GS-PS model for seizure detection by training them on the data
from training patients and apply them to detect seizures on new test patients.
In this case, the subgraph is pre-determined.

###  3.3 GS-DD Model

We next consider the case of learning a student with channel reduction
achieved in a completely data-driven manner. We propose to use a Gumbel-
softmax (GS) channel selection block akin to the approach pursued in [29]. Our
proposed GD-DD model consists of two connected blocks, first, the GS block
that selects the subset of channels/electrodes, followed by the GNN block that
produces a label as shown in Figure 1. The details of the GS block are given
next.

The Gumbel-Softmax EEG channel selection block was first proposed by
Strypsteen and Bertrand [29], where channel selection was acheived through a
learnable layer in the Deep Neural Network (DNN) in an end-to-end
differentiable manner. The Gumbel-softmax layer represents a relaxation of the
discrete selection operation that allows for differentiation [29] [13] [18].
Let $x_{n}$ indicate the feature vector derived from channel $n$, and
$x_{new_{i}}$ indicate the $i$th channel in the reduced set of channels.
During training, the output of each selection neuron $k$ is given by
$x_{new_{k}}=w_{k}^{T}X$, with $w_{k}$ sampled from the concrete distribution
given by [18]:

$\centering
w_{nk}=\frac{\exp((\log\alpha_{nk}+G_{nk})/\beta)}{\sum_{j=1}^{N}\exp((\log\alpha_{jk}+G_{jk})},\@add@centering$
(8)

with $G_{nk}$ independent and identically distributed samples from the Gumbel
distribution and $\beta\in(0,+\infty)$ the temperature parameter of the
concrete distribution. The effective subset of input node features is computed
as $X_{new}=w^{T}X$. The temperature parameter $\beta$ controls the extent of
this relaxation from the one-hot selection: as $\beta$ approaches 0, the
distribution becomes more discrete, the sampled weights converges to one-hot
vectors. The continuous relaxation allows $w$ to be jointly optimized with
model parameters, and to match the channel selection to the target model. The
most pertinent EEG channels are thereby selected without prior expert
knowledge or the need for manual feature selection. The learnable parameters
$\alpha$ of this distribution are jointly optimized with with the other
network weights. At the end of training, the selection layer is made to select
discrete channels by hard-thresholding the entries of $w_{k}$ so that they
select only $K$ channels as $\displaystyle w_{nk}=\begin{cases}1&\text{if
$n={\arg\max}_{j}\alpha^{*}_{jk}$}\\\ 0&\text{otherwise},\\\ \end{cases}$,
where $\alpha^{*}$ is the learned matrix after training. We note that during
test time, the GS block takes the form of a fixed linear matrix multiplication
$W$ that acts to select the electrode channels. We also note that unlike the
pre-selected case presented in Section 3.2, GS-DD model learns a data-driven
subgraph.

Figure 1: Proposed approach

In order to obtain a data-driven channel selection, we use the Gumbel softmax
channel selection block as part of our GNN pipeline shown in Figure 1. In
particular, we apply the GNN on the reduced subgraph obtained by selecting
only a subset of input EEG channel signals $X_{new}$ and that uses the
adjacency matrix $A_{new}$ corresponding to the selected channels. As
discussed above, the GS block is parameterized by a learnable matrix
$\alpha\in\mathbb{R}^{N\times K}$, where $N$ is the total number of
electrodes, and $K$ is the number of electrodes we wish to keep after
reduction. When being fed a sample $X$, the selection block sample a weight
matrix $W\in\mathbb{R}^{N\times K}$ from the concrete distribution following
Equation (8). This can be viewed as a softmax operation, which produces a
weight matrix whose elements summing to one as continuous relaxation of one-
hot vectors. In our experiments, we use a similar method as in paper [29].
During the training, we set $\beta(t)=\beta_{s}(\beta_{e}/\beta_{s})^{B}$,
decreasing in an exponential manner where $B$ is the total number of training
epochs. In particular, $\beta(t)$ is the temperature parameter at epoch $t$,
$\beta_{s}$ and $\beta_{s}$ are respectively the starting and ending $\beta$.
In our settings, $\beta_{s}=100$, $\beta_{s}=0.001$. As we noted before, while
the complete set of electrodes is indeed used during training of the student
GNN, this is not the case during test time as the $W$ matrix will be set to
ones and zeros, thereby not requiring any measurements from the non-selected
electrodes.

Channel consolidation We note that, though we force the weight matrix to
select a reduced set of channels, it is possible that a given channel is
chosen multiple times since we have not actively enforced that there is no
duplication. In order to discourage duplicate channels, we minimize the total
loss regularized with the penalty given by [29]:
$\displaystyle\Omega(P)=\lambda\sum_{n=1}^{N}\mathrm{ReLU}(\sum_{k=1}^{K}p_{nk}-\tau)$,
where $\mathrm{ReLU}(\cdot)$ is the rectified linear unit, $\lambda$ is the
weight of the regularization loss, and $\tau$ the threshold parameter. During
training, we set $\tau(t)=\tau_{s}(\tau_{e}/\tau_{s})^{B}$, decreasing in an
exponential manner. In our settings, $\tau_{s}=3$, $\tau_{s}=1.1$. $\lambda$
is set to be $5$ to control the strength of the regularization.

Then, we learn the GS-DD model with the regularized student loss, to obtain a
seizure detection model that is global and applicable to any patient.

###  3.4 PS-DD model

Epileptic seizures vary significantly between individuals and personalized
models could be beneficial in taking into account their unique patterns and
characteristics. This motivates us to extend our previous model to a
personalized setting to for simultaneous electrode reduction and seizure
detection for every single patient. As with the GS-PS and GS-DD models
proposed in Sections 3.2 and 3.3, our aim here is to arrive at light-weight
models for seizure detection that use only a subset of electrode channels
using KD, but personalized to the patient. As with GS-DD model, we let the
channels be selected in a data driven manner. Our hypothesis is that both
knowledge-distillation and personalized models have an important role to play
in improving the seizure detection performance, particularly in the cases when
the available data is scarce. The PS-DD model is in its essence the same as
the GS-DD model in the architecture, with the crucial difference that the
model is now trained in a patient-specific manner. This means that the PS-DD
model also learns a data-driven subgraph for every patient.

##  4 Numerical Experiments

###  4.1 Settings

Dataset We apply our models for the task of seizure detection on the data from
the Temple University Hospital EEG Seizure Data Corpus (TUSZ) [20], which is
one of the most popularly used, and diverse datasets that consists of over 100
patients of a wide range of ages ($8$-$80$) with different seizure types,
e.g., focal seizure, tonic seizure, generalized non-specific seizure, and
myoclonic seizure for long durations. The data is in the form of 19 channel
EEG recordings in the 10-20 electrode placement system. As our work deals with
the problem of seizure detection, no distinction is made between seizure types
and all seizures were grouped into one class, resulting in a binary
classification problem. The selected seizure (ictal) segments ranged between 5
and 60 seconds in length. Corresponding interictal segments of the same length
were selected that ended one minute before seizure onset, following the
methodolgy pursued in [9]. This resulted in a balanced dataset of 50% seizures
and 50% nonseizure segments. The segments are taken sequentially without
overlap. Every segment is then divided into windows of 5 seconds for both the
classes. All selected segments were then split into five-second windows. The
TUH dataset has two separate sets of recordings for train and for dev, which
correspond to different set of patients for training and test, respectively.
Similarly to the literature, we use only the patients from train for training
models, and the test patients from dev for testing the learnt models on which
the performance is reported. Finally, we have a total of 14382 samples for
training and 4529 samples for testing, each sample being a 5 second window of
multi-channel EEG signal.

Data preprocessing As customary in EEG signal processing, each sample is then
filtered with a Butterworth bandpass filter of order 5 between 0.5 and 50 Hz
to remove the artifacts and noise. Similarly to [25], the features were
calculated for each EEG channel: energy of the signal filtered in frequency
bands (from 0.5 Hz to 30 Hz with a bandwidth of 3 Hz and from 30 Hz to 50 Hz
with a bandwidth of 10 Hz), Hjorth complexity and mobility, decorrelation
time, L2-norm of the approximation and detail coefficients obtained from a
six-level wavelet decomposition using a Daubechies db4 wavelet, log amplitudes
of the non-negative frequency components. This results in 647 features in
total for each sample/window. The features are then normalized component-wise
and taken as input $\mathbf{x}$ to the GNN along with the distance based
adjacency matrix.

Training data In order to train the teacher, no distinction is made between
patients or segments and the entire training data is used to train the
teacher. All the samples from all the test patients are used as test data. For
the training the global models of GS-PS, and GS-DD, we use the data of all
training patients during training and data from all test patients for testing.
On the other hand, since the PS-DD model is trained for each patient
separately, the training and test data segments are obtained by splitting the
segments of the given patient randomly. Further, in order to understand the
effect of personalization, we divide the patients into three bands based on
the amount of data segments they possess as shown in Table 1.

Table 1: Three bands of patients. Data Bands |  Number of Segments N |  Batch Size |  Epoch |
---|---|---|---|---
Rare Data | $4\leq N<20$ | 2 | 20 |
Mid Data | $20\leq N<100$ | 16 | 100 |
Rich Data | $N\geq 100$ | 64 | 100 |

Model Training We use a two-layer GCN network with $32$ hidden nodes in each
hidden layer as the teacher model. It is trained with a batch size of 64 and a
learning rate of $10^{-5}$. The student network in all the three cases of GS-
PS, GS-DD, and PS-DD, is a light-weight model with just one-layer GCN of only
1 hidden node. We note that the number of parameters to learn in the student
is just $3\%$ of that of the teacher. Each of the three models is trained and
tested both with and without KD in order to determine the contribution of the
teacher knowledge. As described in Equation (6), the KL divergence loss is
used as the distillation loss function and the binary cross-entropy loss is
used for the student loss function. The hyperparameters in the total loss are
obtained by performing $5$-fold cross-validation. We set $T=5$, and $\delta$
values are set to $0.1,0.5,0.8$. For GS-DD, we consider the case of $K=4$
channels to compare the performance with that of GS-PS using the four temporal
channels. For the PS-DD model, we use $K=2$ electrodes for every patient.

Evaluation metrics Following [5, 2], we evaluate the performance of the three
models using two standard metrics: f1-score and the Area Under the curve of
the Receiver Operating Characteristic (AUROC), which are used regularly in
evaluating seizure detection. In all the cases, the performance is averaged
over the different test patients.

###  4.2 Detection Performance results

We now report the performance of the different approaches.

GS-PS model The performance of the teacher and the global student with the
pre-selected temporal channels is presented in Table 2. In the pre-selected
student, we observe that KD significantly improves the performance in terms of
the f1-score that tends to be comparable to that of the teacher.

GS-DD model Unlike in the temporal channel pre-selection case, we see that the
performance remains relatively constant to the different levels of KD. This is
probably because the GS selection already results in a high performance, and
the teacher does not offer notable improvement.

PS-DD model In the case of a personalized student GNN with only two electrodes
(that we call PS-DD 2), we observe that the performance improves as $\delta$
is increased, meaning more emphasis is given to the patient’s data over the
teacher’s knowledge, with the highest performance obtained at $\delta=0.8$. On
the other hand, we also observe that completely relying on the patient’s data
and not using the teacher ($\delta=1$) reduces the performance. Further, we
note that the performance of the student even without teacher’s knowledge
($\delta=1$) is generally much better than that of the teacher or the global
student. This in turn supports our intuition and hypothesis that
personalization also plays a significant role in improving seizure detection
performance. In the two plots in Figure 3, we depict the distributions of test
f1 and AUROC of all test patients in the circumstances with or without KD,
respectively for the PS-DD model. The averaged performances are indicated in
numbers in the figures. The dashed red/green lines show the general
performances of models without personalization. When trained on the general
population, we obtain the test f1 of models with and without KD as $0.7$ and
$0.4$, respectively. Whereas after personalization, the average test f1 are
improved by $16\%$ and $50\%$ to around $0.8$, corresponding to with and
without KD, respectively. This shows that by tackling the diversity in EEG
seizure data on a large population, personalization has the potential to
improve seizure detection. The average test AUROC is improved by $8\%$ to
above $0.8$. The detailed results are reported in Table 2. However, the
average performance with KD is only slightly higher than the average
performance without KD in both the metrics. This in turn motivated us to look
into the performance in the three data bands individually next.

Table 2: Test Results with Different Models

Model w/o KD w/ KD Channel Personalization – $\delta=0.1$ $\delta=0.5$
$\delta=0.8$ Selection f1 auroc f1 auroc f1 auroc f1 auroc Teacher – 0.689
0.781 – – – – – – GS-PS $\times$ 0.401 0.755 0.683 0.766 – – – – GS-DD
$\times$ 0.690 0.763 0.695 0.761 0.697 0.763 0.693 0.764 PS-DD $\checkmark$
0.788 0.814 0.755 0.777 0.784 0.829 0.795 0.829

Figure 2: The effectiveness of personalization: With personalization (P), both
test f1 and test rocauc are significantly improved on average.

Figure 3: The effectiveness of Knowledge Distillation (KD) in different bands

###  4.3 Performance analysis

To better understand the effectiveness of our models, we do a detailed
performance analysis by further dividing patients into three bands based on
the number of seizure segments (rare-data band, mid-data band, rich-data band)
and delve into the performances, respectively as shown in Table 1. In Table 3,
we report the seizure detection results when the model training relies
differently on the new patient data to different levels given by $\delta=0.1$,
$\delta=0.5$ and $\delta=.8$, respectively, in (6). The setting of
$\delta=0.8$ corresponds to the case where the student training relies more
heavily on unseen patient-specific data than the teacher. Figure 3 shows the
differences in the percentages of cases in each band where KD boosted the
model performance (in terms of test f1 and test AUROC). Overall, KD helps
$72\%$ (47 out of 65) patients in the rare-data band improve their model
testing performances. But only $49\%$ (26 out of 53) patients in the mid-data
band and $54\%$ (18 out of 33) patients benefit from the teacher. In general,
we observe the tendency that patients with scarce data benefit the most from
KD. This gives us the motivation to further delve into the rare-data band
case.

In the rare-data band, we notice that we constantly encounter four patients
with the lowest performance that bias the overall performance significantly.
It turns out that these four cases correspond to the patients with the least
training data. We refer to these cases as the four ”extremes” in our
experiments. Since the TUSZ dataset is rather diverse and we wish to see the
averaged performance without a strong bias, we chose to exclude the extremes
out and recompute the performance metrics. We notice from Table 4, that the
performance improves overall by excluding the extremes, and the best
performance is obtained when $\delta=0.8$. This indicates that the
effectiveness of KD in personalized settings widely varies with the amount of
data each patient possesses, and potentially across the patient types (since
the dataset includes different types of seizures that we do not currently
account for) and also varies with the change of the weight of student loss
$\delta$. In our experiments, $\delta$ is 0.8 gave the best scores on an
average. A more exhaustive approach would be to compute personalized models
with personalized $\delta$, but that is beyond the scope of the current work.

Table 3: PS-DD Test Results on Different Bands

w/o KD w/ KD Data Bands Personalization – $\delta=0.1$ $\delta=0.5$
$\delta=0.8$ f1 auroc f1 auroc f1 auroc f1 auroc Rare Data $\checkmark$ 0.786
0.791 0.783 0.783 $0.790^{*}$ $0.816^{*}$ 0.798* 0.827* Mid Data $\checkmark$
0.791 0.837 0.726 0.756 0.774 0.819 0.786 0.833 Rich Data $\checkmark$ 0.790
0.821 0.749 0.800 0.786 0.829 0.801* $0.828^{*}$

Figure 4: PS-DD Test Results ($\delta=0.8$) on Rare-Data Band. The right
column shows the results with 4 patients with extremely sparse data and poor
performances excluded (Patient ID: ”00005672”, ”00008706”, ”00006535”,
00004596”)
Table 4: PS-DD Test Results on Rare-Data Bands Model | Extremes $\times$ | Extremes $\checkmark$
---|---|---
$\delta$ | Personalization | f1 | auroc | f1 | auroc
0.1 | $\checkmark$ | 0.794 | 0.806 | 0.783 | 0.783
0.5 | $\checkmark$ | 0.813 | 0.860 | 0.790 | 0.816
0.8 | $\checkmark$ | 0.832 | 0.871 | 0.798 | 0.827
1 (no KD) | $\checkmark$ | 0.816 | 0.833 | 0.786 | 0.791

Effectiveness of Knowledge Distillation when lacking informative
channels/signals To further test the effectiveness of both personalization and
KD, we select to keep we arbitrarily select to keep only signals from channels
FP1, FP2 that belong to the frontal region, which are suggested to be the less
informative region for epileptic seizure detection. The Gumbel-Softmax channel
selection block is not involved in this section. The experiment is conducted
on the rare data band, with the hypothesis that the combination of
personalization and KD can help compensate for the adverse situation brought
by a) lack of data, and b) lack of informative channels. With only
personalization but no KD, 53.8% (35 out of 65) patients’ test f1 and AUROC
score still exceed 0.65, yielding fairly good performances. In the rest of not
the ideal personalized situations, 90% (27 out of 30 patients) benefit from
the teacher. With even the alleged least informative channels, we get 53.8% of
the cases with rather promising results. For the rest of the cases, the
integrated application of personalization and KD has been observed to be
effective for detecting epileptic seizures. We thus see that the combination
leverages the strengths of both techniques to provide highly accurate results
in scarce data scenarios.

Hierarchical Clustering of Patients We now investigate if the different
patients naturally show clusters when the learnt electrode channels are used
to cluster the patients. We use hierarchical clustering on the learnt
selection matrices $W$. Hierarchical clustering is a method of cluster
analysis that builds a hierarchy of clusters by successively splitting or
merging smaller clusters based on the Euclidean distance between clusters. The
result of hierarchical clustering is shown as a dendrogram that shows the
hierarchy of clusters in Figure 5. We observe that there are no clearly
significant clusters emerging except for a large cluster and outliers, which
could be because the patients and seizure signals in the TUSZ dataset are
quite diverse. We also note that we have made no distinction between seizure
types (about 6 of widely varying number of samples per type) in our analysis
which might explain the single big cluster. While some of the outliers
corresponded to patients with rare disease (Rasmussens’ syndrome), it is
unclear if the outliers show specific signature characteristics that separate
them clinically from the main cluster. Further, we see that the main cluster
diameter is relatively large indicating that there is significant variability
in the selected channels across the different patients. In future work, we
plan to pursue alternative clustering strategies and features and also
mitigating the diversity by, for example, filtering out only the focal seizure
signals.

Figure 5: Hierarchical clustering of patients based on their Gumbel-Softmax
channel selection patterns

##  5 Conclusions and future work

We proposed an approach to transfer the knowledge from a pre-trained GNN-based
seizure detection to the case when the number of measurement electrodes is
reduced. We showed that it is possible to obtain models that are (i) light-
weight (requiring just a $3\%$ of the sophisticated network), and (ii) work on
reduced electrodes (requiring as low as only $10\%$ of the original
electrodes), yet offer superior performance in seizure-detection, particularly
in the personalized setting. The approach resulted in patient-specific choice
of the reduced set of electrodes. Our experiments demonstrated the merit of
both knowledge distillation and personalization, particularly when dealing
with patients with scarce data. We observe that there is a trade-off between
the use of prior information (teacher) and patient-specific data: although
teacher-knowledge is necessary, the relative importance should be higher on
the patient-specific data for maximum performance. We believe that these
results show that our approach can provide meaningful insights and guidelines
in the practical setting where there is need to move from full scalp electrode
measurements to reduced form factor measurements, such as personalized
wearable devices. We have currently restricted our analysis to a relatively
simple GNN teacher model and used the graph given by physical placement of
electrodes. The quality of the teacher and the graph used both translate into
the quality of the student model, and hence, we believe that a more
sophisticated GNN could be employed to further improve overall performance. In
the future, it would also be interesting to look into multi-class seizure
classification and identify the different types of seizures.

## References

  * [1] Ahmedt-Aristizabal, D., Fernando, T., Denman, S., Petersson, L., Aburn, M.J., Fookes, C.: Neural memory networks for seizure type classification. In: 2020 IEEE Engineering in Medicine & Biology Society (EMBC). pp. 569–575. IEEE (2020)
  * [2] Asif, U., Roy, S., Tang, J., Harrer, S.: Seizurenet: Multi-spectral deep feature learning for seizure type classification. In: Kia, S.M., Mohy-ud Din, H., Abdulkadir, A., Bass, C., Habes, M., Rondina, J.M., Tax, C., Wang, H., Wolfers, T., Rathore, S., Ingalhalikar, M. (eds.) Machine Learning in Clinical Neuroimaging and Radiogenomics in Neuro-oncology. pp. 77–87. Springer International Publishing, Cham (2020)
  * [3] Blom, J.L., Anneveldt, M.: An Electrode Cap Tested. Electroencephalography and clinical neurophysiology 54(5), 591–4 (1982)
  * [4] Chen, Y., Bian, Y., Xiao, X., Rong, Y., Xu, T., Huang, J.: On self-distilling graph neural network. CoRR abs/2011.02255 (2020), https://arxiv.org/abs/2011.02255
  * [5] Covert, I.C., Krishnan, B., Najm, I., Zhan, J., Shore, M., Hixson, J., Po, M.J.: Temporal graph convolutional networks for automatic seizure detection. In: Machine Learning for Healthcare Conference. pp. 160–180. PMLR (2019)
  * [6] Deng, X., Zhang, Z.: Graph-free knowledge distillation for graph neural networks. In: Zhou, Z. (ed.) Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021. pp. 2321–2327. ijcai.org (2021). https://doi.org/10.24963/ijcai.2021/320, https://doi.org/10.24963/ijcai.2021/320
  * [7] Feng, K., Li, C., Yuan, Y., Wang, G.: Freekd: Free-direction knowledge distillation for graph neural networks. In: Zhang, A., Rangwala, H. (eds.) KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 - 18, 2022. pp. 357–366. ACM (2022). https://doi.org/10.1145/3534678.3539320, https://doi.org/10.1145/3534678.3539320
  * [8] Fürbass, F., et al.: Automatic Multimodal Detection for Long-Term Seizure Documentation in Epilepsy. Clinical Neurophysiology 128(8), 1466–1472 (2017). https://doi.org/https://doi.org/10.1016/j.clinph.2017.05.013, https://www.sciencedirect.com/science/article/pii/S1388245717301980
  * [9] Gabeff, V., Teijeiro, T., Zapater, M., Cammoun, L., Rheims, S., Ryvlin, P., Atienza, D.: Interpreting deep learning models for epileptic seizure detection on EEG signals. Artif. Intell. Medicine 117, 102084 (2021). https://doi.org/10.1016/j.artmed.2021.102084, https://doi.org/10.1016/j.artmed.2021.102084
  * [10] Gou, J., Yu, B., Maybank, S.J., Tao, D.: Knowledge distillation: A survey. International Journal of Computer Vision 129, 1789–1819 (2021)
  * [11] Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
  * [12] Iešmantas, T., Alzbutas, R.: Convolutional neural network for detection and classification of seizures in clinical data. Medical & Biological Engineering & Computing 58(9), 1919–1932 (2020)
  * [13] Jang, E., Gu, S., Poole, B.: Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 (2016)
  * [14] Jasper, H.H.: The ten-twenty electrode system of the international federation. Electroencephalogr. Clin. Neurophysiol. 10, 370–375 (1958)
  * [15] Joshi, C.K., Liu, F., Xun, X., Lin, J., Foo, C.: On representation knowledge distillation for graph neural networks. CoRR abs/2111.04964 (2021), https://arxiv.org/abs/2111.04964
  * [16] Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
  * [17] Kwan, P., Brodie, M.J.: Definition of Refractory Epilepsy: Defining the Indefinable? The Lancet Neurology 9(1), 27–29 (2010). https://doi.org/10.1016/S1474-4422(09)70304-7
  * [18] Maddison, C.J., Mnih, A., Teh, Y.W.: The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712 (2016)
  * [19] Maganti, R.K., Rutecki, P.: EEG and Epilepsy Monitoring. Continuum (Minneapolis, Minn.) 19(3), 598–622 (2013). https://doi.org/10.1212/01.CON.0000431378.51935.d8
  * [20] Obeid, I., Picone, J.: The Temple University Hospital EEG Data Corpus. Frontiers in Neuroscience 10 (2016). https://doi.org/10.3389/fnins.2016.00196
  * [21] Raghu, S., Sriraam, N., Temel, Y.e.a.: EEG based multi-class seizure type classification using convolutional neural network and transfer learning. Neural Networks 124, 202–212 (2020)
  * [22] Rahmani, A., Venkitaraman, A., Frossard, P.: A meta-gnn approach to personalized seizure detection and classification. CoRR abs/2211.02642 (2022). https://doi.org/10.48550/arXiv.2211.02642, https://doi.org/10.48550/arXiv.2211.02642
  * [23] Roy, S., Asif, U., Tang, J., Harrer, S.: Seizure type classification using eeg signals and machine learning: Setting a benchmark. In: 2020 IEEE Signal Processing in Medicine and Biology Symposium (SPMB). pp. 1–6. IEEE (2020)
  * [24] Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G.: The graph neural network model. IEEE transactions on neural networks 20(1), 61–80 (2008)
  * [25] Schiratti, J.B., Le Douget, J.E., Le Van Quyen, M., Essid, S., Gramfort, A.: An Ensemble Learning Approach to Detect Epileptic Seizures from Long Intracranial EEG Recordings. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 856–860 (2018). https://doi.org/10.1109/ICASSP.2018.8461489
  * [26] Shafer, M.P.O.: What Is Epilepsy? (2014), https://www.epilepsy.com/learn/about-epilepsy-basics
  * [27] Siddiqui, M.K., Morales-Menendez, R., Huang, X., Hussain, N.: A review of epileptic seizure detection using machine learning classifiers. Brain informatics 7(1), 1–18 (2020)
  * [28] Sopic, D., Aminifar, A., Atienza, D.: e-Glass: A Wearable System for Real-Time Detection of Epileptic Seizures. In: 2018 IEEE International Symposium on Circuits and Systems (ISCAS). pp. 1–5 (2018). https://doi.org/10.1109/ISCAS.2018.8351728
  * [29] Strypsteen, T., Bertrand, A.: End-to-end learnable eeg channel selection for deep neural networks with gumbel-softmax. Journal of Neural Engineering 18(4), 0460a9 (2021)
  * [30] Tang, S., Dunnmon, J.A., Saab, K., Zhang, X., Huang, Q., Dubost, F., Rubin, D., Lee-Messer, C.: Self-Supervised Graph Neural Networks for Improved Electroencephalographic Seizure Analysis. In: Proceedings on the International Conference on Learning Representations (2022)
  * [31] Yang, Y., Qiu, J., Song, M., Tao, D., Wang, X.: Distilling Knowledge from Graph Convolutional Networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7074–7083 (2020)
  * [32] Zhang, C., Liu, J., Dang, K., Zhang, W.: Multi-Scale Distillation from Multiple Graph Neural Networks. Proceedings of the AAAI Conference on Artificial Intelligence 36(4), 4337–4344 (2022). https://doi.org/10.1609/aaai.v36i4.20354, https://ojs.aaai.org/index.php/AAAI/article/view/20354
  * [33] Zhou, S., et al.: Distilling Holistic Knowledge with Graph Neural Networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10387–10396 (2021)