# Data-Driven Symbol Detection for Intersymbol Interference Channels with
Bursty Impulsive Noise

Boris Karanov, Chin-Hung Chen, Yan Wu, Alex Young,
and Wim van Houtum Manuscript received xx, xxxx; revised xx, xxxx.
(Corresponding author: Boris Karanov)B. Karanov, C.-H. Chen and W. v. Houtum
are with the Signal Processing Systems (SPS) Group, Electrical Engineering
Department, Eindhoven University of Technology, 5600MB Eindhoven, The
Netherlands. All authors are also with NXP Semiconductors, High Tech Campus
46, 5656AE Eindhoven, The Netherlands. (e-mails<EMAIL_ADDRESS><EMAIL_ADDRESS>yan.wu<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We developed machine learning approaches for data-driven trellis-based soft
symbol detection in coded transmission over intersymbol interference (ISI)
channels in presence of bursty impulsive noise (IN), for example encountered
in wireless digital broadcasting systems and vehicular communications. This
enabled us to obtain optimized detectors based on the Bahl-Cocke-Jelinek-Raviv
(BCJR) algorithm while circumventing the use of full channel state information
(CSI) for computing likelihoods and trellis state transition probabilities.
First, we extended the application of the neural network (NN)-aided BCJR,
recently proposed for ISI channels with additive white Gaussian noise (AWGN).
Our investigation highlighted that, although suitable for estimating
likelihoods via labeling of transmission sequences, the BCJR-NN method does
not provide a framework for learning the trellis state transitions. In
addition to detection over the joint ISI and IN states we also focused on
another scenario where trellis transitions are not trivial: detection for the
ISI channel with AWGN with inaccurate knowledge of the channel memory at the
receiver. Our results showed that, without access to the accurate state
transition matrix, the BCJR-NN performance significantly degrades in both
settings. To this end, we devised an alternative approach for data-driven BCJR
detection based on the unsupervised learning of a hidden Markov model (HMM).
The BCJR-HMM allowed us to optimize both the likelihood function and the state
transition matrix without labeling. Furthermore, we demonstrated the viability
of a hybrid NN and HMM BCJR detection where NN is used for learning the
likelihoods, while the state transitions are optimized via HMM. While reducing
the required prior channel knowledge, the examined data-driven detectors with
learned trellis state transitions achieve bit error rates close to the optimal
full CSI-based BCJR, significantly outperforming detection with inaccurate
CSI.

###### Index Terms:

Electromagnetic interference, estimation, intersymbol interference, machine
learning, neural networks, signal detection, signal processing.

##  I INTRODUCTION

Machine learning (ML) is revolutionizing various aspects in the design of the
next-generation communication systems by enabling powerful tools for data-
driven optimization. A particularly appealing application area, subject to
extensive research in recent years, is the development of advanced ML-based
digital signal processing (DSP) algorithms for communication scenarios where
the optimal transceiver DSP is not available, for e.g., due to the lack of
sufficient channel knowledge and/or implementation complexity constraints [1].
More specifically, the vast majority of recent approaches exploit the
universal function approximation capabilities of neural networks (NN) and deep
learning (DL) [2, 3]. For example, end-to-end DL methods for the joint
optimization of the DSP chain via a fully NN-based transceiver has been
recently considered and experimentally verified for both wireless [4, 5] and
optical fiber communications [6, 7, 8]. Nevertheless, continuous end-to-end
learning on an actual transmission link requires advanced solutions for
optimization of the NN transmitter which introduces substantial complexity
overhead [9, 10]. The most common application of NNs and DL for signal
processing in communication systems is for the data-driven optimization of a
specific receiver DSP function, e.g., for equalization, detection or decoding
[11, 12, 13]. Training of an NN at the receiver side does not require the
backpropagation of gradients through the physical channel and is thus a
relatively less complex task which typically involves supervised learning via
labeled transmission sequences. For instance, the authors of [12] proposed an
advanced bidirectional recurrent NN (BRNN) sequence detection scheme which
performs close to the optimal maximum likelihood sequence detector. Notably,
to alleviate the computational demands of employing a fully neural network-
based DSP module, model-based deep learning approaches have been considered,
where a simple NN (e.g. a feedforward NN) augments a conventional signal
processing algorithm. For example an NN-aided digital fiber nonlinearity
compensation scheme was proposed in [11]. More recently, for the task of
symbol detection, NN-based approaches for data-driven implementation of the
trellis-based Viterbi [14] and Bahl-Cocke-Jelinek-Raviv (BCJR) [15] algorithms
were proposed [18, 17, 16]. Investigated for finite-state stationary
intersymbol interference (ISI) channels with additive white Gaussian noise
(AWGN), these detectors achieved error rate performances close to the optimal
full channel state information (CSI)-based BCJR and Viterbi processing.
Importantly, these receivers enabled learning of channel likelihoods directly
from labeled transmission sequences, circumventing the often impractical
requirement for perfect CSI. Moreover, the authors showed that the NN-based
BCJR and Viterbi symbol detectors present a superior data-driven alternative
in terms of training data utilization and computational complexity compared to
the state-of-the-art BRNN receiver.

In more detail, the optimal symbol detector based on the maximum a posteriori
probability (MAP) criterion can be realized via BCJR — an efficient message-
passing algorithm which uses channel likelihoods and state transition
probabilities to compute posterior beliefs. As proposed in [17], the NN-based
BCJR (denoted BCJR-NN in our paper) combines a small feedforward NN, trained
via received sequences labeled with the transmitted symbols, and a Gaussian
mixture density estimator to compute the channel likelihoods for the BCJR
algorithm. The scheme requires state transition probabilities to be provided
as additional domain knowledge or separately counted in the labeled data.

However, there are communication channel models where transitions are not
trivial and optimizing them in a data-driven manner is particularly suitable.
A practically-relevant example is symbol detection over a finite-state ISI
channel with AWGN where the symbol memory of the BCJR algorithm is greater
than the channel memory — a scenario that occurs in real-world systems with
over-estimated states for “worst-case” channel conditions. Another example is
the transmission over ISI channels affected by impulsive noise (IN). Such
noise can be found in powerline communications, digital broadcasting systems
as well as vehicular communications [19, 20, 21, 22, 23, 24]. A commonly used
assumption in such systems is the Middleton Class A noise model [25, 26]. More
recently, to account for the bursty behavior of the IN, the Markov-Middleton
noise model has been proposed [27]. Such a finite-state model exhibits a non-
trivial state transition matrix governed by impulsive noise parameters. The
performance of message-passing algorithms such as BCJR have been investigated
for optimal detection over the Markov-Middleton channel, showing significant
improvement over detection based on the AWGN assumption [27, 28]. Note that in
these cases full prior knowledge of the IN parameters is required for
obtaining the optimal detector.

Our work focused on the machine learning perspective of BCJR symbol detection
1) when knowledge of the channel memory is inaccurate, and 2) over joint ISI
and impulsive noise states. Developing data-driven solutions for these
communication scenarios is important because they yield a non-trivial state
transition matrix for the trellis-based detector which is often unavailable or
impractical to provide as prior knowledge. Moreover, labeling of the channel
states is also difficult, especially in the presence of IN whose source is
typically external to the system. This prompted us, in addition to extending
the application of the NN-based BCJR, to consider a fundamentally different
perspective by interpreting the learnable detector as a hidden Markov model
(HMM) [29]. In particular, we optimized the observation likelihood function
and the state transition matrix parameters of the HMM in an unsupervised
fashion using received symbol sequences and the Baum-Welch algorithm [30, 31].
These were then utilized directly for BCJR symbol detection. Expectation-
maximization approaches such as the Baum-Welch algorithm have previously been
used for learning likelihoods in ISI communication channels with AWGN for
example in [32] and more recently in [33] where the authors investigated joint
channel estimation and symbol detection schemes. Conversely, designing data-
driven detection for the applications of interest in our study puts a strong
emphasis on learning the trellis state transition probabilities. Furthermore,
we investigated the feasibility of a hybrid HMM-NN detection approach to BCJR
(BCJR-HMM-NN) where the NN-computed likelihoods are used together with an HMM-
optimized state transition matrix.

This paper is organized as follows: Sec. II gives a comprehensive overview of
HMMs — a perspective we adopt for interpreting the proposed data-driven BCJR
detector designs. Next, the communication channel model is presented in Sec.
III. The proposed machine learning-based detection schemes are described in
Sec. IV, while their numerical performance is reported in Sec. V. Section VI
concludes the paper.

##  II HMMs and Computing Posterior Beliefs via the Forward-Backward Algorithm

Figure 1:  HMM representation as an undirected graph showing the factorization
and conditional independencies implied by the model.

An HMM is a statistical model that captures the temporal behavior of a
sequence of latent variables (_states_) from a Markov process via a sequence
of state-dependent observable variables (_observations_) [29, 35]. To
highlight the conditional independencies and factorization implied by an HMM,
Fig. 1 shows its representation as an undirected graph. The model consists of
an observed sequence $\left(\ldots,y_{t-1},y_{t},y_{t+1}\ldots\right)$ where
at a given time $t$ an observation $y_{t}$ is produced by a state $s_{t}$. In
particular, for the state and observation sequences of length $T$, the HMM
represents the factorization

$\begin{split}p(s_{1},s_{2},\hbox
to10.00002pt{.\hss.\hss.},s_{T},y_{1},y_{2},\hbox
to10.00002pt{.\hss.\hss.},y_{T})=&p(s_{1})p(y_{1}|s_{1})\\\
&\times\prod_{t=2}^{T}p(s_{t}|s_{t-1})p(y_{t}|s_{t}).\end{split}$

We see that the probability of an observation at a given time depends only on
the state that produced it, such that

$p(y_{t}|s_{1},\hbox to10.00002pt{.\hss.\hss.},s_{t},\hbox
to10.00002pt{.\hss.\hss.},s_{T},y_{1},\hbox
to10.00002pt{.\hss.\hss.},y_{t},\hbox
to10.00002pt{.\hss.\hss.},y_{T})=p(y_{t}|s_{t}).$

Furthermore, the Markovianity of the underlying state generation process
dictates that the probability of being at state $s_{t}$ at time $t$ depends
only on the state at the previous time:

$p(s_{t}|s_{1},\hbox to10.00002pt{.\hss.\hss.},s_{t-1})=p(s_{t}|s_{t-1}).$

It is worth mentioning here that, for the purposes of the presented work, the
HMM is specified over a finite set of $Q$ possible discrete states
$\mathcal{S}=\\{s^{1},\ldots,s^{Q}\\}$, while the observation variables follow
a continuous distribution. Figure 2 shows a state diagram representation of an
HMM. The model consists of a _state transition matrix_ parameter

$\mathbf{P}_{\text{tr.}}=\begin{bmatrix}p(s^{1}|s^{1})&\ldots&p(s^{Q}|s^{1})\\\
\vdots&\ddots&\vdots\\\ p(s^{1}|s^{Q})&\ldots&p(s^{Q}|s^{Q})\end{bmatrix},$

where each element represents the probability $p(s^{j}|s^{i})$ of moving from
state $s^{i}$ to $s^{j}$ in one time step. The second HMM parameter is the
observation likelihood function $p(y|s^{j})$ describing the probability of an
observation being generated from a state $s^{j}$. Note that the
characterization of an HMM also includes a distribution $p(s)$ over the
initial states which can be obtained from the transmission model or learned
together with the other HMM parameters.

Figure 2:  State diagram representation of a 4-state HMM highlighting the
model parameters: the observation likelihood function and the state transition
matrix.

The HMM is a tree-structured graph and the sum-product message-passing
algorithm (belief propagation) can be used to perform efficient inference
[36]. In particular, the belief propagation variant specialized to the
structure of an HMM is called the _forward-backward algorithm_ [29, 35]. In
the following, we describe the procedure for obtaining the marginal
distribution for a latent state given the full sequence of observations, i.e.,
computing the posterior belief

$\begin{split}p(s_{t}|y_{1},y_{2},\hbox
to10.00002pt{.\hss.\hss.},y_{T})&\propto p(y_{1},y_{2},\hbox
to10.00002pt{.\hss.\hss.},y_{T},s_{t})\\\ &=\sum_{s_{t-1}}p(y_{1},y_{2},\hbox
to10.00002pt{.\hss.\hss.},y_{T},s_{t-1},s_{t}).\end{split}$ (1)

Using the chain rule of probability and the conditional independencies in the
HMM graphical model, we can write the joint distribution from the right-most
side of (1) as

$\begin{split}p(y_{1},y_{2},\hbox
to10.00002pt{.\hss.\hss.},y_{T},s_{t-1},s_{t})=&p(y_{1},y_{2},\hbox
to10.00002pt{.\hss.\hss.},y_{t-1},s_{t-1})\\\ &\times
p(s_{t}|s_{t-1})p(y_{t}|s_{t})\\\ &\times p(y_{t+1},y_{t+2},\hbox
to10.00002pt{.\hss.\hss.},y_{T}|s_{t}).\end{split}$ (2)

We first identify a _branch metric_ term defined as

$\gamma_{t}(y_{t},s_{t-1},s_{t})=p(s_{t}|s_{t-1})p(y_{t}|s_{t}).$ (3)

Moreover, (2) entails a _forward_ probability which, exploiting the
factorization associated with the graph structure, can be recursively updated
via messages $\alpha$ defined as

$\begin{split}\alpha_{t}(s_{t})&=p(y_{1},y_{2},\hbox
to10.00002pt{.\hss.\hss.},y_{t},s_{t})\\\
&=\sum_{s_{t-1}}\alpha_{t-1}(s_{t-1})\gamma_{t}(y_{t},s_{t-1},s_{t}),\end{split}$
(4)

for $\forall t\\!=\\!\\{2,\hbox to10.00002pt{.\hss.\hss.},T\\}$, where
$\alpha_{1}(s_{1})=p(y_{1},s_{1})$. Correspondingly, _backward_ probability is
computed via messages $\beta$ as

$\begin{split}\beta_{t}(s_{t})&=p(y_{t+1},y_{t+2}\hbox
to10.00002pt{.\hss.\hss.},y_{T}|s_{t})\\\
&=\sum_{s_{t+1}}\gamma_{t}(y_{t+1},s_{t},s_{t+1})\beta_{t+1}(s_{t+1}),\end{split}$
(5)

for $\forall t=\\{1,\hbox to10.00002pt{.\hss.\hss.},T-1\\}$ with
$\beta_{T}(s_{T})=1$. Substituting (3), (4) and (5) in (2), we can now write
the joint distribution of interest as the product of the forward and backward
probabilities and the branch metric:

$p(y_{1},y_{2},\hbox
to10.00002pt{.\hss.\hss.},y_{T},s_{t-1},s_{t})\\!=\\!\alpha_{t-1}(s_{t-1})\gamma_{t}(y_{t},s_{t-1},s_{t})\beta_{t}(s_{t}).$
(6)

To illustrate this message-passing process, Fig. 3 shows a trellis diagram
representation of an HMM. From a communication system perspective this
“$\alpha$-$\beta$-$\gamma$” forward-backward method for computation of the
posterior beliefs results in the well-known _BCJR algorithm_ [15] which we
used for maximum a-posteriori (MAP) symbol detection as further explained in
Sec. IV.

Figure 3:  Trellis diagram representation of a 4-state HMM illustrating the
message passing mechanism in the forward-backward algorithm.

##  III Communication Channel

In this paper we consider a wireless communication channel subject to
impulsive interference which can be found for example in digital broadcasting
systems and vehicular communications [20, 22, 23, 24]. The channel we consider
is characterized by the presence of inter-symbol interference (ISI) due to the
multipath signal propagation and bursty impulsive noise whose power fluctuates
in time. In particular, the input-to-output relationship for the channel is
given by

$y_{t}=\sum_{l=1}^{L}h_{l}x_{t-l+1}+z_{t},$ (7)

where $x_{t}\in\mathcal{X}$ is the transmitted symbol at time
$t\in\\{1,2,\ldots T\\}$ from a constellation set $\mathcal{X}$, with $T$
being the transmission frame length. The channel output at time $t$ is denoted
by $y_{t}$. The ISI response is given by the set of taps
$\\{h_{1},...,h_{L}\\}$ with $L$ being the finite ISI memory. We assume an
additive interference component which is independent of the transmitted
symbols and denoted by $z_{t}$.

###  III-A Intersymbol Interference

We adopt the ISI model from [17, 18, 34] where the application of data-driven
NN-based BCJR symbol detection has been investigated. In this model the
channel coefficients follow an exponentially decaying profile and, as in [34],
are normalized to unit power, i.e.,

$h_{l}=\frac{e^{-\eta(l-1)}}{\sqrt{\sum_{l=1}^{L}|e^{-\eta(l-1)}|^{2}}},\qquad
l={1,...,L},$

where $\eta$ determines the decay rate. From (7) we can identify
$|\mathcal{X}|^{L}$ possible ISI states
$\mathbf{s}^{\dagger}_{t}\triangleq\left(x_{t},x_{t-1},\hbox
to10.00002pt{.\hss.\hss.},x_{t-L+1}\right)$. ISI state transitions are driven
by the transmitted symbols, which are assumed equiprobable, i.e., symbol
$x_{t}$ is transmitted at time $t$ with probability $\frac{1}{|\mathcal{X}|}$.
Therefore, the probability of being at state $\mathbf{s}^{\dagger}_{t}$ at
time $t$ depends only on the previous state $\mathbf{s}^{\dagger}_{t-1}$ as

$\begin{split}&p(\mathbf{s}^{\dagger}_{t}|\mathbf{s}^{\dagger}_{t-1})=\begin{cases}\frac{1}{|\mathcal{X}|},&(\mathbf{s}^{\dagger}_{t},\mathbf{s}^{\dagger}_{t-1})\in\mathcal{S}^{\dagger}\\\
0,&\text{otherwise}.\end{cases}\end{split}$ (8)

Here $\mathcal{S}^{\dagger}$ denotes the pairs
$(\mathbf{s}^{\dagger}_{t},\mathbf{s}^{\dagger}_{t-1})$ corresponding to all
state transitions driven by the transmitted symbol $x_{t}$. Thus, in the
absence of other effects, the ISI-affected symbol for each state becomes

$\mu_{\mathbf{s}^{\dagger}_{t}}=\sum_{l=1}^{L}h_{l}x_{t-l+1},$ (9)

This is a finite-state time-invariant model of the ISI. In [17, 18], the
authors extended this model to account for channel taps with random variations
in time. This was done by introducing a Gaussian-distributed time-varying
distortion to $h_{l}$ with zero mean and variance $\sigma_{h}^{2}$ in presence
of which the ISI-affected symbol can be expressed as

$\tilde{\mu}_{\mathbf{s}^{\dagger}_{t}}=\sum_{l=1}^{L}\left(h_{l}+\epsilon_{l,t}\right)x_{t-l+1},$
(10)

where $\epsilon_{l,t}\sim\mathcal{N}(0,\sigma_{h}^{2})$. In [17] such a model
for the received symbols was assumed during the training stage of the NN-based
BCJR detector as well as for likelihood computations in the conventional BCJR
detection and it was referred to as _CSI uncertainty_. In [34] further
investigation on the performance of the BCJR detection scheme for rapidly
time-varying ISI channels with AWGN was carried out. In our work we also
considered such an ISI scenario in the presence of impulsive interference.

###  III-B Impulsive Interference

We model the behavior of the interference $z_{t}$ using the Markov-Middleton
impulsive noise model [27] which allows us to account for the bursty
(correlated) nature of the impulsive noise events. Recently, the impulsive
behavior of electric vehicle (EV)-induced interference, for e.g., originating
from an EV power converter or a switch, in the digital audio broadcasting
(DAB) band was investigated on a real world test-bed, showing a good agreement
with the Markov-Middleton noise behavior [37]. In particular, under the
Markov-Middleton model the zero-mean noise sample $z_{t}$ is distributed as

$p(z_{t})=\sum_{j=0}^{N-1}\frac{p_{j}}{\sqrt{2\pi\sigma^{2}_{j}}}\exp\left(-\frac{z_{t}^{2}}{2\mathbf{\sigma}^{2}_{j}}\right),$

where

$p_{j}=\frac{e^{-A}A^{j}/j!}{\sum_{j=0}^{N-1}e^{-A}A^{j}/j!},\hskip
5.69046pt\text{and}\hskip
5.69046pt\sigma^{2}_{j}=\sigma^{2}\frac{j/A+\Gamma}{1+\Gamma},$ (11)

with $N$ denoting the number of noise levels. Here we associate with an
interference state $\tilde{s}^{j}$ the variance $\sigma^{2}_{\tilde{s}^{j}}$
for the $j$-th noise level given by the right-hand side of (11). The model
parameters $A$, $\Gamma$, and $\sigma^{2}$ represent the impulsive index, the
background-to-impulsive noise ratio and the total power of the interference
$z_{k}$, respectively. Note that since we considered the BPSK modulation
format with constellation points $\mathcal{X}=\\{-1,1\\}$ and normalized the
ISI channel taps to unit power, the signal-to-interference-and-noise ratio at
the receiver becomes $\text{SINR}=1/\sigma^{2}$. Also note that
$\tilde{s}^{0}$ corresponds to background AWGN. When $N=1$ there is only
background noise present in the system and thus we use the signal-to-noise
ratio (SNR) metric. Furthermore, the Markov-Middleton impulsive model involves
a parameter $r\in[0,1]$ that establishes correlation between noise samples. In
particular, the probability of being in interference state $\tilde{s}_{t}^{j}$
at time $t$ depends only on the previous state $\tilde{s}_{t-1}^{i}$ with
transitions governed by

$\begin{split}&p(\tilde{s}_{t}^{j}|\tilde{s}_{t-1}^{i})=\begin{cases}r+(1-r)p_{j},&\tilde{s}^{i}=\tilde{s}^{j}\\\
(1-r)p_{j},&\text{otherwise}.\end{cases}\end{split}$ (12)

##  IV Receiver Design

In this section we describe the proposed methods for data-driven maximum a
posteriori probability (MAP) symbol detection for the ISI channel with bursty
IN. The MAP symbol detector is based on the BCJR algorithm and we investigate
both NN and HMM approaches for its machine learning-based implementation. We
first start with the description of the BCJR algorithm for optimal inference
over the joint ISI and IN channel states. Using our notation from the
definition of the channel in Sec. III we identify a communication system state
as the pair of the ISI process state and the impulsive noise process state at
a given time instance $t$, i.e,
$\mathbf{s}_{t}\triangleq\left(\mathbf{s}^{\dagger}_{t},\tilde{s}_{t}\right)$.
Note that since the ISI and impulse noise processes are assumed independent we
obtain a set $\mathcal{S}$ of system states which has the cardinality of
$|\mathcal{S}|=Q=N\cdot|\mathcal{X}|^{L}$, where $N$ is the number of noise
levels, $|\mathcal{X}|$ is the cardinality of the set of modulation symbols
and $L$ is the ISI symbol memory. Based on the channel model from (7), we can
factorize the joint distribution of the sequences of system states and
received symbols as

$\begin{split}p(\mathbf{s}_{1},\mathbf{s}_{2},\hbox
to10.00002pt{.\hss.\hss.},\mathbf{s}_{T},y_{1},y_{2},&\hbox
to10.00002pt{.\hss.\hss.},y_{T})=p(\mathbf{s}_{1})|p(y_{1}|\mathbf{s}_{1})\\\
&\times\prod_{t=2}^{T}p(\mathbf{s}_{t}|\mathbf{s}_{t-1})p(y_{t}|\mathbf{s}_{t}),\end{split}$
(13)

where for the ease of notation we assumed that the received symbol at $t=1$
has a full ISI state. This can be ensured via a guard band of $L-1$
transmitted symbols at the beginning of the transmission frame. Going further,
we can express the likelihood of an output symbol given a channel state by

$p(y_{t}|\mathbf{s}_{t})=\frac{1}{\sqrt{2\pi\sigma^{2}_{\mathbf{s}_{t}}}}\exp\left(-\frac{\left(y_{t}-\mu_{\mathbf{s}_{t}}\right)^{2}}{2\sigma^{2}_{\mathbf{s}_{t}}}\right).$
(14)

We note that for the considered channel model the received symbol follows a
Gaussian distribution
$y_{t}\sim\mathcal{N}(\mu_{\mathbf{s}_{t}},\sigma^{2}_{\mathbf{s}_{t}})$,
whose mean and variance parameters are determined by the ISI and the impulsive
interference states, i.e., (9) and (11), respectively.

Figure 4:  Schematic of the communication system using channel model-based or
data-driven symbol detector based on the forward-backward algorithm (BCJR
detection). The neural network can be used in the data-driven estimation of
the channel likelihoods, while both the channel likelihood estimation and the
state transitions can be obtained by an HMM optimization.

We construct a state transition matrix for the system by taking the Kronecker
product of the transition matrices for the ISI and the implusive interference
defined by (8) and (12), respectively. Thus, we obtain the probability of the
system being at the joint ISI and IN state $\mathbf{s}_{t}$ at time $t$ to be
governed by the previous system state as

$\begin{split}&p(\mathbf{s}_{t}|\mathbf{s}_{t-1})=\begin{cases}\frac{r+(1-r)p_{j}}{|\mathcal{X}|},&(\mathbf{s}^{\dagger}_{t},\mathbf{s}^{\dagger}_{t-1})\\!\in\\!\mathcal{S}^{\dagger}\hskip
2.13387pt\text{and}\hskip 2.13387pt\tilde{s}^{i}\\!=\\!\tilde{s}^{j}\\\
\frac{(1-r)p_{j}}{|\mathcal{X}|},&(\mathbf{s}^{\dagger}_{t},\mathbf{s}^{\dagger}_{t-1})\\!\in\\!\mathcal{S}^{\dagger}\hskip
2.13387pt\text{and}\hskip 2.13387pt\tilde{s}^{i}\\!\neq\\!\tilde{s}^{j}\\\
0,&\text{otherwise}.\end{cases}\end{split}$ (15)

###  IV-A Optimal symbol detection with the BCJR algorithm

The optimal symbol detector based on the MAP criterion that minimizes the
symbol error rate is given by

$\hat{x}_{t}\\!=\\!\operatorname*{\mathrm{argmax}}_{x_{t}\in\mathcal{X}}p(x_{t}|y_{1},\hbox
to10.00002pt{.\hss.\hss.},y_{T})\\!=\\!\operatorname*{\mathrm{argmax}}_{x_{t}\in\mathcal{X}}p(y_{1},\hbox
to10.00002pt{.\hss.\hss.},y_{T},x_{t}),$

where the joint distribution on the right-hand side of the above equation can
be written as

$p(y_{1},\hbox
to10.00002pt{.\hss.\hss.},y_{T},x_{t})=\sum_{\mathclap{\mathbf{s}_{t},\mathbf{s}_{t-1}\in\mathcal{S}^{\ast}}}p(y_{1},\hbox
to10.00002pt{.\hss.\hss.},y_{T},\mathbf{s}_{t-1},\mathbf{s}_{t}).$ (16)

For ease of notation we denoted with $\mathcal{S}^{\ast}$ the subset of
allowed system state transitions driven by the transmitted symbol $x_{t}$.
Utilizing the factorization from (13), the right-hand side of (16) can be
recursively computed via the $\alpha$-$\beta$-$\gamma$ forward-backward (BCJR)
algorithm from (6), which was extensively described in Sec. II. Note that the
branch metrics as well as the forward and backward probabilities are functions
of the joint ISI and IN channel states. Their computation conventionally
requires full channel knowledge to compute the state transition probability
$p(\mathbf{s}_{t}|\mathbf{s}_{t-1})$ from (15) as well as the channel
likelihood $p(y_{t}|\mathbf{s}_{t})$ from (14). In the following we describe
both the channel model-based and the machine learning approaches to performing
these computations.

Figure 4 shows a schematic of the considered communication system. We
investigated the bit error rate performance for coded transmission. At the
transmitter the information bits are first encoded using a forward error
correcting (FEC) code. The encoded bits are mapped to integer symbols,
interleaved and modulated on transmission symbols $x_{t}$. The symbol sequence
is transmitted through the ISI channel with bursty IN whose model we described
in Sec. III. In the presence of full CSI at the receiver, the likelihoods of
the noisy symbols in the received sequence can be evaluated directly via (14).
To obtain the branch metrics, these are then multiplied by the state
transition probabilities computed from (15). When full CSI is not available,
we resort to data-driven BCJR detection.

###  IV-B Data-Driven NN-based BCJR Detection

The BCJR-NN detector utilizes the received signal in two ways: In training
mode the received noisy symbols
$\left(\ldots,y_{t-1},y_{t},y_{t+1}\ldots\right)$ from a training sequence are
labeled with the corresponding channel states
$\mathbf{s}_{t}=\begin{pmatrix}x_{t},x_{t-1},\ldots,x_{t-L-1},\tilde{s}_{t}\end{pmatrix}$.
The training dataset of $T$ examples, formed as
$\\{y_{t},\mathbf{s}_{t}\\}_{t=1}^{T}$ is used for supervised optimization of
an NN-based estimator for the likelihood $p(y_{t}|\mathbf{s}_{t})$. In
inference mode the learned estimator is used to evaluate the likelihood of the
received symbols in transmission sequences generated independently from the
training ones. In more detail, the NN is employed to classify channel states
using a received symbol, thus yielding an approximation
$p^{\theta}(\mathbf{s}_{t}|y_{t})$ where $\theta$ denotes the NN parameter set
of weight matrices and bias vectors for each layer. Employing a softmax output
layer, the parameters $\theta$ are optimized by minimizing the average cross-
entropy loss over the training dataset, i.e.,

$\mathcal{L}(\theta)=-\frac{1}{T}\sum_{t=1}^{T}\log{p^{\theta}(\mathbf{s}_{t}|y_{t})},$

via stochastic gradient descent (GD). Furthermore, as proposed in [17], a
Gaussian mixture model is utilized to approximate the marginal distribution
$p^{\phi}(y_{t})$ from the received training sequence. The channel likelihoods
required for the BCJR algorithm can then be computed as

$p^{\theta,\phi}(y_{t}|\mathbf{s}_{t})=\frac{p^{\theta}(\mathbf{s}_{t}|y_{t})p^{\phi}(y_{t})}{p(\mathbf{s}_{t})}.$

Note that the marginal probability of the joint channel states should be
provided, based on the transmission model. Importantly, in order to carry out
the BCJR algorithm, the state transition probabilities
$p(\mathbf{s}_{t}|\mathbf{s}_{t-1})$ also need to be available as prior
knowledge. Subsequently, the branch metrics for the BCJR detection can be
computed.

###  IV-C Data-Driven HMM-based BCJR Detection

The BCJR-HMM detector utilizes a sequence of unlabeled received symbols
$\left(\ldots,y_{t-1},y_{t},y_{t+1}\ldots\right)$ to learn the distribution
parameters of the likelihood function as well as to optimize the state
transition matrix. This machine learning approach views the design of the
symbol detector from the prism of the hidden Markov model framework described
in Sec. II. Given an observation sequence
$\left(\ldots,y_{t-1},y_{t},y_{t+1}\ldots\right)$ and the number of possible
states, we can carry out an optimization for the parameters of an HMM. In such
a way we obtain a _data-driven model_ of the underlying Markov process of
interest. The HMM learning procedure involves a combination of the previously
discussed computation of posterior beliefs via the forward-backward algorithm
(expectation step) and a maximization step which updates the model parameters.
This expectation-maximization method for updating the HMM parameters is also
known as the _Baum-Welch algorithm_ [30]. We start by describing the learning
of the state transition probabilities. First, we define the posterior
probability of being in state $\mathbf{s}^{i}$ at time $t-1$ and transitioning
to state $\mathbf{s}^{j}$ at time $t$ given the observed sequence as

$\begin{split}\xi_{t}(\mathbf{s}_{t-1}^{i},\mathbf{s}_{t}^{j})&=p(\mathbf{s}^{i}_{t-1},\mathbf{s}^{j}_{t}|y_{1},\hbox
to10.00002pt{.\hss.\hss.},y_{T})\\\ &=\frac{p(y_{1},\hbox
to10.00002pt{.\hss.\hss.},y_{T},\mathbf{s}^{i}_{t-1},\mathbf{s}^{j}_{t})}{p(y_{1},\hbox
to10.00002pt{.\hss.\hss.},y_{T})}\\\
&=\frac{\alpha_{t-1}(\mathbf{s}^{i}_{t-1})\gamma_{t}(y_{t},\mathbf{s}^{i}_{t-1},\mathbf{s}^{j}_{t})\beta_{t}(\mathbf{s}^{j}_{t})}{\sum_{j=1}^{Q}\alpha_{T}(\mathbf{s}^{j}_{T})}.\end{split}$

Note that for the numerator in the final expression we used the factorization
of the joint distribution from (6), while in the denominator we employed the
forward probability from (4) computed at time $t=T$ to obtain the marginal
probability of the full observation sequence. The estimate of the state
transition probability can then be obtained by the ratio of the expected
number of transitions from $\mathbf{s}^{i}$ to $\mathbf{s}^{j}$ (over all $t$)
and the total expected number of transitions from state $\mathbf{s}^{i}$ as

$\begin{split}\hat{p}(\mathbf{s}^{j}|\mathbf{s}^{i})&=\frac{\sum_{t=2}^{T}\xi_{t}(\mathbf{s}_{t-1}^{i},\mathbf{s}_{t}^{j})}{\sum_{t=2}^{T}\sum_{j=1}^{Q}\xi_{t}(\mathbf{s}_{t-1}^{i},\mathbf{s}_{t}^{j})}.\end{split}$

We now explain the update of the observation likelihood function. For this we
are first interested in the posterior belief of being in state
$\mathbf{s}^{j}$ at time $t$ given by

$\begin{split}\xi_{t}(\mathbf{s}_{t}^{j})&=p(\mathbf{s}^{j}_{t}|y_{1},\hbox
to10.00002pt{.\hss.\hss.},y_{T})\\\ &=\frac{\sum_{i=1}^{Q}p(y_{1},\hbox
to10.00002pt{.\hss.\hss.},y_{T},\mathbf{s}^{i}_{t-1},\mathbf{s}^{j}_{t})}{p(y_{1},\hbox
to10.00002pt{.\hss.\hss.},y_{T})}\\\
&=\frac{\sum_{i=1}^{Q}\alpha_{t-1}(\mathbf{s}^{i}_{t-1})\gamma(y_{t},\mathbf{s}^{i}_{t-1},\mathbf{s}^{j}_{t})\beta_{t}(\mathbf{s}^{j}_{t})}{\sum_{j=1}^{Q}\alpha_{T}(\mathbf{s}^{j}_{T})}.\end{split}$
(17)

In this work, due to the channel model described in Sec. III, we considered
observation likelihood function that follows a Gaussian distribution. It is
worth noting that different choice of distribution (e.g., a Gaussian mixture
[35]) can be made and its parameters can be updated in a similar fashion.
Consequently, the likelihood of an observation is given by (14). Using (17),
the mean of the observation likelihood function for state $\mathbf{s}^{j}$ can
be updated as

$\begin{split}\hat{\mu}_{\mathbf{s}^{j}}=\frac{\sum_{t=1}^{T}\xi_{t}(\mathbf{s}^{j}_{t})y_{t}}{\sum_{t=1}^{T}\xi_{t}(\mathbf{s}^{j}_{t})}.\end{split}$

Similarly, estimation of the variance is performed as

$\begin{split}\hat{\sigma}^{2}_{\mathbf{s}^{j}}=\frac{\sum_{t=1}^{T}\xi_{t}(\mathbf{s}^{j}_{t})(y_{t}-\hat{\mu}_{\mathbf{s}^{j}})^{2}}{\sum_{t=1}^{T}\xi_{t}(\mathbf{s}^{j}_{t})}.\end{split}$

After the symbol detection phase, the soft outputs, expressed by the joint
probability from (16), for the channel model-based as well as the data-driven
BCJR detectors are appropriately deinterleaved and fed to the FEC decoder
which recovers the information bits. The obtained BER performance of the
investigated systems is reported next.

##  V Performance Results

TABLE I: Simulation parameters FEC scheme |  $(171,133)$ conv. code
---|---
Code rate | $1/2$
Code memory |  $7$ bits
Modulation scheme | BPSK ($|\mathcal{X}|=2$)
Symbol sequence length ($T$) | 500000
Monte Carlo iterations | 500
Channel memory ($L$) |  $1$ or $2$ sym.
Channel taps decay rate ($\eta$) | $1$
Channel taps deviation ($\sigma_{h}^{2}$) |  $0$ or $0.1$
Impulse noise levels ($N$) |  $1$ or $2$
Impulsive index ($A$) | $0.8$
Backgr.-to-impulse noise ratio ($\Gamma$) |  $0.01$ or $0.1$
Total noise power ($\sigma^{2}$) | varied
Impulse correlation ($r$) | 0.98
Number of detector states ($Q$) |  $N|\mathcal{X}|^{L}$ or $|\mathcal{X}|^{L}$
NN layers |  $1\times 100$ (Sigmoid)
$100\times 50$ (ReLU)
$50\times Q$ (Softmax)
NN optimizer | Adam (learning rate $=0.01$)
GD iterations (NN) | 20000
EM iterations (HMM) | 1500

Table I lists the simulation parameters. We used a rate $1/2$ binary
$(171,133)$ convolutional code with memory depth of 7 bits [38] which is
standard for digital broadcasting systems. As in [17] and [27], we considered
the BPSK modulation format with transmitted symbol alphabet
$\mathcal{X}=\\{-1,1\\}$. The transmitted sequence consisted of $5\cdot
10^{5}$ symbols and $500$ different transmissions were simulated when
evaluating the BER of the communication system at a specific SNR/SINR. The
impulsive noise parameters were adopted from [27]. Nevertheless, depending on
the investigated communication scenario, channel parameters were varied
accordingly. The neural network for the BCJR-NN detector followed the three
layer architecture from [17] with sigmoid, rectified linear unit (ReLU) and
final softmax activation. Stochastic GD optimization of the NN was performed
over 20000 iterations using the Adam optimizer with learning rate of 0.01. The
number of EM algorithm iterations for optimizing the HMM parameters was set to
1500.

###  V-A Intersymbol Interference Channel with AWGN

Figure 5:  SER versus SNR for uncoded BPSK transmission over a time-invariant
ISI ($L=2$ and $\sigma^{2}_{h}=0$) channel with AWGN for systems with channel
model-based or data-driven BCJR detection. Figure 6: BER versus SNR for coded
BPSK transmission over a time-invariant ISI ($L=2$ and $\sigma^{2}_{h}=0$)
channel with AWGN for systems with channel model-based or data-driven BCJR
detection.

We start by evaluating the symbol error rate (SER) performance of the BCJR
detectors for uncoded transmission over the ISI channel (symbol memory of
$L=2$) and AWGN ($N=1$). In particular, as also done in [17], we considered
the error rate performance for transmission over the time-invariant ISI and
obtained the channel likelihoods in two different ways: 1) using the same
time-invariant ISI model for both the full CSI-based BCJR and the optimization
of the data-driven detectors and 2) using the rapidly time-varying model from
(10) with $\sigma^{2}_{h}=0.1$ for this (referred to as _CSI uncertainty_ in
[17]). Note that for such a channel the BCJR state transitions, given by (8),
are trivial and can be readily provided as prior knowledge for the channel
model-based and NN detectors. Figure 5 shows the obtained SERs as a function
of SNR for the different detection methods. As first reported in [17], we see
that the BCJR-NN achieves SER performance close the optimal full CSI-based
BCJR detector regardless of whether it was trained on the time-invariant or
the time-varying channels. Our results show that the BCJR-HMM exhibits similar
robustness to channel variations. In contrast, the SER of the channel model-
based BCJR where likelihoods are computed with $\sigma^{2}_{h}=0.1$ is
significantly degraded at higher SNRs.

We extend the investigation of symbol detection over such channels for the
case of coded transmission with the $(171,133)$ convolutional code. Figure 6
shows the BER versus SNR curves for systems employing the investigated symbol
detectors. For this setting we can also observe that the BCJR-NN and the BCJR-
HMM detection schemes achieve BERs close to the algorithm with perfect CSI
regardless of whether they were optimized via the time-invariant or the time-
varying ISI model with $\sigma^{2}_{h}=0.1$. Furthermore, at 1 dB SNR the
data-driven solutions outperform by a factor of 2 the BCJR detection with
inaccurate CSI — an improvement which becomes more pronounced as the SNR
increases.

###  V-B Impact of Detector States on the System Performance

Figure 7: BER versus SNR for coded BPSK transmission over an AWGN channel for
BCJR detection based on different number of states. TABLE II: Likelihood
function parameters and state transition matrix for BPSK over an AWGN channel
($10$ dB SNR) learned in three different independent trials (left, center and
right) using a 4-state HMM. Steady-state distributions are also listed.
Conversely, (8) needs to be used for the other $L=2$ detectors.

black!30!white$\hat{\mu}$ black!0!white$-1$ black!0!white$-1$ black!0!white$1$
black!0!white$1$ black!30!white$\hat{\sigma}^{2}$ black!0!white$0.1$
black!0!white$0.1$ black!0!white$0.1$ black!0!white$0.1$ black!15!white
$\mathbf{s}_{t-1}$ $\mathbf{s}_{t}$ black!15!white$\mathbf{s}^{1}$
black!15!white$\mathbf{s}^{2}$ black!15!white$\mathbf{s}^{3}$
black!15!white$\mathbf{s}^{4}$ black!15!white$\mathbf{s}^{1}$ 0.33 0 0 0.67
black!15!white$\mathbf{s}^{2}$ 0.25 0.49 0.13 0.13
black!15!white$\mathbf{s}^{3}$ 0.3 0.2 0.32 0.18
black!15!white$\mathbf{s}^{4}$ 0.29 0.21 0.16 0.34 black!15!white$\pi$
black!15!white0.295 black!15!white0.204 black!15!white0.127
black!15!white0.374

black!30!white$\hat{\mu}$ black!0!white$-1$ black!0!white$-1$ black!0!white$1$
black!0!white$1.7$ black!30!white$\hat{\sigma}^{2}$ black!0!white$0.1$
black!0!white$0.1$ black!0!white$0.1$ black!0!white$0.0001$ black!15!white
$\mathbf{s}_{t-1}$ $\mathbf{s}_{t}$ black!15!white$\mathbf{s}^{1}$
black!15!white$\mathbf{s}^{2}$ black!15!white$\mathbf{s}^{3}$
black!15!white$\mathbf{s}^{4}$ black!15!white$\mathbf{s}^{1}$ 0.06 0.7 0.24 0
black!15!white$\mathbf{s}^{2}$ 0.02 0.47 0.51 0 black!15!white$\mathbf{s}^{3}$
0.03 0.47 0.5 0 black!15!white$\mathbf{s}^{4}$ 1 0 0 0 black!15!white$\pi$
black!15!white0.026 black!15!white0.476 black!15!white0.498 black!15!white0

black!30!white$\hat{\mu}$ black!0!white$-1$ black!0!white$-1$
black!0!white$-1$ black!0!white$1$ black!30!white$\hat{\sigma}^{2}$
black!0!white$0.1$ black!0!white$0.1$ black!0!white$0.1$ black!0!white$0.1$
black!15!white $\mathbf{s}_{t-1}$ $\mathbf{s}_{t}$
black!15!white$\mathbf{s}^{1}$ black!15!white$\mathbf{s}^{2}$
black!15!white$\mathbf{s}^{3}$ black!15!white$\mathbf{s}^{4}$
black!15!white$\mathbf{s}^{1}$ 0.28 0.72 0 0 black!15!white$\mathbf{s}^{2}$
0.005 0.345 0.65 0 black!15!white$\mathbf{s}^{3}$ 0 0.05 0.27 0.68
black!15!white$\mathbf{s}^{4}$ 0 0.13 0.37 0.5 black!15!white$\pi$
black!15!white0.001 black!15!white0.129 black!15!white0.369
black!15!white0.501

Next, we considered a practically-relevant communication scenario where the
number of detector states is over-estimated, for example to accommodate
operation in a “worst-case” ISI scenario. In particular, we devise a simple
setting of coded BPSK transmission over an AWGN channel where the BCJR symbol
detectors are provisioned for ISI with $L=2$, i.e., they have
$Q=|\mathcal{X}|^{L}=4$ states. Figure 7 compares the BER performance of the
different detection schemes in such a scenario. As a reference baseline we use
the channel model-based $L=1$ BCJR detector where states are the two BPSK
symbols with transition probabilities of $0.5$. Compared to the reference
system, we observe a degraded BER performance for the conventional BCJR as
well as the BCJR-NN both operated with $L=2$. This is due to the fact that
these detectors rely on the channel model-based computation of the state
transition probabilities from (8), which is performed with the inaccurate ISI
knowledge of $L=2$ and hence is not matched to the actual transmission
channel. On the other hand, our results show that the BER performance of the
BCJR-HMM is not significantly affected by the inaccurate number of detector
states. Since the BCJR-HMM detection scheme uses the received symbol sequence
to optimize both the parameters of the channel likelihood function as well as
the state transition matrix, it manages to mitigate the effects of over-
estimating the number of states by accordingly adjusting the state
transitions. To emphasize this, Table II shows examples of the optimized state
transition matrices and likelihood distribution parameters which we obtained
in three independent optimization attempts. On the left-hand side we see that
the learning algorithm separated the four available states into two groups of
two with means of the channel likelihood distribution equivalent to the BPSK
symbols (i.e., $\hat{\mu}=-1$ for $\mathbf{s}^{1}$ and $\mathbf{s}^{2}$, and
$\hat{\mu}=1$ for $\mathbf{s}^{3}$, $\mathbf{s}^{4}$) and variances equal to
the inverse of the linear SNR. Importantly, we observe that only certain
transitions between states were allowed. To gain further insight on the
optimized state transition matrix, the table also shows the corresponding
steady-state distribution $\pi$ which allows us to see that the two groups
have total probabilities of approximately $0.5$ which is in correspondence
with the state transitions in the optimal $L=1$ detector. Similar grouping
behavior can also be recognized in the middle and right-hand side examples.
Note that for the former, the optimization converged to an unused state
$\mathbf{s}^{4}$ with an outlier mean and very small variance with no incoming
transitions allowed, while for the latter, three states were grouped together.

To further examine the importance of the state transition matrix, we utilized
the NN-based channel likelihood estimator with state transitions optimized via
HMM. The BER performance curve when employing this BCJR-HMM-NN scheme is also
shown in Fig. 7. Interestingly, we observe that such a detection approach
improves the performance of the BCJR-NN and allows operation close the the
optimal full CSI-based BCJR due to the optimized transition matrix. Such a
joint HMM-NN scheme can be particularly promising in applications with non-
trivial channel likelihood functions where the powerful computational
capabilities of the NN are particularly beneficial.

###  V-C Intersymbol Interference Channel with Bursty IN

Figure 8: BER versus SINR for coded BPSK transmission over a time-invariant
ISI ($L=2$ and $\sigma^{2}_{h}=0$) channel with bursty IN ($N=2$) for systems
with channel model-based or data-driven BCJR detection. Solid lines correspond
to $\Gamma=0.01$, while dotted lines to $\Gamma=0.1$.

In this section we consider coded transmission over ISI ($L=2$) channel with
bursty IN ($N=2$). The BER performance for the proposed systems with different
detectors as a function of the SINR is shown in Fig. 8. To investigate
different regimes of impulsive interference, the figure shows results for two
different background-to-interference noise ratios $\Gamma=0.01$ and
$\Gamma=0.1$. More specifically, we consider the cases where the number of
states in the BCJR-HMM and the BCJR-NN detectors are matched to the number of
channel states ($L=2,N=2$) and compare their performance with the channel
model-based BCJR with perfect CSI ($L=2,N=2$) as well as conventional BCJR
detection designed for the ISI channel with AWGN (the _BCJR, $L=2,N=1$_
system). We see that for the ISI with IN channel the BERs of the two data-
driven receivers follow closely that of the optimal BCJR detection for both
values of $\Gamma$. Conversely, there is a strong degradation in the system
performance if detection is performed under the ISI with AWGN assumption,
i.e., where only the total noise variance $\sigma^{2}$ is considered in
computing the likelihoods and the state transitions are governed solely by the
ISI process. The receivers for digital broadcasting systems are often based on
the AWGN assumption and crucially the proposed machine learning solutions
provide an opportunity to tailor their performance in a data-driven manner
when impulsive interference is present. While the BCJR-HMM detector manages to
fully adapt to the channel characteristics without prior knowledge of the IN,
the BCJR-NN requires labeling of the IN states for training as well as
providing the state transition matrix. These requirements could be impractical
since the sources of impulsive interference are typically external to the
communication system. To further investigate the scenario of employing neural
networks for the ISI channel with IN, we consider the BCJR-NN with $L=2,N=1$
where states are designed assuming only the ISI effect, while the training
data for the NN-based likelihood estimator is generated by the ISI channel
with IN. To simulate this scenario we labeled only the ISI states during the
supervised learning. The performance of such a system, which does not rely on
prior knowledge of the IN, is also shown in Fig. 8.

Figure 9: Normalized channel (observation) likelihoods for the time-invariant
ISI ($L=2$ and $\sigma^{2}_{h}=0$) channel with IN ($N=2$) computed by the
BCJR-NN and the conventional BCJR detectors with $L=2,N=1$.

Our results indicate that, while inferior to the detectors which accurately
handle the IN states, the BCJR-NN ($L=2,N=1$) outperforms the conventional
BCJR with $L=2,N=1$. When the number of states is reduced, the Gaussian
assumption for the channel likelihoods is not necessarily accurate, hence the
NN-based computation which learns the distribution directly from data provides
a better estimate. To exemplify this, Fig. 9 illustrates the (normalized)
channel likelihoods obtained by the reduced-state NN and conventional AWGN
detection methods at $3$ dB SINR for $\Gamma=0.01$. Indeed, we see that the
NN-learned distribution deviates from a Gaussian for each of the
$Q=|\mathcal{X}|^{L}=4$ states. Another interesting observation from the BER
results in Fig. 8 is that the gains achieved by the reduced-state BCJR-NN over
its conventional counterpart are smaller for $\Gamma=0.1$ compared to the case
of $\Gamma=0.01$. In the extreme case of $\Gamma=1$ the Gaussian assumption
for the likelihoods for these 4-state detectors becomes accurate since the
background and interference noises have identical variances. This results in
identical BER performance which we verified in our simulations.

Figure 10: BER versus SINR for coded BPSK transmission over a rapidly time-
varying ISI ($L=2$ and $\sigma^{2}_{h}=0.1$) channel with IN ($N=2$).

Next, we consider the impact of rapidly time-varying ISI behavior on the
detectors’ performance under bursty IN. Note that such a scenario for the ISI
with AWGN channel and NN-based BCJR detection was considered as a practical
use case in [34] (denoted by the authors as Case 5). Similarly here, to
simulate such a system in presence of IN we set the deviation of the ISI taps
to $\sigma^{2}_{h}=0.1$ for both the optimization and transmission channels.
Figure 10 shows the BER performance as a function of SINR for the BCJR-HMM and
BCJR-NN detectors and compares them to the conventional BCJR where channel
likelihoods were also computed with $\sigma^{2}_{h}=0.1$. Note that all
detectors were provided with the accurate $Q=N|\mathcal{X}|^{L}=8$ number of
states for this investigation. Compared to the BER performances over the time-
invariant ISI model with IN, shown previously, our results indicate a
considerable degradation in the examined systems performances due to the rapid
ISI variations. Nevertheless, we notice that the data-driven detectors can
provide significantly lower BERs at all examined SINRs compared to the
conventional BCJR approach which cannot adapt to the changes in the ISI
channel coefficients. Such an increased robustness to channel variations
exhibited by the data-driven approaches is of great practical importance and
thus presents a major direction in our future work.

##  VI Conclusions

We considered two fundamentally different machine learning approaches — based
on the supervised learning of neural networks as well as the unsupervised
optimization of hidden Markov models, for the implementation of data-driven
BCJR symbol detection in communication scenarios characterized by non-trivial
state transitions. To model these systems we focused on transmission over ISI
channels with bursty IN as well as symbol detection with over-estimated
trellis states in AWGN. Our results showed that obtaining the state transition
matrix in a data-driven manner via the HMM framework is important for avoiding
significant performance penalties. On the other hand, in settings where the
channel likelihoods follow a non-Gaussian distribution, for e.g., when the IN
states are not accounted for in the detector trellis, the NN-based estimator
shows superior performance compared to the conventional detector operating on
the AWGN assumption. Moreover, our work demonstrates the advantageous
performance of a hybrid HMM-NN-based approach to learning the state
transitions and the channel likelihoods for BCJR. Importantly, we extended our
investigation of the data-driven detectors to rapidly time-varying ISI in the
presence of IN and showed that these methods are significantly more robust to
channel variations compared to the conventional detector.

## References

  * [1] O. Simeone, “A very brief introduction to machine learning with applications to communication systems,” IEEE Trans. Cogn. Commun. Netw., vol. 4, no. 4, pp. 648-664.
  * [2] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016.
  * [3] K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,” Neural Networks, 1989.
  * [4] T. O’Shea and J. Hoydis, “An introduction to deep learning for the physical layer,” IEEE Trans. Cogn. Commun. Netw., vol. 3, no. 4, pp. 563-575, 2017.
  * [5] S. Dörner, S. Cammerer, J. Hoydis and S. ten Brink, “Deep learning based communication over the air,” IEEE J. Sel. Topics Signal Process., vol. 12, no. 1, pp. 132-143, 2018.
  * [6] B. Karanov, M. Chagnon, F. Thouin, T. A. Eriksson, H. Bülow, D. Lavery, P. Bayvel and L. Schmalen, “End-to-end deep learning of optical fiber communications,” IEEE/Optica J. Lightw. Technol., vol. 36, no. 20, pp. 4843-4855, 2018.
  * [7] B. Karanov, D. Lavery, P. Bayvel and L. Schmalen, “End-to-end optimized transmission over dispersive intensity-modulated channels using bidirectional recurrent neural networks,” Opt. Express, vol. 27, no. 14, pp. 19650-19663, 2019.
  * [8] B. Karanov, L. Schmalen and A. Alvarado, “Distance-agnostic auto-encoders for short reach fiber communications,” in Optical Fiber Communications Conference (OFC), 2021, pp. 1-3.
  * [9] F. A. Aoudia and J. Hoydis, “Model-free training of end-to-end communication systems,” IEEE J. Sel. Areas Commun., vol. 37, no. 11, pp. 2503-2516, 2019.
  * [10] B. Karanov, M. Chagnon, V. Aref, D. Lavery, P. Bayvel and L. Schmalen, “Concept and Experimental demonstration of optical IM/DD end-to-end system optimization using a generative model,” in Proc. Optical Fiber Communications Conference and Exhibition (OFC), 2020, pp. 1-3.
  * [11] C. Häger and H. D. Pfister, “Physics-based deep learning for fiber-optic communication systems,” IEEE J. Sel. Areas Commun., vol. 39, no. 1, pp. 280-294, 2021.
  * [12] N. Farsad and A. Goldsmith, “Neural network detection of data sequences in communication systems,” IEEE Trans. Signal Process., vol. 66, no. 21, pp. 5663-5678, 2018.
  * [13] E. Nachmani, E. Marciano, L. Lugosch, W. J. Gross, D. Burshtein and Y. Be’ery, “Deep learning methods for improved decoding of linear codes,” IEEE J. Sel. Top. Signal Process., vol. 12, no. 1, pp. 119-131, 2018.
  * [14] G. D. Forney, “Maximum-likelihood sequence estimation of digital sequences in the presence of intersymbol interference,” IEEE Trans. Inform. Theory, vol. 18, no. 3, pp. 363-378, 1972.
  * [15] L. R. Bahl, J. Cocke, F. Jelinek, and J. Raviv, “Optimal decoding of linear codes for minimizing symbol error rate,” IEEE Trans. Inform. Theory, vol. 20, no. 2, pp. 284–287, 1974.
  * [16] N. Shlezinger, N. Farsad, Y. C. Eldar and A. J. Goldsmith, “ViterbiNet: A deep learning based Viterbi algorithm for symbol detection,” IEEE Trans. Wireless Commun., vol. 19, no. 5, pp. 3319-3331, 2020.
  * [17] N. Shlezinger, N. Farsad, Y. C. Eldar and A. J. Goldsmith, “Learned factor graphs for inference from stationary time sequences,” IEEE Trans. Signal Process., vol. 70, pp. 366-380, 2022.
  * [18] N. Shlezinger, J. Whang, Y. C. Eldar and A. G. Dimakis, “Model-based deep learning,” Proc. IEEE, vol. 111, no. 5, pp. 465-499, 2023.
  * [19] Q. Shan et al., “Estimation of impulsive noise in an electricity substation,” IEEE Trans. Electromagn. Compat., vol. 53, no. 3, pp. 653-663, 2011.
  * [20] M. Sanchez, L. de Haro, M. Ramon, A. Mansilla, C. Ortega and D. Oliver, ”Impulsive noise measurements and characterization in a UHF digital TV channel,” in IEEE Trans. on Electromagn. Compat., vol. 41, no. 2, pp. 124-136, 1999.
  * [21] M. Mostafa, “Stability proof of iterative interference cancellation for OFDM signals with blanking nonlinearity in impulsive noise channels,” IEEE Signal Process. Lett., vol. 24, no. 2, pp. 201-205, 2017.
  * [22] P. Yang, Y. L. Guan, X. B. Liu and Z. Liu, “An improved hybrid turbo equalizer for single carrier transmission with impulsive noise and ISI,” IEEE Trans. Veh. Technol., vol. 66, no. 11, pp. 9852-9861, 2017.
  * [23] H. Oh and H. Nam, “Design and performance analysis of nonlinearity preprocessors in an impulsive noise environment,” in IEEE Trans. Veh. Technol., vol. 66, no. 1, pp. 364-376, 2017.
  * [24] S. Liu, F. Yang, W. Ding and J. Song, “Double kill: Compressive-sensing-based narrow-band interference and impulsive noise mitigation for vehicular communications,” IEEE Trans. Veh. Technol., vol. 65, no. 7, pp. 5099-5109, 2016.
  * [25] D. Middleton, “Statistical-physical model of electromagnetic interference,” IEEE Trans. Electromagn. Compat., vol. EMC-19, no. 3, pt. 1, pp. 106–127, 1977.
  * [26] D. Middleton, “Non-Gaussian noise models in signal processing for telecommunications: new methods an results for class A and class B noise models,” IEEE Trans. Inf. Theory, vol. 45, no. 4, pp. 1129-1149, 1999.
  * [27] G. Ndo, F. Labeau and M. Kassouf, “A Markov-Middleton model for bursty impulsive noise: Modeling and receiver design,” IEEE Trans. Power Del., vol. 28, no. 4, pp. 2317-2325, 2013.
  * [28] A. Mirbadin, A. Vannucci, G. Colavolpe, R. Pecori and L. Veltri, “Iterative receiver design for the estimation of Gaussian samples in impulsive noise,” Appl. Sci., vol. 11, no. 2: 557, 2021.
  * [29] L. R. Rabiner, “A tutorial on hidden Markov models and selected applications in speech recognition,” Proc. IEEE, vol. 77, no. 2, pp. 257-286, 1989.
  * [30] L. E. Baum, “An inequality and associated maximization technique in statistical estimation for probabilistic functions of Markov processes,” Inequalities III: Proceedings of the 3rd Symposium on Inequalities, pp. 1-8, 1972.
  * [31] L. R. Welch, “Hidden Markov models and the Baum-Welch algorithm,” IEEE Information Theory Society Newsletter, vol. 53, no. 4, pp. 1&10-13, 2003.
  * [32] G. K. Kaleh and R. Vallet, “Joint parameter estimation and symbol Detection for linear or nonlinear unknown channels,” IEEE Trans. Commun., vol. 42, no. 7, pp. 2406-2413, 1994.
  * [33] L. Schmid, T. Raviv, N. Shlezinger, and Laurent Schmalen, “Blind channel estimation and joint symbol detection with data-driven factor graphs,” ArXiv preprint arXiv:2401.12627, 2024.
  * [34] C.-H. Chen, B. Karanov, W. v. Houtum, W. Yan, A. Young and A. Alvarado, “On the robustness of deep learning-aided symbol detectors to varying conditions and imperfect channel knowledge,” in Proc. IEEE Wireless Communications and Networking Conference (WCNC), 2024, pp. 1-6.
  * [35] D. Jurafsky and J. H. Martin, Speech and Language Processing (2nd Edition). Upper Saddle River, NJ, USA, Prentice-Hall, Inc., 2009.
  * [36] C. M. Bishop, Pattern Recognition and Machine Learning. New York, NY, USA, Springer, 2006.
  * [37] C.-H. Chen, W.-H. Huang, B. Karanov, Y. Wu, A. Young, W. v. Houtum, “Analysis of impulsive interference in digital audio broadcasting systems in electric vehicles,” in Proc. Symposium on Information Theory and Signal Procssing in the Benelux (SITB), 2024.
  * [38] K. S. Gilhousen, J. A. Heller, I. M. Jacobs, and A. J. Viterbi, “Coding systems study for high data rate telemetry links,” NASA Rep., Jan. 1971, prepared under Contract NAS2-6024, Linkabit Corp., San Diego, CA, USA, pp. 235.


\Sym^{\bullet}\left(\coprod_{j\leq r}U_j\right)\cong \prod_{j\leq r}\Sym^{\bullet}(U_j)
\]

we can reduce the claim that (<ref>) is an isomorphism to the special case in which $X=U_i$.  This case in turn follows from the special case in which $X=\BA^2$, by restriction.  By Corollary <ref> each of the morphisms $\Phi_{\BA^2,n}$ is a split inclusion, and the inclusion of the BPS sheaf, and the claim regarding $\Psi_{\BA^2}$ follows from the PBW theorem (Proposition <ref>).  We have thus produced the isomorphism <ref>.
Applying $\tau^{\leq 0}$ to (<ref>) produces a PBW isomorphism

\[
\Sym_{\cup}\left(\Delta_{n,*}\underline{\BQ}_{X,\vir}\right)\cong \tau^{\leq 0}p_*\BD\underline{\BQ}_{\Mst_{\bullet}(X)}
\]

and the claim that this is in fact an isomorphism of algebras reduces to the claim that $\tau^{\leq 0}p_*\BD\ul{\BQ}_{\Mst_{\bullet}(X)}$ is a commutative algebra object.  Again, we may work analytically locally and reduce to the claim that $\Up\tau^{\leq 0}p_*\BD\BQ_{\Mst_{\bullet}(\BA^2)}$ is a commutative algebra object.  With notation as in <ref>, this is equivalent to the claim that $\DTS_{\Pi_Q,\bullet}$ is an Abelian Lie algebra object, for $Q$ the Jordan quiver with one vertex and one loop.  For this the argument is given in <ref>: the Lie algebra morphism is induced by a morphism

\begin{equation}
\label{finlo}
\cup_*(\Delta_{m,*}\underline{\BQ}_{\mathbb{A}^2,\vir}\boxtimes\Delta_{n,*}\underline{\BQ}_{\mathbb{A}^2,\vir})\rightarrow \Delta_{m+n,*}\underline{\BQ}_{\mathbb{A}^2,\vir}
\end{equation}

where the source and the target are semisimple mixed Hodge modules with differing supports, so that (<ref>) is indeed zero.

[1]
J. Alper, J. Hall, and D. Rydh.
A Luna étale slice theorem for algebraic stacks.
Ann. Math., 191(3):675–738, 2020.

[2]
V. Baranovsky.
The variety of pairs of commuting nilpotent matrices is irreducible.
Transform. groups, 6(1):3–8, 2001.

[3]
O. Ben-Bassat, C. Brav, V. Bussi, and D. Joyce.
A “Darboux theorem” for shifted symplectic structures on derived
Artin stacks, with applications.
Geom. & Top., 19(3):1287–1359, 2015.

[4]
A. Beilinson, J. Bernstein, and P. Deligne.
Faisceaux pervers.
Astérisque, 100, 1983.

[5]
T. Bridgeland.
Equivalences of triangulated categories and Fourier–Mukai
Bull. London Math. Soc., 31(1):25–34, 1999.

[6]
T. Bozec and O. Schiffmann.
Counting absolutely cuspidals for quivers.
Mathematische Zeitschrift, 292(1-2):133–149, 2019.

[7]
W. Crawley-Boevey.
Geometry of the moment map for representations of quivers.
Compos. Math., 126(3):257–293, 2001.

[8]
K. Corlette.
Flat $G$-bundles with canonical metrics.
J. Differ. Geom., 28(3):361–382, 1988.

[9]
B. Davison.
Cohomological Hall algebras and character varieties.
Int. J. Math., 27(07).

[10]
B. Davison.
The critical CoHA of a quiver with potential.
Quart. J. Math., 68(2):635–703, 2017.

[11]
B. Davison.
The integrality conjecture and the cohomology of preprojective
<https://arxiv.org/abs/1602.02110v3>, 2017.

[12]
B. Davison.
Purity of critical cohomology and Kac's conjecture.
Math. Res. Lett., 25(2):469–488, 2018.

[13]
B. Davison.
BPS Lie algebras and the less perverse filtration on the
preprojective CoHA.
<https://arxiv.org/abs/2007.03289>, 2020.

[14]
B. Davison.
Purity and 2-Calabi–Yau categories.
<https://arxiv.org/abs/2106.07692>, 2021.

[15]
MAA. de Cataldo, T. Hausel, and L. Migliorini.
Topology of Hitchin systems and Hodge theory of character varieties:
the case $A_1$.
Ann. Math., 175(3):1329–1407, 2012.

[16]
Mark Andrea A de Cataldo and Davesh Maulik.
The perverse filtration for the Hitchin fibration is locally
Pure Appl. Math. Q., 16(5):1444–1464, 2020.

[17]
P. Deligne.
Théorie de Hodge: II.
Publ. Math. IHÉS, 40:5–57, 1971.

[18]
P. Deligne.
Théorie de Hodge: III.
Publ. Math. IHÉS, 44:5–77, 1974.

[19]
G. Dobrovolska, V. Ginzburg, and R. Travkin.
Moduli spaces, indecomposable objects and potentials over a finite
<http://arxiv.org/abs/1612.01733>, 2016.

[20]
B. Davison and S. Meinhardt.
Cohomological Donaldson–Thomas theory of a quiver with potential
and quantum enveloping algebras.
Invent. Math., 221:777–871, 2020.

[21]
S. Donaldson.
Twisted harmonic maps and self-duality equations.
Proc. London Math. Soc., 55:127–131, 1987.

[22]
C. Felisetti.
Intersection cohomology of the moduli space of Higgs bundles on a
genus 2 curve.
J. Inst. Math. Jussieu, page 1–50, 2021.

[23]
C. Felisetti and M. Mauri.
P=W conjectures for character varieties with symplectic resolution.

[24]
T. Hausel.
Kac's conjecture from Nakajima quiver varieties.
Invent. Math., 181(1):21–37, 2010.

[25]
N. Hitchin.
The self-duality equations on a Riemann surface.
Proc. London Math. Soc., 3(1):59–126, 1987.

[26]
T. Hausel, E. Letellier, and F. Rodriguez-Villegas.
Positivity for Kac polynomials and DT-invariants of quivers.
Ann. Math., 177:1147–1168, 2013.

[27]
T. Hausel and F. Rodriguez-Villegas.
Mixed Hodge polynomials of character varieties.
Invent. Math., 174:555–624, 2008.

[28]
V. Kac.
Root systems, representations of quivers and invariant theory.
In Invariant theory, pages 74–108. Springer, 1983.

[29]
B. Keller.
Calabi–Yau triangulated categories.
in Trends in representation theory of algebras and related
topics (editor A. Skowronski), pages 467–489, 2008.

[30]
T. Kinjo.
Dimensional reduction in cohomological Donaldson–Thomas theory.
<https://arxiv.org/abs/2102.01568>, 2021.

[31]
T. Kinjo and N. Koseki.
Cohomological $\chi$-independence for Higgs bundles and
Gopakumar–Vafa invariants.
<https://arxiv.org/abs/2112.10053>, 2021.

[32]
T. Kinjo and N. Masuda.
Global critical chart for local Calabi–Yau threefold.
<https://arxiv.org/abs/2112.10052>, 2021.

[33]
M. Kashiwara and P. Schapira.
Sheaves on manifolds, volume 292 of Grundlehren der
Mathematischen Wissenschaften [Fundamental Principles of Mathematical
Springer-Verlag, Berlin, 1990.

[34]
M. Kontsevich and Y. Soibelman.
Cohomological Hall algebra, exponential Hodge structures and motivic
Donaldson–Thomas invariants.
Commun. Number Theory Phys., 5, 2011.

[35]
M. Kapranov and E. Vasserot.
The cohomological Hall algebra of a surface and factorization
<https://arxiv.org/abs/1901.07641>, 2019.

[36]
M. Mauri.
Intersection cohomology of rank two character varieties of surface
<https://arxiv.org/abs/2101.04628>, 2021.

[37]
A. Minets.
Cohomological Hall algebras for Higgs torsion sheaves, moduli of
triples and sheaves on surfaces.
Sel. Math. New Set., 26:1–67, 2020.

[38]
S. Meinhardt and M. Reineke.
Donaldson–Thomas invariants versus intersection cohomology of
quiver moduli.
J. für die Reine und Angew. Math. (Crelles Journal),
2019(754):143–178, 2019.

[39]
D. Maulik and J. Shen.
Cohomological $\chi$-independence for moduli of one-dimensional
sheaves and moduli of Higgs bundles.
arXiv preprint arXiv:2012.06627, to appear Geom.& Top., 2020.

[40]
D. Maulik and Y. Toda.
Gopakumar–Vafa invariants via vanishing cycles.
Invent. Math., 213(3):1017–1097, 2018.

[41]
A. Polishchuk.
Abelian varieties, theta functions and the Fourier transform.
Cambridge University Press, 2003.

[42]
J. Ren and Y. Soibelman.
Cohomological Hall Algebras, Semicanonical Bases and
Donaldson–Thomas Invariants for 2-dimensional Calabi–Yau Categories (with
an Appendix by Ben Davison).
In Algebra, Geometry, and Physics in the 21st Century, pages
261–293. Springer, 2017.

[43]
C. Simpson.
Moduli of representations of the fundamental group of a smooth
projective variety I.
Publ. Math. IHÉS, 79:867–918, 1994.

[44]
S. Schlegel-Mejia.
BPS cohomology for rank 2 degree 0 Higgs bundles (and more).

[45]
F. Sala and O. Schiffmann.
Cohomological Hall algebra of Higgs sheaves on a curve.
Algebraic Geom., 7(3):346–376, 2020.

[46]
O. Schiffmann and E. Vasserot.
Cherednik algebras, W-algebras and the equivariant cohomology of the
moduli space of instantons on $\mathbb{A}^2$.
Publ. Math. IHÉS, 118(1):213–342, 2013.

[47]
Y. Toda.
Gopakumar-Vafa invariants and wall-crossing.
<https://arxiv.org/abs/1710.01843>, 2017.

[48]
Y. Yang and G. Zhao.
On two cohomological Hall algebras.
Proc. Roy. Soc. Edinburgh A, pages 1–27, 2016.

[49]
Y. Yang and G. Zhao.
The cohomological Hall algebra of a preprojective algebra.
Proc. London Math. Soc., 116(5):1029–1074, 2018.