# An End-to-End Khmer Optical Character Recognition using Sequence-to-Sequence
with Attention

Rina Buoy† Sokchea Kor ‡ Nguonly Taing†
†Techo Startup Center (TSC)
‡Royal University of Phnom Penh (RUPP)
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Keywords: Khmer, OCR, NLP, Deep Learning, RNN, CNN, Attention, Seq2Seq

##  1 Introduction

###  1.1 Optical Character Regognition (OCR)

One of the artificial intelligence (AI) paradigms is to develop a machine that
can mimic the ability of human recognition. In term of visual perception and
understanding text, the computer is still at the infancy level, compared with
human [1].

To make a text machine-readable, it can be either converted manually or
digitally extracted by mean of Optical Character Recognition (OCR), from the
digital image of the document [2].

OCR is the science of extracting analyzable and editable data from the scanned
documents or images. The OCR technology has been evolving over the last 8
decades. The early phase of OCR development was mainly contributed by large
tech players. The recent advancement of artificial intelligence, particularly
deep learning has allowed researchers from various spectrum to devise OCR
algorithms that can achieve higher accuracy levels [2].

Although OCR technology for English and other high-resource languages has been
developed over the last 8 decades [2], the earlier OCR work on the Khmer
language was around the year 2005.

###  1.2 Khmer Writing System

Khmer (KHM) is the official language of the kingdom of Cambodia. The Khmer
script is used in the writing system of Khmer and other minority languages
such Kuay, Tampuan, Jarai Krung, Brao and Kravet. Khmer language and writing
system were hugely influenced by Pali and Sanskrit in early history [3, 4].

Unlike Latin-based languages, the Khmer language has a complex writing system.
One or two consonants can be stacked below an initial consonant using the
alternate form (aka Coeng - foot in English) to form a consonant cluster [3].
Khmer writing also uses diacritical signs which are placed above a consonant.
Dependent vowels cannot stay alone by themselves and have to be attached to an
initial consonant [5]. Orthographically, a dependent vowel can be placed to
the left, right, above, below, or around a base consonant [6].

Therefore, Khmer scripts require complex rendering layout which is not the
case of Latin-based writing systems. A few examples of Khmer words are
selected for illustration purpose as given in Figure 1.

A complete Khmer OCR system needs to recognize all characters, given the
complexity of Khmer writing.

Figure 1: A few examples of Khmer words illustrating complex glyph layout

##  2 Related Work

###  2.1 OCR for Khmer Language

One of the early work in Khmer OCR was done by [7]. The proposed method was a
variant of instance-based classifiers. The authors used wavelet descriptors to
extract features (coefficients) from images in the training set and built a
template for each character. For a given new input image, a set of wavelet
coefficients was extracted and matched against all the training templates. The
input image was then assigned to the class with the smallest Euclidean
distance.

[8], from PAN Localization Cambodia team, proposed a complete workflow for
recognizing Khmer text. The four-steps workflow included pre-processing,
segmentation, recognition and mapping. The pre-processing step included line
separation and character block segmentation. Blocks of characters were then
segmented into atomic shapes namely: Main Body, SuperScript, SubScript,
CCDown, and CC (Complex Character). Discrete cosine transform was used to
extract features from the atomic shapes for classification task. The
recognized shapes were finally mapped to produce valid Khmer text. The average
recognition rate was reported to be 96.34

[9] applied artificial neural network in recognizing Khmer characters. The
approach applied to individual character recognition. The authors proposed
two-steps recognition pipeline. An input image (20 by 20 pixels) was first
passed into a self-organizing network which grouped the input image into one
of the 9 classes. Each class had one multilayer neural network which
classified the input image to one of 82 Khmer characters including consonant,
vowels and numbers.

[10] proposed a convolution neural network (CNN) classifier in recognizing
ancient Khmer characters from palm leaf manuscript. The proposed CNN
architecture was applied to individual character recognition and is composed
of 3 convolutional blocks and a linear classifier. The classifier output a
vector of 106 elements representing character classes. The accuracy on the
test set was reported around 95.96%.

[11] applied support vector machine SVM algorithm to recognize Khmer
characters. The proposed pipeline was composed of four steps - character
segmentation, feature extract, classification and character re-assembling. The
reported character classification accuracy was about 98% for various font
sizes.

[1] experimented both multilayer and convolutional neural networks to
recognize Khmer consonants. A CNN-based model was compared against artificial
neural network (ANN)-based classifier with full feature set and ANN-based
classifier with reduced feature set. The CNN model achieved up to 94.85%
average accuracy.

[12] fined tune a pre-trained Tesseract OCR engine for khmer Unicode and
legacy Lemon fonts. Tesseract is an end-to-end multilingual OCR engine.
Tesseract uses deep convolutional recurrent neural network architecture with
connectionist temporal classification (CTC) loss. Tesseract learns feature
representation automatically via one convolutional layer followed by multiple
stacked LSTMs[17]. Tesseract can recognize a text-line image. [12] reported
the accuracy of 90% on the trained fonts.

###  2.2 Encoder-Decoder Networks

Although deep neural networks (DNNs) are very good models in computer vision
or natural language processing, DNNs are not able to handle inputs and targets
of variable length. This limitation prevents DNNs from being applied to
certain tasks such as speech recognition and machine translation, in which
sequence lengths are not fixed [13].

Recurrent neural networks (RNNs), on the other hand, can encode an input
sequence of unknown length to produce a fixed-dimension representation of the
input sequence. Thus, [13] proposed an RNN known as called encoder to encode
the input sequence of variable length to a fixed-dimension vector
representation, commonly known as a context vector. The context vector was
passed to another RNN known as decoder to extract the information and generate
the output sequence any length.

[13] applied the encoder-decoder network also known as Seq2Seq to the English-
French machine translation task. [13] achieved a BLEU score of 34.81 which
was, at that time, the best result by neural machine translation. [13] made
the observation that the the Long Short-Term Memory (LSTM) architecture
performed robustly even on long sequences although previous study suggested
otherwise.

[14] suggested that using a single fix-dimension context is a bottleneck for
the encoder-decoder network. To remove this bottleneck, [14] proposed the
attention mechanism to allow the decoder to selectively use only parts of the
input sequence that are useful for predicting a target word.

The key distinction between [13] and [14] is that [13] proposed a single
fixed-length vector for the whole input sentence, while [14] proposed to
encode the input sequence as a sequence of vectors and the decoder chooses a
subset of these vectors adaptively via attention mechanism. while decoding the
translation.

The experimental results from [14] suggested that the proposed approach
outperformed the conventional encoder–decoder significantly on the English-to-
French translation task.

OCR task can be viewed as another machine translation task. The input to the
OCR systems is an input image of a text-line with unknown length. The output
of the OCR systems is a character sequence of unknown length. Therefore, the
solutions to machine translation task can also be applied to OCR task.

[15] proposed an end-to-end encoder-decoder network for recognizing printed
text. [15] used long short-term memory (LSTM) as both word image reader
(encoder) and word image predictor (decoder). [15] reported the label rate of
0.84 % on the annotated English word images.

[16] proposed an end-to-end convolutional recurrent neural network with
connectionist temporal classification (CTC) loss for Bengali handwritten
words. [16] experimented with various CNN feature extractors such as DenseNet,
Xception, NASNet, and MobileNet and different recurrent neural networks such
as LSTM and GRU. [16] reported 0.091 character error rate (CER) and 0.273 word
error rate (WER) performed using DenseNet121 model with GRU recurrent layer.

In this paper, we attempt to approach the Khmer OCR task by using an encoder-
decoder network. We propose an end-to-end, attention-based encoder-decoder
network (Seq2Seq). The encoder is a convolutional recurrent neural network
with skip (residual) connections while the decoder is a recurrent neural
network and a linear classifier. During decoding, the attentive decoder uses
an attention mechanism to search for the relevant encoder outputs. Gated
Recurrent Unit (GRU) is used in both the decoder and encoder. The entire OCR
network can read a text-line image of arbitrary length and produces a
character sequence of unknown length. Being an end-to-end solution, the
proposed solution does not need any pre-processing, feature extraction and
post-processing steps.

The overall contributions of this work to Khmer OCR are as follows:

  * •

We introduce one of the first end-to-end solutions to Khmer OCR.

  * •

The proposed solution outperforms the current state-of-art Tesseract engine
for Khmer language.

##  3 End-to-End Khmer OCR Model

We present the end-to-end Khmer OCR model using a Seq2Seq network with an
attention mechanism. The end-to-end OCR model does not require pre-processing
steps such as character separation and post-processing steps such as character
mapping or assembling.

The proposed Seq2Seq network takes a text-line input image via the encoder
network. The encoder network passes the encoded information via a context
vector to the decoder network to decode one character at a time until end-of-
sentence token is reached. There is no limit on the number of characters in
the input text-line image that the encoder can read and the decoder can
produce. The attention mechanism allows the network performs robustly even
with an input image with very long text-line.

The high-level architecture of the end-to-end, attention-based Seq2Seq network
for Khmer OCR is given in Figure 2. Figure 2 shows the decoding process of the
second timestep while the decoding process of the timestep is being grayed
out. Detailed description of the encoder, decoder and attention is provided in
the subsequent sections.

Figure 2: The end-to-end, attention-based Seq2Seq network for Khmer OCR

###  3.1 Encoder Network

The encoder network is composed of a series of convolutional blocks with skip
connections and a layer of GRU units. The encoder network is composed of the
followings:

  1. 1.

A convolutional layer with 3x3 filter, 1x1 padding and 1x1 stride. This
convolutional layer has input channel of 1 and output channel of 16. Batch
normalization and ReLU are applied to the resulting feature maps.

$O_{conv1}=Conv2D(img)$ (1)

where:

     * •

$O_{conv1}$: are the feature maps.

     * •

$img$: is the text-line input image.

  2. 2.

The first convolution layer is followed by 3 layers of the residual blocks.
Each layer has 2 residual blocks. There are two types of the residual blocks:

     * •

The residual block without downsampling is shown in Figure 4. There is no
change to the dimension of the feature maps within this residual block.

     * •

The residual block with downsampling is shown in Figure 4. In the first
convolution of the residual block, 2x1 stride is applied and thus, the height
of the resulting feature maps is reduced by half. Therefore, the input feature
maps to the residual block must be downsampled before being adding to the
output feature maps from the last convolution in the residual block.

$O_{resnet}=ResLayers(O_{conv1})$ (2)

     * •

$O_{resnet}$: are the feature maps.

     * •

$ResLayers$: are the 3 layers of residual blocks.

The input feature maps to the first layer of residual blocks have 16 channels
and the number of channels is maintained across the first layer. The second
and third layer of the residual blocks increase the depth of the feature maps
twice in each layer. That means after the third layer of the residual blocks,
the feature maps have 64 channels.

  3. 3.

Average pooling with 8x8 filter and 8x6 stride is applied to the feature maps
from the last layer of residual blocks, followed by a 2D dropout layer. The
resulting feature maps are reshaped into a width-by-channel format by
flattening the height into the channel dimension.

$O_{avgpooling}=AvgPooling2D(O_{resnet})$ (3)
$O_{dropout2D}=Dropout2D(O_{avgpooling})$ (4)

  4. 4.

Finally, the width-by-channel input is fed to a layer of bidirectional GRU
units. The GRU recurrent network produces encoder hidden states - $H$ =
{$h_{1}$,$h_{2}$,…,$h_{T}$}. An encoder hidden state, $h_{t}$
($[h_{t}^{\rightarrow};h_{T-t-1}^{\leftarrow}]$) is a concatenated forward and
backward hidden state, at a timestep, $t$ in the input sequence (i.e. width of
the final feature maps). The last hidden state of the forward GRU layer,
$h_{T}^{\rightarrow}$ is fed to a dropout layer and then, a linear layer
followed by a $tanh$ activation function to produce a hidden state vector $z$
as its initial hidden state, $s_{0}$ for the decoder GRU network.

$H,h_{T}^{\rightarrow}=EncoderGRU(O_{dropout2D})$ (5)

$z=Tanh(LinearLayer(dropout(h_{T}^{\rightarrow})))$ (6)

In a nutshell, the encoder network take an input image with a text-line and
returns a sequence of hidden states, $H$ and the decoder’s initial hidden
state, $s_{0}$ which is $z$.

Figure 3: The architecture of the encoder network Figure 4: The residual
blocks - with and without downsampling

###  3.2 Attention

At each decoding step $i$, the decoder needs to search the relevant
information from the encoder’s hidden states adaptively. This can be done by
calculating a weighted average context vector, $c_{i}$, over the encoder’s
hidden states [14].

$c_{i}=\sum_{j=1}^{T_{x}}\alpha_{ij}*h_{j}$ (7)

where:

  * •

$T_{x}$ is the length of the input sequence.

  * •

$c_{i}$ is the weighted average context vector at decoding step $i$.

  * •

$\alpha_{ij}$ is the attention weight for the encoder’s hidden state at time
$j$ and the decoder’s previous hidden state at time $i$.

  * •

$h_{j}$ is the encoder’s hidden state at time $j$.

$\alpha_{ij}$ can be computed using the below formula [14].

$\alpha_{ij}=\frac{e^{e_{ij}}}{\sum_{k=1}^{T_{x}}e^{e_{ik}}}$ (8)

$e_{ij}=v^{T}_{a}tanh(W_{a}s_{i-1}+U_{a}h_{j})$ (9)

where:

  * •

$v_{a}$, $W_{a}$, and $U_{a}$ are weight matrices.

  * •

$s_{i-1}$ is the decoder’s previous hidden state.

###  3.3 Decoder Network

The decoder network is a single layer GRU network followed by a linear
classifier. The inputs to the GRU network are:

  * •

$d(y_{t})$, the previous decoded target that is one-hot encoded. For $t=1$,
$y_{0}$ is a special start-of-sentence (SOS) token.

  * •

$s_{t-1}$, the decoder’s previous hidden state.

  * •

$c_{t}$, the weighted average context vector at decoding step $t$.

$d(y_{t})$ and $c_{t}$ are concatenated as a single input vector to the GRU
layer. Together with the previous hidden state, $s_{t-1}$, the GRU layer
produces a new hidden state, $s_{t}$.

$s_{t}=DecoderGRU([d(y_{t}),c_{t}],s_{t-1})$ (10)

Next,$s_{t}$, $d(y_{t})$ and $c_{t}$ are concatenated as a single input vector
to the linear classifier to predict the target, $y_{t+1}$.

$y_{t+1}=LinearClassifer([s_{t},c_{t},d(y_{t})])$ (11)

The decoding process is repeated until $y_{t+1}$ is a special end-of-sentence
(EOS) token

##  4 Experiments

The proposed solution was applied to Khmer OCR task. Each input image contains
a text-line of Khmer words, phrases or sentences from Khmer Asian Language
Treebank (ALT) 111https://www2.nict.go.jp/astrec-att/member/mutiyama/ALT/ .
The dataset was generated by using text2image 222developed by Tesseract team
tool for a number of common Khmer fonts. The rendered images were augmented
with speckle noise, dilation/erosion and rotation. The proposed solution was
benchmarked against the state-of-art OCR tool for Khmer language which is
Tesseract.

###  4.1 Dataset Details

The train dataset consists of 92,213 words, phrases or sentences. The
text2image was used to generate the corresponding images for a number of Khmer
fonts including OS, OS Siemreap, OS Battambang, OS Bokor, OS Freehand, and OS
Fasthand. The images for different fonts for the same word is given in Figure
5. The test dataset consists of 3000 images.

The text2image generates images with variable width and height, depending on
the word or phrase or sentence length and the presence of subscript
consonants, vowels and diacritics. Therefore, the input images are scaled to a
common height of 64 pixels that givens the final height of 2 after the
convolutional layers in the encoder network. According to [17], Tesseract uses
a common height of 48 pixels.

Figure 5: Different input images of a Khmer word for the different fonts

###  4.2 Training Details

The following hyper-parameters were used to train the proposed Seq2Se2 model.

  * •

$outdim_{decoder}$ is the output dimension of the linear classifier in the
decoder network. $outdim_{decoder}$ is equal to the number of Khmer characters
including vowels (dependent + independent), consonants, diacritics, and
numbers.

  * •

$hiddendim_{encoder}$ is the hidden dimension of the GRU layer in the encoder.
300 was used.

  * •

$hiddendim_{decoder}$ is the hidden dimension of the GRU layer in the decoder.
300 was used.

  * •

$Dropout_{encoder}$ is the dropout probability in the encoder network. 0.2 was
used.

  * •

$Dropout_{decoder}$ is the dropout probability in the decoder network. 0.5 was
used.

  * •

$Optimizer$ is Adam optimizer with initial learning rate (LR) of 0.001.

  * •

$LR\ schedule$: the learning rate was decreased by half every 10 epochs.

  * •

$Batch\ size$ is the number of input images loaded per optimization iteration.
32 was used.

  * •

$Teacher\ Forcing\ Ratio$ is the probability of using the actual target for
$y_{t}$ instead of the decoder’s prediction at time $t$. The value of 0.5 was
used.

  * •

$Cross\ Entropy$ loss function was used.

The proposed Seq2Seq model was implemented in PyTorch 333adapted from
https://github.com/bentrevett/pytorch-seq2seq and the model was trained on
Google Colab virtual machine.

##  5 Results and Discussions

###  5.1 Evaluation Metrics - CER and WER

The trained model was evaluated on the test set of 3000 images. The model
achieved the perplexity of 1.037 and average cross-entropy loss of 0.037.
Character and word error rate were also computed. The character error rate
(CER) is the ratio of unrecognized characters over the total number of
characters in the target words. The word error rate(WER) is defined similarly.
CER was computed using Levenshtein distance.

The CER is defined as below [16]:

$CER=\frac{S+I+D}{N}$ (12)

where:

  * •

$S$ is number of substitutions.

  * •

$I$ is number of inserts.

  * •

$D$ is number of deletes.

  * •

$N$ is number of ground-truth characters .

The WER is defined as below:

$WER=\frac{count\ of\ incorrect\ samples}{count\ of\ samples}$ (13)

In this study, WER is a conservative error metric since each sample image can
at minimum a word and at maximum a sentence.

The model achieved a CER of 1% and 9% WER on the test set while Tesseract gave
3% CER and 26% WER.

###  5.2 Error Analysis

We investigated the errors produced by Tesseract and the proposed model. The
following observations were made:

  * •

Some of the erroneous cases by Tesseract were due to that Tesseract was likely
to produce symbols such as @, #, \ that do not present in the ground truth.
This is not the case of the proposed model since the proposed model outputs
only Khmer characters and symbols only.

  * •

Some of the erroneous cases by the proposed model were due to that the model
was likely to produce repeated characters before EOS token was reached. This
is not the case of Tesseract since the Tesseract does not have separate
encoder and decoder networks.

  * •

Most of the erroneous cases by Tesseract and the proposed model were due to
that some Khmer characters have similar glyphs. The pairs of Khmer characters
with similar glyphs are given in Figure 6. These pairs were identified by
aligning the erroneous cases and their corresponding ground truths.

Figure 6: Pairs of Khmer characters with similar glyphs

###  5.3 Visualizing Attention

Figure 7, Figure 8 and Figure 9 are the input images along with the attention
heatmaps produced by the model. The attention heatmaps show parts of the input
images at which the decoder pays attention at each step.

Figure 7: Example 1 - Input image and attention heatmap Figure 8: Example 2 -
Input image and attention heatmap Figure 9: Example 3 - Input image and
attention heatmap

##  6 Printed Khmer Text Extraction Application

###  6.1 PDF Text Extraction

The trained model was used to extract text from a PDF document. The sample PDF
text is given in Figure 10 444obtained from http://krou.moeys.gov.kh/.

Since the trained model accepts input as a text-line image. The following
image processing steps were required.

  * •

Thresholding was applied by using Otsu’s thresholding. The resulting image
after thresholding is given in Figure 11.

  * •

Dilution was then applied to increase the boundaries of regions of foreground
pixels. The resulting image after dilution is given in Figure 12.

  * •

Contour detection was used to detect the borders of the connected characters.
The detected contours are shown in Figure 13.

Each detected contour was, finally, fed into the trained OCR model to extract
the digital text. The recognized text, corresponding to Figure 10, is given in
Figure 14.

Figure 10: Sample Khmer text in PDF format Figure 11: After thresholding
Figure 12: After dilution Figure 13: The detected contours Figure 14: The
recognized text, corresponding to Figure 10

###  6.2 Camera Text Extraction

We applied the same above extraction processes for text-line images from a
smart phone’s camera. The camera text-line images are given in Figure 15 and
Figure 16, illustrating the presence of background and text deformation
effects.

The extracted texts from the proposed model and Tesseract are given in Figure
17 and Figure 18. Figure 17 and Figure 18 suggest that both the proposed model
and Tesseract are able to recognize text in camera images with acceptable
accuracy.

Figure 15: Sample Khmer text from a smart phone’s camera Figure 16: Another
ample Khmer text from a smart phone’s camera Figure 17: The recognized text by
the proposed model and Tesseract Figure 18: The recognized text by the
proposed model and Tesseract

##  7 Conclusion and Future Work

This paper presents an end-to-end Khmer OCR system which utilizes an encoder-
decoder (Seq2Seq) network. The encoder is composed of layers of convolutional
residual blocks and a layer of GRU units. The decoder consists of a layer of
GRU units and a linear classifier. The decoder uses attention mechanism to
adaptively select parts of the encoder’s outputs that are relevant for
predicting the target character. The proposed model was trained a collection
of text-line images generated from ALT dataset by using open-source text2image
tool with degrading effects. The experiment results suggested that the
proposed solution outperformed the current state-of-art Tesseract engine for
Khmer OCR by achieving a CER of 1% vs 3% and a WER of 9% vs 26% We also
present necessary image processing steps to extract text-line images from a
PDF document before running the OCR system.

A robust OCR system must be able to recognize text in presence of text
deformations, and noisy background. The system must be also font-invariant. In
the future, we would like to train the model on more complex, augmented text-
line images and additional Khmer fonts are included.

## References

  * [1]  Bayram Annanurov and Norliza Mohd Noor.  Khmer handwritten text recognition with convolution neural networks.  ARPN Journal of Engineering and Applied Sciences, 2018.
  * [2]  Jamshed Memon, Maira Sami, Rizwan Ahmed Khan, and Mueen Uddin.  Handwritten optical character recognition (ocr): A comprehensive systematic literature review (slr).  IEEE Access, 8:142642–142668, 2020.
  * [3]  Rina Buoy, Nguonly Taing, and Sokchea Kor.  Joint khmer word segmentation and part-of-speech tagging using deep learning, 2021.
  * [4]  Makara Sok.  Phonological Principles and Automatic Phonemic and Phonetic Transcription of Khmer Words.  PhD thesis, Payap University, 2016.
  * [5]  Rina Buoy, Nguoly Taing, and Sokchea Kor.  Khmer word segmentation using bilstm networks.  4th Regional Conference on OCR and NLP for ASEAN Languages (ONA 2020), 2020.
  * [6]  Chenchen Ding, Masao Utiyama, and Eiichiro Sumita.  Nova: A feasible and flexible annotation system for joint tokenization and part-of-speech tagging.  ACM Trans. Asian Low-Resour. Lang. Inf. Process., 18(2), December 2018.
  * [7]  C. Chey, P. Kumhom, and K. Chamnongthai.  Khmer printed character recognition by using wavelet descriptors.  International Journal of Uncertainty , Fuzziness and Knowledge-Based Systems, 1, 2005.
  * [8]  Ing Lenleng and Ahmed Muaz.  Khmer optimical character recognition (ocr).  PAN Localization Cambodia, 1, 2015.
  * [9]  Hann Meng and Daniel Morariu.  Khmer character recognition using artificial neural network.  APSIPA, 1, 2014.
  * [10]  Dona Valy, Michel Verleysen, Sophea Chhun, and Jean-Christophe Burie.  A new khmer palm leaf manuscript dataset for document analysis and recognition: Sleukrith set.  In Proceedings of the 4th International Workshop on Historical Document Imaging and Processing, HIP2017, page 1–6, New York, NY, USA, 2017\. Association for Computing Machinery.
  * [11]  Pongsametrey Sok and Nguoly Taing.  Support vectir machine(svm)-based classifier for khmer character-set recognition.  APSIPA, 2014.
  * [12]  Kim Sokphyrum, Suos Samak, and Javier Sola.  Khmer ocr finte tune engine for unicode and legacy fonts using tesseract 4.0 with deep neural network.  Optical character recognition for complex scripts and Natural language processing for ASEAN Languages, 1, 2019.
  * [13]  Ilya Sutskever, Oriol Vinyals, and Quoc V. Le.  Sequence to sequence learning with neural networks, 2014.
  * [14]  Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate, 2014.
  * [15]  Devendra Kumar Sahu and Mohak Sukhwani.  Sequence to sequence learning for optical character recognition, 2015\.
  * [16]  Farisa Benta Safir, Abu Quwsar Ohi, M. F. Mridha, Muhammad Mostafa Monowar, and Md. Abdul Hamid.  End-to-end optical character recognition for bengali handwritten words, 2021.
  * [17]  Bernhard Liebl and Manuel Burghardt.  On the accuracy of crnns for line-based ocr: A multi-parameter evaluation, 2020.


# Reconstruction of 3-Axis Seismocardiogram from Right-to-Left and Head-to-
Foot Components Using A Long Short-Term Memory Network ††thanks: *This work
was supported by Mississippi State University’s Office of Research and
Economic Development through the Quick Grants Program.

Mohammad Muntasir Rahman  Dept of Ag. & Biological Engineering
Mississippi State University
Mississippi State, MS 39762, USA
<EMAIL_ADDRESS>Amirtahà Taebi Corresponding author:
<EMAIL_ADDRESS>Dept of Ag. & Biological Engineering
Mississippi State University
Mississippi State, MS 39762, USA
<EMAIL_ADDRESS>
###### Abstract

This pilot study aims to develop a deep learning model for predicting
seismocardiograms (SCGs) in the dorsoventral direction from the SCG signals in
the right-to-left and head-to-foot directions (SCGx and SCGy). The dataset
used for the training and validation of the model was obtained from 15 healthy
adult subjects. The SCG signals were recorded using tri-axial accelerometers
placed on the chest of each subject. The signals were then segmented using
electrocardiogram R waves, and the segments were downsampled, normalized, and
centered around zero. The resulting dataset was used to train and validate a
long short-term memory (LSTM) network with two layers and a dropout layer to
prevent overfitting. The network took as input 100-time steps of SCGx and
SCGy, representing one cardiac cycle, and outputted a vector that mapped to
the target variable being predicted. The results showed that the LSTM model
had a mean square error of 0.09 between the predicted and actual SCG segments
in the dorsoventral direction. The study demonstrates the potential of deep
learning models for reconstructing 3-axis SCG signals using the data obtained
from dual-axis accelerometers.

###### Index Terms:

Seismocardiography, heart vibration, signal reconstruction, deep learning,
LSTM network.

##  I Introduction

Cardiovascular diseases (CVDs) are the leading cause of death in the United
States, claiming the life of one person every 34 seconds, resulting in a
staggering 2,544 deaths per day based on 2020 data [1]. Beyond the devastating
human toll, this places an immense strain on healthcare systems and society,
with significant economic costs associated with treatment and lost
productivity. The early detection of cardiac abnormalities is crucial for
achieving better outcomes for patients with CVD, and improving diagnostic
methods and accessibility is a key step in this direction. Current diagnostic
methods, including non-invasive techniques such as electrocardiography (ECG),
medical imaging, and cardiac catheterization, can aid in identifying CVDs.
Advancements in technology have also led to the development of new diagnostic
options, such as wearable and remote monitoring systems, which can provide
continuous monitoring of patients’ cardiovascular health outside of
traditional healthcare settings. Seismocardiography (SCG) is another technique
that noninvasively monitors cardiovascular activity by measuring
cardiovascular-induced vibrations on the chest [2, 3, 4]. These vibrations
result from a range of cardiac activities, including valve opening and
closing, isovolumetric contraction, blood ejection, and rapid left ventricle
filling [3, 5, 6, 7]. Unlike other non-invasive techniques such as ECG and
pulse oximetry, which focus on the electrical activity of the heart and the
blood oxygen level, respectively, SCG provides complementary insights into the
mechanical activity of the heart [8, 5, 9]. With its ability to evaluate these
mechanical activities, SCG has the potential to offer valuable diagnostic
information for cardiac conditions such as heart failure, myocardial
infarction, ischemia, and hemorrhage, as changes in the mechanical function of
the heart can be an early indication of these diseases [10, 11, 12, 13, 14,
15]. As a result, SCG can enhance our understanding of the cardiac function
and contribute to the development of more accurate diagnostic tools for
patients with CVD. SCG signals are commonly measured using accelerometers that
are placed on the chest surface. These signals are typically measured in three
directions of right-to-left, head-to-foot, and dorsoventral. In that regard,
while single or dual-axis accelerometers may be used for SCG measurement,
three-axis accelerometers are more informative as they offer a more
comprehensive understanding of the motion of the heart and chest wall [8].

This study aims to answer the question: “Can we generate three-axis SCG
measurements using a dual-axis accelerometer?” More specifically, is it
possible to generate SCG vibrations in the $z$ direction using the
measurements from the $x$ and $y$ axes of a dual-axis accelerometer? In this
work, we propose a deep neural network model based on long short-term memory
(LSTM) to predict the SCG component in the dorsoventral direction (SCGz) using
the vibrations in right-to-left and head-to-foot directions (SCGx, SCGy). LSTM
models have been successfully applied to a wide range of time series-related
problems, including but not limited to stock price prediction, energy load
forecasting, weather forecasting, speech recognition, and natural language
processing. In this paper, we designed a regression model based on a stacked
LSTM neural network with two layers to process the SCGx and SCGy sequences
corresponding to a single cardiac cycle and generate an output sequence of
SCGz of the same length. The use of a stacked LSTM network can potentially
improve the accuracy of the predictions by allowing the network to learn more
complex temporal patterns in the data.

##  II Materials and Methods

Figure 1: (a) A representation of a LSTM cell. (b) Stacked LSTM network
architecture.

###  II-A Long Short-Term Memory Network

The basic building block of an LSTM cell consists of a memory cell and three
gates: the input gate, the forget gate, and the output gate. The memory cell
is responsible for storing information over time and passing it forward
through the sequence. The input gate controls how much new information is
added to the cell state, the forget gate controls how much old information is
removed from the cell state, and the output gate controls how much information
from the cell state is used to compute the hidden state. Fig. 1a shows a
schematic diagram of an LSTM unit. At each time step $t$, the LSTM cell takes
as input the current input sequence $x_{t}$, the previous hidden state
$h_{t-1}$, and the previous cell state $c_{t-1}$. Using these inputs, the LSTM
cell generates the current hidden state $h_{t}$ and the updated cell state
$c_{t}$. Let

$\displaystyle i_{t}=\sigma(W_{x}^{(i)}x_{t}+W_{h}^{(i)}h_{t-1}+b^{(i)})$ (1)
$\displaystyle f_{t}=\sigma(W_{x}^{(f)}x_{t}+W_{h}^{(f)}h_{t-1}+b^{(f)})$ (2)
$\displaystyle o_{t}=\sigma(W_{x}^{(o)}x_{t}+W_{h}^{(o)}h_{t-1}+b^{(o)})$ (3)
$\displaystyle\tilde{c}_{t}=tanh(W_{x}^{(c)}x_{t}+W_{h}^{(c)}h_{t-1}+b^{(c)})$
(4)

where $i_{t}$, $f_{t}$, and $o_{t}$ are the input, forget, and output gates,
respectively, and $\tilde{c}_{t}$ provides the change contents, then the
updated cell state $c_{t}$ and hidden state $h_{t}$ are computed as:

$\displaystyle c_{t}=f_{t}\odot c_{t-1}+i_{t}\odot\tilde{c}_{t}$ (5)
$\displaystyle h_{t}=o_{t}\odot tanh(c_{t})$ (6)

where $\odot$ performs element-wise product.

###  II-B Study Population

The study enrolled a total of 15 participants (including 4 female), with no
prior history of cardiovascular diseases (age: 25.93 $\pm$ 10.65 year, height:
171.31 $\pm$ 9.22 cm, weight: 74.83 $\pm$ 22.83 kg, body mass index: 25.29
$\pm$ 6.58 kg/m2). The study sample was diverse, with participants from
various racial and ethnic backgrounds, including 53.3% White, 20% Black, 20%
Asian, and 6.7% mixed. The Mississippi State University Institutional Review
Board approved the study protocol.

###  II-C Data Acquisition Protocol

To minimize any potential movement artifacts that could affect the quality of
the data, all subjects were instructed to lay supine on a bed without
additional body movements. Three triaxial accelerometers (356A32, PCB
Piezotronics, Depew, NY) were attached to three locations on the sternum
including the manubrium, the fourth costal notch, and the xiphoid process. The
accelerometer outputs were amplified using a signal conditioner (482C, PCB
Piezotronics, Depew, NY) with a gain factor of 100 to increase the signal-to-
noise ratio. The amplified signals were then recorded using a data acquisition
system (416, iWorx Systems, Inc., Dover, NH), with a sampling frequency of
5000 Hz. A microphone was connected to the system and tapped at the beginning
and end of each recording. These taps were then located in the sound signals
to identify the start and end of the intended part of the recording. An ECG
module was also used to simultaneously record the ECG signal. Data were
collected from all accelerometers during a 15-second breath-hold at the end of
inhalation and exhalation, and 2 additional minutes of normal breathing.

###  II-D SCG Dataset

To prepare the dataset, the following pre-processing steps were carried out to
eliminate noise from the raw signals. The first step involved applying a
moving average filter to smooth the SCG signals. This step helps reduce high-
frequency noise in the data, making it easier to detect underlying patterns or
features. Subsequently, a band-pass filter was applied to the accelerometer
outputs with cutoff frequencies of 1 and 30 Hz. This eliminated the low-
frequency respiration vibrations and the higher-frequency SCG components above
30 Hz.

SCG signals were then segmented using the ECG R waves as reference points. To
detect the R waves, we utilized Pan-Tompkins algorithm [16]. We then computed
the average duration of the cardiac cycle for each subject from the ECG RR
intervals. This information was then used to determine the window size to
segment the SCG signals for each subject. Specifically, we set the start of
the window to 1/4 of the average cardiac cycle duration before the R wave and
the end of the window to 3/4 of the average cardiac cycle duration after the R
wave, resulting in consistent SCG segments. After segmenting the SCG signals
for each cardiac cycle, the segments were downsampled to a fixed number of
sample points (100 points in this study). This allowed us to obtain consistent
segment lengths across all samples for different subjects, ensuring that the
input data for the LSTM model was uniform. This is crucial because the neural
network takes one segment at a time as input, and the original segments may
have varied lengths for different subjects (due to different cardiac cycle
durations). By treating each SCG segment as a separate sample in the dataset,
the neural network can learn patterns and features specific to each cycle,
which can be useful for predicting SCG signals in the $z$-direction.

Next, the segments were normalized to have values between -1 and 1 to ensure
that the input data falls within a suitable range for activation functions.
Then, the mean value was subtracted from each segment to center the signals
around zero. This step was useful in mitigating any DC offset or baseline
drift present in the signal that could affect the performance of the neural
network. In the last step, the pre-processed and normalized SCG segments from
all subjects were combined to form a single dataset, which was used for
training, validating and testing the neural network model. A total of 7492 SCG
segments were used for training and validation and the model was tested on 475
segments. This dataset was expected to capture the common patterns and
features present in the SCG signals of the population studied, and was thus
representative of the entire cohort.

###  II-E Network Architecture and Training

Fig. 1b provides an overview of a stacked LSTM network’s architecture employed
in this study. A stacked LSTM refers to the use of multiple LSTM layers, one
on top of the other which allows learning more complex patterns and
relationships in the data. The network has two stacked LSTM layers, each with
a hidden size of 512. The hidden size refers to the number of hidden units or
neurons in the LSTM layer, which determines the capacity of the network to
capture complex patterns in the data. The network processes 100-time steps of
SCGx and SCGy as a sample input, which corresponds to approximately one
cardiac cycle. Utilizing dual-axis rather than single-axis SCG data may help
the network to capture more information and learn the relationship between the
input (SCGx and SCGy) to predict the SCGz and make accurate predictions on
new, unseen data. The input then flows into the stacked LSTM layers. The final
layer outputs a vector $h_{i}$, which is subsequently fed into a fully
connected layer featuring 100 output neurons. This fully connected layer maps
the input vector to a set of output values that correspond to the target
variable predicted by the model. To prevent overfitting and improve the
generalization, a dropout layer is added after each LSTM layer. The dataset
was divided into training, validation, and testing sets to train and evaluate
the network. Since we collected data from three accelerometers for each
subject, we separated breath-hold data captured at the end of inhalation and
at the end of exhalation from the accelerometer attached to the xiphoid
process to test the model. The remaining data were used for training and
validation. The training set, which contained 90% of the data, was used to
determine the weight and bias parameters through forward and backward
propagation during the training process. The validation set, which contained
10% of the data, was used to evaluate the model performance during the
training process. We trained the network with a maximum of 1000 epochs and
used early stopping to prevent overfitting during training. Early stopping is
a regularization technique to monitor the performance of the model on the
validation set during training and stop the training when the performance on
the validation set no longer improves. The initial learning rate was 0.001,
the learning rate was reduced using learning rate decay when the validation
set stopped improving for a specified number of epochs. This technique helped
the model converge to a better solution and avoid getting stuck in local
minima.

TABLE I: Accuracy of the model in terms of MSE Subject | Exhale-MSE | Inhale-MSE | Total-MSE
---|---|---|---
Subject 01 | 0.064 | 0.083 | 0.07
Subject 02 | 0.037 | 0.115 | 0.07
Subject 03 | 0.094 | 0.096 | 0.09
Subject 04 | 0.069 | 0.071 | 0.07
Subject 05 | 0.087 | 0.150 | 0.11
Subject 06 | 0.055 | 0.062 | 0.05
Subject 07 | 0.107 | 0.057 | 0.08
Subject 08 | 0.070 | 0.163 | 0.11
Subject 09 | 0.079 | 0.063 | 0.07
Subject 10 | 0.073 | 0.087 | 0.08
Subject 11 | 0.189 | 0.098 | 0.14
Subject 12 | 0.058 | 0.077 | 0.06
Subject 13 | 0.129 | 0.107 | 0.11
Subject 14 | 0.071 | 0.082 | 0.07
Subject 15 | 0.140 | 0.092 | 0.11
All Subjects | 0.087 | 0.096 | 0.09
Figure 2: Mean square error between the predicted and actual SCGz segments in
the test set. Figure 3: Predicted versus actual SCGz segments for Subjects 6
and 12.

##  III Results and Discussion

Our goal was to investigate the possibility of predicting SCGz from the SCG
signals in $x$ and $y$ directions through the use of an LSTM neural network.
We used the testing set to evaluate the model’s performance. The accuracy of
the model in predicting SCGz based on SCGx and SCGy was assessed by comparing
the predicted and actual SCG signal in the $z$ direction, using mean-square
error (MSE) as the measure of accuracy. Table I shows the model’s performance
in terms of MSE, while Fig. 2 provides a box plot to illustrate the results
for each subject’s end-exhalation and end-inhalation data, as well as the
total performance based on all end-exhalation and end-inhalation data. The box
plot shows the median, interquartile range, and outliers of the prediction.
The prediction MSE for each subject varied from 0.05 to 0.14, with an MSE of
0.09 for all subjects combined. Four samples of the predicted and actual SCGz
segments from subjects 6 and 12 are presented in Fig. 3. The close resemblance
between the predicted and actual SCGz signals suggests that there is a
correlation between the SCG components in the $x$, $y$, and $z$ directions,
and that our LSTM network was able to learn it.

##  IV Conclusion

The study investigated the possibility of predicting SCGz from SCG components
in $x$ and $y$ directions using an LSTM neural network. Results showed that
the predicted SCGz closely matched the actual signal, indicating a
relationship between the SCG components in the three directions. The study
provides insights into the potential of using a dual-axis accelerometer to
monitor SCG signals in all three directions, which can improve cardiovascular
monitoring methods.

## References

  * [1] C. W. Tsao, A. W. Aday, Z. I. Almarzooq _et al._ , “Heart disease and stroke statistics—2023 update: A report from the american heart association,” _Circulation_ , vol. 147, no. 8, pp. e93–e621, 2023.
  * [2] A. Taebi and H. A. Mansy, “Time-frequency distribution of seismocardiographic signals: A comparative study,” _Bioengineering_ , vol. 4, no. 2, p. 32, 2017\.
  * [3] J. M. Zanetti and D. M. Salerno, “Seismocardiography: a technique for recording precordial acceleration,” in _Computer-Based Medical Systems-Proceedings of the Fourth Annual IEEE Symposium_. IEEE Computer Society, 1991, pp. 4–5.
  * [4] M. M. Rahman, J. Cook, and A. Taebi, “Non-contact heart vibration measurement using computer vision-based seismocardiography,” _Scientific Reports_ , vol. 13, no. 1, p. 11787, 2023.
  * [5] A. Taebi, B. E. Solar, A. J. Bomar _et al._ , “Recent advances in seismocardiography,” _Vibration_ , vol. 2, no. 1, pp. 64–86, 2019.
  * [6] A. Mann, P. Thibbotuwawa Gamage, B. Kakavand, and A. Taebi, “Exploring the impact of sensor location on seismocardiography-derived cardiac time intervals,” _Journal of Engineering and Science in Medical Diagnostics and Therapy_ , 2024.
  * [7] A. Taebi and H. A. Mansy, “Analysis of seismocardiographic signals using polynomial chirplet transform and smoothed pseudo wigner-ville distribution,” in _2017 IEEE Signal Processing in Medicine and Biology Symposium (SPMB)_. IEEE, 2017, pp. 1–6.
  * [8] O. T. Inan, P.-F. Migeotte, K.-S. Park _et al._ , “Ballistocardiography and seismocardiography: A review of recent advances,” _IEEE journal of biomedical and health informatics_ , vol. 19, no. 4, pp. 1414–1427, 2014.
  * [9] J. Cook, M. Umar, F. Khalili, and A. Taebi, “Body acoustics for the non-invasive diagnosis of medical conditions,” _Bioengineering_ , vol. 9, no. 4, p. 149, 2022.
  * [10] K. Sørensen, S. E. Schmidt, A. S. Jensen _et al._ , “Definition of fiducial points in the normal seismocardiogram,” _Scientific reports_ , vol. 8, no. 1, pp. 1–11, 2018.
  * [11] U. Ha, S. Assana, and F. Adib, “Contactless seismocardiography via deep learning radars,” in _Proceedings of the 26th Annual International Conference on Mobile Computing and Networking_ , 2020, pp. 1–14.
  * [12] K. Tavakolian, A. P. Blaber, A. Akhbardeh _et al._ , “Estimating cardiac stroke volume from the seismocardiogram signal,” _CMBES Proceedings_ , vol. 33, 2010.
  * [13] M. Pänkäälä, T. Koivisto, O. Lahdenoja _et al._ , “Detection of atrial fibrillation with seismocardiography,” in _2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_. IEEE, 2016, pp. 4369–4374.
  * [14] M. M. H. Shandhi, J. Fan, J. A. Heller _et al._ , “Estimation of changes in intracardiac hemodynamics using wearable seismocardiography and machine learning in patients with heart failure: a feasibility study,” _IEEE Transactions on Biomedical Engineering_ , vol. 69, no. 8, pp. 2443–2455, 2022\.
  * [15] D. M. Salerno and J. Zanetti, “Seismocardiography for monitoring changes in left ventricular function during ischemia,” _Chest_ , vol. 100, no. 4, pp. 991–993, 1991.
  * [16] H. Sedghamiz, “Matlab implementation of pan tompkins ecg qrs detector,” _Code Available at the File Exchange Site of MathWorks_ , 2014.