Generative Adversarial Network for Handwritten Text
Bo Ji Tianyi Chen
cs.LG cs.CV


Genetic Deep Learning for Lung Cancer Screening
    Hunter Park 

Innovation Dx Inc.

hpark@innovationdx.com
Connor Monahan 

Innovation Dx Inc. 

cmonahan@innovationdx.com

    today
=============================================================================================================================






[1]Zhejiang University, Email: mailto:gsorris27@gmail.comgsorris27@gmail.com
[2]Microsoft Cloud & AI, Email: mailto:Tianyi.Chen@microsoft.comTianyi.Chen@microsoft.com





Generative adversarial networks (GANs) have proven hugely successful in variety of applications of image processing. However, generative adversarial networks for handwriting is relatively rare somehow because of difficulty of handling sequential handwriting data by Convolutional Neural Network (CNN). In this paper, we propose a handwriting generative adversarial network framework (HWGANs) for synthesising handwritten stroke data. The main features of the new framework include: (i) A discriminator consists of an integrated CNN-Long-Short-Term-Memory (LSTM) based feature extraction with Path Signature Features (PSF) as input and a Feedforward Neural Network (FNN) based binary classifier; (ii) A recurrent latent variable model as generator for synthesizing sequential handwritten data. The numerical experiments show the effectivity of the new model.  Moreover, comparing with sole handwriting generator, the HWGANs synthesize more natural and realistic handwritten text.





§ INTRODUCTION


Learning generative sequential models is a long-standing machine learning challenge and historically in the domain of dynamic Bayesian networks (DBNs) such as Hidden Markov Model (HMMs), then further handled by Recurrent Neural Network. Among various  sequential data, handwritten text has served for thousands of years as one of our primary means of communication. Nowadays, following the rapid development of electronic engineering, human being utilizes more and more stylus-based devices, like smartphone, whiteboard, or tablet to write text as digital ink which are easily processed and manipulated. However, despite promising progress in handwriting recognition, synthesizing handwritten text in digital ink is still relatively less exploited. Even though there exist a few well-designed RNN-based generative models <cit.> for handwritten text, it has been observed that sometimes, the synthesized texts of these models 
are not as natural as human-being's normal written text. 

On the other hand, generative adversarial networks (GANs) are a merging technique proposed in 2014 <cit.> to learn deep representation without numerous annotated training data. In literature, GANs consist of an expert, known as Discriminator 𝒟, and a forger, known as Generator 𝒢 <cit.>.  The forger creates forgeries with the aim of faking realistic images, while the expert aims at distinguishing between genuine and faked images. This architecture achieves the generation of high reality, consequently the representation learned by GANs may be used in a variety of applications of image processing, while currently less applied on sequential data generation. Therefore, a common analogy is to think of bringing the idea of GANs into handwritten text generation of digital ink in order to achieve more realistic handwritten text than handwritten generator alone. 


In this paper, we propose a generative adversarial architecture to generate realistic handwritten strokes. The following bulleted points summarize our key contributions.


  * We design a discriminator consisting of a compact CNN-LSTM based feature extraction, followed by an auxiliary feedforward neural network to distinguish between genuine and forgery hand-writing. The input of this feature extraction is encoded handwritten strokes following Path Signature Features <cit.>. 

  * We construct an adversarial architecture by combining the designed discriminator and the handwritten generator proposed by Alex Grave <cit.>, referred as  throughout the whole remaining paper. 

  * We use  to obtain more realistic English handwritten text than . In general,  generate handwritten texts which distribute more spatially uniformly than . On the other hand, the handwritten samples synthesized by  are typically more neat and sometimes possess more diverse styles than .


The remainder of this paper is organized as follows. First, in Section <ref>, we briefly review some related work about handwritten text generation. Next, in Section <ref>, we move on to describe our proposed handwritten GANs (), followed by  numerically demonstrating the effectiveness of  in Section <ref>. And we finally conclude this paper in Section <ref>. 



§ RELATED WORK


Handwritten Text Image GANs: The GANs proposed on 2019 <cit.> aims at generating realistic images of handwritten texts, which is naturally a fit for Optical Character Recognition (OCR). The authors use bidirectional LSTM recurrent layers to get an embedding of the word to be rendered, and then feed it to the generator network. They also modify the standard GAN by adding an auxiliary network for text recognition. However, since this approach can not directly synthesize handwritten text of digital ink, although its generated images are realistic, an additional effective Ink Grab algorithm is further required for the conversion from image to digital stroke.  

Variational RNN (VRNN) Handwritten Generator: VRNN <cit.> explores the inclusion of latent random variables into the hidden state of a RNN by combining a recurrent variational autoencoder (VAE), which is effective modeling paradigm to recover complex multimodal distributions over the non-sequential data space <cit.>. This VRNN has been numerically demonstrated by generating speech and handwritten data, while their synthesized handwritten data are not realistic enough.  

Alex Grave's Handwritten Generator: Alex Grave proposed an RNN based generator model to mimic handwriting data <cit.>, referred as  throughout the whole paper. For each timestamp,  encodes prefix sampled path to produce a set of parameters of a probability distribution of next stroke point, then sample the next stroke point given this distribution.  There are two variants of , i.e., handwritten predictor and handwritten synthesizer, where the later one has the capability to synthesize handwritten text for given text. 







§ PROPOSED ADVERSARIAL APPROACH


In this section, we present our proposed adversarial approach for digital ink handwritten text generation. As standard architectures of GANs,  comprise a discriminator 𝒟 and a generator 𝒢. Discriminator 𝒟 is used to distinguish realistic handwritten text from forgery ones, and described in Section <ref>.  In Section <ref>,  we summarize generator 𝒢, which is designed to forge realistic handwritten text to cheat discriminator adversarially. 

For a more rigorous description, let us introduce the notations used throughout the whole remaining paper. Denote H={S_1,S_2, ⋯, S_n} as a handwritten text instance, which consists of a sequence of strokes, where S_i refers as the i-th stroke of current handwritten instance. One stroke S comprises a sequential stroke points, i.e., S={(x_1,y_1,s_1), (x_2,y_2,s_2), ⋯, (x_m,y_m,s_m)}, where (x_i, y_i, s_i) is i-th stroke point with (x_i, y_i) as its coordinate axis and s_i∈{0,1} as pen up/down status respectively to indicate the starting or ending stroke point. 




 §.§ Discriminator


Discriminator 𝒟 is essentially a binary classifier to predict handwritten text of digital ink as a forgery or not. Consequently, feature extraction is important for successful classification. Since the target handwritten data H is sequential, handling the task by recurrent network seems a natural choice. However, digital ink representations of the same handwritten text, especially the number of stroke points, may be varied dramatically, among different underlying ink representation algorithms, and may further cause significant classification discrepancy by  recurrent network. To solve this issue, rendering stroke points into binary images seems a solution, while lots of useful information like order and evolution of stroke points are lost after rendering. In order to balance rendering technique and sequential property of handwritten data, we uniformly sample the stroke points and further utilize Path Signature Features (PSF) which has been demonstrated the effectiveness in online handwritten text recognition <cit.>. 

Path signature feature is defined over a path, and calculated up to 2nd order <cit.>. More specifically, for each smallest stroke point path {(x_i,y_i), (x_i+1, y_i+1)} consisting of 2 consecutive points, the signature feature p_i,i+1=(p_i,i+1^(0), p_i,i+1^(1), p_i,i+1^(2))∈ℝ^7 is calculated as follows: 

    p_i,i+1^(0)   =1∈ℝ
    
    p_i,i+1^(1)   =(x_i+1,y_i+1)-(x_i, y_i)∈ℝ^2
    
    p_i,i+1^(2)   =p_i,i+1^(1)⊗ p_i,i+1^(1)∈ℝ^4

where ⊗ represents Kronecker matrix product. It follows the definition of PSF (<ref>) that PSF encodes geometrical and order information of stroke points into seven path signature maps. 

These seven path signature maps extracted as described above are stacked to form a seven-channel tensor denoted as T_I, which serves as the input of a following CNN-LSTM binary classification model, as illustrated in Figure <ref>. The CNN-LSTM model consists of a CNN and an LSTM, whose configurations are displayed in Table <ref>. CNN serves to encode extracted path signature feature into a matrix, since the input 3D tensor possesses variable widths, consequently results in that the output matrix of CNN possesses variable columns. Then each column of this encoded matrix is further fed into an LSTM cell sequentially, an FNN is assembled on the top of last output of the LSTM to calculate the probability of current instance as genuine written text. Thus loss function of discriminator is naturally selected as binary cross entropy loss. The whole procedure is shown in Figure <ref>.
















 §.§ Generator


Generator 𝒢 is basically stacked LSTMs with sampling layers for generating strokes points from a certain initial stroke point. In this paper, we consider two architectures of generator designed for prediction and synthesis tasks, denoted as 𝒢_p and 𝒢_s proposed by Alex <cit.>, illustrated as Figure <ref>. The prediction generator 𝒢_p only requires the network to generate random handwritten strokes, while the synthesis 𝒢_s generator further asks the model to produce  handwriting texts given prescribed text of user. 







Although these two generators are used in different scenarios, they share numerous things in common. First of all, both of them are stacked LSTMs with outputs as parameters of an appropriate probability distribution to predict the status of next stroke point given current generated stroke path. For this probability distribution, Alex <cit.> proposed to use a mixture of bivariate Gaussian distribution associated with a Bernoulli distribution to represent the distribution of pen up/down. Our model incorporates the same idea, more specifically the outputs of last layer of LSTMs represent the parameters of distribution for x-axis and y-axis offsets between current and next stroke point, and a parameter to indicate if next stroke point ends up stroke path or not.

Secondly, both generators update their parameters by maximizing log-likelihood of generated stroke sequences. During the adversarial training, they utilize the signal from the discriminator described in Section <ref> to adjust network parameters. Since the discriminator predicts the probability of handwritten text as genuine, the generator aims at maximizing this signal by using generated fake handwritten texts.  Thirdly, both 𝒢_p and 𝒢_s use biased sampling <cit.> to achieve higher quality of handwriting generation. In terms of the input of discriminator formed as Path Signature Features, the generated handwritten strokes are further processed as PSF features. 








Different from the prediction generator 𝒢_p, the inputs of synthesis generator 𝒢_s requires not only the stroke point data but also the text information, since the synthesis network needs to know what texts to generate. The text is an one-hot vector processed from the input string, and input to LSTM layers with window parameters to constrain the synthesis network.






§ EXPERIMENTS


In this section, we numerically demonstrate the superiority of the adversarial network we proposed than .  At first, we describe the experimental settings in Section <ref>, then in Section <ref> we move on to presenting numeric results generated from both  and  to illustrate the advantages of our model.



 §.§ Experimental Settings


Both of the prediction network 𝒢_p and synthesis network 𝒢_s are trained on IAM dataset <cit.>, which contains 80 distinct characters, of which we only choose 54 characters, including 26 capital letters, 26 lower case letters, 1 space character and a place holder for unknown character. Each stroke point of data possesses  x-axis and y-axis offset from the previous stroke point and a binary end-of-stroke indicator.  The source code is available at <https://github.com/orris27/hw_gan>.

We use 20 bivariate Gaussian mixture components at the output layer to parameterize the distribution of next point's coordinates and a Bernoulli distribution to determine pen up/down. A mixture of 5 Gaussian distributions was employed for estimating window parameters. The bias for biased sampling is set to 3.0. Architecture configuration details of discriminator 𝒟 and two generators 𝒢_p, 𝒢_s are shown in Table <ref> to <ref> respectively. The initial learning rates for both generator and discriminator are set as 0.001, then decaying exponentially during training, and optimizer is selected as Adam. For synthesis network 𝒢_s, we utilize 25 stroke points to represent one character. In order to ensure that two inputs for the discriminator do not differ significantly in the manner of size during the adversarial training, we resize the PSF features extracted from IAM dataset and those sampled from generator to similar sizes. To avoid exploding gradients, gradients of generator parameters are normalized to magnitude as one, thus step size is fully determined by learning rate, and gradient estimation serves providing an ascent search direction.

It is widely acknowledged that training GANs is relatively difficult <cit.>, and we employ multiple tricks  to train GANs more effectively. Firstly, we normalize PSF features between -1 and 1. Secondly, label smoothing technique is applied to both generator and discriminator during adversarial training <cit.>. Besides these, discriminator 𝒟 keeps trained with positive samples from IAM dataset and negative samples from generators 𝒢_s and 𝒢_p supervisedly during adversarial training, since a challenge that we face when training the model is to handle the pool capability of discriminator to distinguish realistic handwritten texts generated by 𝒢_s and 𝒢_p. More specifically,  in the prediction network, the positive and negative samples fed for discriminator are randomly chosen from IAM dataset and randomly produced by 𝒢_p, while in the synthesis network, we first randomly select a set of handwritten text instances with corresponding texts as positive samples, and then generate a set of fake data based on these texts as negative samples. 










 §.§ Numerical Results







In this section, experimental results are present to further demonstrate the advantages of our  to . In general, the  can produce competitive handwritten samples in prediction task, more promisingly, it can generate handwritten samples of which stroke points tend to distribute more spatially uniformly than those sampled from , and the font styles of handwritten text from  are more neat and sometimes more diverse than those of . Note that generating more diverse font styles of handwritten text is crucial for many related tasks, e.g. handwritten text with more diverse font styles may result in a more robust handwritten recognition system, which requires a large amount of annotated data to be trained.  





We first describe the performance of our model on prediction tasks compared with . The handwriting instances are sampled with bias as 3.0. As shown in Figure <ref>, handwriting samples produced by  are very competitive to that of Alex, which demonstrates the excellent abilities of learning handwriting structure of GANs. Since the criteria of judging the quality of handwriting data varies from person to person and there is no universal standard, it is not suitable to claim that one is better than the other. However, the samples from two networks are slightly different in font styles, since IAM is the only dataset for training both models, it implies that their learning strategies are not the same and may be applied in different scenarios.



We then compare our model with  in synthesis task. As an overview, the results of both models are shown in Figure <ref>. Generally speaking, we can see that  may produce more realistic, more natural and more neat handwritten texts than . More detailed comparisons are followed in the remaining section.



Given that the value of bias used in biased sampling has a large impact on the results, we next test both models under different sampling biases. In Figure <ref>, the top handwritten samples are from , and the bottom samples are from  under sampling biases as 4, 5, 7 and 10 respectively. The results of  distribute almost more spatially uniformly, and neat on every bias. On the other hand, it seems that  may produce more diverse handwritten text styles, like the “A".



Besides the above, we also investigate both models based on different prescribed sentences with a fixed sampling bias as 5.0, and present the results in Figure <ref>, which further proves the advantages of our model observed on previous experiments are consistent.




§ CONCLUSIONS


We present a new Generative Adversarial Network architecture  for synthesizing handwritten text of digital ink.  The method comprises two main components, a discriminator consisting a PSF feature extractor, followed by a CNN-LSTM binary classifier to distinguish realistic and forgery handwritten data, and a generator following the architectures of Alex Grave's to produce either random handwritten characters or sentences with prefixed text. In particular, the numerical results show that  are generally better than  in terms of uniformness of spatial distribution of stroke points, neatness and diversity of font style. This phenomenon illustrates that adversarial training is beneficial for generating more realistic handwritten text data. 

unsrt
 



