In this paper, we propose a Seed-Augment-Train/Transfer (SAT) framework that contains a synthetic seed image dataset generation procedure for languages with different numeral systems using freely available open font file datasets. This seed dataset of images is then augmented to create a purely synthetic training dataset, which is in turn used to train a deep neural network and test on held-out real world handwritten digits dataset spanning five Indic scripts, Kannada, Tamil, Gujarati, Malayalam, and Devanagari. We showcase the efficacy of this approach both qualitatively, by training a Boundary-seeking GAN (BGAN) that generates realistic digit images in the five languages, and also qualitatively by testing a CNN trained on the synthetic data on the real-world datasets. This establishes not only an interesting nexus between the font-datasets-world and transfer learning but also provides a recipe for universal-digit classification in any script. Transfer learning from the synthetic realm to the real-world has elicited a lot of attention in the machine learning community recently (See [1; 2; 3]), which typically entails three steps: generating large volumes of synthetic training data (which is often a relatively cheap process), synthesizing it using a domain specific recipe to address the reality gap BID3 , and training real-world deploymentworthy machine learning models. As seen in [1; 2; 3] , deep generative models such as Variational Autoencoders (VAE) and Generative Adversarial Networks (GAN) are often deployed during the synthesizing process. This paper fits squarely into this category of work, where we tackle the problem of absence of MNIST-scale datasets for Indic scripts to achieve high, real-world accuracy digit classification by using synthetic datasets generated by harnessing the Open Font License 1 (OFL) font files freely available on the internet. We begin with a handwritten grid of 32 x 40 digits on a commercial Mead Cambridge Quad Writing Pad, 8-1/2" x 11", Quad Ruled, White, 80 Sheets/Pad book with a black ink Z-Grip Series -Zebra Pen. We then scan the sheet(s) using a Dell -S3845cdn scanner BID2 and use an image segmentation script to slice the scanned image grid to generate 1280 28 x 28 mnist-ized digit images. An example of the raw scanned image for the Devanagri-Hindi script is as shown in FIG1 We now perform a novel 2-step sanity check for MNIST compatibility for the languages in the following manner. Firstly, we pick images belonging to class 0 and pass it through a CNN pre-trained on the MNIST dataset BID4 . Given that the representation of 0 is the same in all the Indic languages and the standard-MNIST, we expect to get high accuracy predictions for this class. Given the Indian roots of the modern Hindu-Arabic numeral system BID5 , we expect to find morphological similarity in the shape of certain digits between the modern digit system (used in MNIST) and the Indic languages considered in this work. We exploit this similarity to perform our second sanity check, which is showcased with a specific example in FIG2 where the shapes of the digits for 3 and 7 in Kannada look similar to 2 in the modern script for the Hindu-Arabic numeral system. Therefore, passing images belonging to class 3 and class 7 in the Kannada dataset through the standard MNIST-trained CNN should result in high rates of correct misclassifications as class 2. For this dataset, we obtained accuracies of 1.0 and (1.0, 0.977), respectively, for the two checks above. Published as a workshop paper at ICLR 2019 There exists a vast body of handwritten as well as synthetic and artistically inscribed textual data freely available on the internet in the form of font files. We found 961 Open Font License (OFL) fonts in the Google fonts repository BID3 alone, each with intra-font variants such as italicized, bold, etc. For the Indic scripts, Red Hat 5 released the Lohit 6 font family that covers 11 languages: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, Telugu. In this paper, we used the Lohit True Type Font (TTF) 7 files (that can be used on both Mac and Windows platforms) to create a seed dataset for each of the languages. We combine the text insertion ability of the Python Imaging Library [7] with the textual glyphs generated using the Lohit font family, as per Appendix Listing 1. Using this script, we have a 5 × 10 array of the 28 × 28 images as shown in Fig 4. We can similarly create a plethora of seed image datasets using a vast array of other fonts from sources such as Google fonts BID6 and Baraha 9 .Figure 4: The Lohit seed dataset portraying 0-9 digits for the five Indic scripts One can argue that the glyph in a seed image is a proper representation of what the digit should look like. However, if our goal is to perform human handwritten digit classification, we need to intelligently distort seed images in order to generate a synthetic dataset realistic enough to transfer well into the real-world data domain. The problem of realistic augmentation for digits has a variety of approaches. One of the most notable works is Simard et al. BID6 , which introduces the Elastic distortion method hypothesized to correspond well to "uncontrolled oscillations of the hand muscles, dampened by inertia." This constitutes our shallow augmentation approach. Once we generate the elastic-distortion augmented dataset, we train an AC-GAN BID7 whose samples constitute the deep augmented synthetic training dataset. Classic image augmentation techniques BID6 include additive noise, intensity shifting and scaling, random rotation, translation, flipping, and warping. As discussed above, we will focus on the Elastic distortion method proposed in BID6 . It is a bi-parametric method with the following control parameters:• α: The random-field parameter • σ: The Gaussian-convolution standard deviation parameterAs recommended in BID6 , we set α = 8 for each of the five Indic scripts and vary the σ parameter and tune until the perturbed images look realistic. For these experiments, we used the implementation in imgaug 10 python library. Fig 5 shows a grid of random images generated using the five Indic seed datasets, with α = 8 and σ varying from 0.01 to 3. The realistic-looking image distortions mimicking the variations caused by the human hand occur in the interval of σ ∈ [1.5, 2.5]. We use this heuristic to generate the Elastic distorted datasets for all the five languages by sampling from σ ∼ U [1.5, 2.5] Figure 5 : The Lohit seed dataset portraying 0-9 digits for the five Indic scripts We complement classic image augmentation techniques using synthetic data generation, where data is generated and sampled from deep generative models such as Generative Adversarial Networks (GANs). GANs can implicitly learn the true data distribution and generate promising samples such that the samples from the learned distribution resemble the true underlying data distribution p data .In order to generate synthetic data with corresponding labelled classes, we leverage the Auxiliary Classifier GAN (ACGAN) in which the generator input can be conditioned to sample from a target class BID7 . ACGAN is trained with the discriminator producing two outputs, one to discriminate between real and fake images X, and the other to classify X in terms of its class, c. During training, we explicitly sample X c from the generator during training for each class c ∼ p c , with the generator parameters adjusted to maximize the superposition of the two components in the objective function, L s and L c as follows. Here L s refers to the log-likelihood of the correct source whereas L c refers to the log-likelihood of the correct class. DISPLAYFORM0 The discriminator, D, is trained to maximize L s + L c whereas the generator, G, is trained to maximize L c − L s . In our experiments we used learning rate l with Adam Optimizer (l = 0.0002, β 1 = 0.5, β 2 = 0.999), and with dropout probability p dropout = 0.3 in the discriminator and batch normalization in the generator as regularization techniques. The generator takes in the Hadamard product between the latent space and the class conditional embedding as an input. ACGAN exhibits promising results in deep class-condition augmentation, as the discriminator now produces a probability distribution over the sources and also a probability distribution over the class labels. With the introduction of class probabilities, p c , more structure is present in the GAN latent space which helps produce higher-quality training samples and improve stability during training. Nonetheless we chose ACGAN as a key deep generative model in the augment step as it generates samples with corresponding class labels, as the class is crucial in measuring classification accuracy for the generated output with regards to the corresponding label. FIG1 the snapshots of images produced by sampling the ACGAN generator after 10 epochs for all the five languages. For each language we see two blocks distanced by the white separator space. The images in the left column are the generated images and the images in the right column are real shallow-synthetic images that the GAN was trained on. We wish to showcase the efficacy of the SAT framework using two approaches. The first approach is more qualitative by training a deep-generative model (BGAN) using only shallow synthetic data, and utilizing the model to generate realistic looking handwritten digit images for the various languages. The second approach is quantitative where we train a Convolutional Neural Network (CNN) solely on synthetic training data, and test the efficacy with both real-world handwritten data as well as doping the training dataset with a small amount of real-world data (∼ 20%). We then computing the classification accuracies in both scenarios. GANs face the limitation where the generated samples have to be completely differentiable with regards to the parameters of the generator, θ, and do not work for discrete data where the gradient is zero almost everywhere, not to mention other issues such as training stability. Boundary-seeking GANs (BGANs) allow for the training of GANs with discrete data by using a policy gradient based on the KL-divergence with importance weights as a reward signal BID8 .In our setting, we focus more on the convexity of the objective function of BGANs. We formulate BGANs in our setting as follows: we have G θ : Z → X as a generator that takes in a latent variable drawn from a prior, z ∼ p z . We also have a neural network F φ : X → R parameterized by φ, with our BGAN objective defined as: DISPLAYFORM0 Typical generator loss of continuous data is a concave function with poor convergence properties since it relies on continuous optimization of the discriminator to stabilize learning. We reformulate the concave optimization as a convex one by training the generator to aim for the decision boundary of the discriminator, which improves stability. In implementation we also use Adam Optimizer (l = 0.0002, β 1 = 0.5, β 2 = 0.999). The results are as shown in FIG2 , 1000, 2000 and 5000. We see the learning setting in as the epochs increase with realistic looking imagery emerging after as few as 5000 epochs. Figure 7: The accuracy comparisons for the five scripts. For the test-accuracy computations over the real-world dataset we trained an off-the-shelf plainvanilla CNN model 11 with cross-entropy loss and the ada-delta optimizer. Fig 7 showcases the results obtained. When we used a purely synthetic training dataset with 140,000 training samples of which 80,000 came from the elastic distortion procedure from section-3 and 60,000 came from the ACGAN, we obtained overall accuracies varying from 60% to 76% for the five different languages (the Purely synthetic bars in the bar-plot). When we augmented this training dataset with merely ∼ 20% of the real-world dataset (a 280-1000 train-test split) the accuracies shot up by a significant amount. We believe this observation has two important facets to it besides the reality-gap: the limited size of the test-set and some idiosyncratic morphological changes in the digits' representation in the lohitfont and the colloquial representation. For example both in Gujarati and Kannada ( FIG1 we see that six is represented rather differently in the synthetic set and in the test set. This seems to get tackled when a small amount of real-world data is introduced into the training data. In the case of Gujarati, the accuracy for digit 6 increases from 0.7% to 95%! As seen in Fig 7 the overall test accuracy increases for every language by a substantial margin for the 'Purely synth + 280 real-world training' dataset option. The confusion matrices for the two training scenarios are as shown in FIG5 We introduced 5 new real-world digit datasets in Indic languages and also a transfer learning framework which we term as Seed-Augment-Train/Transfer (SAT) to perform real-world handwritten digit classification in these languages. Our goal is to draw the attention of the machine learning community to this potentially attractive nexus between the work happening in the synthetic-to-real transfer learning domain and a veritably rich reservoir of semi-synthetic textual glyphs training data freely available in the form of Font files. We showcased the efficacy of this SAT-based approach by perform digit-classification and also training GANs for the digits of the languages considered. We are expanding this foundational effort in several directions: the first entails using a small portion of the dataset for transfer learning, which will help tackle cases in which off-the-shelf image augmenters do not effectively capture human handwriting's natural variations. The second involves generative modeling of the deviations in handwritten digits with regards to the clean seed-synthetic dataset using the extended MNIST dataset (in lieu of the digits themselves). We would then apply this approach to deep augment seed datasets
Is seeding and augmentation all you need for classifying digits in any language? This paper presents new datasets for five languages and proposes a new framework (SAT) for font image datasets generation for universal digit classification.