Recognition of Handwritten Japanese Characters Using Ensemble of
  Convolutional Neural Networks
The Japanese writing system is complex, with three character types of
Hiragana, Katakana, and Kanji. Kanji consists of thousands of unique
characters, further adding to the complexity of character identification and
literature understanding. Being able to translate handwritten Japanese
characters into digital text is useful for data analysis, translation, learning
and cultural preservation. In this study, a machine learning approach to
analyzing and recognizing handwritten Japanese characters (Kanji) is proposed.
The study used an ensemble of three convolutional neural networks (CNNs) for
recognizing handwritten Kanji characters and utilized four datasets of MNIST,
K-MNIST, Kuzushiji-49 (K49) and the top 150 represented classes in the
Kuzushiji-Kanji (K-Kanji) dataset for its performance evaluation. The results
indicate feasibility of using proposed CNN-ensemble architecture for
recognizing handwritten characters, achieving 99.4%, 96.4%, 95.0% and 96.4%
classification accuracy on MNIST, K-MNIS, K49, and K-Kanji datasets
respectively.
Angel I. Solis, Justin Zarkovacki, John Ly, Adham Atyabi
# Recognition of Handwritten Japanese Characters Using Ensemble of Convolutional Neural Networks

###### Abstract

The Japanese writing system is complex, with three character types of Hiragana, Katakana, and Kanji. Kanji consists of thousands of unique characters, further adding to the complexity of character identification and literature understanding. Being able to translate handwritten Japanese characters into digital text is useful for data analysis, translation, learning and cultural preservation. In this study, a machine learning approach to analyzing and recognizing handwritten Japanese characters (Kanji) is proposed. The study used an ensemble of three convolutional neural networks (CNNs) for recognizing handwritten Kanji characters and utilized four datasets of MNIST, K-MNIST, Kuzushiji-49 (K49) and the top 150 represented classes in the Kuzushiji-Kanji (K-Kanji) dataset for its performance evaluation. The results indicate feasibility of using proposed CNN-ensemble architecture for recognizing handwritten characters, achieving 99.4%, 96.4%, 95.0% and 96.4% classification accuracy on MNIST, K-MNIS, K49, and K-Kanji datasets respectively.

 Computer Vision, Convolutional Neural Networks, Ensemble Models, Transfer Learning, Handwriting Recognition, Japanese Handwriting

## I Introduction

The Japanese writing system is unique compared to the English writing system as it has three different character types of _Hiragana_, _Katakana_, and _Kanji_, each serving a different purpose. In Japanese typing system, Hiragana character type is primarily employed for native Japanese function words while Katakana is commonly utilized for words originating in other languages, and Kanji are used to represent characters with Chinese origin that been adapted to the Japanese language. Hiragana and Katakana both contain 46 basic characters, 71 characters including their diacritics. Compared to Kanji, Hiragana and Katakana are relatively simple as they both have characters that point to the same English characters for the sake of translation and pronunciation.

Kanji consists of Chinese "Han" characters. These characters first came about from the pictograms of the words they represent. Over time, Kanji became more simplistic until they became the characters they are today. While there are many several thousands of Kanji characters, roughly 2000 of them are commonly used.

_Kuzushiji_ is the cursive form of Kanji, and despite being used for over 1000 years, most people fluent in Japanese and/or native to Japan, cannot read Kuzushiji documents anymore. This is partly due to the modernization efforts of the Meiji restoration in 1868, the era were the Japanese education system was reformed, which removed Kuzushiji from school curriculum. This has left a substantial Japanese literature that can only be worked on depending on availability of a language expert. An estimated three million Japanese pre-modern books and over one billion records and documents use Kuzushiji [1] and this signifies the importance of developing modern approaches to read, analyze, and translate Kuzushiji documents. Aiming to address the need for tools to read and translate this large corpus of knowledge and information that is tied to history of Japanese culture, research community identified Machine Learning (ML) and Artificial Intelligence (AI) as one of the potential mechanism to develop such tools.

### _Problem statement_

As Kuzushiji continues to be phased out by the school curriculum, scientists and researchers must develop reliable systems to aid in the research and cultural preservation of this literature. Given the complex nature of reading Japanese hand-writing, even when AI and ML are used, it is necessary to study mechanism for automated reading these handwritten Japanese characters using AI and ML. In the absence of strong benchmarks on Japanese handwritten character recognition (JHCR), it is important to develop such benchmarks on existing publicly available datasets, specially for Kuzushiji-Kanji.

The Kuzushiji-Kanji dataset is one if not the most developed dataset aimed at assisting the development of these AI-empowered automated Japanese hand-writing recognition systems. In this project, we provide an initial benchmark performance measure to be used as a stepping stone for future algorithms and methodology aimed at taking on this dataset while assessing the performance of the proposed CNN-ensemble on other publicly available hand-writing datasets as well.

### _Contribution, Novelty and Outline_

The contribution of this work is the development of Ensemble Classifiers composed of 3 Convolutional Neural Networks (CNNs). The feasibility of the proposed CNN-Ensemble method is assessed using four publicly available datasets (see Section III for more information).

The outline of this study is as follows, a brief literature review of the most recent Japanese and Chinese character recognition methods is presented in section II, and description of the datasets used for CNN-Ensemble evaluation are detailed in section III. Section IV introduces the research methodology followed by results on evaluations on four publicly available
datasets in section V. Discussion of the results and study conclusion are discussed in sections V and VII respectively.

## II Related Work

Japanese Handwritten character recognition (JHCR) is an active research area with challenges associated to complexity and variability of Japanese characters. Multiple mechanisms, including Template-based recognition, convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are considered by the research community to address the task of accurately recognizing handwritten Japanese characters.

Template-based recognition, an approach that is traditionally used for JHCR task, is mainly focused on identifying a best match to a given character from a collection of reference templates gathered in a database. Inability to adequately address and deal-with differences in brush stroke orders and shapes are the main limitations of this approach which convinced researchers to explore and consider the use of more advance computer-vision methods based on machine learning and deep learning. Convolutional neural networks (CNNs) have become a popular method for JHCR task owing to its ability to automatically learn and extract deep features from images and being able to address limitations of template-based recognition approaches such as stroke order and shape variations. Das et al. [2] reported success in the character recognition of Japan's Hiragana character set, which is similar to Kanji. Rather than examining characters as a whole, in their approach, each character is passed through a pre-processing and normalization stage and is encircled and split into quadrants. in this method, Center of Gravity (CoG) is found for the whole character and for each quadrant. The proposed algorithm produced feature vectors by searching each quadrant for conjunction points (points where multiple strokes intersect) and endpoints and Euclidean distance is measured between the CoG and the significant features of each quadrant. The proposed method achieved 94.1% recognition rates. Using Japanese cursive kuzushiji character image dataset (Kuzushiji-MNIST or K-MNIST) [3] and CNN, Ueki and Kojima achieved 73.1% classification accuracy on JHCR task [4].

Yang et al. [5] used the CASIA-OLHWDB 1.0 (DB 1.0) and CASIA-OLHWDB 1.1 (DB 1.1) datasets [6] curated by The Institute of Automation, Chinese Academy of Sciences for their Handwritten Chinese Character Recognition (HCCR) study. DB 1.0 dataset is composed of 3740 Chinese characters; each class has 420 examples, and a train-test-split of 336 to 84 is used. The DB 1.1 dataset is composed of 3755 characters; each class has 300 examples, and a train-test-split of 240 to 60 was used. Authors proposed the using combination of a training sampling algorithm based on Leitner's learning box and the integration of a domain-specific knowledge layer in order to improve Chinese Character recognition. Leitner's learning box principle is based on the _"principle of spaced repetition for learning"_ which simply states that things that are harder to learn should appear more frequently. Authors used CNN for their CCR task where training samples are assigned a value that helps to place them in one of three bins of well-recognized (these can be sparsely sampled), confusing (should be sampled more often), and heavily noisy or mislabeled samples (these should rarely be if ever sampled). The proposed approach out-performed state-of-the- art in [7, 8], and [9] by 2.5% on the DB 1.0 dataset, achieving classification accuracy of 97.33% and by 1.4% on the DB 1.1 dataset, achieving 97.06% classification accuracy.

Shi et al. [10] used the HITPU handwritten Chinese character database collected by the Harbin Institute of Technology and Hong Kong Polytechnic University in their HCCR study. The dataset comprises 3755 characters from 200 different writers and a total of 751,000 images. Authors used active shape modeling with landmark labeling/recognition and GA for radical recognition. The radical is a fundamental part of Chinese and Japanese character, which significantly aids in recognizing the overall character. The proposed solution achieved a correct radical matching rate of 97.4%, outperforming state-of-the-art in HCCR ( [11] and [12] ) by roughly 5.9%.

## III Datasets

This study uses the datasets curated by [3]: **Kuzushijl-MNIST**, **Kuzushijl-49** and **K-Kanjl**, referenced as K-MNIST, K-49 and K-Kanji from this point forward, for evaluation and assessment of its proposed CNN-ensemble for JHCR task.

K-MNIST dataset is designed to serve as a replacement for the MNIST dataset, containing 10 classes and following MNIST sample representation format, the K-MNIST samples are represented as \(28\times 28\) grayscale images, total of 70,000 images with a train-test-split of 6 to 1 images per class.

K-49 dataset is a much larger datset in comparison to K-MNIST although it is imbalanced. K-49 dataset contains 48 Hiragana characters and one Hiragana iteration mark, total of 49 classes with samples presented as \(28\times 28\) grayscale images with total of 270,920 sample images. The train-test-split in K-49 dataset is 86 to 14 and the distribution is set in such a way to reflect the class imbalance from actual Japanese literature.

The K-Kanji dataset is an imbalanced dataset with total of 140,426 samples represented as \(64\times 64\) grayscale images with a total of 3,832 classes. The sample distribution ranges from 1766 examples in one class to only a single example in another with some classes having up to 3 different representations due to the original Kanji they are derived from which further

Fig. 1: Sub-sample of K-MNIST, K-49 and K-Kanji dataset.
increases the complexity of this dataset. In this study, only the top 150 most populated classes in the K-Kanji dataset are used.

Figure 1 illustrate samples from 5 different classes of each of these three datasets and Table I presents details about the datsets and their sample sizes and sample distributions in training and testing sets.

## IV Methodology

The ensemble model developed in this study consists of three CNN's as follows:

1. CNN-1 is consisted of a convolutional layer, followed by an average pooling layer, a convolutional layer, an average pooling layer, a functional layer, and a densely connected layer. This model is considered in order to attain a more general understanding of input sample shapes. The reduction in sample dimension size from the two average pooling layers makes each sample more generalizable. However, this would be ineffective without the three convolutional layers which help create enough of a distinction between each sample for the model to be effective on its own.
2. CNN-2 is consisted of two convolutional layers, followed by an average pooling layer, convolutional layer, flatten layer, and a densely connected layer. The architecture is slightly altered and modified for the Kanji dataset by adding a convolutional layer before the flatten layer. This model uses the three convolutional layers to focus its learning on the specific shape of each sample while the pooling layer is used to reduce computational complexity.
3. CNN-3 is the transfer learning model. This model's architecture utilizes a base model that is pre-trained on another dataset. The base model is then used as the head of the transfer learning model before additional layers are appended to it. The mechanism used for transfer learning in CNN-3 model depend on the dataset being used for evaluation (e.g., KMNIST, K49, or Kanji dataset). That is, when CNN-3 model is to be used for evaluating samples in KMNIST dataset, the base model is pre-trained using MNIST dataset and when K-49 is to be used for evaluating the model performance, the CNN-3 base model is pre-trained using KMNIST dataset. Finally, CNN-3 base model is pre-trained by samples of K-49 dataset when the evaluation is done by samples of Kanji dataset. 1. Base Model architecture includes a convolutional layer, followed by a dropout layer, convolutional layer, flatten layer, and a densely connected layer. The architecture is modified for the Kanji dataset by adding an average pooling layer before the flatten layer. This modification is considered due to samples of the Kanji dataset being much larger in size compared to the samples of the other datasets (64\(\times\)64 pixels in K-Kanji dataset versus 28\(\times\)28 pixels in K-MNIST and K-49). The extra average pooling layer is placed to accommodate for the image size differences, reducing computational cost while increasing the accuracy. 2. Additional layers: In CNN-3, the layers of the base model are followed by additional layers of convolutional layer, dropout layer, convolutional layer, average pooling layer, flatten layer, and a densely connected layer. Pre-training the base model allow the model to recognize general image fundamentals. Freezing the weights in the base model helps to retain the knowledge of image fundamentals while the new layers are tuned on the new target domain (training set samples from evaluating dataset). Starting with basic knowledge generally allows transfer learning models to be faster and more accurate than regular models.

Each CNN model (CNN1, CNN-2, and CNN-3) in CNN-Ensemble architecture is connected to a final output layer to allow the accuracy of the output to be evaluated. An illustration of the CNN-ensemble architecture is presented in figure 2.

In this study, the proposed ensemble model is a conglomeration (rather than combination) of CNN-models. As expected from ensemble models, single CNN model architectures are expected to be less accurate in their predictions and conglomerating the models can mitigate single model weaknesses through group voting. The proposed ensemble architecture utilizes the strengths of three different CNN-models to offset incorrect individual predictions and increase overall validation accuracy on each dataset.

The performance of the proposed CCN-ensemble architecture is assessed on all three datasets used in this study (MNIST, K-MNIST, K-49 and K-Kanji), one dataset at the time. Each CNN-model in the proposed CNN-ensemble is trained and validated individually (using the same training and validation sets on each dataset for all three CNN-models) and a voting mechanism based on aggregating votes of all three CNN-models into a single vector and taking the \(maxsum\) of the final vector is used to generate the overall ensemble prediction on the validation set of each dataset.

The results of the individual models and the CNN-ensemble model are presented in table II and further elaborated in section
## V Results

### _MNIST Results_

The three CNN models were tested on the MNIST dataset to gain a performance baseline.

* CNN-1: been able to achieve a 99.00% accuracy on the validation data.
* CNN-2 been able to achieve a 97.30% accuracy on the validation data.
* CNN-3 been able to achieve a 99.03% accuracy on the validation data. The MNIST dataset was used as a benchmark for the other datasets so it did not use any transfer learning.
* CNN-Ensemble produce a 99.35% accuracy on the validation data. The average classification performance across CNN models achieved classification accuracy of 98.35%, indicating that using an ensemble model improved the average model accuracy by 1.0%.

These results can also be found on table II.

### _K-MNIST Results_

The three CNN models been evaluated on the K-MNIST dataset aiming to provide performance comparison between the ensemble model and individual CNN models. The results are as follows:

* CNN-1 achieved a 94.44% accuracy on the validation data.
* CNN-2 achieved a 95.11% accuracy on the validation data.
* CNN-3 achieved a 95.15% accuracy on the validation data. Samples of MNIST dataset been used to pre-train the base model in CNN-3 although marginal performance improvement is observed compare to CNN-1 and CNN-2.
* CNN-Ensemble achieved classification accuracy of 96.37% on the validation data, showing an improvement of 1.47% on average performance achieved across the three CNN models (94.90%) while also outperforming individual CNN models' performances.

These results are also illustrated on table II.

### _K-49 Results_

All three CNN models are also tested on the K-49 dataset, with model 3 using transfer learning from the K-MNIST dataset. The results are as follows:

* CNN-1 achieved a 91.80% accuracy on the validation data.
* CNN-2 achieved a 93.01% accuracy on the validation data.
* CNN-3 achieved a 93.09% accuracy on the validation data. CNN-3 only marginally outperformed CNN models 1 and 2 on the K-49 dataset using transfer learning from the K-MNIST dataset. However, the average training time for the transfer learning model was reduced from 1900 seconds to 987 seconds, or by 48%.
* CNN-Ensemble achieved classification accuracy of 95.04% on the validation data. This model outperformed both average CNN models performances (92.63%, show

Fig. 2: Example CNN ensemble architecture
ing 2.41% performance improvement) and individual CNN models performances.

These results can additionally be found on table II.

### _K-Kanji Results_

K-Kanji dataset is considered as the true target class of this study since, to the best of our knowledge, its performance been never evaluated in literature. This dataset consists of a total of 3832 Kanji characters in a 64\(\times\)64 grayscale images of Kanji handwriting. This dataset consists of a smaller set of 140,426 images and is highly imbalanced where numbers of examples per class can range from 1,766 to even just a single example. For evaluation of the dataset and to assess feasibility of the proposed CNN-Ensemble architecture, only the top 150 populated character classes are considered in this study.

* CNN-1 achieved 92.83% classification accuracy on the validation set.
* CNN-2 achieved 93.49% classification accuracy on the validation set.
* CNN-3 achieved 95.01% classification accuracy on the validation set. The base model in CNN-3 is pre-trained using samples from K-49 dataset. The results indicate the substantial improvement in performance on K-Kanji dataset compared to CNN-1 and CNN-2 models. This is likely due to the increased sample size in K-49 dataset (266,407 samples, see table I for more details) that better accommodated learning through transfer learning (base model in CNN-3), allowing the model to learn more information about the images in the base model.
* CNN-Ensemble achieved 96.43% classification accuracy on the validation data. The average classification performance across 3 CNN models achieved 93.77% classification accuracy. This indicates improvement of 2.65% in CNN-Ensemble compared to average model and outperforming all 3 CNN models. This is likely due to an increased sample size compared to the KMNIST and K-49 datasets. The base model for Kanji included an additional pooling layer because of the increased sample size. This extra layer is likely the contributing factor to the great success of the Kanji transfer learning model.

A summary of these results are presented in table II.

## VI Discussion

In table III, no state-of-the-art performance is provided for K-Kanji dataset since, to the best of our knowledge, this dataset never been considered by research community, possibly due to complexities associated with the dataset, specially the imbalance nature of the dataset which makes it more difficult to effectively train models on smaller sampled character classes.

Table III is augmented from the [16] study to show how a variety of algorithms perform on the MNIST, K-MNIST and K-49 dataset. This table has been updated to show the result of the ensemble architecture and it's performance on the same datasets with the addition of the K-Kanji dataset.

To better understand the performance variations across the 150 classes of the K-Kanji dataset, the testing dataset F1-Scores for each class is illustrated in table IV. The F1-Scores are presented in descending order, ranging from F1-Score of 1 to 0.49. It is noticeable that majority of the classes are lacking adequate number of samples for training and tuning of CNN models which speaks to difficulties in using K-Kanji dataset.

A SHapely Additive explanations (SHAP) analysis is performed on each CNN model used in CNN-Ensemble to better understand the results achieved. SHAP analysis is a methodology used to represent contributions of different features in a given sample in the prediction achieved by a machine learning model [17]. SHAP values, ranging from -1 (commonly represented by blue color in SHAP images) to 1 (commonly represented by red color in SHAP images), indicates the impact of each feature on prediction, by higher SHAPE units indicating higher contribution in achieved prediction. SHAP values are calculate based on acquiring the difference between prediction achieved by machine learning model for a given sample with and without using the feature.

Figure 3 illustrates the results of SHAP analysis on the three CNN models used in this study using three samples, randomly chosen, from each dataset considered in the study, e.g., KMNIST, k-49, and K-Kanji. Brighter red pixels indicate a positive contribution to the classification while the blue pixels indicate negative contribution to the model prediction. The results highlights little to no discernible difference between the models which is an indication of the observed inability of CNN-ensemble model in achieving substantial gain in classification performance compared to individual CNN models, e.g., CNN-1, CNN-2, and CNN-3.

## VII Conclusion

This study discussed the development of a machine learning approach to identify and translate handwritten Japanese characters (Kanji) into its corresponding English translation. It is imperative to develop reliable systems for aiding in the
research and cultural preservation of Kuzushiji literature given that very few individuals are able to translate these documents with millions of such documents still in process of being translated and prepared for use by researches interested in understanding historical context withhold within them.

This study employed an ensemble of three convolutional neural networks (CNNs) and evaluated its performance. One of the aims and main contributions of this study is to develop benchmarks for Japanese handwritten character recognition, using publicly available datasets. To this end, four publicly available datasets of Kuzushiji- MNIST, Kuzushiji-49 and K-Kanji, referred to as as K-MNIST, K-49 and K-Kanji, are used to evaluate the performance of the proposed CNN-Ensemble architecture. The results indicated feasibility of the proposed CNN-Ensemble architecture in correctly predicting various Japanese Handwritten characters, achieving average accuracy

Fig. 3: SHAP Analysis Output of CNN1, CNN-2 and CNN-3 models on three samples for K-MNIST, K-49 and K-Kanji datasets.
of 96% across the datasets used, with individual CNN models showing a range of performance from 91.8% (CNN-1 on K-49 dataset) to 95.15% (CNN-3 on K-MNIST dataset).
