Chinese text classification has received more and more attention today. However, the problem of Chinese text representation still hinders the improvement of Chinese text classification, especially the polyphone and the homophone in social media. To cope with it effectively, we propose a new structure, the Extractor, based on attention mechanisms and design novel attention networks named Extractor-attention network (EAN). Unlike most of previous works, EAN uses a combination of a word encoder and a Pinyin character encoder instead of a single encoder. It improves the capability of Chinese text representation. Moreover, compared with the hybrid encoder methods, EAN has more complex combination architecture and more reducing parameters structures. Thus, EAN can take advantage of a large amount of information that comes from multi-inputs and alleviates efficiency issues. The proposed model achieves the state of the art results on 5 large datasets for Chinese text classification. Recently, Chinese text classification, as an important task of Chinese natural language processing (NLP), is extensively applied in many fields. Deep learning has gotten great results on Chinese text classification. However, the relevant studies are still insufficient compared with English, especially the method of Chinese text representation or encoding. It is considered to be closely related to the result of Chinese text classification models. Specifically, there are some issues in previous representation methods: (i) The word embedding (Le & Mikolov (2014) ; Mikolov et al. (2013) ; Pennington et al. (2014) ) is the most common method to represent the text, but it may become less effective when processing texts with the ambiguous word boundary such as Chinese texts. (ii) The character embedding (Zhang et al. (2015) ) can avoid the word segment. However, using Pinyin characters loses the ideographic ability of Chinese characters, and using Chinese characters requires more training data because there are thousands of Chinese characters that are often used in daily life. (iii) Both the word embedding and the Chinese character embedding are hard to encode some intricate Chinese language phenomena about pronunciations, such as the polyphone and the homophone. We notice that humans have associated the word or character with the corresponding pronunciation and remembered them in the process of learning the language. Thus, when humans read texts in daily life, they spontaneously associate with the corresponding voices. It is very difficult for computers and usually ignored by traditional text classification method. Moreover, using the voice can cope with some representation issues of Chinese characters or words better, The polyphone and the homophone are 2 typical examples. The former means different pronunciations and meanings are from the same character, and the latter means the same pronunciations are from different characters, which are usually used to represent similar meanings in social media. And inspired by recent multimedia domain methods (Gu et al. (2018) ), the extra audio information can obtain better results. However, large amounts of corresponding audio data are required difficultly. Pinyin can precisely express the pronunciation by no more than 6 letters and is easily generated from texts, and it also solves representation issues of Chinese characters or words. There are some typical examples that illustrate these points in detail. Table 1 shows an example (sentence1) of the homophone of social medias.There is a homophone "鸭梨山大" , the pronunciation wǒ zhǐ néng shuō dōng xī shì hǎo dōng xī，1 hào de dìng dān，6 hào cái dào，dìng dān shì liǎng jiàn，yī jiàn yùn dá，yī jiàn zhōng tōng，yùn dá 2 tiān dào，zhōng tōng 6 tiān dào，shāng jiā wán fēn kāi sòng，zhēn shì ràng mǎi jiā yā lí shān dà！ sentence2: 大学英语六级考试：优选真题 标准模拟 没有王长喜好 好 好 用，后悔了 dà xué yīng yǔ liù jí kǎo shì：yōu xuǎn zhēn tí biāo zhǔn mó nǐ méi yǒu wáng zhǎng xǐ hǎo yòng，hòu huǐ le sentence3: 有一点点小(我个人的喜好 好 好)，勉强吧 yǒu yī diǎn diǎn xiǎo (wǒ gè rén de xǐ hào )，miǎn qiǎng ba (Pinyin) and the meaning of which are the same as "压力山大". Table 1 also shows some examples (sentence2 and sentence3) of the polyphone of social medias. The pronunciation (Pinyin) and the meaning of "好" are different in two sentences. Besides,"hào" can represent "好" in sentence3 or "号" in sentence1. In fact, it can represent the pronunciation of dozens of Chinese characters. By those examples, we foucs on some points: In Chinese texts, some intricate language phenomena about pronunciations relatively easier to be recognized by a simple Pinyin encoder than by a complex Chinese character or word encoder. And most of language phenomena about glyph are the opposite. Based on the above points, we propose a new hybrid encoder (including word encoder and Pinyin character encoder) network to obtain better results for Chinese text classification, we call it Extractor-attention network (EAN). Inspired by Transformer (Vaswani et al. (2017) ), we also propose a new structure named the Extractor. The Extractor includes a multi-head self-attention mechanism with separable convolution layers (Chollet (2017) ). In EAN, the Extractor is used to encode the information of Pinyin. Besides, it is repeatedly used to combine word encoder with Pinyin encoder. Compared with previous hybrid encoder methods, our method has relatively simple encoders and a complicated combination part, which uses a deep self-attention mechanism. It makes EAN assign weights between features extracting by each encoder more accurately and avoid huge feature maps. Moreover, we use pooling layers for downsampling and separable convolution layers to compress parameters. Therefore, the Extractor network represent the Chinese text well, improve the classification accuracy, and the computational cost is relatively cheap. The experimental results show that our model outperforms all baseline models on all datasets, and has fewer parameters in comparison to similar works. Our primary contributions (i) Inspired by human language learning and reading, we design a novel method to solve the text representation issue of Chinese text classification, especially the language phenomena about pronunciations such as the polyphone and the homophone. To the best of our knowledge, this is the first time that a hybrid encoding method including Pinyin has been used to solve those language phenomena expression problem. (ii) We propose a new attention architecture named the Extractor to experss Chinese texts information. Besides, to better represent Chinese texts, we design a new hybird encoder method EAN based on the Extractor. We also propose a complex attention method to combine word encoder with Pinyin encoder effectively, which can commendably balance the amount of information transmitted by 2 encoders. (iii) Our method is able to surpass previous methods. It can get the state of the art results on public datasets. This paper proposes a novel attention network, the Extractor-attention network (EAN), for Chinese text classification. Compared to the traditional Chinese text classification methods using only word encoder, our approach uses hybrid encoder including words and Pinyin characters, which takes full advantage of the extra Pinyin information to improve the performance. Moreover, there is a new structure named the Extractor in our work, reduces the number of parameters in EAN and makes it excellent to extract feature. Thus, EAN obtains the state of the art results on 5 public Chinese text classification datasets. Finally, we also analyze the effects of different encoders structures on the method.
We propose a novel attention networks with the hybird encoder to solve the text representation issue of Chinese text classification, especially the language phenomena about pronunciations such as the polyphone and the homophone. This paper proposes an attention-based model consisting of the word encoder and Pinyin encoder for the Chinese text classification task, and extends the architecture for the Pinyin character encoder. Proposal for an attention network where both word and pinyin are considered for Chinese representation, with improved results shown in several datasets for text classification.