Investigating Self-Attention Network for Chinese Word Segmentation
Leilei Gan Yue Zhang
cs.CL

UTF8gkai
Compressing deep quaternion neural networks with targeted regularization
    Riccardo Vecchi, Simone Scardapane, 
 Danilo Comminiello, and Simone ScardapaneAuthors are with the Department of Information Engineering, Electronics, and Telecommunications, Sapienza University of Rome, Via Eudossiana 18, 00184, Rome, Italy.
E-mail: {firstname.lastname}@uniroma1.it.
    December 30, 2023
=================================================================================================================================================================================================================================================================================================


Neural network has become the dominant method for Chinese word segmentation. Most existing models cast the task as sequence labeling, using BiLSTM-CRF for representing the input and making output predictions. Recently, attention-based sequence models have emerged as a highly competitive alternative to LSTMs, which allow better running speed by parallelization of computation. We investigate self attention for Chinese word segmentation, making comparisons between BiLSTM-CRF models. In addition, the influence of contextualized character embeddings is investigated using BERT, and a method is proposed for integrating word information into SAN segmentation. Results show that SAN gives highly competitive results compared with BiLSTMs, with BERT and word information further improving segmentation for in-domain and cross-domain segmentation. Our final models give the best results for 6 heterogenous domain benchmarks.




§ INTRODUCTION


Word segmentation is a necessary pre-processing step for Chinese language processing <cit.>. The dominant method treats Chinese Word Segmentation (CWS) as a sequence labeling problem <cit.>, where neural network models <cit.> have achieved the state-of-the-art results. A representative model <cit.> takes LSTM <cit.> as a feature extractor, and a standard CRF <cit.> layer is used on top of a BiLSTM layer to predict the label sequences.

Recently, self-attention network (SAN) <cit.> has been shown effectiveness for a range of natural language processing tasks, such as machine translation <cit.>, constituency parsing <cit.>, and semantic role labeling <cit.>. Compared with recurrent neural networks (RNNs) <cit.>, SAN has advantages of capturing long-term dependencies and supporting parallel computing more easily. However, its effectiveness on CWS has not been fully investigated in the literature.









We empirically investigate SAN for CWS by building a SAN-CRF word segmentor, studying the influence of global and local attention for segmentation accuracy. Based on the SAN-CRF segmentation model, we investigate two further questions. First, in Chinese, characters are highly polysemantic and the same character can have different meanings in different context. SAN has also been shown a useful method for training contextualized word representations <cit.>. We compare context-independent character representations <cit.> with contextualized character representations in both in-domain and cross-domain CWS evaluation. 

Second, out of vocabulary words, especially domain specific noun entities, raises a challenge for cross-domain CWS. To solve this problem, domain lexicons can be used <cit.> for cross-domain CWS tasks. We consider a novel method for integrating lexicons to SAN for cross-domain CWS, using attention to integrate word information by generalizing words into POS tags, resulting in end-to-end neural type-supervised domain adaptation.













Results on three benchmarks show that SAN-CRF can achieve competitive performance compared with BiLSTM-CRF. In addition, BERT character embeddings are used for both in-domain and cross-domain evaluation. In cross-domain evaluation, the proposed neural type-supervised method gives an averaged error reduction of 30.32% on three cross-domain datasets. Our method gives the best results on standard benchmarks including CTB, PKU, MSR, ZX, FR and DL. To the best of our knowledge, we are the first to investigate SAN for CWS[Code and trained models will be made available].











§ BASELINE

We take BiLSTM-CRF as our baseline, which has been shown giving the state-of-the-art results <cit.>. Formally, given an input sentence with m characters s=c_1, c_2, ..., c_m, where c_i denotes the ith character, the task of character based CWS is to assign each character c_i with a label y_i, where y_i ∈{B, M, E, S} <cit.>. The label B, M, E and S represent the begin, middle, end of a word and single character word, respectively.

For each character c_i, its input representation e_i is the concatenation of unigram character embedding e_c_i and bigram character embedding e_c_ic_i+1. Following <cit.>, we take BiLSTM to capture the context of character in both forward and backward directions.

The forward hidden state 𝐡_i of character c_i is calculated as follows:

    𝐢_i     =  σ(W_ih_i-1 + U_ie_i + b_i) 
    𝐟_i     =  σ(W_fh_i-1 + U_fe_i + b_f) 
    𝐨_i     =  σ(W_oh_i-1 + U_oe_i + b_o) 
    𝐜_i    =  tanh(W_ch_i-1 + U_ce_i + b_c) 
    𝐜_i     =  𝐟_i ⊙𝐜_i - 1 + 𝐢_i ⊙𝐜_i
    𝐡_i     =  𝐨_i ⊙tanh(𝐜_i)

where 𝐢_i, 𝐟_i and 𝐨_i are input, forget and output gates, respectively. σ and ⊙ are element-wise sigmoid and product function respectively while W, U and b are model parameters to learn.

The backward hidden state 𝐡_i can be obtained in a similar way. Thus, the hidden state of character c_i is the concatenation of 𝐡_i and 𝐡_i:

    𝐡_i = 𝐡_i ⊕𝐡_i


In the scoring layer, a CRF is used to consider the dependencies of adjacent labels. The probability of a label sequence y=y_1, y_2, ..., y_n of sentence s is:

    P(y|s) = exp(∑_i=1^n(F(y_i) + L(y_i-1, y_i)))∑_y^'∈ℂ(s)exp(∑_i=1^n(F(y^'_i) + L(y^'_i-1, y^'_i)))

where ℂ(s) is the set of all possible label sequences of sentence s. F(y_i) is the emission score of y_i and L(y_i-1, y_i) is the transition score from y_i - 1 to y_i. 


§ MODEL

Figure <ref> shows our segmentor framework on a input character sequence “(Fellow of the Chinese Academy of Sciences)". The model takes character representation and positional embeddings as input. By matching the input to a word-POS lexicon, word information is investigated by using attention for each character. Multiple layers of self-attention network <cit.> are used as feature extractor to replace BiLSTM in the baseline. Similar to the baseline, we also use a CRF layer on top of the self-attention network to model the dependencies of adjacent labels.


 §.§ Embedding Layer

As shown in Figure <ref>, the embedding layer consists of character embeddings and positional embeddings. The character representation of c_i is the concatenation of unigram character embedding e_c_i and bigram character embedding e_c_ic_i+1.

    𝐱_i^c = e_c_i⊕e_c_ic_i+1

where ⊕ represents concatenation operation.


Because self-attention network does not explicitly consider sequence information, positional embedding is added to the input of self-attention network as follows:

    PE_(pos, 2i)   =  sin(pos/10000^2i/d_model) 
     
    PE_(pos, 2i+1)   = sin(pos/10000^2i/d_model) 
    𝐱_i^c    = 𝐱_i^c + PE

where pos is the position, i is the dimension, d_model is the dimension of output, respectively, and + denotes vector addition.


 §.§ Self-Attention Network

We extend the model of <cit.> for the SAN segmentor. The model has multiple identical layers, each of which is composed of a multi-head self-attention sub-layer and a position-wise fully connected feed-forward network.


Multi-head self-attention is used to exchange information directly between positions in the sequence. First, for single-head self-attention, the representations of a sequence X is computed by scaled dot-product attention as follows:

    Attention(X)     = Attention(Q, K, V) 
        =softmax(QK^T√(d_k))V

where Q=W^T_QX, K=W^T_KX, V=W^T_VX are query, key and value vectors, respectively, and W^T_Q, W^T_K, W^T_V are parameters.

Local Self-Attention   In order to investigate the effect of long-term dependencies on CWS task, we propose a local self-attention, which only attends to surrounding positions for each character instead of all positions in the sequence. The intuition is that long-term dependencies may bring more noise than information in a sequence labeling task <cit.>. The local self-attention is denoted as:

    L-Attention(X)     = Attention(Q, K, V) 
        =(softmax(QK^T√(d_k)) ⊙ W) V

where W is a matrix to control the self-attention inner a window and its element W_ij is denoted as:

    W_ij =
    	 
    	1    j - i <= WS 
     
    	-∞   otherwise

Here WS is the window size. 

Multi-Head Self-Attention is used, which linearly maps Q, K and V into multiple versions Q_i, K_i and V_i and then concatenates the outputs of different head_i as follows:

    MH(X) = Concat(head_1, ..., head_n)W^o

where 

    head_i = Attention(QW_i^Q, KW_i^K, VW_i^V)
 
W^o, W_i^Q, W_i^K and W_i^V are parameters.


On top of the multi-head attention sub-layer, a fully connected feed-forward network (FNN) is applied to each position. FNN is composed of two linear transformations with a ReLU activation.

    FNN(x) = W_2ReLU(0, xW_1 + b_1) + b_2






§ RICH CHARACTER AND WORD FEATURES

We incorporate rich character and word features into SAN model. Specifically, pre-trained contextualized character representation is introduced as well as a word-based neural type-supervised domain adaptation method.



 §.§ BERT Character Representation

BERT <cit.> is trained from a large scale corpora by a deep bidirectional Transformer using masked LM tasks. Usages of BERT can be divided into feature-based and fine-tuning methods. The former fixes all model parameters and directly extracts character features from the pre-trained model, while the latter jointly fine-tunes all parameters on downstream tasks. We take the latter method, feeding the input sequence of characters into BERT and use the top layer output as character representation. Development experiments show that fine-tuning BERT embeddings give higher results than the feature-based method.

Formally, character c_i is represented using pre-trained BERT embedding according to the whole sentence.

    𝐞_i^c = 𝐞^c(c_i)

where 𝐞^c denotes a pre-trained BERT character embedding.



 §.§ Integrating Word-POS Lexicon for Type-Supervision

We integrate word information into SAN to handle rare words in cross-domain settings. Following the definition by <cit.>, we describe this model in a cross-domain setting only, where C_s denotes a set of annotated source-domain sentences, and ξ_t denotes an annotated target-domain lexicon, in which each word is associated with one POS tag. The domain adaptation model is firstly trained on C_s, and makes use of ξ_t when performing domain adaptation. In practice, our method can be used in in-domain settings also.

As shown in Figure <ref>, for each character c_i in the input sentence, the set of all character subsequences that match words in the external lexicon 𝔻 is denoted as 𝐰^i = {w_b_1, e_1, w_b_2, e_2, ..., w_b_m, e_m}. Here b_k and e_k are the start and end index of the matched words in the sentence, where e_k >= i and b_k <= i. Word embeddings should intuitively be used for encoding w_b_k, e_k. However, for characters forming domain specific words, there may not be readily available embeddings. POS embeddings can be used as alternative unlexicalized features of words embeddings. We introduce how to integrate POS embeddings as word information from both prediction and training.


Prediction   During testing, we match character subsequences in a given input sentence to a word-POS lexicon ξ_t. For all matching subspans, we find a vector representation by first performing a lookup action to a word embedding table, and then using the corresponding POS embedding to represent the word if no word embedding is available for the subspan.

    𝐱^w_b_k, e_k = 𝐩^w(p_i)    if w_b_k, e_k∈ξ_t
    𝐞^w(w_b_k, e_k)    otherwise

where 𝐞^w and 𝐩^w are word and POS embedding lookup tables, respectively. 

Training  Training is performed on a source domain corpus only. We do not fine-tune word embeddings. The key task for knowledge transfer is the learning of POS embeddings, which offer a generalized representation for words not in the embedding lexicon. To this end, we randomly replace words in the training data  with their gold-standard POS tags as follows:

    P(w_b_k, e_k) = min(1, √(tf(w_b_k, e_k)))

where f(w_b_k, e_k) is the frequency of w_b_k, e_k in the training data and t is a chosen threshold.

The representation of w_b_k, e_k is:

    𝐱^w_b_k, e_k=𝐩^w(p_i)      rb < P(w_b_k, e_k)
    𝐞^w(w_b_k, e_k)    otherwise

where rb is a random number and p_i is the gold-standard POS tag of w_b_k, e_k.


Considering the positional information of characters in the word, the set of POS tags can be denoted in combination with segmentation labels: P_t_b={p_1b, p_2m, p_1e, p_1s, ..., p_ne}. The difference between P_t and P_t_b is that for c_j and matched word w_b_i, e_i, if c_j is the first, middle or end character of w_b_i, e_i, the corresponding POS tag of w_b_i, e_i is p_ib, p_im and p_ie, respectively. Figure <ref> shows the difference between p_t and p_t_b through an example.

For each character c_i, we integrate dictionary word information by augmenting its embedding with a word context vector h_i, which is the weighted sum over 𝐱_b_k, e_k^w for all spans (b_k, e_k) that contain c_i. In particular,

    𝐡_i = ∑α_ik𝐱_b_k, e_k^w

where the weight for each context word is:

    α_ik    = attention(𝐱^c_i, 𝐱^w_b_k, e_k) 
        = exp(score(𝐱^c_i, 𝐱^w_b_k, e_k))/∑_k=1^m(exp(score(𝐱^c_i, 𝐱^w_b_k, e_k)))

Considering computation efficiency, the score function is:

    score(𝐱^c_i, 𝐱^w_b_k, e_k) = 𝐱^c_iW 𝐱^w_b_k, e_k

where W is parameters. The output of the attention layer is the concatenation of the character embedding 𝐱_i^c and the context vector 𝐡_i:

    𝐱_i = 𝐱_i^c ⊕𝐡_i




 §.§ Decoding and Training

For decoding, the Viterbi algorithm <cit.> is used to find the highest scored label sequence y^* over a input sentence.

Given a training set with N samples, the loss function is negative log-likelihood of sentence-level with L_2 regularization:

    Loss = -∑_i=1^Nlog(P(y_i|s_i)) + λ/2||Θ||^2





§ EXPERIMENTS

We carry out an extensive set of experiments to investigate the effectiveness of SAN-CRF and the proposed neural type-supervised domain adaptation method across different domains under different settings. F1-value is taken as our main metric. 



 §.§ Datasets

We separately evaluate the proposed model in in-domain and cross-domain settings. For in-domain evaluation, CTB6 (Chinese Tree Bank 6.0), PKU and MSR are taken as the datasets. The train/dev/test split of CTB6 follows <cit.>, while the split of PKU and MSR are taken from the SIGHAN Bakeoff 2005 <cit.>. For cross-domain evaluation, PKU is used as the source domain, and three Chinese novel datasets including DL (DouLuoDaLu), FR (FanRenXiuXianZhuan) and ZX (ZhuXian) <cit.> are used as target domains. Following <cit.>, we collect target-domain lexicons from Internet Encyclopedia[https://baike.baidu.com/item/诛仙/12418][https://baike.baidu.com/item/凡人修仙传/54139][https://baike.baidu.com/item/斗罗大陆/5313]. Table <ref> shows the statistics of the datasets. 



 §.§ Experimental Settings

Table <ref> shows the values of model hyper-parameters. For the SAN CWS model, we use the Adam <cit.> optimizer with β_1=0.9, β_2=0.98, ϵ=10^-9. Following <cit.>, we increase the learning rate linearly for the first warmup_steps steps, and then decrease it proportionally. The value of warmup_steps is set to 1000. When BERT is used for character embeddings, the learning rate is set to 5e-6. For the baseline model, we use stochastic gradient descent (SGD) follwing <cit.>, and the initial learning rate is set to 0.001, which gives better development results.

 

Character and Word Embedding   The pre-trained word embedding size is 200, which is based on word co-occurrence and the directions of word pairs <cit.>, and the word length is restricted to 4. we use the topmost layer output as character embedding of the pre-trained Chinese Simplified BERT model with 12 layers, 768 hidden units and 12 heads[https://github.com/huggingface/pytorch-pretrained-BERT]. Besides that, the bigram embeddings and character unigram embeddings used for attending words are the same as <cit.>.


















 §.§ Development Experiments

We perform development experiments on the CTB6 development dataset to investigate the influence of hyper-parameters of self-attention network for CWS, and compare the performance of SAN, especially local self-attention, with BiLSTM. In addition, we evaluate the effect of utilizing of BERT for CWS models. 

Figure <ref> shows the iteration curve of F1-value against the number of training iterations with different configurations.“_Bigram" is the model using both unigram and bigram information, and “_BERT" is the model replacing the word2vec character unigram representation with BERT. “SAN" represents the original self-attention network and “L-SAN" represents local self-attention network. “BiLSTM" is our baseline model, which uses a bidirectional LSTM as feature extractor.

Width and Depth   <cit.> shows that increasing the number of layers can improve the performance of English-to-German translation. We investigate the effect of number of layers on CWS, by increasing the number of layers from 1 to 6 while fixing the number of heads to 8. The results are listed in Table <ref>. The model achieves the best F1-value 0.956 within 3 layers, after which the performance decreases with the increasing of layers. The F1-value decreases to 0.931 when the number of layers is set to 6. We fix the number of layers to 2 for the remaining experiments.

We vary the number of heads in multi-head self-attention to investigate its effect on CWS performance. The number of layers and dimension of head is fix to 2 and 64, respectively. As shown in Table <ref>, with increasing number of heads from 2 to 16, the performance does not vary too much. We fix the number of heads to 8 for the remaining experiments.

Effect of Local Attention  As Figure <ref> shows, the performance of “L-SAN_Bigram" gives much better results compared to “SAN_Bigram", which suggests that long-term dependencies can bring more noise than useful information. The proposed local self-attention network model can achieve the competitive results comparing with the baseline BiLSTM model. 

Effect of BERT  By replacing word2vec character embeddings with BERT, both BiLSTM and L-SAN models can reach the best F1-value within several epochs, with a significant improvement, which proves that context-dependent word representation can benefit CWS task.



 §.§ Final Results

In-Domain Results  We evaluate our model on three news datasets, including CTB, PKU and MSR. The main results and the recent state-of-the-art models are listed in Table <ref>. Compared with the baseline “BiLSTM+CRF" model, the proposed “L-SAN+CRF" model can achieve similar results, which proves that self-attention network can be a competitive feature extractor for CWS besides recurrent neural network. When replacing word2vec character embedding with BERT, the “BiLSTM+CRF" model gives 41.3%, 30.6% and 31.0% error reduction on CTB6/PKU/MSR, respectively,  and the “L-SAN+CRF" model has 41.3%, 32.7% and 41.3% error reductions on three in-domain datasets, respectively. Finally, “L-SAN+CRF" slightly outperforms “BiLSTM+CRF" when using BERT as unigram character representation.

Cross-Domain Results  We evaluate our model on the three cross-domain datasets, including ZX, FR and DL. The main results and three state-of-the-art models are listed in Table <ref>. “t" means neural type-supervised method is used to learn POS embedding and domain-specific words is generalized to corresponding tag. In “t_b", we learn different POS embeddings for different positions in a word. 

As shown in Table <ref>Cross domain evaluation, the F1-value of “L-SAN+CRF+BERT" has an average 0.7 improvement compared with the state-of-the-art results <cit.> in ZX and FR without using <cit.>'s domain adaptation techniques. This may be because ZX, FR and DL are all Chinese novels which contain a large number of noun entities and their wring styles are different from news domain. The result shows that BERT has rarely less effect on cross-domain CWS compared with strong domain adaptation methods. The “L-SAN+CRF+BERT+t" model has 21.15%, 25.96% and 1.54% error reduction on ZX/FR/DL datasets, respectively, which shows that the proposed neural type-supervised method can handle out of vocabulary words more effective. For characters within a word, instead of sharing the same POS embedding of the word, we further distinguish POS embeddings of characters according to their position in a word. The “L-SAN+CRF+BERT+t_b" gives 33.65%, 32.69% and 24.62% on three datasets, respectively. We believe that this is due to more supervision information.



 §.§ Analysis

Sentence Length  We compare the baseline model and local self-attention network model, as well as the two models with BERT input representation on different sentence lengths. Figure <ref> shows the F1-value on CTB6 test dataset. The two models without using BERT show a similar performance-length curve, which reaches a peak at around 30-character sentences and decreases when sentence length over 90. One possible reason is that very short sentences are rare while long sentences are semantically more challenging. However, the two models using BERT both show more stable performance-length curves, which shows that contextualized BERT representation can stabilize performance against sentence length.


Noun Entity Segmentation  Noun entities raise a key problem for cross-domain CWS. Table <ref> shows the three models segmentation results on 15 noun entities with the highest frequency of three datasets. M1 and M2 represent “L-SAN+CRF+BERT" and “L-SAN+CRF+BERT+t", respectively, while M3 represents “L-SAN+CRF+BERT+t_b". As the table shows, the average precision of MI is 0.55. By using neural type-supervised domain adaptation method, the average precision of M2 has a improvement of 0.18 in absolute value.

Some person names are incorrectly segmented by M2, such as “戴沐白(person name)" and “张小凡(person name)". When incorporating the positional information of character in the word, the average segmentation precision improves further and most noun entities can be correctly segmented, except the word “叶知秋(person name)". The reason is that the domain lexicon does not contain “叶知秋". This shows that our method makes effective use of domain lexicons.

Case Study  We use two examples of neural type-supervised domain adaptation for illustrated discussion. In example 1, “L-SAN+CRF+BERT" fails to handle the domain entity noun “韩立(person name)" while the two neural type-supervised domain adaptation method segment it correctly. For example 3, only “L-SAN+CRF+BERT+t_b" segments it correctly. One possible reason is that it maybe difficult to distinguish between “戴沐白 (person name)", which is a domain specific entity noun and “白虎 (white tiger)", which is a common noun.



§ RELATED WORK


Chinese Word Segmentation  <cit.> extract features based on character representation by using LSTM or GRU. <cit.> propose a transition-based neural model which can utilize the word-level features. <cit.> trains character embedding with word-based context information on auto-segmented data. <cit.> exploit the effectiveness of rich external resources through multi-task learning. For cross-domain CWS, <cit.> propose a type-supervised domain adaptation approach for joint CWS and POS-tagging, which shows a competitive result compared to token-supervised methods. <cit.> investigate CWS for Chinese novels, proposing a method which can automatically mine noun entities for novels using a double-propagation algorithm. <cit.> investigate how to integrate external dictionary into CWS models. Similar to <cit.> and <cit.>, our work uses domain lexicon. The difference is we utilize POS embeddings through an end-to-end neural method.


Self-Attention Network  Self-attention network <cit.> was first proposed for machine translation. <cit.> and <cit.> use SAN for the task of semantic role labeling, which can directly capture the relationship between two arbitrary tokens in the sequence. <cit.> incorporate linguistic information through multi-task learning, including dependency parsing, part-of-speech and predicate detection. <cit.> propose multi-dimensional attention as well as directional information, achieving the state-of-the-art results on natural language inference and sentiment analysis tasks. <cit.> show that a novel encoder based on self-attention can lead to state-of-the-art results for the constituency parsing task. Along with this strand of work, we study the influence of global and local attention for CWS and build a SAN-CRF word segmentor, which gives competitive results compared with BiLSTMs.



Contextualized word representation  Context-dependent word representations pre-trained from large-scale corpora have received much recent attention. ELMo <cit.> is based on recurrent neural networks language models. OpenAI GPT <cit.> builds a left-to-right language model with a multi-layer multi-head self-attention networks, which can handle long-term dependencies better compared to recurrent networks. Different from OpenAI GPT, BERT <cit.> uses a deep bidirectional Transformer pre-trained on Masked LM. Our work investigates the effect of contextualized character representation on both in-domain and cross-domain CWS under a unified SAN framework.



§ CONCLUSION

We investigated self-attention network for Chinese word segmentation, demonstrating that it can achieve comparable results with recurrent network methods. We found that local attention gives better results compared to standard SAN. Under SAN, we also investigate the influence of rich character and word features, including BERT character embeddings and a neural attention method to integrate word information into character based CWS. Extensive in-domain and cross-domain experiments show that the proposed SAN method archives state-of-the-art performance on both in-domain and cross-domain Chinese word segmentation datasets.






acl_natbib

