Hybrid Neural Tagging Model for Open Relation Extraction
Shengbin Jia Yang Xiang
cs.CL


Neural Open Relation Tagging Model]Hybrid Neural Tagging Model for Open Relation Extraction




This is the corresponding author


  Tongji University
  College of Electronic and Information Engineering
  Shanghai
  China
shengbinjia,shxiangyang@tongji.edu.cn

This work is supported by the National Natural Science Foundation of China under Grant No.71571136, and the 2019 Tencent Marketing Solution Rhino-Bird Focused Research Program.



	
Open relation extraction (ORE) remains a challenge to obtain a semantic representation by discovering arbitrary relation tuples from the unstructured text. Conventional methods heavily depend on feature engineering or syntactic parsing, they are inefficient or error-cascading. Recently, leveraging supervised deep learning structures to address the ORE task is an extraordinarily promising way. However, there are two main challenges: (1) The lack of enough labeled corpus to support supervised training; (2) The exploration of specific neural architecture that adapts to the characteristics of open relation extracting. In this paper, to overcome these difficulties, we build a large-scale, high-quality training corpus in a fully automated way, and design a tagging scheme to assist in transforming the ORE task into a sequence tagging processing. Furthermore, we propose a hybrid neural network model (HNN4ORT) for open relation tagging. The model employs the Ordered Neurons LSTM to encode potential syntactic information for capturing the associations among the arguments and relations. It also emerges a novel Dual Aware Mechanism, including Local-aware Attention and Global-aware Convolution. The dual aware nesses complement each other so that the model can take the sentence-level semantics as a global perspective, and at the same time implement salient local features to achieve sparse annotation. Experimental results on various testing sets show that our model can achieve state-of-the-art performances compared to the conventional methods or other neural models.
	









	<ccs2012>
	<concept>
	<concept_id>10010147.10010178</concept_id>
	<concept_desc>Computing methodologies Artificial intelligence</concept_desc>
	<concept_significance>500</concept_significance>
	</concept>
	
	<concept>
	<concept_id>10010147.10010178.10010179</concept_id>
	<concept_desc>Computing methodologies Natural language processing</concept_desc>
	<concept_significance>300</concept_significance>
	</concept>
	
	<concept>
	<concept_id>10010147.10010178.10010179.10003352</concept_id>
	<concept_desc>Computing methodologies Information extraction</concept_desc>
	<concept_significance>100</concept_significance>
	</concept>
	<concept>
	<concept_id>10002951.10003260.10003277</concept_id>
	<concept_desc>Information systems Web mining</concept_desc>
	<concept_significance>300</concept_significance>
	</concept>
	
	<concept>
	<concept_id>10002951.10003260.10003277.10003279</concept_id>
	<concept_desc>Information systems Data extraction and integration</concept_desc>
	<concept_significance>100</concept_significance>
	</concept>
	
	
	</ccs2012>


[500]Computing methodologies Artificial intelligence
[300]Computing methodologies Natural language processing
[100]Computing methodologies Information extraction
[300]Information systems Web mining
[100]Information systems Data extraction and integration











[
    Yang Xiang
    , 
==============











§ INTRODUCTION


Open relation extraction (ORE) <cit.> is an important NLP task for discovering knowledge from unstructured text. It does not identify pre-defined relation types like the Traditional relation extraction (TRE) <cit.>. Instead, it aims to obtain a semantic representation that comprises a arbitrary-type relational phrase and two argument phrases. Such as tracking a fierce sports competition, the news reporting works should find unpredictable relation descriptions (e.g., (Gatlin, again won, the Championships). ORE systems can satisfy such personalized scenarios by discovering resourceful relations without pre-defined taxonomies. 
A robust open relation extractor, especially working with limited well-annotated data, can greatly prompt the research not only in knowledge base construction <cit.>, but also in a wide variety of applications <cit.>, for example, intelligent questions and answers <cit.>, text comprehension <cit.>. ORE has gained consistent attention.

However, this task is constrained by the lack of labeled corpus or inefficient extraction models. Conventional methods are usually based on pattern matching to carry out unsupervised or weak-supervised learning. The patterns are usually syntax-relevant. They
are handcrafted by linguists <cit.> or bootstrapped depend on syntax analysis tools <cit.>. As for the former, the manual cost is high and the scalability is poor. The latter heavily depends on syntax analysis tools, and the cascading effect caused by parsing errors is serious.

Deep learning-based methods are popular and have achieved good accomplishments in various information extraction tasks <cit.>. Recently, a few people are trying to solve the ORE task with supervised neural network methods, especially the neural sequence tagging models <cit.>. 
Nevertheless, there are many challenges and difficulties. 
(1) There are few public large-scale labeled corpora for training supervised ORE learning models.  And labeling a training set manually with a large number of relations is heavily costly. 
(2) It's always challenging for open-ended tasks to produce supervised systems. The specific neural architecture that adapts to the characteristics of ORE should be explored urgently. 
Normal sequence labeling models encode word-level context information as assigning each word a tag to indicate the boundary of adjacent segments <cit.>. However, relation extraction is to only mark relational words corresponding to arguments, multiple relations involved in one sentence only focus on fractional aspects of the text. How to get the relevant information among the arguments and a relation,
and how to focus on the key parts of a sentence according to a certain relation, are worthy of special attention.



We overcome the above challenges by the following works. 
As for the first challenge, we build a large-scale, high-quality corpus in a fully automated way. We verify that the dataset has good diversification and can motivate models to achieve promising performances. 
Besides, we design a tagging scheme to transform the extraction task into a sequence tagging problem. A triple corresponds to its unique tag sequence, thus overlapping triples in a sentence can be presented simultaneously and separately.

As for the second challenge, to adapt to the characteristics of open relation extraction, we present a hybrid neural network model (HNN4ORT) for open relation tagging. The model has two highlights: 
(a) It employs the Ordered Neurons LSTM (ON-LSTM) to learn temporal semantics, while capturing potential syntactic information involved in natural language, where syntax is important for relation extraction to acquire the associations between arguments and relations;
(b) We propose the Dual Aware Mechanism, including Local-aware Attention and Global-aware Convolution. The two complement each other and realize the model focusing on salient local features to achieve sparse annotation while  considering a sentence-level semantics as a global perspective.

In summary, the main contributions of this work are listed as follows:

	
	
  * We transform the open relation extraction task into a sequence tagging processing and design a tagging scheme to address multiple overlapping relation triples. 
	
	
  * We present a hybrid neural network model (HNN4ORT), especially the Dual Aware Mechanism, to adapt to the characteristics of the ORE task.
	
	
  * We construct an automatic labeled corpus [The dataset can be obtained from https://github.com/TJUNLP/NSL4OIE.] in favor of adopting the supervised approaches for the ORE task. It is simple to construct and is larger-scale than other existing ORE corpora.

	
  * Experimental results on various testing sets show that our model can produce state-of-the-art performances compared to the conventional methods or other neural models.
	


The rest of this article is organized as follows. In Section 2, we review some related work. Then we detail the process of preparing the training corpus in Section 3. In Section 4, we present our model in detail. In Section 5 and 6, we conduct and analyze experiments on multiple datasets. Section 7, concludes this work and discusses future research directions.




§ RELATED WORKS



Relation extraction can be divided into two branches, including traditional relation extraction and open relation extraction.  

Traditional relation extraction (TRE) <cit.> can be regarded as a classification task which is committed to identifying pre-defined relation taxonomies between two arguments, and undefined relations will not be found. The mainstream methods were based on neural systems to recognize relations by using the large-scale knowledge bases to provide distant supervision <cit.>.

As for the Open relation extraction (ORE) <cit.>, most of the existing methods used patterns-based 
matching approaches by leveraging linguistic analysis. 
In general, the patterns were generalized by handcrafting <cit.> or semi-supervised learning <cit.> such as bootstrapping. These manual patterns were higher-accuracy than those produced automatically. However, they were heavy-cost and poor-efficiency.

Many extractors, such as TextRunner <cit.>, WOE^pos <cit.>, Reverb <cit.>, focused on efficiency by restricting the shallow syntactic parsing to part-of-speech tagging and chunking. Meanwhile, many approaches designed complex patterns from complicated syntactic processing, especially dependency parsers, such as WOE^parse <cit.>, PATTY <cit.>, OLLIE <cit.>, Open IE-4.x <cit.>, MinIE <cit.> and so on. These extractors could get significantly better results than the extractors based on shallow syntax. However, they heavily relied on the syntactic parsers. Many papers <cit.> analyzed the errors made by their extractors and found that parser errors account for a large even the largest part of the whole. Parsing errors restrained the extracting performances and would produce a serious error cascade effect.
Taking a certain strategy to refine the extraction results, should be an effective way to gather high-quality relation triples.

The sequence tagging tasks, such as Chinese word segmentation (CWS), Part-of-speech tagging (POS), and Named entity recognition (NER), require to assign representative labels for each word in a sentence. Conventional models were linear statistical models, which included Hidden Markov models <cit.>, Maximum entropy models <cit.>, and Conditional random fields (CRF) <cit.> and so on. Neural methods mapped input sequences to obtain fixed dimensional vector representations by various neural networks <cit.>, then predicted the target sequences from the vectors using a layer with Softmax activation function <cit.> or a special CRF layer <cit.>.

There were a few examples that applied neural models to open information extraction tasks.
According to the Machine translation mechanism, the extraction process was converted into text generation. Zhang et al. <cit.> extracted predicate-argument structure phrases by using a sequence to sequence model. Cui et al. <cit.> proposed a multi-layered encoder-decoder framework to generate relation tuples related sequences with special placeholders as a marker. 
Similarly, Bhutani et al. <cit.> used the encoder-decoder method to generate relation triples, but only from the limited question and answer pairs. In addition, Zheng et al. <cit.> creatively designed the model to transform the traditional relation extraction into relation sequence tagging. Later, Stanovsky et. al. <cit.> formulated the ORE task as a sequence tagging problem. They applied the LSTM with a Softmax layer to tag each word. However, we try to design more effective semantic learning frameworks to annotate relations. Besides, we have a larger training set of about several hundred thousand, but they have only a few thousand.




§ TRAINING CORPUS AND TAGGING SCHEME



There are few public large-scale labeled datasets for the ORE task. Stanovsky and Dagan <cit.> created an evaluation corpus by an automatic translation from QA-SRL annotations <cit.>. It only contains 10,359 tuples over 3200 sentences. Then, Stanovsky et al. <cit.> further expand 17,163 labeled open tuples from the QAMR corpus <cit.>. However, the accurate has declined. 

Therefore, we adopt a mechanism of bootstrapping by using multiple existing open relation extractors without having to resort to expensive annotation efforts. Currently, the training set contains 477,701 triples. And it is easy to get expanded. We also design a tagging scheme to automatically annotate them.






 §.§ The Correct Relation Triples Collecting


We use three existing excellent and popular extractors (OLLIE <cit.>, ClausIE <cit.>, and Open IE-4.x <cit.>) to extract relation triples from the raw text[The original text is produced from the WMT 2011 News Crawl data, at http://www.statmt.org/lm-benchmark/.]. These extractors have their expertise, so to ensure the diversity of extraction results. If a triple is obtained simultaneously by the three extractors, we should believe that the triple is correct and add it to our corpus.

All the extractors are based on the dependency parsing trees, so the extracted relational phrase may be a combination product. That is, the distance between the adjacent words in the phrase may be distant in the original sentence sequence. Moreover, the adjacency order of the words in a triple may be different from that in the sentence. For example, from the sentence “He thought the current would take him out, then he could bring help to rescue me", we can get a triple (the current, would take out, him). 
Thus, we define some word order constraints: The arguments in a triple (Argument1, Relation, Argument2) are ordered. All words in Argument1 must appear before all words in Relation. All the words contained in Argument2 must appear after the Relation words. The order of words appearing in Relation must be consistent with them appearing in the original sentence sequence. In addition, each relational word must have appeared in the original sentence and is not the modified word or the added word. These constraints is to make a model easy to use sequence tagging.

We randomly sample 100 triples from the dataset to test the accuracy. The accuracy performance is up to 0.95. The sampling result verifies the validity of the above operating. In addition, it represents that the extraction noises caused by syntactic analysis errors can also be well filtered out.

In order to build a high-quality corpus for model training, we are committed to the high-accuracy of extractions at the expense of the recall. The constructed dataset is imperfect, but it still has acceptable scalability. The experimental sections prove the effectiveness of the dataset.




 §.§ Automatically Sequence Tagging


Tagging is to assign a special label to each word in a sentence <cit.>. We use the BIOES annotation (Begin, Inside, Outside, End, Single) that indicates the position of the token in an argument or the relational phrase. It has been reported that this annotation is more expressive than others such as BIO <cit.>.
The words appearing outside the scope of a relation triple will be labeled as “O”.
As for a relation triple, the arguments and relational phrase could span several tokens within a sentence respectively. Thus, the arguments and relation-phrase need to be division and tagged independently.

In the general tagging scheme <cit.>, only one label will be assigned to each word in sentences. However, A sentence may contain multiple, overlapping triples. An argument can have multiple labels when it belongs to different triples. As a result, the normal tagging program cannot handle the case where a relation is involved in multiple triples, or an argument is related to multiple relations.

To cope with such multiplicity, we design an overlap-aware tagging scheme that can assign multiple labels for each word. Each triple will correspond to its unique tag sequence.
Figure <ref> is an example that a sentence is tagged by our scheme. 
Giving an argument pair, there can only be one relation between the pair in a sentence [The coordination of verbs in a sentence should be considered as one complete relational phrase.]. Therefore, after the argument tags being pre-identified, the sequence of relational tags is uniquely determined.

When using models to predict, we extract the candidate argument pairs in advance [Candidate arguments recognition is easy to do through many existing methods, so we can add it to the pre-processing process. We use the methods in the ClausIE <cit.> and Open IE-4.x <cit.> to pre-identify the arguments in sentences.], then transform them into the argument embedding as model inputs to identify the relationship between them. 




 §.§ From Tag Sequence to Extracted Results


From the tag sequence  Tags Sequence1 in Figure <ref>, “The America" and “Trump" can be combined into a triple whose relation is “President". Because the relation role of “The America" is “1" and “Trump" is “2", the final result is (The America, President, Trump). The same applies to (Trump, will visit, the Apple), (the Apple, founded by, Steven Paul Jobs).






§ METHODOLOGY 



We provide a detailed description of our model (HNN4ORT). It is illustrated in Figure <ref>. Our model has the following superiorities:


  1 Employ the Ordered Neurons LSTM (ON-LSTM) to learn temporal semantics, while capturing potential syntactic information in natural language.

  2 Design Dual Aware Mechanism, including Local-aware Attention and Global-aware Convolution. It emphasizes focusing on local tagging but with a sentence-level semantics as a global perspective.




 §.§ ON-LSTM Network



Natural language can be defined as sequence. However, the underlying structure of language is not strictly sequential. It should be hierarchically structured that be  determined by a set of rules or syntax <cit.>. 

This syntactic information is critical to the relation extraction task. There are strict semantic associations and formal constraints that the head argument is the agent of the relation and the tail argument is the object of the relation. 
Integrating syntax-semantics into a neural network can encode better representations of natural language sentences. 

Recurrent neural network (RNN), and in particular it's variant  the Long short-term memory network (LSTM), is good at grasping the temporal semantics of a sequence. They are useful for all kinds of tagging applications <cit.>. However, RNNs  explicitly impose a chain structure on a sentence.  The chain structure is contrary to the potential hierarchical structure of language, resulting in
that RNN (including LSTM) models may be difficult to effectively learn potential syntactic information <cit.>.

Therefore, we employ a new RNN unit, ordered neurons LSTM (ON-LSTM) <cit.>. It enables model to performing tree-like syntactic structure composition operations without destroying its sequence form. For a given sentence S={x_1, x_2, …, x_t}, the ON-LSTM returns the representation 𝐇={𝐡_1, 𝐡_2, … , 𝐡_t} about the sequence S. 

The ON-LSTM uses a special inductive bias for the standard LSTM. This inductive bias promotes differentiation of the life cycle of information stored inside each neuron: high-ranking neurons will store long-term information, while low-ranking neurons will store short-term information. 

The ON-LSTM uses an architecture similar to the standard LSTM. 
An LSTM unit is composed of a cell memory and three multiplicative gates (i.e. input gate, forget gate, and output gate) which control the proportions of information to forget and to pass on to the next time step <cit.>. The forget gate f_t and input gate i_t in ON-LSTM are identical with them in LSTM, where they are used to control the erasing and writing operation on cell states c_t, as,


    𝐟_t  = σ( 𝐖_f𝐱_t + 𝐔_f𝐡_t - 1 + b_f)


    𝐢_t  = σ( 𝐖_i𝐱_t + 𝐔_i𝐡_t - 1 + b_i).


As for the cell memory c_t, it may be difficult  for a standard LSTM to discern the hierarchy information, since the gates in the LSTM act independently on each neuron. Therefore, the ON-LSTM enforces the order where the cell in each neuron should be updated.
The updated information in cells is allocated by the activation function cumax(). The contribution of the function is to produce the vectors of master input gate ĩ_t and master forget gate f̃_t ensuring that when a given neuron is updated, all of the neurons that follow it in the ordering are also updated.


    𝐜̂_t  = tanh( 𝐖_c𝐱_t + 𝐔_c𝐡_t - 1 + b_c)


    𝐟̃_t  = cumax( 𝐖_f̃𝐱_t + 𝐔_f̃𝐡_t - 1 + b_f̃)


    𝐢̃_t  = cumax( 𝐖_ĩ𝐱_t + 𝐔_ĩ𝐡_t - 1 + b_ĩ)


    ω_t  = 𝐟̃_t ∘𝐢̃_t


    𝐜_t  = ω_t ∘( 𝐟_t∘𝐜_t - 1 + 𝐢_t∘𝐜̂_t ) + ( 𝐟̃_t - ω_t )∘𝐜_t - 1 + (𝐢̃_t - ω_t )∘𝐜̂_t


According to the output gate o_t and cell memory c_t, the output of the hidden state 𝐡_t is


    𝐨_t  = σ( 𝐖_o𝐱_t + 𝐔_o𝐡_t - 1 + b_o)


    𝐡_t  = 𝐨_t∘tanh( 𝐜_t).


In the above formulas, 𝐖 and 𝐔 are the trainable parameter matrixes, b is the bias. σ is the logistic sigmoid function, ∘ denotes the Hadamard product.





 §.§ Local-aware Attention



The RNN-based encoder generates the equal-level representation for each token of the entire sentence regardless of its contexts. However, a sentence may present an argument from various aspects, and various relations only focus on fractional aspects of the text.

Given the target arguments, not all of words/phrases in its text description are useful to model a specific fact. Some of them may be important for predicting relation, but may be useless for other relations. Empirically, relational words tend to appear in the neighborhoods of arguments.
Therefore, it is necessary for a model to pay more attention to the relevant parts of a sentence, according to different target relation. We propose a Local-aware Attention Network according to the attention mechanism <cit.>, as shown in Figure <ref>(B-1).

After being given a candidate argument pair a, the attention weight for each token x_t of the sentence is defined as α_t, which is


    α_t=exp(𝐕_t)/∑_i=1^Texp(𝐕_i)



    𝐕_t(a)= tanh(𝐖_a𝐡_t+𝐔_a𝐚).


In the above formula, 𝐖_a, 𝐔_a are parameters matrices. The 𝐚 is the embeddings of the candidate argument pair. It is the key information that tells the model what the expected destination is, so that the same sentence can get different attention states for different predicting relations. The 𝐡_t is obtained by the bidirectional ON-LSTM network. The ON-LSTM processes the sequence S in both directions and encodes each token x_t into a fixed-size vector representation 𝐇=[𝐡_1, 𝐡_2, …, 𝐡_t], by being calculated as

    𝐡_t=ON-LSTM_fw(𝐡_t-1,𝐜_t) ⊕ON-LSTM_bw(𝐡_t+1,𝐜_t),

where 𝐡_t-1 is the hidden representation of the forward ON-LSTM (ON-LSTM_fw) at position t-1, and 𝐡_t+1 are the hidden representation of the backward ON-LSTM (ON-LSTM_bw) at position t+1, ⊕ denotes concatenation operation. 


We apply the attention weight α_t to 𝐡_t, resulting in an updated representation 𝐇^(a), as


    𝐇^(a)= {α_1𝐡_1, α_2𝐡_2, …, α_t𝐡_t}.






 §.§ Global-aware Convolution



Normal sequence labeling models cannot fully encode sentence-level information but encode word-level context information as assigning each word a tag to indicate the boundary of adjacent segments <cit.>. However, we believe that the Local-aware Attention mechanism can play a better role only when it is provided with a comprehensive sentence-level context.
Therefore, in this section, we propose a Global-aware Convolution network. It and the Local-aware Attention network complement each other and form the Dual Aware Mechanism.

We use the stacked Convolutional neural network to learn global information from a whole sentence since it owns powerful spatial perception ability <cit.>.
In the process of recognizing a piece of text, the human brain does not recognize the entire text at the same time, but firstly perceives each local feature in the text, and then performs a comprehensive operation at a higher level to obtain global information. To simulate this learning process of the human brain, we design the Global-aware Convolution network, as shown in Figure <ref>(B-2).

Fragment-level feature maps can be derived from a set of convolutions that operate on the embeddings of a sentence S. Convolution is an operation between a vector of weights and the embeddings. The weight matrix is regarded as the filter for the convolution <cit.>. We apply a set of convolutional filters 𝐖_l and bias terms b_l to the sentence as per equation <ref>, to learn a representation.

    𝐡_t^l=RELU(𝐖_l·[𝐱_t, …, 𝐱_t+l]+b_l),  l = 2i+1     i=1,2,...

where 𝐡_t^l is used to represent vectors at time t, which includes all phrases of length l started with x_t.
i indicates the id of layers.

By stacking layers of convolutions of increasing filter width, we can expand the size of the effective input width to cover the most of length of a sequence.
Each iteration takes as input the result of the upper convolution layer. The kernel size of each current layer is 2 more than that of the upper layer.

After convoluting, a global max-pooling operation is applied that stores only the highest activation of each filter. It is used for feature compression and extracts the main features. 
So far, the final output vector with a fixed length can be obtained as sentence-level global features.




 §.§ Hybrid Framework


Figure <ref> illustrates the framework of our model. The main components of the framework are the Triplet Embedding Layer, Dual Aware Encoder, and Tag Decoder.


We adopt triplet embeddings to map each token to a vector, as shown in Figure <ref>(A).
Given a sentence S as a sequence of tokens (i.e., words of the sentence), each token x_t is represented by 
a set of embeddings (𝐱_t^(w), 𝐱_t^(a), 𝐱_t^(p)), where 𝐱_t^(w) represents the word embedding, 𝐱_t^(a) denotes the embedding of candidate argument pair, and 𝐱_t^(p) shows the Part-of-speech (POS) embedding. We consider the POS information into our model since it plays an important role in the relation extraction processing. 

Next,  the token representation vectors 𝐒={𝐱_1, 𝐱_2, …, 𝐱_t} are sent to the Dual Aware Encoder. One side, these embeddings are fed into the module of Local-aware Attention, as shown in Figure <ref>(B-1), outputting the features 𝐅^(l).
Meanwhile, after giving input 𝐒, we execute the deep convolution module (B-2) to output global-aware features 𝐅^(g).
Therefore, the encoder returns a representation 𝐅 = [ 𝐅^(l); 𝐅^(g)].

Subsequently, the upper output 𝐅 is fed into the Tag Decoder to jointly yield the final predictions for each token, as shown in Figure <ref>(C). 
We take advantage of the sentence-level tag information by a Conditional random fields (CRF) decoding layer.
 
Notably, there are strong dependencies across output labels. It is beneficial to consider the correlations between labels in neighborhoods. The CRF can efficiently use past and future tags to predict the current tag. Therefore, it is a common way to model label sequence jointly using a CRF layer <cit.>.

In detail, for an output label sequence y=  ( y_1, y_2, …, y_t), we define its score as

    s ( S, y  )= ∑_i=0^T𝐀_y_i,y_i+1+∑_i=1^T𝐏_i,y_i,

where 𝐀 is a matrix of transition scores such that 𝐀_i,j represents the score of a transition from the tag i to tag j. y_0 and y_n that separately means the start and the end symbol of a sentence. We regard 𝐏 as the matrix of scores outputted by the upper layer. 𝐏_i,j corresponds to the score of the j^th tag of the i^th word in a sentence. 

We predict the output sequence that gets the maximum score given by

    y^* =ỹ∈ Y_Sargmaxs ( S,ỹ ),

where Y_S represents all possible tag sequences including those that do not obey the BIOES format constraints.






§ EXPERIMENTS AND DISCUSSIONS



In this section, we present the experiments in detail. We evaluate various models with Precision (P), Recall (R) and F-measure (F1). 



 §.§ Experimental Setting


Testing set. To satisfy the openness and effectiveness of the experiments, we gather four testing sets from the previously published works. They should be close to nature and independent of the training set. Table 1 presents the details of the four datasets.

Firstly, the Reverb dataset is obtained from <cit.> which consists of 500 sentences with manually labeled 1,765 extractions. The sentences are obtained from Yahoo. Next, the Wikipedia dataset includes 200 random sentences extracted from Wikipedia. And we collect 605 extractions manual labeled by Del Corro <cit.>. Then, the NYT dataset contains 578 triples extracted from 200 random sentences in the New York Times collection. It is also created by Del Corro et al.
In addition, Stanovsky and Dagan <cit.> present the OIE2016 dataset which is an opened benchmark for ORE. It contains 10,359 tuples over 3200 sentences.







Hyperparameters. We implement the neural network by using the Keras library[https://github.com/keras-team/keras]. 
The training set and validation set contain 395,715 and 81,986 records, respectively. 
The batch size is fixed to 256. We use early stopping <cit.> based on performance on the validation set. The number of LSTM units is 200 and the number of feature maps for each convolutional filter is 200. Parameter optimization is performed with Adam optimizer <cit.>. The initial learning rate is 0.001, and it should be reduced by a factor of 0.1 if no improvement of the loss function is seen for some epochs. Besides, to mitigate over-fitting, we apply the dropout method <cit.> to regularize models.
We use three types of embeddings as inputs. Word embedding is pre-trained by word2vec <cit.> on the corpora. Its dimension is 300. Part-of-speech (POS) embedding is obtained by using the TreeTagger <cit.> which is widely adopted to annotate the POS category, containing 59 different tags. It is 59 dimensions one-hot vectors. Besides, we represent the tag information of arguments as argument embedding, by using 10 dimensions one-hot vectors.




 §.§  Experimental Results 



We report the results [In order to avoid the distortion of argument recognition errors to the final performance, we only recognize the correctness of the relational phrases in a triple.][As for a relation extracted by Reverb, OLLIE, ClausIE, and Open IE-4.x, only when its confidence is greater than 0.5 can it be judged that the relations are correct.] of various models to work on the Reverb dataset, Wikipedia dataset, and NYT dataset, as shown in Table <ref> and Figure <ref>. We can draw the following conclusions:

(1) We discuss various popular neural structures and do lots of experiments to adapt to the open relation tagging task. It is valuable to understand the effectiveness of different neural architectures for the ORE task and to help readers reproduce experiments. These methods include the unidirectional LSTM network (UniLSTM) <cit.>; the bidirectional LSTM network with a Softmax classifier (LSTM), or with  
a CRF classifier (LSTM-CRF) <cit.>; the CNN network with a Softmax classifier (CNN), or with  
a CRF classifier (CNN-CRF) <cit.>; the combination of the LSTM and CNN (LSTM-CNN-CRF) <cit.> and so on.

Our model HNN4ORT outperforms all the above methods according to F1. It shows the effectiveness of its hybrid neural network architecture. Remarkably, previous models (such as LSTM-CNN-CRF <cit.>) have improved performances by integrating CNN and LSTM. They usually simply concatenate the feature vectors learned from the CNN and LSTM at each time-step.  However, we propose the Dual Aware Mechanism which achieves better performances.

In addition, we analyze the effects of various networks. Compared to unidirectional LSTM (UniLSTM), bidirectional LSTM (LSTM) is obviously superior to UniLSTM in F1 on average, since it can capture richer temporal semantic information. LSTM is better than CNN in recall and F1, however, CNN takes better precision. In addition, from the comparison between LSTM and LSTM-CRF, ON-LSTM and ON-LSTM-CRF, CNN and CNN-CRF, we can get a unanimous conclusion that the CRF layer can greatly improve model performances than a Softmax layer.






(2) We compare neural sequence models with conventional methods. When the three models used to construct corpus (OLLIE, ClausIE, and Open IE-4.x) being regarded as baselines, our model HNN4ORT achieves better recall and F1. Especially, our model achieves a 16.1% improvement in F1 over the best baseline model ClausIE.

In addition, many of the neural sequence learning methods outperform conventional syntax-based methods. The conventional methods may be of high accuracy, but they can't learn rich enough patterns resulting in a lower recall. In addition, patterns is hard and inflexible, however, the syntax structures of sentences are ever-changing. The neural network-based models can learn deep sentence semantics and syntactic information, to achieve better precision and recall. 

(3) We verify the effectiveness of each module of our model HNN4ORT. 
First, we test the capability of the ON-LSTM. To our knowledge, it is the first time to be used for sequence tagging. The experimental results show that the ON-LSTM is better than LSTM about 2 % in F1 on average. This shows that the potential syntactic information captured by ON-LSTM is very useful for open relation annotation.

Second, we use the ON-LSTM with the CRF classifier (ON-LSTM-CRF) as the basic model, to test the performances of Dual Aware Mechanism.
After the Local-aware Attention network is integrated into the ON-LSTM-CRF network (+Local-Aware), the effect is increased by about 1 %. After the Global-aware Convolution network is integrated into the ON-LSTM-CRF network (+Global-Aware), the effect is increased by about 1.5 %. This proves that both modules are valid. 

Furthermore, when the dual modules are adopted at the same time, the model increases by more than 2 % on the average of F1. It shows that the Dual Aware Mechanism is highly reasonable that global sentence-level information and local focusing information complement each other for handling open relation extraction.


(4) We use the OIE2016 dataset to evaluate the precision and recall of different systems [The model proposed in <cit.> isn't shown here, because it used the OIE2016 dataset as a training set. We have evaluated it in Table <ref>. Its model structure is the same as the BiLSTM network with a Softmax output layer (LSTM, i.e. the 4th line).]. The precision-recall curves are shown in Figure <ref>, and the Area under precision-recall curve (AUC) for each system is shown in Figure <ref>. It is observed that our model HNN4ORT has a better precision than the three conventional models in the most recall range. The HNN4ORT is learned from the bootstrapped outputs of the extractions of the three conventional systems, while the AUC score is better than theirs. It shows that the HNN4ORT has fixed generalization ability after training on the training set. Although the precision of the neural model En-Decoder <cit.> is better than that of the HNN4ORT, the recall of it has been maintained in a lower range than that of the HNN4ORT. In addition, the HNN4ORT achieves the best AUC score of 0.487, which is significantly better than other systems. In particular, the AUC score of the HNN4ORT is two times more than that of the En-Decoder model.

(5) To sum up, the performances of our model HNN4ORT on the four datasets are stable and superior. It indicates that the model has good robustness and scalability.













§ PERFORMANCE ANALYSIS AND DISCUSSIONS






 §.§ Evaluation of Training Corpus


According to whether the three extractors that are used to construct the training set can correctly identify each test instance, we classify the test instances into four parts: all three extractors identify correctly (ATE), existing two extractors identify correctly (ETE), only one extractor identify correctly (OOE), no one extractor identify correctly (NOE). 

As shown in Figure <ref>, the model HNN4ORT identifies these instances in the ATE with an accuracy close to 1. It implies that the model can learn the training set data features well.  In addition, from the perspective of a single extractor, the model HNN4ORT can acquire the extracting ability of this single extractor after training on our training set. The correct results extracted by this single extractor, in addition to appearing in the ATE, will also appear in the ETE and OOE. These instances outside the ATE own the cognate regularity (data distribution) with those in the ATE. Therefore, the HNN4ORT has a certain recognition in three parts (ATE, EYTE, and OOE). The accuracy is highest on the ATE, follows by it on the ETE, and worse on the OOE (but still above 0.7).

In particular, the model can produce certain results in the NOE where such kind of instances may be little or barely appear in the training set. It shows that the model has strong generalization ability and can obtain more powerful recognition capability beyond the any single extractor of the three extraction tools.
In addition, as shown in Table <ref>, the model has achieved remarkable performances in various testing sets, although these testing sets and training set come from different sources. Based on the above phenomena, it can be concluded that the quality of training corpus should be acceptable.

Other than this, to maintain the high quality of the training set, we only select triples from the ATE. Although the ATE occupies a small proportion of the output of the extractors, the instances in the training set are considerable and are extracted from a large-scale text. This ensures that the training set has good diversification and passable quality.






 §.§ Analysis of  Embeddings

We evaluate the effect of various embeddings, as shown in Table <ref>. When the model HNN4ORT only uses word embeddings, the results are worse than that of the model with word embeddings and POS embeddings. We certificate that the POS features play a great role in promoting model performances and improve the F1 by 4.1%. In addition, when the input embeddings are adjusted along with model training, the effect is better and the F1 is increased by 1.4%.









 §.§ Analysis of Overlapping Triples

As shown in Table <ref>, The statistical results show that overlapping triples account for a large proportion in various datasets, all of which are above 30%. By using our overlap-aware tagging scheme, models can identify multiple, over-lapping triples, so that they have a better recall capacity. We perform the model HNN4ORT to identify overlapping triples, with good results on all three datasets.








 §.§ Error Analysis


To find out the factors that affect the performance of ORE tagging models, we analyze the tagging errors of the model HNN4ORT as Figure <ref> shown. There are four major types of errors. The 30.3% relations are missed by the model. And 22.1% of the extractions are abandoned because their corresponding tag sequences violate the tagging scheme. The two types of errors mainly limit the increase in recall. In addition, the model may wrongly determine the start or end position of a relational phrase. As a result, the relation will be recognized as falseness. Such phenomena affect model precision.










§ CONCLUSIONS



Open relation extraction is an important NLP task. The effect of conventional methods is not satisfactory. Therefore, we are committed to using advanced deep learning models to solve this task. In this paper, we construct a training set automatically and design a neural sequence tagging model to extract open relations. Taking the ON-LSTM and Dual Aware Mechanism as the essence, we propose a hybrid neural sequence tagging model (NHH4ORT). The experimental results show the effectiveness of our method. Compared with conventional extractors or other neural models, our approach achieves state-of-the-art performances on multiple testing sets. We also analyze the quality of the automatically constructed training set, and conclude that it should be highly acceptable. In future work, we will consider to study a more efficient annotation scheme and use it to deal with n-ary relation tuples.



ACM-Reference-Format-Journals



