A Hybrid Neural Network Model for Commonsense Reasoning
Pengcheng He Xiaodong Liu Weizhu Chen Jianfeng Gao
cs.CL

The Wang-Landau Algorithm as Stochastic Optimization and Its Acceleration
    Jun S. Liu
    December 30, 2023
=========================================================================


This paper proposes a hybrid neural network (HNN) model for commonsense reasoning. 
An HNN consists of two component models, a masked language model and a semantic similarity model, which share a BERT-based contextual encoder but use different model-specific input and output layers. 
HNN obtains new state-of-the-art results on three classic commonsense reasoning tasks, pushing the WNLI benchmark to 89%, the Winograd Schema Challenge (WSC) benchmark to 75.1%, and the PDP60 benchmark to 90.0%.
An ablation study shows that language models and semantic similarity models are complementary approaches to commonsense reasoning, and HNN effectively combines the strengths of both.
The code and pre-trained models will be publicly available at <https://github.com/namisan/mt-dnn>.





§ INTRODUCTION




Commonsense reasoning is fundamental to natural language understanding (NLU). As shown in the examples in Table <ref>, in order to infer what the pronoun “they” refers to in the first two statements, one has to leverage the commonsense knowledge that “demonstrators usually cause violence and city councilmen usually fear violence.” Similarly, it is obvious to humans what the pronoun “it” refers to in the third and fourth statements due to the commonsense knowledge that “An object cannot fit in a container because either the object (trophy) is too big or the container (suitcase) is too small.”



In this paper, we study two classic commonsense reasoning tasks: the Winograd Schema Challenge (WSC) and Pronoun Disambiguation Problem (PDP) <cit.>. 
Both tasks are formulated as an anaphora resolution problem, which is a form of co-reference resolution, where a machine (AI agent) must identify the antecedent of an ambiguous pronoun in a statement. 
WSC and PDP differ from other co-reference resolution tasks <cit.> in that commonsense knowledge, which cannot be explicitly decoded from the given text, is needed to solve the problem, as illustrated in the examples in Table <ref>.

Comparing with other commonsense reasoning tasks, such as 
COPA <cit.>, 
Story Cloze Test <cit.>,
Event2Mind <cit.>,
SWAG <cit.>, 
ReCoRD <cit.>, and so on, 
WSC and PDP better approximate real human reasoning, can be easily solved by native English-speaker <cit.>, and yet are challenging for machines. For example, the WNLI task, which is derived from WSC, is considered the most challenging NLU task in the General Language Understanding Evaluation (GLUE) benchmark <cit.>. Most machine learning models can hardly outperform the naive baseline of majority voting (scored at 65.1) [See the GLUE leaderboard at <https://gluebenchmark.com/leaderboard>], including BERT <cit.> and Distilled MT-DNN <cit.>.

While traditional methods of commonsense reasoning rely heavily on human-crafted features and knowledge bases <cit.>, we explore in this study machine learning approaches using deep neural networks (DNN). Our method is inspired by two categories of  DNN models proposed recently. 

The first are neural language models trained on large amounts of text data. <cit.> proposed to use a neural language model trained on raw text from books and news to calculate the probabilities of the natural language sentences which are constructed from a statement by replacing the to-be-resolved pronoun in the statement with each of its candidate references (antecedent), and then pick the candidate with the highest probability as the answer. <cit.> showed that a significant improvement can be achieved by fine-tuning a pre-trained masked language model (BERT in their case) on a small amount of WSC labeled data.

The second category of models are semantic similarity models. <cit.> formulated WSC and PDP as a semantic matching problem, and proposed to use two variations of the Deep Structured Similarity Model (DSSM) <cit.> to compute the semantic similarity score between each candidate antecedent and the pronoun by (1) mapping the candidate and the pronoun and their context into two vectors, respectively, in a hidden space using deep neural networks, and (2) computing cosine similarity between the two vectors.  The candidate with the highest score is selected as the result.

The two categories of models use different inductive biases when predicting outputs given inputs, and thus capture different views of the data. While language models measure the semantic coherence and wholeness of a statement where the pronoun to be resolved is replaced with its candidate antecedent, DSSMs measure the semantic relatedness of the pronoun and its candidate in their context. 

Therefore, inspired by multi-task learning <cit.>, we propose a hybrid neural network (HNN) model that combines the strengths of both neural language models and a semantic similarity model. 
As shown in Figure <ref>, HNN consists of two component models, a masked language model and a deep semantic similarity model. The two component models share the same text encoder (BERT), but use different model-specific input and output layers. The final output score is the combination of the two model scores. 
The architecture of HNN bears a strong resemblance to that of Multi-Task Deep Neural Network (MT-DNN) <cit.>, which consists of a BERT-based text encoder that is shared across all tasks (models) and a set of task (model) specific output layers.  
Following <cit.>, the training procedure of HNN consists of two steps: (1) pretraining the text encoder on raw text [In this study we use the pre-trained BERT large models released by the authors.], and (2) multi-task learning of HNN on WSCR which is the most popular WSC dataset, as suggested by <cit.>.

HNN obtains new state-of-the-art results with significant improvements on three classic commonsense reasoning tasks, pushing the WNLI benchmark in GLUE to 89%, 
the  benchmark [<https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WS.html>] 
<cit.> to 75.1%, 
and the PDP-60 benchmark [<https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/PDPChallenge2016.xml>] 
to 90.0%.
We also conduct an ablation study which shows that language models and semantic similarity models provide complementary approaches to commonsense reasoning, and HNN effectively combines the strengths of both.









§ THE PROPOSED HNN MODEL




The architecture of the proposed hybrid model is shown in Figure <ref>. The input includes a sentence S, which contains the pronoun to be resolved, and a candidate antecedent C. The two component models, masked language model (MLM) and semantic similarity model (SSM), share the BERT-based contextual encoder, but use different model-specific input and output layers. The final output score, which indicates whether C is the correct candidate of the pronoun in S, is the combination of the two component model scores.



 §.§ Masked Language Model (MLM)



This component model follows <cit.>.
In the input layer, a masked sentence is constructed using S by replacing the to-be-resolved pronoun in S with a sequence of N  tokens, where N is the number of tokens in candidate C.

In the output layer, the likelihood of C being referred to by the pronoun in S is scored using the BERT-based masked language model P_mlm(C|S). If C={c_1...c_N} consists of multiple tokens, log P_mlm (C|S) is computed as the average of log-probabilities of each composing token:

    P_mlm(C|S)= exp ( 1/N∑_k=1...Nlog P_mlm(c_k|S)  ).




 §.§ Semantic Similarity Model (SSM)



In the input layer, we treat sentence S and candidate C as a pair (S,C) that is packed together as a word sequence, where we add the  token as the first token and the  token between S and C.

After applying the shared embedding layers, we obtain the semantic representations of S and C, denoted as 𝐬∈ℝ^d and 𝐜∈ℝ^d, respectively. We use the contextual embedding of  as 𝐬. Suppose C consists of N tokens, whose contextual embeddings are 𝐡_1,...,𝐡_N, respectively. The semantic representation of the candidate C, 𝐜, is computed via attention as follows:

    α_k = softmax (𝐬^⊤𝐖_1 𝐡_k/√(d)),


    𝐜 = ∑_k=1...Nα_k ·𝐡_k.

where 𝐖_1 is a learnable parameter matrix, and α is the attention score. 

We use the contextual embedding of the first token of the pronoun in S as the semantic representation of the pronoun, denoted as 𝐩∈ℝ^d. 
In the output layer, the semantic similarity between the pronoun and the context is computed using a bilinear model:

    Sim(C,S) = 𝐩^⊤𝐖_2 𝐜,

where 𝐖_2 is a learnable parameter matrix.
Then, SSM predicts whether C is a correct candidate (i.e., (C,S) is a positive pair, labeled as y=1) using the logistic function:

    P_ssm(y=1|C,S) = 1/1 + exp(-Sim(C,S)).


The final output score of pair (S,C) is a linear combination of the MLM score of Eqn. <ref> and the SSM score of Eqn. <ref>:

    Score(C,S) =1/2 [P_mlm(C|S) + P_ssm(y=1|C,S)].




 §.§ The Training Procedure


We train our model of Figure <ref> on the WSCR dataset, which consists of 1886 sentences, each being paired with a positive candidate antecedent and a negative candidate. 

The shared BERT encoder is initialized using the published BERT uncased large model <cit.>. We then finetune the model on the WSCR dataset by optimizing the combined objectives:

    ℒ_mlm + ℒ_ssm + ℒ_rank,

where ℒ_mlm is the negative log-likelihood based on the masked language model of Eqn. <ref>, and ℒ_ssm is the cross-entropy loss based on semantic similarity model of Eqn. <ref>. 

ℒ_rank is the pair-wise rank loss. 
Consider a sentence S which contains a pronoun to be resolved, and two candidates C^+ and C^-, where C^+ is correct and C^- is not. We want to maximize Δ = Score(S,C^+) - Score(S,C^-), where Score(.) is defined by Eqn. <ref>. We achieve this via optimizing a smoothed rank loss:

    ℒ_rank = log (1 + exp(-γ (Δ + β))),

where γ∈ [1, 10] is the smoothing factor and β∈ [0, 1] the margin hyperparameter. In our experiments, the default setting is γ = 10, and β = 0.6.

We construct two tasks to solve the problem, one is language model task() as in <cit.>, the other is semantic matching task(). 
While   is robust to solve this problem, it lacks good explanation to the results.   is designed to produce meaningful semantic representations for the antecedent 𝐀 and pronoun 𝐏 with regarding to the context of the sentence 𝐒.

  uses contextualized attentive pooling (CAP) to produce the representation for antecedent, denoted as 𝐑_𝐀𝐍, and uses first token pooling (FTP) to produce the representation of the pronoun, denoted as  𝐑_𝐏. We will demonstrate that   performs well to capture semantic meaning of the antecedent and pronoun which will help a lot in WNLI task.



Our final model is trained via modified multi-task learning() which combines the benefits of   and . 



 §.§ Language Model Task


Following <cit.> we use the same setting to construct the input. Each input 𝐒 is separated into three sub-sentences according to the pronoun, that is, the sub-sentence at the left of the pronoun, 𝐒_𝐋, the pronoun 𝐏, the sub-sentence at the right of the pronoun, 𝐒_𝐑, as shown in (<ref>). 
Given a candidate 𝐀, it will be tokenized into n sub-words {a_k}_× n.  first construct a new sentence Ŝ by replace 𝐏 with n [MASK] tokens, as shown in (<ref>). 


    𝐒 = {𝐒_𝐋, 𝐏, 𝐒_𝐑}



    Ŝ = {𝐒_𝐋, {[𝐌𝐀𝐒𝐊]}_× n, 𝐒_𝐑}


Then it will produce a language model probability of 𝐀 conditioned on Ŝ, as shown in (<ref>)


    p = ℙ(𝐀|Ŝ) 
        =√(∏_k ℙ(a_k|Ŝ))


Instead of using hinge loss as the training objective function, we use a smoothed ranking loss as the objective function, as shown in (<ref>)


    L_LMT =    -log(p^+) + 
       log(1+e^(p^- - p^+ + β_1))


Where β_1 ∈ [0,1] are hyper-parameters chosen by grid search and we usally use 0.5 as the default value. With the modification, the final loss and gradient will be smoother which makes the model easier to be trained. We will show in the experiment that it will improve the performance of .



 §.§ Semantic Matching Task






   





















 first produces the semantic representations for the given antecedent and pronoun. Then it evaluates the matching score via a semantic matching function. Finally it determines if a candidate is the antecedent of the pronoun by compare the score with a threshold, e.g. binary classification, or with the score of other candidates, e.g. ranking.
 contains five major parts: input encoding, antecedent encoding, pronoun encoding, semantic matching and objective function.



  §.§.§ Input Encoding


Given a sentence 𝐒 and antecedent candidate 𝐀, we will treat 𝐀 as a separate sentence and pair it with 𝐒 together as sentence pairs. Following <cit.>, these two sentences are packed together with a special separator token [SEP]. Add Example The type id of the first sentence 𝐒 is assigned as 0, while the type id of the second sentence 𝐀 is assigned as 1.

    𝐄_𝐬_𝐣 = 𝐏_𝐬_𝐣 + 𝐓_0 + 𝐖_𝐬_𝐣



    𝐄_𝐚_𝐤 = 𝐏_𝐚_𝐤 + 𝐓_1 + 𝐖_𝐚_𝐤


The token embedding of the jth token s_j in 𝐒 is calculated via  (<ref>) while the token embedding of kth token a_k in 𝐀 is calculated via (<ref>). Where 𝐏_𝐬_𝐣,𝐄_𝐚_𝐤∈ R^1 × d  correspond to position bias embedding of s_j and a_k, 𝐖_𝐬_𝐣,𝐖_𝐚_𝐤∈ R^1 × d correspond to sub-word embedding of  s_j and a_k, while 𝐓_0, 𝐓_1∈ R^1 × d correspond to the sentence type embedding of type 0 and type 1.



  §.§.§ Antecedent Encoding

We denote BERT<cit.> encoder's last layer output of token [CLS] and a_k as 𝐇_[𝐂𝐋𝐒],𝐇_𝐚_𝐤∈ R^1 × d correspondingly.
  uses 𝐇_[𝐂𝐋𝐒] as a contextualized representation for the whole sentence. It is used to do contextualized attentive pooling(CAP) over {𝐇_𝐚_𝐤| k ∈[0,n) }. Bi-linear attention is used to calculate the embedding of antecedent via (<ref>) and (<ref>). 


    α_k=𝐇_[𝐂𝐋𝐒]𝐐 𝐇^⊺_𝐚_𝐤/√(d)



    𝐑_𝐀𝐍=∑_kα_k · 𝐇_𝐚_𝐤


Where 𝐐∈ R^d  × d is a learnable projection parameter. α_k is the attention score of antecedent token a_k. 𝐑_𝐀𝐍 the the final embedding of the antecedent 𝐀.



  §.§.§ Pronoun Encoding

For the given pronoun 𝐏 in 𝐒, it is tokenized into M sub-word tokens {pn_m} where m is the index of the sub-word token. We denote BERT<cit.> encoder's last layer output of tokens {pn_m} as {𝐇_𝐩𝐧_𝐦}. We just simply use first token pooling(FTP) to produce the representation of 𝐏. e.g. 

    𝐑_𝐏=𝐇_𝐩𝐧_0




  §.§.§ Semantic Matching

To evaluate the semantic matching score of the two representations, we apply two matching function. One is bi-linear model (<ref>), where 𝐌∈ R^d × d is a learnable parameter. 


    S_bi(𝐑_𝐏, 𝐑_𝐀𝐍)=𝐑_𝐏𝐌𝐑^⊺_𝐀𝐍







The final output of  is is normalized to make the output in the same range as the output of (<ref>) in , e.g. [0,1].
When bi-linear is applied, (<ref>) is used to normalize the output. 

    Ŝ_bi = 1/1+e^-S_bi







For WNLI task, we also scan a binary threshold on the score using WNLI training data, so that we can do binary  classification and don't need to extract other candidates from the premise sentence as in <cit.>.



  §.§.§ Objective Function

We use (<ref>) to calculate the loss.

    L_SMT =    1/2 log(1+e^-γ_2 S^+_bi) +
       1/2 log(1+e^γ_2 ( S^-_bi + β_2 ))


Where  S^+,-_bi∈[ -1, 1] are corresponding to the matching score of positive example and negative example calculated via (<ref>), γ_2 ∈ [1,10], β_2 ∈ [0,1] is the hyper-parameter chosen via grid search. We usally use 1, 0 as the default value of  γ_2 and β_2

When using cosine as the similarity function, the objective function choosing is crucial for the model to converge well. Hence we design a new loss as (<ref>) when cosine similarity is used.


    L_SMT = e^γ_2 (1-S^+_cos)^2 + e^γ_2 (1+S^-_cos)^2

Where  S^+,-_cos∈[ -1, 1] are corresponding to the matching score of positive example and negative example calculated via (<ref>).

We will show in experiment that, while bi-linear matching performs well generally, cosine performance better for WNLI task.



 §.§ Modified Multi-Task Learning





While  and  can solve the problem from different views of the same problem, we introduce  to combine the benefits of the two.
The structure of is shown in figure <ref>. Typical multitask learning usually tries to solve different problems with different objects in the same pass and use one output as the final output of each problem. During training, the tasks are scheduled randomly at mini-batch level or instance level. Our setting is different from typical multitask leaning in several ways,

    
  * First, the two tasks in our setting try to solve the same problem using the same training data from different views. 
    
  * Second, the two tasks are grouped together for each input-label pair instead of shuffled randomly during training. 
    
  * Finally, during inference we will combine both of the two tasks' output to produce the final output instead of just use one of the outputs.

Add picture to show how the multitask learning works


  §.§.§ Input

Suppose each training instance is composed of sentence 𝐒, a pair of candidates {𝐀^+,𝐀^-} corresponding to positive candidate and negative candidate. In , each instance will generate a pair of sentences {Ŝ^+_𝐥𝐦𝐭, Ŝ^-_𝐥𝐦𝐭} as in <ref>. In , each instance will generate a pair of sentences {Ŝ^+_𝐬𝐦𝐭, Ŝ^-_𝐬𝐦𝐭} as in <ref>. In , those two pairs will be grouped together as the input, e.g. {Ŝ^+_𝐥𝐦𝐭, Ŝ^-_𝐥𝐦𝐭,Ŝ^+_𝐬𝐦𝐭, Ŝ^-_𝐬𝐦𝐭}. This is different from typical multi-task learning which often shuffles the instances of different tasks randomly.



  §.§.§ Output

In , the output of  and  are added together as the ensemble output of , Ŝ_M-MTL, as shown in (<ref>)

    Ŝ_M-MTL = Ŝ +  p/2

Where p is the output of , calculated via (<ref>), Ŝ is the output of , calculated via (<ref>).



  §.§.§ Objective Function

A new ranking loss L_joint is constructed as (<ref>).

    L_joint = log(1+e^γ_3 (Ŝ^-_M-MTL-Ŝ^+_M-MTL+β_3 ))

Where γ_3 ∈ [1,10], β_3 ∈ [0,1] are hyper-parameters chosen via grid search. We usually use 10, 0.6 as their default values.

As we already have ranking loss in (<ref>), we remove the ranking loss part in (<ref>), which simplifies it as 

    L'_LMT = -log(p^+)


The final loss L is a sum of these three (<ref>).

    L_M-MTL = L_SMT + L'_LMT + L_joint










§ EXPERIMENTS


We evaluate the proposed  on three commonsense benchmarks:  <cit.>, PDP60[<https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/PDPChallenge2016.xml>] and WNLI. 
WNLI is derived from WSC, and is considered the most challenging NLU task in the GLUE benchmark <cit.>.





 §.§ Datasets



Table <ref> summarizes the datasets which are used in our experiments. 

Since the  and PDP60 datasets do not contain any training instances, following <cit.>, we adopt the WSCR dataset <cit.> for model training and selection. 

WSCR contains 1886 instances (1322 for training and the rest as dev set). 
Each instance is presented using the same structure as that in WSC. 




For the WNLI instances, we convert them to the format of  as illustrated in Table <ref>: we first detect pronouns in the premise using spaCy[<https://spacy.io>]; then given the detected pronoun, we search its left of the premise in hypothesis to find the longest common substring (LCS) ignoring character case. Similarly, we search its right part to the LCS; by comparing the indexes of the extracted LSCs, we extract the candidate. A detailed example of the conversion process is provided in Table <ref>.










 §.§ Implementation Detail


Our implementation of  is based on the PyTorch implementation of BERT[<https://github.com/huggingface/pytorch-pretrained-BERT>]. All the models are trained with hyper-parameters depicted as follows unless stated otherwise.
The shared layer is initialized by the BERT uncased large model. Adam <cit.> is used as our optimizer with a learning rate of 1e-5 and a batch size of 32 or 16. The learning rate is linearly decayed during training with 100 warm up steps. We select models based on the dev set by greedily searching epochs between 8 and 10. The trainable parameters, e.g., 𝐖_1 and 𝐖_2, are initialized by a truncated normal distribution with a mean of 0 and a standard deviation of 0.01.
The margin hyperparameter, β in Eqn. <ref>, is set to 0.6 for MLM and 0.5 for SSM, and γ is set to 10 for all tasks.  
We also apply SWA <cit.> to improve the generalization of models. All the texts are tokenized using WordPieces, and are chopped to spans containing 512 tokens at most.






 §.§ Results



We compare our  with a list of state-of-the-art models in the literature, including BERT <cit.>, GPT-2 <cit.> and DSSM <cit.>. The brief description of each baseline is introduced as follows.

    
  * BERTLARGE-LM <cit.>: This is the large BERT model, and we use MLM to predict a score for each candidate following Eq <ref>.
    
  * GPT-2 <cit.>: During prediction, We first replace the pronoun in a given sentence with its candidates one by one.  We use the GPT-2 model to compute a score for each new sentence after the replacement, and select the candidate with the highest score as the final prediction. 
   
  * BERTWiki-WSCR and BERTWSCR <cit.>: These two models use the same approach as BERTLARGE-LM, but are trained with different additional training data. For example, BERTWiki-WSCR is firstly fine-tuned on the constructed Wikipedia data and then on WSCR. 
   BERTWSCR is directly fine-tuned on WSCR.
  
  * DSSM <cit.>: It is the unsupervised semantic matching model trained on the dataset generated with heuristic rules.
  
  * : It is the proposed hybrid neural network model.
















































































The main results are reported in Table <ref>. Compared with all the baselines,  obtains much better performance across three benchmarks. 
This clearly demonstrates the advantage of the  over existing models. 
For example,  outperforms the previous state-of-the-art BERT_Wiki-WSCR model with a 11.7% absolute improvement (83.6% vs 71.9%) on WNLI and a 2.8% absolute improvement (75.1% vs 72.2%) on  in terms of accuracy.
Meanwhile, it achieves a 11.7% absolute improvement over the previous state-of-the-art BERT_LARGE-LM model on PDP60 in accuracy. 
Note that both BERT_Wiki-WSCR and BERT_LARGE-LM are using language model-based approaches to solve the pronoun resolution problem. 
On the other hand, We observe that DSSM without pre-training is comparable to BERT_LARGE-LM which is pre-trained on the large scale text corpus (63.0% vs 62.0% on WSC and 75.0% vs 78.3% on PDP60). 
Our results show that , combining the strengths of both DSSM and BERT_WSCR, has consistently achieved new state-of-the-art results on all three tasks.
 
 

To further boost the WNLI accuracy on the GLUE benchmark leaderboard, we record the model prediction at each epoch, and then produce the final prediction based on the majority voting from the last six model predictions. 
We refer to the ensemble of six models as _ensemble in Table <ref>. _ensemble brings a 5.4% absolute improvement (89.0% vs 83.6%) on WNLI in terms of accuracy. 









 §.§ Ablation study





In this section, we study the importance of each component in  by answering following questions: 

How important are the two component models: MLM and SSM?

To answer this question, we first remove each component model, either SSM or MLM, and then report the performance impact of these component models. 
Table <ref> summarizes the experimental results. It is expected that the removal of either component model results in a significant performance drop. 
For example, with the removal of SSM, the performance of  is downgraded from 77.1% to 74.5% on WNLI. Similarly, with the removal of MLM,  only obtains 75.1%, which amounts to a 2% drop. All these observations clearly demonstrate that SSM and MLM are complementary to each other and the  model benefits from the combination of both. 

Figure <ref> gives several examples showing how SSM and MLM complement each other on WNLI. We see that in the first pair of examples, MLM correctly predicts the label while SSM does not. This is due to the fact that “the roof repaired” appears more frequently  than “the tree repaired” in the text corpora used for model pre-training. 
However, in the second pair, since both “the demonstrators” and “the city councilment” could advocate violence and neither occurs significantly more often than the other,  SSM is more effective in distinguishing the difference based on their context. 
The proposed HNN, which combines the strengths of these two models, can obtain the correct results in both cases.





















Does the additional ranking loss help?


As in Eqn. <ref>, the training objective of  model contains three losses. The first two are based on the two component models, respectively, and the third one, as defined in Eqn. <ref>, is a ranking loss based on the score function in Eqn. <ref>. At first glance, the ranking loss seems redundant. 
Thus, we compare two versions of HNN trained with and without the ranking loss.


Experimental results are shown in Table <ref>. We see that without the ranking loss, the performance of  drops on three datasets: WNLI, WSCR and WSC. On the PDP60 dataset, without the ranking loss, the model performs slightly better. However, since the test set of PDP60 includes only 60 samples, the difference is not statistically significant. 
Thus, we decide to always include the ranking loss in the training objective of HNN.





Is the WNLI task a ranking or classification task?










The WNLI task can be formulated as either a ranking task or a classification task. To study the difference in problem formulation, we conduct experiments to compare the performance of a model used as a classifier or a ranker. For example, given a trained HNN, when it is used as a classifier we set a threshold to decide label (0/1) for each input. When it is used as a ranker, we simply pick the top-ranked candidate as the correct answer.
We run the comparison using all three models HNN, MLM and SSM. 
As shown in Figure <ref>, the ranking formulation is consistently better than the classification formulation for this task. 
For example, the difference in the HNN model is about absolute 2.5% (74.6% vs 77.1%) in terms of accuracy. 































































§ CONCLUSION


We propose a hybrid neural network (HNN) model for commonsense reasoning.
HNN consists of two component models, a masked language model and a deep semantic similarity model, which share a BERT-based contextual encoder but use different model-specific input and output layers.

HNN obtains new state-of-the-art results on three classic commonsense reasoning tasks, pushing the WNLI benchmark to 89%, the  benchmark to 75.1%, and the PDP60 benchmark to 90.0%. We also justify the design of HNN via a series of ablation experiments.

In future work, we plan to extend HNN to more sophisticated reasoning tasks, especially those where large-scale language models like BERT and GPT do not perform well, as discussed in <cit.>. 






§ ACKNOWLEDGMENTS

We would like to thank Michael Patterson from Microsoft for his help on the paper. 







acl_natbib




