TASK DEFINITION: In this task, you will be presented with a context from an academic paper and a question separated with a 
. You have to answer the question based on the context.
PROBLEM: To tackle this issue, we investigate how to improve the generalization of BERT-based MCQA models with the constraint of limited training data using four representative MCQA datasets: DREAM, MCTest, TOEFL, and SemEval-2018 Task 11. 
 Question: What four representative datasets are used for bechmark?

SOLUTION: DREAM, MCTest, TOEFL, and SemEval-2018 Task 11

PROBLEM: We carried out a reliability study for the proposed scheme using two pairs of expert annotators, P1 and P2.  
 Question: do they use a crowdsourcing platform?

SOLUTION: No

PROBLEM: As Garimella et al. BIBREF23 have made their code public , we reproduced their best method Randomwalk on our datasets and measured the AUC ROC, obtaining a score of 0.935. An interesting finding was that their method had a poor performance over their own datasets. This was due to the fact (already explained in Section SECREF4) that it was not possible to retrieve the complete discussions, moreover, in no case could we restore more than 50% of the tweets. So we decided to remove these discussions and measure again the AUC ROC of this method, obtaining a 0.99 value. Our hypothesis is that the performance of that method was seriously hurt by the incompleteness of the data. We also tested our method on these datasets, obtaining a 0.99 AUC ROC with Walktrap and 0.989 with Louvain clustering. 
 Question: What are the state of the art measures?

SOLUTION:
Randomwalk Walktrap Louvain clustering

