The Story Cloze Test (SCT) is a recent framework for evaluating story comprehension and script learning. There have been a variety of models tackling the SCT so far. Although the original goal behind the SCT was to require systems to perform deep language understanding and commonsense reasoning for successful narrative understanding, some recent models could perform significantly better than the initial baselines by leveraging human-authorship biases discovered in the SCT dataset. In order to shed some light on this issue, we have performed various data analysis and analyzed a variety of top performing models presented for this task. Given the statistics we have aggregated, we have designed a new crowdsourcing scheme that creates a new SCT dataset, which overcomes some of the biases. We benchmark a few models on the new dataset and show that the topperforming model on the original SCT dataset fails to keep up its performance. Our findings further signify the importance of benchmarking NLP systems on various evolving test sets.
Introduction Story comprehension has been one of the longestrunning ambitions in artificial intelligence (Dijk, 1980; Charniak, 1972) . One of the challenges in expanding the field had been the lack of a solid evaluation framework and datasets on which comprehension models can be trained and tested. Mostafazadeh et al. (2016) introduced the Story Cloze Test (SCT) evaluation framework to address * This work was performed at University of Rochester. this issue. This test evaluates a story comprehension system where the system is given a foursentence short story as the 'context' and two alternative endings and to the story, labeled 'right ending' and 'wrong ending.' Then, the system's task is to choose the right ending. In order to support this task, Mostafazadeh et al. also provide the ROC Stories dataset, which is a collection of crowd-sourced complete five sentence stories through Amazon Mechanical Turk (MTurk). Each story follows a character through a fairly simple series of events to a conclusion. Several shallow and neural models, including the state-of-the-art script learning approaches, were presented as baselines (Mostafazadeh et al., 2016) for tackling the task, where they show that all their models perform only slightly better than a random baseline suggesting that richer models are required for tackling this task. A variety of new systems were proposed (Mihaylov and Frank, 2017; Schenk and Chiarcos, 2017; Schwartz et al., 2017b; Roemmele et al., 2017) as a part of the first shared task on SCT at LSDSem'17 workshop (Mostafazadeh et al., 2017) . Surprisingly, one of the models made a staggering improvement of 15% to the accuracy, partially due to using stylistic features isolated in the ending choices (Schwartz et al., 2017b) , discarding the narrative context. Clearly, this success does not seem to reflect the intent of the original task, where the systems should leverage narrative understanding as opposed to the statistical biases in the data. In this paper, we study the effect of such biases between the ending choices and present a new scheme to reduce such stylistic artifacts. The contribution of this paper is threefold: (1) we provide an extensive analysis of the SCT dataset to shed some light on the ending data characteristics (Section 3) (2) we develop a new strong classifier for tackling the SCT that uses a variety Context Right Ending Wrong Ending Ramona was very unhappy in her job. She asked for a raise, but was denied. The refusal prompted her to aggressively comb the want ads. She found an interesting new possibility and set up an interview. She was offered the new job at a higher salary. Ramona had no reason to want to change jobs anymore. The teacher was walking with a stack of papers. Outside started to rain. When the teacher tried to walk down a few steps, she ended up falling. The papers flew out of her hands and landed on the ground. A passer-by helped her up and helped her collect the papers. The teacher got up and walked home leaving the papers behind. (Chaturvedi et al., 2017) fails to keep up the performance on our new dataset. We discuss the implications of this experiment to the greater research community in terms of data collection and benchmarking practices in Section 6. All the code and datasets for this paper will be released to the public. We hope that the availability of the new evaluation set can further support the continued research on story understanding. Related Work This paper mainly extends the work on creating the Story Cloze Test set (Mostafazadeh et al., 2016) , hereinafter SCT-v1.0. The SCT-v1.0 dataset was created as follows: full five-sentence stories from the ROC Stories corpus were sampled, then, the initial four sentences were shown to a set of MTurk 2 crowd workers who were prompted to author 'right' and 'wrong' endings. Mostafazadeh et al. (Mostafazadeh et al., 2016) give special care to make sure there were no boundary cases for 'right' and 'wrong' endings by implementing extra rounds of data filtering. The resulting SCT-v1.0 dataset had a validation (hereinafter, SCT-v1.0 Val) and a test set (SCT-v1.0 test), each with 1,871 cases. Furthermore, we conducted an extensive ngram analysis, using word tokens, characters, partof-speech, and token-POS (similar to Schwartz et al. (Schwartz et al., 2017b) ) as features. We see char-grams such as "sn't" and "not" appear more commonly in the 'wrong endings', suggesting heavy negation. In 'right endings', pronouns are used more frequently versus proper nouns used in 'wrong endings'. Artifacts such as 'pizza' are common in 'wrong endings,' which could suggest that for a given topic, the authors may replace an object in a right ending with a wrong one and quickly think up a common item such as pizza to create a 'wrong' one. An extensive analysis of these features, including the n-gram analysis, can be found in the supplementary material. Model Following the analysis above, we developed a Story Cloze model, hereinafter EndingReg, that only uses the ending features while disregarding the story context for choosing the right ending. We expanded each Story Cloze Test case's ending options into a set of two single sentences. Then, for each sentence, we created the following features: 1. Number of tokens 2. VADER composite sentiment score 3. Yngve complexity score 4. Token-POS n-grams 5. POS n-grams 6. Four length character-grams All n-gram features needed to appear at least five times throughout the dataset. The features were collected for each five-sentence story and then fed into a logistic regression classifier. As an initial experiment, we trained this model using the SCT-v1.0 validation set and tested on the SCT-v1.0 test set. An L2 regularization penalty was used to enforce a Gaussian prior on the feature-space, where a grid search was conducted for hyper-parameter tuning. This model achieves an accuracy of 71.5% on the SCT-v1.0 dataset which is on par with the highest score achieved by any model using only the endings. Table 3 shows the accuracies ob-tained by models using only those particular features. We achieve minimal but sometimes important classification using token count, VADER, and Yngve in combination alone, better classification using POS or char-grams alone, and best classification using n-grams alone. By combining all of them we achieve the overall best results. Table 3 : Classification results on SCT-v1.0 using each of the feature sets designated in the columns. Data Collection Based on the findings above, a new test set for the SCT was deemed necessary. The premise of predicting an ending to a short story, as opposed to predicting say a middle sentence, enables a more systematic evaluation where human can agree on the cases 100%. Hence, our goal was to come up with a data collection scheme that overcomes the data collection biases, while keeping the original evaluation format. As the data analysis revealed, the token count, sentiment, and the complexity are not as important features for classification as the ending n-grams are. We set the following goals for sourcing the new 'right' and 'wrong' endings. They both should: 1. Contain a similar number of tokens 2. Have similar distributions of token n-grams and char-grams 3. Occur as standalone events with the same likelihood to occur, with topical, sentimental, or emotion consistencies when applicable. First, we crowdsourced 5,000 new five-sentence stories through Amazon Mechanical Turk. We prompted the users in the same manner described in Mostafazadeh et al. (2016) . In order to source new 'wrong' endings, we tried two different methods. In Method #1, we kept the original ending sourcing format of Mostafazadeh et al., but imposed some further restrictions. This was done by taking the first four sentences of the newly collected stories and asking an MTurker to write a 'right' and 'wrong' ending for each. The new restrictions were: 'Each sentence should stay within the same subject area of the story,' and 'The number of words in the Right and Wrong sentences should not differ by more than 2 words,' and 'When possible, the Right and Wrong sentences should try to keep a similar tone/sentiment as one another.' The motivation behind this technique was to reduce the statistical differences by asking the user to be mindful of considerations. In Method #2, we took the five sentences stories and prompted a second set of MTurk workers to modify the fifth sentence in order to make a resulting five-sentence story non-sensible. Here, the prompt instructs the workers to make sure the new 'wrong ending' sentence makes sense standalone, that it does not differ in the number of words from the original sentence by more than three words, and that the changes cannot be as simple as e.g., putting the word 'not' in front of a description or a verb. As a result, the workers had much less flexibility for changing the underlying linguistic structures which can help tackle the authorship style differences between the 'right' and 'wrong' endings. The results in Table 4 , which show classification accuracy when using EndingReg on the two new data sources, show that Method #2 is a slightly better data sourcing scheme in reducing the bias, since the EndingReg model's performance is slightly worse. The set was further filtered through human verification similar to Mostafazadeh et al. (2016) . The filtering was done by splitting each SCT-v1.0's two alternative endings into two independent five-sentence stories and asking three different MTurk users to categorize the story as either: one where the story made complete sense, one where the story made sense until the last sentence and one where the story does not make sense for another reason. Stories were only selected if all the three MTurk users verified that the story with the 'right ending' and the corresponding story with the 'wrong ending' were verified to be indeed right and wrong respectively. This ensured a higher quality of data and eliminating boundary cases. This entire process resulted in creating the Story Cloze Test v1.5 (SCT-v1.5) dataset, consisting of 1,571 stories for each validation and test sets. Method #1 Method #2 EndingReg 0.709 0.695 cogcomp 0.649 0.641 Table 4 : Comparison of initial data sourcing methods n − gram char − gram P OS SCT-v1.0 13.9 12.4 16.4 SCT-v1.5 7.0 6.3 7.5 Table 5 : Standard deviation of the word and character n-gram counts, as well as the part of speech (POS) counts, between the right and wrong endings. Results In order to test the decrease in n-gram bias, which was the most salient feature for the classification task using only the endings, we compare the variance between the n-gram counts from SCT-v1.0 to SCT-v1.5. The results are presented in Table  5 , which indicates the drop in the standard deviations in our new dataset. Table 6 shows the classification results of various models on SCT-v1.5. The drop in accuracy of the EndingReg model between the SCT-v1.0 and SCT-v1.5 shows a significant improvement on the statistical weight of the stylistic features generated by the model. Since the main features used are the token length and the various n-grams, this suggests that the new 'right endings' and 'wrong endings' have much more similar token n-gram, pos n-gram, postoken n-gram and char-gram overlap. Furthermore, the CogComp model's performance has significantly dropped on SCT-v1.5. Although this model seems to be using story comprehension features such as event sequencing, since the endings are included in the sequences, the biases within the endings have influenced the predictions and the weak performance of the model in SCT-v1.5 suggest that this model had picked up on the biases of SCT-v1.0 as opposed to really understanding the context. In particular, the posterior probabilities for each ending choice using their features are quite similar on the SCT-v1.5. These results place the classification accuracy of this top performing model on par with or worse than the models that did not use the ending features of the old SCT-v1.0 dataset (Mostafazadeh et al., 2017) , which suggest that the gap that once was held by models using the ending biases seems to be corrected for. Al-   though we did not get to test all the other models published on SCT-v1.0 directly, we predict similar trends. It is important to point out that the 64.4% performance attained by our EndingReg model is still high for a model which completely discards the context. This indicates that although we could correct for some of the stylistic biases, there are some other hidden patterns in the new endings that would not have been accounted for without having the EndingReg baseline. This showcases the importance of maintaining benchmarks that evolve and improve over time, where systems should not be optimized for particular narrow test sets. We propose the community to report accuracies on both SCT-v1.0 and SCT-v1.5, both of which still have a huge gap between the best system and the human performance. Conclusion In this paper, we presented a comprehensive analysis of the stylistic features isolated in the endings of the original Story Cloze Test (SCT-v1.0). Using that analysis, along with a classifier we developed for testing new data collection schemes, we created a new SCT dataset, SCT-v1.5, which overcomes some of the biases. Based on the results presented in this paper, we believe that our SCT-v1.5 is a better benchmark for story comprehension. However, as shown in multiple AI tasks (Ettinger et al., 2017; Antol et al., 2015; Jabri et al., 2016; Poliak et al., 2018) , no collected dataset is entirely without its inherent biases and often the biases in datasets go undiscovered. We believe that evaluation benchmarks should evolve and improve over time and we are planning to incrementally update the Story Cloze Test benchmark. All the new versions, along with a leader-board showcasing the stateof-the-art results, will be tracked via CodaLab https://competitions.codalab.org/ competitions/15333. The success of our modified data collection method shows how extreme care must be given for sourcing new datasets. We suggest the next SCT challenges to be completely blind, where the participants cannot deliberately leverage any particular data biases. Along with this paper, we are releasing the datasets and the developed models to the community. All the announcements, new supplementary material, and datasets can be accessed through http://cs. rochester.edu/nlp/rocstories/. We hope that this work ignites further interest in the community for making progress on story understanding.
1 2 4 5 6 7
Introduction Related Work Model Data Collection Results Conclusion
Next Steps
We believe that evaluation benchmarks should evolve and improve over time and we are planning to incrementally update the Story Cloze Test benchmark.
Stay tuned for updates on the dataset and SOTA models via http:// cs.rochester.edu/nlp/rocstories/
We expect to release the final dataset, along with reporting the performance of the most recent SCT 1.0 SOTA models on the new dataset, shortly after
Allen, Omid B Tackling the Biasesakhshandeh,
We believe that evaluation benchmarks should evolve and improve over time and we are planning to incrementally update the Story Cloze Test benchmark.
Stay tuned for updates on the dataset and SOTA models via http:// cs.rochester.edu/nlp/rocstories/
We expect to release the final dataset, along with reporting the performance of the most recent SCT 1.0 SOTA models on the new dataset, shortly after
Allen, Omid B Tackling the Biasesakhshandeh,