User-generated content such as the questions on community question answering (CQA) forums does not always come with appropriate headlines, in contrast to the news articles used in various headline generation tasks. In such cases, we cannot use paired supervised data, e.g., pairs of articles and headlines, to learn a headline generation model. To overcome this problem, we propose an extractive headline generation method based on learning to rank for CQA that extracts the most informative substring from each question as its headline. Experimental results show that our method outperforms several baselines, including a prefix-based method, which is widely used in real services.
Introduction Community question answering (CQA) is a service where users can post their questions and answer the questions from other users. The quality of a CQA service is inevitably linked to how many questions are answered in a short amount of time. To this end, the headline of a posted question plays a key role, as it is the first thing users see in a question list or push notification on a smartphone. However, questions in a CQA service do not always have appropriate headlines because the questions are written by various users who basically do not have any specialized knowledge in terms of writing such content, in contrast to news articles written by professional editors. In fact, the biggest CQA service in Japan, Yahoo! Chiebukuro 1 , does not even provide an input field for headlines in the submission form of questions, as general users do not have enough patience and tend not to post questions if even just one required field is added to the form. This service alternatively uses the prefix of a question as its headline as in Figure 1(a) , where the headline " (Nice to meet you. Thank you in advance. At work ...)" is created from the prefix of the content. Obviously, this headline is uninformative because  of the lack of actual content, which is related to an initial unrequited love of a woman in the workplace. Figure 1(b) shows how ineffective such an uninformative headline is for a smartphone push notification, where users would have practically zero motivation to click on it and answer since they cannot imagine what kind of question is being asked. This negative effect has been confirmed on a commercial CQA service (Section 2). In this work, we take an extractive approach for improving uninformative headlines. Although there have recently been many studies on abstractive headline generation (as described in Section 6), we do not follow any of these approaches because the content we deal with has no correct headlines and also the abstractive methods can yield erroneous output. This latter issue is important in practical terms because correct output is critical for a commercial service. If by some chance an erroneous headline, or one including politically incorrect phrases, is sent to all users, the service's credibility can be lost instantly. Therefore, we formalize our task as an extraction problem of a fixed-length substring from a question as its headline. In this setting, we can assume that outputs never include such errors caused by the service, as the outputs are substrings of user-posted questions. Note that they have no coherence errors in selecting multiple sentences as in normal extractive summarization tasks. While it is true that these outputs might contain inappropriate expressions authored by users, this type of error is beyond our scope since it is a different inevitable problem. Furthermore, the situation of "the service generated an inappropriate expression by itself" is significantly worse than the situation of "a user posted an inappropriate question to the service, and the service displayed it". Therefore, it is difficult to directly use abstractive methods for commercial services from a business standpoint. Our approach involves preparing headline candidates and ranking them. The formal description is as follows. Let q be a target question to be translated into a headline. We first prepare a set S(q) of headline candidates from the question q. Note that the set S(q) is restricted to a set of fixed-length substrings given a length n, i.e., S(q) ⊆ {x | x ⪯ q, |x| = n}, where x ⪯ q means that x is a substring of q. Then we extract the best headline that maximizes a score function f q (x), which represents the "headline-ness" score of a candidate x ∈ S(q) with respect to the target question q, as follows: argmax x∈S(q) f q (x). (1) To ensure simplicity of implementation and understandability from users, we use a set of the fixed-length prefixes of all sentences in a question q as the candidate set S(q) (Section 3). Because the problem is to select the best candidate from among several targets, the score function f q (x) is naturally trained by learning to rank (Section 4). The main contributions of this paper are as follows. • We report empirical evidence of the negative effect of uninformative headlines on a commercial CQA service (Section 2). We additionally show that our task can reduce uninformative headlines by using simple dictionary matching, which dramatically improves the average answer rate of questions by as much as 2.4 times. • We propose an extractive headline generation method based on learning to rank for CQA (Section 4) that extracts the most informative substring (prefix of mid-sentence) from each question as its headline. To the best of our knowledge, our work is the first attempt to address such a task from a practical standpoint, although there have been many related studies (Section 6). Experimental results show that our method outperforms several baselines, including the dictionary-based method (Section 5). • We create a dataset for our headline generation task (Section 3), where headline candidates extracted from questions are ranked by crowdsourcing with respect to "headline-ness", that is, whether or not each headline candidate is appropriate for the headline of the corresponding question. Negative Effect of Uninformative Headlines We conducted A/B testing on the push notifications of smartphones in collaboration with Yahoo! Chiebukuro, as shown in Figure 1 (b). We first prepared a dictionary of typical first sentences that cause uninformative headlines. This dictionary was manually selected from frequent first sentences and consists of 913 sentences including greetings such as " (Good morning)", " Changed uninformative Unchanged uninformative Informative 0.75% 0.31% 0.45% Table 1 : Average answer rates of three question groups in A/B testing. (Good afternoon)", and " (Nice to meet you)", and fixed phrases such as " (Can I ask you something)", " (Please tell me)", and " (Thank you in advance)". We assumed that a question with a prefix match in the dictionary has an uninformative prefix headline and classified such questions into an uninformative group. For convenience, we also classified the other questions into an informative group, although they might include not so informative headlines. We further randomly divided the uninformative group into two equal groups: changed and unchanged. In the changed uninformative group, each headline is extracted as the prefix of the first (informative) sentence that does not match with the dictionary, which is the same as DictDel explained in Section 5.2. The unchanged group remains in uninformative. For comparison of these groups, we used the average answer rate over notified questions in each group as an evaluation measure, defined as Average answer rate = No. of questions answered from the notification No. of notified questions . (2) Note that we use a percentage expression (%) for easy reading. Table 1 shows the evaluation results of the A/B testing during a 1-month period (Feb. 2 -Mar. 4, 2018) , where about three million questions were sent to users. Comparing the unchanged uninformative group with the informative group, we can see that the average answer rate of the uninformative questions, 0.31%, is actually lower than that of the informative questions, 0.45%. Comparing the changed and unchanged uninformative groups, the average answer rate of the changed questions, 0.75%, is much higher than that of the unchanged questions, 0.31%. This means that even a simple dictionary-based method can dramatically improve the quality of the uninformative headlines, i.e., by as much as 2.4 times. We confirmed that the difference is statistically significant on a one-tailed Wilcoxon signed-rank test (p < 0.05). Note that the average answer rate represents a conversion rate (or rate of target actions), which is more important than a click-through rate (or rate of initial actions). The average answer rate is one of the most important indicators for a CQA service, while the click-through rate can be meaninglessly high if the service sends headlines that are fake or too catchy. We should point out that low answer rates are sufficient for the service, since it has 44M users: i.e., each question has an average of 2.4 answers, as the service currently has 189M questions and 462M answers. Dataset Creation We created a dataset for our headline generation task based on the Yahoo! Chiebukuro dataset 2 , which is a dataset including questions and answers provided from a Japanese CQA service, Yahoo! Chiebukuro. We first prepared headline candidates from this dataset as in Section 3.1 and then conducted a crowdsourcing task specified in Section 3.2. In Section 3.3, we report the results of the crowdsourcing task. Preparation of Headline Candidates We extracted only questions from the Chiebukuro dataset and split each question into sentences by using punctuation marks (i.e., the exclamation (" "), question (" "), and full stop (" ") marks). We regarded 20 Japanese characters that are basically extracted from each sentence as a headline candidate x ∈ S(q) in Eq. (1), since this setting is used for push notifications in the actual service in Figure 1 (b). More specifically, the headline candidate is created as follows: 1. If the sentence is the first one in the question, we extract the first 19 characters and put an ellipsis mark (" ") at the end. (a) Example of our crowdsourcing task. Posted Question: Nice to meet you, I am a man in my 30s. Please give me your advice on a pressing concern I have. A dog kept in the next house barks from morning to night. Neighbors have given the owner cautions against it, but there is no improvement. This area has only private houses, not rented houses, so I cannot move out. However, I will go crazy if I have to keep enduring this. How can I effectively manage this problem? ... Please give me your advice on a pressing concern ... (b) English translation of left example. Figure 2 : Examples of (a) our crowdsourcing task and (b) its English translation. In the case where the length of a candidate is less than 20 characters, we include some of the next sentence in order to maximize use of display space. We included questions with more than five sentences for the purpose of efficiently collecting ranking information. All told, we prepared 10,000 questions containing more than five headline candidates each. Figure 2 shows an example of our crowdsourcing task (a) and its English translation (b). This task involves a posted question and headline candidates corresponding to the question. We asked workers to select the best candidate from options after reading the posted question. A relative evaluation, where workers select the best candidate, was used instead of an absolute evaluation, where workers select a score from 0 to 10 for each candidate, because we wanted to obtain as accurate a headline as possible, and it might be difficult for workers to select an appropriate absolute score. The workers were instructed as follows (English translation): Crowdsourcing Task Various candidate headlines are listed as options for a posted question in a Q&A service. After reading the question, please select the best option from the list so that users can guess the content of the question and distinguish it from other ones. Please remove uninformative ones such as greetings, self-introductions, and unspecific expressions. We explain how to judge the appropriateness of each candidate by means of the example in Figure 2 . After examining the posted question, we can assume that the most important content is "he is annoyed by the barking of a dog kept in the next house". On the basis of this assumption, option 6 is the best one, since the headline "A dog kept in the next house barks from morning" is enough to help answerers deduce that "the questioner is annoyed by the sound". Option 1 is inappropriate because although the answerers might be able to guess that "the questioner cannot move out", this matter is not the central one. Option 2 is uninformative because it consists merely of greetings and self-introduction, and option 3, while a question sentence, is unspecific. Option 4 enables answerers to guess that this question is related to an issue involving pets, but they cannot grasp the specific content. Option 5 specifies a likely damage due to the trouble, but the reason (trouble) is more important for the answering than the result (damage). Option 7 directly shows that "the questioner is annoyed and wants some advice", but answerers cannot understand why he/she is annoyed. The detailed implementation of our task is as follows. First we randomly sorted the candidates of each question (shown in Figure 2 (a)) to avoid position bias by the workers. We included ten actual questions and a dummy question so that workers would always have to answer one dummy question per every ten actual questions. A dummy question is a question with a clear answer inserted to eliminate fraud workers (i.e., workers who randomly select answers without actually reading them). Each question was answered by ten workers so each headline candidate had a vote score from 0 to 10 representing whether or not the candidate was appropriate for a headline. This task took nine days and was answered by 1,558 workers. As a result, our dataset consists of 10,000 questions, each of which has more than five headline candidates with accompanying "headline-ness" scores. Analysis of Crowdsourcing Results We analyzed our dataset to determine how much room for improvement our task has compared to the prefix headline. It is well known that the first sentence can be a strong baseline in many summarization tasks, so the prefix headline is also expected to perform well. Figure 3 shows the statistical information based on sentence position that includes the (a) ratio of the true best (most voted) candidates, (b) average votes, and (c) average rank (in order of votes) over the candidates at each sentence position. Looking at the true best candidates (a), the ratio 61.8% for the 1st sentence clarifies the effectiveness of the prefix headline, as expected. Conversely, we still have room for improvement for the prefix headline up to 38.2%. Our goal is to improve the uninformative headlines of 38.2% while keeping the remaining 61.8% unchanged. The other two figures (b) and (c) also support the above discussion. Furthermore, we qualitatively checked the crowdsourcing quality. Workers successfully eliminated uninformative candidates including greetings and self-introductions, while one or two workers sometimes chose ones that included a fixed phrase such as "Please tell me ...". This is probably because workers had different criteria regarding the "unspecific expressions" described in the instructions. Since we cannot enumerate all of the concrete bad examples, we ignore this phenomenon with the expectation that a learning algorithm will reduce its negative effect. Proposed Method In this section, we explain how to construct a headline generation model from the dataset presented in Section 3. We took a ranking approach, i.e., learning to rank, for our task, rather than a simple regression one, since estimating absolute scores is not required for our purpose. Even if two headline candidates (of different questions) have the same expression, their votes can be significantly different since the votes in our dataset are based on relative evaluation. For example, the best candidate for Figure 2 was No. 6, but it might not be selected in other questions such as "A dog kept in the next house barks from morning. Does anybody know why dogs generally want to bark?". Learning to rank is an application of machine learning that is typically used for ranking models in information retrieval systems. The ranking models are basically learned from a supervised dataset consisting of triples (q, x, y), where q is a user's query, x is a document corresponding to q, and y is a relevance score of x with respect to q. In this work, we formalize our task by regarding q, x, and y, as a posted question, a headline candidate, and a voted score in our dataset, respectively. We used a pairwise ranking method that is also implemented as an instance of the well-known SVM tools LIBLINEAR  and LIBSVM (Kuo et al., 2014) . We used a linear model based on LIBLINEAR, an L2-regularized L2-loss linear rankSVM, for the experiments. Let D be a dataset that consists of triples including a posted question q, a headline candidate x, and a voted score y, i.e., (q, x, y) ∈ D. In the pairwise ranking method, we train a ranking model as a binary classifier that determines whether the condition y i > y j is true or false for two candidates x i and x j in the same question (q i = q j ). Specifically, we first define the index pairs of positive examples by P = {(i, j) | q i = q j , y i > y j , (x i , y i , q i ) ∈ D, (x j , y j , q j ) ∈ D}. Note that we do not need to consider the negative examples N = {(j, i) | (i, j) ∈ P } since they yield the same formula as P in the optimization process. The training of the pairwise ranking method is achieved by solving the following optimization problem using the set P of the index pairs: min w 1 2 w ⊤ w + C ∑ (i,j)∈P ℓ(w ⊤x i − w ⊤x j ), (3) where w is a weight vector to be learned,x i is a feature vector extracted from a headline candidate x, and C is the regularization parameter. The function ℓ is a squared hinge loss, which is defined as ℓ(d) = max(0, 1 − d) 2 . Finally, we define the score function in Eq. (1) as f q (x) = w ⊤x , wherex can be created by using q as well as x. This score means the relative "headline-ness" of x. Experiments Basic Settings The basic settings of the experiments are as follows. We split our dataset into training and test sets consisting of 9,000 and 1,000 examples, respectively. We used an implementation 3 based on LIBLIN-EAR for training our ranking model, i.e., a linear L2-regularized L2-loss rankSVM model , as described in Section 4. The regularization parameter was optimized by cross validation and set as C = 0.125. The feature vector for a headline candidate consists of three kinds of features: bag-of-words, embedding, and position information. The bag-of-words feature is a sparse vector of 30,820 dimensions based on the tf-idf scores of nouns, verbs, interjections, conjunctions, adverbs, and adjectives in a candidate, where we used a Japanese morphological analyzer, MeCab 4 (Kudo et al., 2004) , with a neologism dictionary, NEologd 5 (Toshinori Sato and Okumura, 2017). The embedding feature is a dense vector of 100 dimensions based on a doc2vec model (Le and Mikolov, 2014) trained with all 3M sentences in the Chiebukuro dataset using the Gensim tool 6 . The position feature is a binary vector of ten dimensions, where each dimension represents the coarse position (or coverage) of a headline candidate for a question. Specifically, we equally split a question (character sequence) into ten parts and set one to each dimension if and only if the corresponding part overlaps a candidate. For example, candidate No. 2 in Figure 2 had a position feature (1, 1, 0, · · · , 0), since the candidate covers the first 2/10 of the whole question. Similarly, No. 6 and No. 3 had (0, 0, 1, 1, 0, · · · , 0) and (0, · · · , 0, 1), respectively. For constructing the feature vector of each headline candidate, we used the previous and next candidates in sentence order, in addition to the target candidate. This is based on the idea that near candidates might have useful information for the target candidate. Finally, we prepared each feature vector by concatenating nine feature vectors, i.e., the above three kinds of features for three candidates, and normalizing them. Compared Methods We compared our method, MLRank, with the baselines listed below. Prefix, DictDel, and Random are simple baselines, while Prefix and DictDel are practically strong. ImpTfidf, SimTfidf, SimEmb, and LexRank are unsupervised baselines, and SVM and SVR are supervised ones. • Prefix: Selects the first candidate in sentence order. • DictDel: Selects the first (informative) candidate that does not match in the dictionary of uninformative headlines (Section 2). • Random: Randomly selects a candidate. • ImpTfidf: Selects the most important candidate with the highest tf-idf value, where a tf-idf value is calculated by the sum of the elements in a bag-of-words feature (described in Section 5.1). • SimTfidf: Selects the most similar candidate to the original question, which is calculated by the cosine similarity between the bag-of-words features (in Section 5.1) of each candidate and the question. • SimEmb: An embedding-based variation of SimTfidf with embedding features (in Section 5.1). • LexRank: Selects the candidate with the highest score based on LexRank 7 (Erkan and Radev, 2004) , which is a widely used unsupervised extractive summarization method based on the PageRank algorithm. The graph expression of each question was constructed on the basis of cosine similarity of the tf-idf vectors corresponding to candidates. • SVM: Selects the candidate with the highest confidence based on a model learned as a classification task, where candidates with nonzero votes were labeled as positive. This setting was the best in our preliminary experiments. We used the L2-regularized L2-loss support vector classification model (C = 0.0156) in LIBLINEAR. The other settings were the same as those described in Section 5.1. • SVR: Selects the candidate with the highest predicted votes based on a model learned as a regression task, where the target variable is the number of votes. We used the L2-regularized L2-loss support vector regression model (C = 0.0625) in LIBLINEAR. The other settings were the same as above. • MLRank: Proposed method described in Section 4. Evaluation Measures We defined three evaluation measures for evaluating each method on our headline generation task. Change Rate from Prefix Headline We measured how much each method changed the default prefix headline in order to determine the effect of application to an actual CQA service. We defined this measure as change rate from the prefix headline, as follows: Change rate = No. of questions where the best candidate is not the prefix headline No. of all questions . Clearly, the change rate of the default method that selects the prefix headline is 0%. If the value is small, the effect on the service will be small, but if the value is higher than the ideal change rate of 38.2% (Section 3), there can be side effects even if the average result is good. A higher change rate up to the ideal rate is desirable from a practical standpoint. Winning Rate against Prefix Headline We measured how much each method won against the prefix headline to directly assess the quality of changed headlines. We defined this measure as winning rate against the prefix headline, as follows: Winning rate = No. of questions where the best candidate got more votes than the prefix headline No. of questions where the best candidate is not the prefix headline . (5) We did not consider the first candidate (in sentence order) selected by each method, which is the same as the prefix headline, since they obviously have the same number of votes. Average Votes We measured how appropriate the candidates selected by each method are in order to determine the overall performance. We defined this measure as average votes, as follows: Average votes = Sum of votes for the best candidates for all questions No. of questions . Note that the average votes score is different from the average votes score for position (Figure 3(b) ) in that the former is the average over the selected candidates while the latter is the average over the candidates at a sentence position. This measure is related to (normalized) discounted cumulative gain (DCG), which is widely used as an evaluation measure of ranking models. We often use DCG@k for evaluating top-k rankings, and the above definition acutally corresponds to DCG@1. According to a well-known paper (Järvelin and Kekäläinen, 2002) in the information retrieval field, DCG is appropriate for graded-relevance judgments like our task, while precision (described below) is appropriate for binary-relevance ones. Average votes is expected to be more appropriate than precision for our task because we want "averagely better headlines than default ones" rather than "best ones" from a practical standpoint. Precision Precision is a widely used evaluation measure for classification tasks, and we added it to support an evaluation based on average votes. We defined it with respect to the best candidate, i.e., precision@1, as follows: Precision = No. of questions where the best candidate had the maximum votes No. of questions . (7) Table 2 shows examples of headlines generated by the prefix method, Prefix, and the proposed method, MLRank. Looking at the first example, we can see that our method successfully eliminated the selfintroduction phrase ("I am a 27-year-old woman"). The headline (right) of our method allows answerers to know that the questioner is discouraged about how to encounter men from the phrase "little chance of new encounters with men", while the headline (left) of the prefix method lacks this important clue. The second and third examples show similar effects to the first example. In the second one, although there are few clues about what the question is with the prefix method, our method correctly included the important clue ("honeymoon"). In the third one, our method appropriately eliminated the uninformative long phrase ("I am sorry if the category is wrong"), which is not a frequent fixed phrase. The fourth example shows a slightly challenging case, where both headlines make it difficult to understand the question. However, the headline of our method included the term "winning bidder", so at least the answerer can assume that the question is about some sort of auction trouble. The fifth example is a clearly successful result, where our method extracted the main question point about "welfare pension" as a headline. These results qualitatively demonstrate the effectiveness of our method. Results Qualitative Analysis Quantitative Analysis We compared our method MLRank with the baselines in Section 5.2 on the headline generation task for our dataset in Section 3. Table 3 shows the evaluation results based on the change rates, winning rates, average votes, and precision. Looking at the average votes and precision, which represent the overall performances, our method MLRank clearly performed the best among all methods. We confirmed that Table 3 : Evaluation results of our headline generation task for proposed method MLRank and baselines. the relative improvement of the average votes of our method MLRank against every baseline including the prefix method Prefix is statistically significant on the basis of a one-tailed Wilcoxon signed-rank test (p < 0.01). The change and winning rates of our method are 9.9% and 94.9%, respectively. This means that our method detected 9.9% of the uninformative headlines and improved them with the high accuracy of 94.9%. In other words, our method could successfully improve the overall performance while simultaneously avoiding any negative side effects. The ideal results (Ref) based on correct labels suggest that our method still has room for improvement, especially for the change rate. The results of the other baselines are as follows. Not surprisingly, the prefix method Prefix performed well. This is consistent with the fact that the first sentence can be a good summary in many summarization tasks. The random method Random performed the worst, also as expected. The dictionarybased deletion method DictDel was relatively useful, although the change rate was small. The reason the winning rate of DictDel is relatively low compared with MLRank is that there are some cases where a combination of uninformative expressions can yield likely clues. For example, the self-introduction "I am a newbie of this forum" itself is basically uninformative for a question, but a combination with additional information such as "I am a newbie of this forum. Where can I change the password ..." can be more informative than only the additional information "... Where can I change the password since I forgot it after ..." because the combination specifies the target site by the expression "this forum". The unsupervised methods, SimTfidf, SimTfidf, SimEmb, and LexRank, which are widely used for summarization tasks, performed worse than the prefix method Prefix. Although the change rates are higher than our method, the winning rates are lower. In other words, they yielded many bad headlines. These results suggest that supervised learning specialized to the target task would be required. Comparing the important sentence extraction method ImpTfidf and the similarity-based summarization method SimTfidf, we found that SimTfidf performed better. This implies that the content information of each question is useful for our headline generation task, as is the case with other summarization tasks. The similarity-based method SimEmb with embeddings performed worse than our expectation. The reason seems to be that it was difficult to obtain meaningful document embeddings from long questions. The graph-based method LexRank had a similar performance to SimTfidf, because LexRank tends to select a candidate similar to the question when only one candidate was selected. The supervised methods, SVM and SVR, performed relatively well compared to the unsupervised methods, but they did not outperform the strong simple baselines, Prefix and DictDel, nor our method MLRank. These results support the appropriateness of our approach. Related Work In this section, we briefly explain several related studies from two aspects: headline generation task and CQA data. As discussed below, our work is the first attempt to address an extractive headline generation task for a CQA service based on learning to rank the substrings of a question. After Rush et al. (2015) proposed a neural headline generation model, there have been many studies on the same headline generation task (Takase et al., 2016; Chopra et al., 2016; Kiyono et al., 2017; Ayana et al., 2017; Raffel et al., 2017) . However, all of them are abstractive methods that can yield erroneous output, and the training for them requires a lot of paired data, i.e., news articles and headlines. There have also been several classical studies based on nonneural approaches to headline generation (Woodsend et al., 2010; Alfonseca et al., 2013; Colmenares et al., 2015) , but they basically addressed sentence compression after extracting important linguistic units such as phrases. In other words, their methods can still yield erroneous output, although they would be more controllable than neural models. One exception is the work of Alotaiby (2011) , where fixed-sized substrings were considered for headline generation. Although that approach is similar to ours, Alotaiby only considered an unsupervised method based on similarity to the original text (almost the same as SimTfidf in Section 5.2), in contrast to our proposal based on learning to rank. This implies that Alotaiby's method will also not perform well for our task, as shown in Section 5.4. There have been several studies on extractive summarization (Kobayashi et al., 2015; Yogatama et al., 2015) based on sentence embeddings, but they were basically developed for extracting multiple sentences, which means that these methods are almost the same as SimEmb in Section 5.2 for our purpose, i.e., extraction of the best candidate. This also implies that they will not be suitable for our task. Furthermore, recent sophisticated neural models for extractive summarization (Cheng and Lapata, 2016; Nallapati et al., 2017) basically require large-scale paired data (e.g., article-headline) to automatically label candidates, as manual annotation is very costly. However, such paired data do not always exist for real applications, as in our task described in Section 1. There have been many studies using CQA data, but most of them are different from our task, i.e., dealing with answering questions (Surdeanu et al., 2008; Celikyilmaz et al., 2009; Bhaskar, 2013; Nakov et al., 2017) , retrieving similar questions (Lei et al., 2016; Romeo et al., 2016; Nakov et al., 2017) , and generating questions (Heilman and Smith, 2010) . Tamura et al. (2005) focused on extracting a core sentence and identifying the question type as classification tasks for answering multiple-sentence questions. Although their method is useful to retrieve important information, we cannot directly use it since our task requires shorter expressions for headlines than sentences. In addition, they used a support vector machine as a classifier, which is almost the same as SVM in Section 5.2, and it is not expected to be suitable for our task, as shown in Section 5.4. The work of Ishigaki et al. (2017) is the most related one in that they summarized lengthy questions by using both abstractive and extractive approaches. Their work is promising because our task is regarded as the construction of short summaries, but the training of their models requires a lot of paired data consisting of questions and their headlines, which means that their method cannot be used to our task. Conclusion We proposed an extractive headline generation method based on learning to rank for CQA that extracts the most informative substring in each question as its headline. We created a dataset for our task, where headline candidates in each question are ranked using crowdsourcing. Our method outperformed several baselines, including a prefix-based method, which is widely used for cases where the display area is limited, such as the push notifications on smartphones. The dataset created for our headline generation task will be made publicly available 8 . Although our task is basically designed for extractive summarization, this dataset can also be used for abstractive summarization as a side information for training abstractive models. In future work, we will investigate how effectively our method can perform in practical situations, e.g., push notifications. In addition, we will consider how to improve the change rate of our method while keeping its winning rate and how to create a useful dataset even if removing the length limitation.
1 2 3 3.1 3.2 3.3 4 5.1 5.2 5.3 6 7
Introduction Negative Effect of Uninformative Headlines Dataset Creation Preparation of Headline Candidates Crowdsourcing Task Analysis of Crowdsourcing Results Proposed Method Basic Settings Compared Methods Evaluation Measures Related Work Conclusion
Crowdsourcing Results
Ratio of questions whose prefix headlines were most voted
Room for improvement for prefix headline was up to
Improve uninformative headlines of 38.2%
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.
Ratio of questions whose prefix headlines were most voted
Room for improvement for prefix headline was up to
Improve uninformative headlines of 38.2%
Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved.