# Best-Answer Prediction in Q&A Sites Using User Information

Rafik Hadfi ID 111Department of Social Informatics, Kyoto University, Japan
222Correspondence: Rafik Hadfi (rafik.hadfi@i.kyoto-u.ac.jp). Tel.:
+81-075-753-4821 Ahmed Moustafa333Department of Computer Science, Nagoya
Institute of Technology, Japan Kai Yoshino††footnotemark:  Takayuki Ito ID
††footnotemark:

###### Abstract

Community Question Answering (CQA) sites have spread and multiplied
significantly in recent years. Sites like Reddit, Quora, and Stack Exchange
are becoming popular amongst people interested in finding answers to diverse
questions. One practical way of finding such answers is automatically
predicting the best candidate given existing answers and comments. Many
studies were conducted on answer prediction in CQA but with limited focus on
using the background information of the questionnaires. We address this
limitation using a novel method for predicting the best answers using the
questioner’s background information and other features, such as the textual
content or the relationships with other participants. Our answer
classification model was trained using the Stack Exchange dataset and
validated using the Area Under the Curve (AUC) metric. The experimental
results show that the proposed method complements previous methods by pointing
out the importance of the relationships between users, particularly throughout
the level of involvement in different communities on Stack Exchange.
Furthermore, we point out that there is little overlap between user-relation
information and the information represented by the shallow text features and
the meta-features, such as time differences.

Keywords. Community Q&A, Natural Language Processing, Machine Learning,
Feature Engineering, Cross-platform Prediction, Topic modelling, Stack
exchange

##  1 Introduction

Community Question Answering (CQA) sites play an integral part in our daily
lives. People are extensively using sites like Quora or Stack Exchange to find
answers to their questions and to share their opinions. As a result, CQA sites
have attracted a lot of attention from researchers in social sciences,
information retrieval, and machine learning.

An important aspect in CQA platforms is the quality of the content and its
relevance to any potential users. Such platforms have several functions to
assess the quality of the answers, such as voting, or by using the reputation
of its users. Such functions motivate experts to share their knowledge and to
answer questions in exchange for more visibility and fame [1]. In this
context, the distinguishing function of modern CQA platforms is their ability
to select the best answer. Such a function enables questioners to mark one
answer as the best reply to their particular question.

While manually choosing the best answer is sufficient to reach a conclusion,
there are many questions for which it is difficult to distinguish the best
answer [2]. For instance, a large number of conflicting answers renders the
task of finding the best candidate challenging. In this case, the automatic
selection of the best answer can reduce the burden on the questioner and
stimulate the community. It is therefore worthwhile to automatically search
for the best answer, and it has in fact attracted a lot of attention [2, 1].

Despite its practicality, the automatic prediction of the best answer faces a
number of challenges. The main challenges are the lack of clear answers to
many questions and the large number of participants that might be involved in
the discussion. Most of the research in the question answering area does not
fully acknowledge those key aspects. For instance, in open-domain question
answering, the answers usually exist in open datasets and domains such as
Wikipedia. Other similar systems have directly or indirectly indefinite
answers in their data or knowledge models [3, 4, 5].

Conceptually, CQA systems adopt a simple model where the discussion is centred
around a question that is posted by a user and a number of answers provided by
the community members [2]. In addition, there is often no clear answer or
conclusion to the question, and the questioner has to elect one after seeing
all the answers of the users. In this sense, the questioner is a key important
factor in determining the best answer regardless of the objective evaluation
of the answer.

A common problem with the existing studies on the prediction of the best
answer is the assumption that the questioners population is uniform. It is
highly unlikely that the questioner shares the same belief and knowledge with
the candidate answerers. For instance, the current models are general and
assume that the majority of the questioners are set as a reference to the
questioners. Such models do not sufficiently handle the background information
about the questioner.

The main goal of our work is to address the problem of best answer
identification and prediction by combining linguistic, user information, and
relationship features. To this end, we set two goals. The first is to increase
the performance of automatic best-answer selection based on the proposed set
of features. We believe that with the help of AI, future online Q&A platforms
should be able to automatically predict the best answers to questions that
lack clear answers. The second goal is the identification of the factors or
features that lead to best answers. This goal will contribute to the area of
feature engineering in Natural Language Processing (NLP) domains [6].

In the following, we treat the best answer prediction as a binary
classification problem and propose a method that integrates the features used
in previous research as well as the background information of the questioner.
Our contributions are twofold. Firstly, we focus on the individual features of
the questioner as well as the relationships among all the questioners/users.
Secondly, we focus on the importance of each questioner/user participation in
the community.

The rest of the paper is structured as follows. In the next section, we survey
the existing literature on CQA. In Section 3, we present the methods. Section
4 presents our experiments and results. Section 5 discusses the results.
Finally, we conclude and highlight the future directions.

##  2 Related Work

Community Question Answering (CQA) platforms aim at providing high-quality
answers to online users. Incomplete or unreliable answers are often returned
and the users are usually forced to browse multiple answers in search for the
best one. Existing automatic prediction methodologies attempt to find the best
answers by using multiple criteria and techniques. Such methodologies are
usually based on binary prediction or on ranking the candidate answers. In the
following, we cover some of the important methodologies such as ranking, deep
learning, expert recommendation, and finally, the user-centred approach that
we are adopting in our work.

Machine learning ranking (MLR) has been extensively used in building models to
rank web pages, to retrieve online information, and to predict the best
answers in CQA systems. MLR methods distinguish between point-wise, pair-wise,
and list-wise ranking approaches. The point-wise approach operates on single
documents by scoring them according to classification or regression functions
obtained from training data. The document scores are then taken as the
standard of documents ranking [7]. The pair-wise approach considers the
context of the documents and takes the comparison results as the standard to
rank the answers [8, 9]. Finally, the list-wise approach does not convert the
ranking problem into classification or regression instances, but optimises the
ranking results according to predefined evaluation criteria [10]. The authors
in [11], for instance, proposed a pair-wise method that ranks the answers
based on numerous historical documents. They later applied preference
relationships using a deep learning framework. The results showed that the
proposed scheme outperformed several state-of-the-art baselines in answer
ranking tasks.

When using machine learning, CQA answer ranking benefits from the recent
progress in deep learning methods. For instance, reference [12] combined
bidirectional long short term memory (bi-LSTM) with CNN to construct Q&A text
embedding. On the other hand, reference [13] proposed an AP-BILSTM model to
perform feature weighting of the answers and questions and improve the model
performance. Reference [14] proposed a novel attention mechanism named
Dynamic-Clip Attention that filters the noise in the attention matrices and
mines the semantic relevance of word-level vectors. The authors in reference
[15, 16, 17] proposed the IWAN, CARNN, and MCAN models respectively by
combining attention mechanism and LSTM, where CARNN was an improvement of
IWAN. A comprehensive review on deep learning methods applied to answer
selection could be found in [18].

Other approaches find the best questions by exploiting the expertise of the
users in the given topic. For instance, [19] extracted the expertise from
topics learnt from the content and the tags of the questions and the answers.
In practice, the topics of the questions are often general whereas the
question tags are specific and therefore more informative. Our method exploits
this type of information by focusing on the user expertise.

Other methodologies focus on the features of the users. For instance,
reference [2] proposed a method that integrates user-related features such as
the reputation captured from features such as answer ratings or age. For
instance, the method in [20] used 72 features, including the textual content
of the question and the history of the questioner and answerer (number of
questions they resolved and the time since their last registration) in order
to predict the questioner satisfaction. These methods focus primarily on the
answerer’s information, and less on the questioner. The authors in [2] did not
use the information of the questioner, and the ratio of the feature amount
between answerer and questioner used by [20] was 21:4, which indicates a clear
bias.

Other than individually focusing on the users, it is possible to look at their
social interactions since they are involved in the same thread of discussion.
In this context, reference [21] proposed a framework that encodes social
interaction cues in the community in addition to the content of question-
answer in order to improve the CQA tasks. This framework used information in
heterogeneous social networks. Reference [22] developed an asymmetric ranking
network learning method with Deep Recurrent Neural Networks by using a
heterogeneous asymmetric CQA network in order to consider the authority of the
answerers of the question. This heterogeneous asymmetric CQA network is
composed by embedding questions, answers, and the users who posted the
answers. Despite the usage of the network of features between users, the
information about the questioner is neglected.

##  3 Methods

The proposed method is an extension of [23] with the addition of new features
that lead to significant improvements in the prediction accuracy. The details
of the proposed method are explained as follows.

###  3.1 Adopted Discussion Model

We start from the hierarchical discussion model used in CQA and illustrated in
Figure 1.

ptQuestionptAnswer AptAnswer BptComment 1ptComment 2ptAnswer CptComment 3
Figure 1: Question Answering Model

The model starts from the question located at the root node of the tree. Below
the question, answers are provided by the users (Answers A, B, and C). Users
could also comment below each answer by posting messages (Comments 1, 2, and
3). The question is formulated by the author who could expect people to have
divergent points of view. Answers respond to the question and only one
question could be chosen as the final solution. Note that comment 1 is
expected to be directly related to answer B. Questioning is supposed to end
once the best answer is chosen by the questioner. The proposed model is a
simplification of the issue-based information system (IBIS) [24] in the sense
that both models contain a sequence that starts with a problem followed by a
hierarchy of responses and comments.

In this discussion model, the choice of the answer depends mainly on the
questioner’s choice. In this sense, the author of the question will not
necessarily choose the correct answer. Such an answer will later be upvoted or
downvoted by the community, which could indicate the absolute correctness of
the answer. In our proposed method, incorporating the user features accounts
for such discrepancies through the use the histories of votes combined with
the acceptance rates. As for the used datasets, questions and answers are
usually curated to account for the wrong answers and avoid the inconsistencies
that might undermine the predictors.

###  3.2 Relevant Features

The proposed method combines five types of features. We will use these
features to train a classification model to predict the best answer to a
question. The features of interest are described below.

  * •

Content features are composed of shallow and textual features. These features
are based on the assumption that the quality of the answer sentence and the
degree of association between the question and the answer are important
factors to be selected as the best answer.

  * •

User features are composed of the answerer, questioner, and user-relation
features. These user features reflect the intrinsic quality of the users by
capturing either their personal history or their interactions with other
members of the community. The user features are based on the assumption that
different people will conceive their answer differently, which affects the
selection of the best answer. Note that the answerer features affect the
quality of the users that answered, the questioner features affect the quality
of the users who questioned, and the user-relation network reflects the
interactions between the users and the effect on the final quality of the
answers.

In the next section, we cover these two main features in detail.

###  3.3 Content Features

####  3.3.1 Shallow Features

Shallow features (S) [25] do not employ semantic or syntactic parsing such as
sentence length [26] or word length [27]. They are proven to be effective in
assessing properties such as the ease of reading, or the usefulness [28]. To
the best of our knowledge, the most complete approach that uses shallow
features is found in [23]. The work combines the linguistic and meta features
that have been emphasized in previous research. In addition, the authors
assign ranks after computing numerical values in order to indicate the
relative importance. The shallow features follow this adopted approach. The
prediction features are listed bellow.

  * Meta features

    * –

Age

    * –

Rating score

  * Linguistic features

    * –

Length

    * –

Word count

    * –

Number of sentences

    * –

Longest sentence in characters

    * –

Average words per sentence

    * –

Average characters per word

    * –

Contains hyperlinks or not

    * –

Answer count per thread

  * Vocabulary features

    * –

Normalised Log Likelihood

    * –

Flesch–Kincaid grade

Note that for each feature, there exists a ranked version [23]. In addition to
the linguistic features and meta-features, reference [23] employed two
features relevant to the vocabulary used in the questions and answers and are
useful in estimating the readability of an answer. These features are the
Normalised Log Likelihood and the Flesch–Kincaid grade.

The Normalised Log Likelihood ($LL_{n}$) is a probabilistic approach that
measures to what extent the lexicon in an answer is distant from the
vocabulary used in the whole forum community [23, 25, 2, 29]. In the
following, we define $LL_{n}$ as in (1).

$\displaystyle
LL_{n}(s)=\frac{1}{U_{s}}\sum_{w_{s}}C(w_{s})\log(P(w_{s}|Voc))$ (1)

Given a sentence $s$ in an answer, $P(w_{s}|Voc)$ is the probability of
occurrence of the word $w_{s}$ according to the background corpus $Voc$, and
$C(w_{s})$ is the number of times the word $w_{s}$ occurs in $s$. The
normalisation factor $U_{s}$ is the number of unique words occurring in $s$.
While the authors in [23] argued that $LL_{n}$ is insignificant, we took the
hypothesis that the corpus used in their experiment was too large to be
applied individually to each answer. To improve $LL_{n}$, we propose to use a
smaller corpus that is well suited for each answer. While $LL_{n}$ uses the
vocabulary of the whole forum community as a corpus, we propose to use the
vocabulary used in the question thread as a corpus. In the case where we allow
multiple threads for one single question, all the content of the threads will
be compacted into one corpora where messages are temporally ordered below the
root node, the question.

The Flesch–Kincaid grade ($F-K$) is a readability metric introduced by [28]
and is defined as in (2).

$\displaystyle\begin{split}F-K_{p}(awps_{p},asps_{p})=0.39\times
awps_{p}+11.8\times asps_{p}-15.59\end{split}$ (2)

For any given post $p$, the average number of words per sentence ($awps_{p}$)
and the average number of syllables per word ($asps_{p}$) are calculated and
added together. The ranked versions of the rating score and age show the
highest importance, indicating that it is important to answer relatively good
answers as quickly as possible.

To further represent the shallow textual information, we have also employed
Tag features extracted from the HyperText Markup Language (HTML) tags. HTML
tags are used to visually highlight the question and answers. In the proposed
features, the number of specific HTML tags that appear in the answers are
calculated and used as features. We distinguish the three following features.

  * •

Quote represents the number of text elements between a “quote tag” in an
answer. The “quote tag” is used to represent quotation from other sources or
questions.

  * •

Contains represents the number of quotations originating from the question and
used in an answer. A quotation candidate is first selected from the text
between “quote tag” and then checked whether it is contained in a question or
not.

  * •

Strong represents the number of text elements between “strong tag” in an
answer text. The “strong tag” is used to emphasise the text by showing it in
bold.

Assuming that readability may affect the questioner incentive to choose the
best answer, we decided to use the above-mentioned 3 features.

####  3.3.2 Textual Features

The textual features (T) aim at estimating the quality and the associations
between questions and answers. They are extensively used in [30, 31] and can
be studied using several analytical tools. To analyse such features, we rely
on Latent Dirichlet Allocation (LDA) [32] to extract the proportions of topics
contained in a document from an input text. These proportions are initially
described using probability distributions on questions and answers. Then,
information metrics are applied to those distributions to estimate the
relevance between question and answer. LDA was trained on the dataset
described in Section 4.1 and tuned with coherence [33]. We then employed the
model with the highest coherence. We particularly calculated 3 types of
probabilistic metrics and 2 types of vectorial metrics between question and
answer. Table 1 describes the features used in the LDA method.

Table 1: Latent Dirichlet allocation measures Feature | Description
---|---
$D_{KL}(Q||A)$ | KLD from question to answer
$D_{KL}(A||Q)$ | KLD from answer to question
$JSD(Q||A)$ | JSD between question and answer
$R^{2}(Q,A)$ |  $R^{2}$-score between question and answer
$Cos(Q,A)$ | Cosine similarity between question and answer

The measures $D_{KL}(Q||A)$ and $D_{KL}(A||Q)$ are the Kullback-Leibler
divergences between question and answer. We treat $D_{KL}(Q||A)$ as the new
information from the answer about the question and $D_{KL}(A||Q)$ as the
information difference from the answer to the question. We also used Jensen-
Shannon divergence (JSD), $R^{2}$-score, and Cosine similarity, to represent
topic similarity between question and answer.

###  3.4 User Features

The features of the users are the most important features in CQA in addition
to the textual features [2, 20]. Next, we will detail these features and how
they contribute to better predictions.

####  3.4.1 Answerer Feature

The answerer features (AF) are the most emphasised user features. The question
and answer history as well as the public profile descriptions are usually
available in all major CQA platforms.

Table 2 shows the user features that we used. These features were obtained
from the data collected using the official API of Stack Exchange [34]. Here,
reputation is a rough measurement of how much the CQA community trusts the
user. reputation can be earned by posting interesting questions and quality
answers. If one user has a reputation of 200 or more, at least on one of Stack
Exchange sites, the user will start with 101 reputation as a bonus when he
logs into a new site. The features, bronze, silver, and gold, represent the
badges obtained by gaining reputation through questions and answers. Badges
are difficult to get in the order of gold, silver, and bronze. In addition,
the counts down_vote_count and up_vote_count represent the numbers of upvotes
and downvotes that the user got from other users. These counts could either
raise or lower the reputation of the user.

Table 2: User Features Feature | Description
---|---
reputation | User’s reputation
bronze | Number of bronze badges
silver | Number of silver badges
gold | Number of gold badges
q_count, a_count | Posted questions and answers
up_vote_count | Number of upvotes
down_vote_count | Number of downvotes
view_count | Number of views from other users
accept_rate | Rate of selection as the best answer

####  3.4.2 Questioner feature

We propose to use the questioner features (Q) for the first time in the
context of best answer prediction. Q includes the information of the
questioner. The questioner is in fact the most important person that can
assess if a conclusion is valuable or not. Here, Q relies on features similar
to the answerer features but originating from the questioner.

####  3.4.3 Difference Feature

In addition to the A and Q features, we have also used the margins between
user features to measure user similarity. The difference is obtained by
subtracting the numerical value of the A feature from each numerical value of
Q features. This feature is applied only when both A and Q features are used.

####  3.4.4 User-Relation Feature

The user-relation (UR) feature was proposed to represent the relationships
between users. These graphical features are illustrated on Figure 2 and
represent the types of edges when the connections between users are
represented as a network. These features are extracted from questions,
answers, and comments in the discussion model described in Section 3.1.

QuestionerOther
UsersAnswerer$aqSendEdge$$qaSendEdge$$qUserSendEdge$$qUserGetEdge$$aUserSendEdge$$aUserGetEdge$
Figure 2: Network of User Relationship Features

###  3.5 Percent Rank Features

Another feature we looked at is the percent rank features (PR), and which
illustrates feature scaling. In machine learning, feature scaling is useful
when the range of values in the feature set is variable. Here, we employed
normalisation as a kind of feature scaling. The rank is divided by the number
of answers so that the biggest rank value becomes 1 and the smallest rank
value approaches 0. The PR method can be applied to all features that possess
ranks. In this work, we apply PR to all the features explained in Section 3.

##  4 Experimental Results

Since the goal of our study is to identify the combination of features that
could link the threads, meta, linguistic, vocabulary, textual, relationship,
and user information all at once, we focused on a dataset that was previously
used for a similar task but without looking at the relationships between users
[2, 25, 35]. Additional evaluations of our metrics with respect to other
datasets are required in order to generalize our findings.

###  4.1 The Stack Exchange Dataset

The dataset of our study was collected using the official API of Stack
Exchange [34] and is described in Table 3. Stack Exchange is a group of CQA
websites on diverse topics with each site covering a particular theme. Sites
in Stack Exchange use a common system in which the best answer is selected.
The existence of specific themes as well as the best-answer selection
functionality are important criteria for this study. When the themes are
separated, we assumed that the users are prompted to participate only on the
sites of their themes of interests. In this case, the best-answer selection
functionality is necessary to obtain an undisputed conclusion. In sum, the
Stack Exchange dataset has the following advantages.

  1. 1.

The answer rate in all sites is 100%.

  2. 2.

The average number of answers per question in all sites is around 2.2.

  3. 3.

The ease of participation whereby people do not need a technical background or
a deep knowledge on the topic.

Table 3: Stack Exchange datasets and Q&A distributions Site | Question | Answer | Positive/Negative Ratio
---|---|---|---
workplace | 11254 | 40722 | $\sim 1:4$
english | 34624 | 109337 | $\sim 1:3$

The positive/negative ratio in Table 3 shows that all sites are imbalanced and
there are many answers that are not chosen as the best answer. The dataset of
each site is composed of questions and answers, including one best-answer for
each question. If the dataset presents low answer rates, it should be curated
to account for other external features such as the information about the
author. This consideration is important when the proposed approach is extended
to other CQA platforms.

###  4.2 The Experimental Setting

For the experiment, we prepared and carried out classifications with the four
following types of classifiers.

  1. 1.

Support Vector Machines (SVMs), consist in a set of supervised learning
approaches used for classification, regression, and outliers detection.

  2. 2.

Random Forests (RFs), can be used for tasks such as regression and
classification by relying on sets of small decision trees with their own local
estimators.

  3. 3.

Multivariate Adaptive Regression Splines (MARS), are algorithms for complex
non-linear regression problems.

  4. 4.

Light Gradient Boosting Machine (LGB), is a gradient boosting framework that
relies on tree-based learning algorithms.

The choice of these four classifiers is founded on their proven ability in
binary classification problems and particularly when treating major CQA
features [23].

Herein, the classifiers were evaluated using a five-fold cross-validation
approach with the input dataset being divided into five disjoint partitions.
In our case, five folds allowed us to apply the greedy selection procedure in
a realistic time. From the five partitions, four training partitions were
divided and used to adjust the classifier hyper-parameters, and one partition
was used as a test set. To analyse the predictive power of the features, we
trained the model on several subsets of features with a greedy selection
procedure [36]. We started by separately testing each feature and picking the
performant one, then, we maintained that feature while combining it with all
the remaining features to select the best combination. The process is repeated
until all features are included. The greedy strategy allows us to find a
locally optimal choice at each stage with the hope of finding a global optimum
in a reasonable time.

We finally compared the Area Under the Curve (AUC) performance values of the
classifiers when using the proposed methods and when using the method of [23]
as a baseline. We employed AUC because of its independence from thresholds.

###  4.3 Results

Table 4 shows the average AUC values for the different classifiers and
combinations of features. Letters refer to answerer features (A), questioner
features (Q), shallow features (S), textual features (T), percent rank
features (PR), and user-relation features (UR). The best combinations of
features have the highest AUC value and are shown in bold.

Table 4: Average AUC for feature groups.

[htbp]  Feature Group SVM RF MARS LGB S [23] 0.876 0.873 0.898 0.945 S+PR
0.866 0.869 0.917 0.945 T+PR 0.710 0.713 0.794 0.824 A+PR 0.708 0.729 0.811
0.857 Q+PR 0.500 0.530 0.614 0.628 UR+PR 0.664 0.683 0.774 0.822 S +T+PR 0.870
0.874 0.920 0.948 S +A+PR 0.873 0.873 0.921 0.948 S +Q+PR 0.872 0.874 0.921
0.948 S +UR+PR 0.873 0.875 0.922 0.950 S +UR+T+PR 0.870 0.875 0.919 0.950 S
+UR+A+PR 0.872 0.872 0.923 0.951 S +UR+Q+PR 0.873 0.874 0.922 0.951 S
+UR+A+Q+PR 0.873 0.860 0.923 0.951 S +UR+A+T+PR 0.870 0.867 0.924 0.951 All
0.870 0.852 0.924 0.952

The most predictive feature is the feature group S. This group contains 22
features that mainly capture the shallow and meta features. These features are
very important factors in predicting the quality of the answer. On the other
hand, Q features alone are expectedly the worst performing group. The nature
of the questioner should influence the answer and does not contribute
substantially without the answer information. Similar remark could be made for
the UR group.

The performance of the S features has not improved except for MARS when
compared to the baseline, which confirms that Normalised Log Likelihood is not
important as pointed out and the effect of scaling is small. Interesting
results emerged when combining features in pairs. That is, the performance
improved mostly by combining the S features with the UR features that had
previously shown the second worst performance. In addition, the A features and
the Q features have a large performance difference when used alone. However,
there is no significant performance difference when combined with the S
features.

These results show that the S features, the A features, and the Q features
have overlapping information while the UR features have less. The combinations
of three or more features made it clear that the A features have the greatest
effect on the improvement, albeit with a slight difference, and the effects
are smaller in the order of T and Q. Finally, when all the features were used
with LGB, the AUC performance reached a maximum of 95.2%. We used a t-test to
verify the difference in AUC values between the case where all features were
used in LGB and the baseline. It was confirmed that the difference is
significant. For instance, the obtained AUC scores have outperformed the
averaged AUC performances of two related baselines. The first one is that of
[23] which focused on thread, meta, linguistic and vocabulary feature ranking
applied to Stack Overflow datasets. The second baseline [25, 2] looked also at
the same set features as [23] but used ten-fold cross-validation in addition
to a cross-site training.

We summarise our findings below.

  * •

Shallow Features are the ones with higher predictive potential compared to
other features. In particular, the time lag between the answer and the
question and the rating of the answer were important, and they showed higher
importance than the features proposed in this paper.

  * •

Among the proposed features, the User-Relation Features, particularly the
number of posts by the questioner and the answerer, showed improved
performance over other features when combined with the Shallow Features.

  * •

There was a large difference in performance when used alone, but when combined
with other features, Textual Features, Answerer Features, and Questioner
Features showed a small difference in performance improvement, which is
smaller compared to User-Relation Features.

##  5 Discussion

###  5.1 The Proposed Features

Since the LGB showed higher AUC performance than the other classifiers, it
could be considered as the most suitable for our prediction problem. Besides
the greedy combination of features, to discover which single feature
valuations give the best prediction, we compared the average weight of each
feature value of the higher-order LGB and its type when all features were
used. This comparison of average weights is illustrated in Figure 3.

Figure 3: Important Features of LGB

The rating score and age of the S features show high weights as it was also
found in [23]. This evidently indicates the importance of the evaluation of
other users and the speed of response. In addition, the proposed features show
the weights that follow the baseline features. We consider that these features
play a supplementary role in dealing with cases that cannot be explained by
the features of the baseline.

In the following, we describe the consideration for the main values of each
feature.

####  5.1.1 Textual and Answerer Feature

The textual feature that had the highest importance was given by $R^{2}(Q,A)$.
Here, $R^{2}(Q,A)$ was used to calculate topic similarity between question and
answer. The difference from $Cos(Q,A)$ is the use of average value. The
original purpose of using the average value in $R^{2}$-score was to measure
the relative residual between the predicted value and the measured value.
However, the proposed method uses the topic probability distribution for both
the predicted value and the measured value. Therefore, the average of
predicted value and the measured value is 1 divided by the number of topics.
In other words, the $R^{2}$ score of the proposed features calculates how
similar the topic distribution of the answer is to the topic distribution of
the question rather than the uniform distribution. It was shown that when
compared to a uniform distribution, such similarity is more important than the
absolute similarity such as with $Cos(Q,A)$. The features $up\\_vote\\_count$
and $down\\_vote\\_count$ showed high weights while $reputation$ did not show
higher comparative importance. From these results, it can be said that it is
more important for the answerer to consider good and bad evaluations
separately than comprehensive evaluations.

####  5.1.2 Questioner Features

Two Q features showed highest weights: The reputation, and the view count of
the questioner. The reputation of the questioner was mainly compared with the
value near to 101. As we described in Section 4.1, 101 is one of the initial
reputation values and represents inactive users. view_count was compared with
multiple values and no bias was observed. Both reputation and view_count are
values that increase mainly by the virtue of the questioner’s posting to the
community. That is, certain or more active participation of the questioner in
the community was found to be an important factor for the prediction of best
answers.

####  5.1.3 User-Relation (UR) Feature

From the UR features, qUserSendEdge, aUserSendEdge showed high weights. The
feature qUserSendEdge was compared to a relatively low numbers (at most 30),
and aUserSendEdge was compared to high numbers (at least 66 and at most 966).
The feature qUserSendEdge was compared to a relatively low number, suggesting
the existence of some participation of the questioner in the community. That
is, certain or more active participation of the questioner in the community
was valued with respect to user relationships too. Similarly, qUserSendEdge
was compared to high numbers, suggesting that it was checking the answerer’s
quality. Users with a lot of posts can be said to be actively participating in
the community and aUserSendEdge can be used to measure the degree of
participation independently from the other users when compared to features
reputation, up_vote_count, and down_vote_count.

##  6 Conclusions and Future Work

We explored new features for the prediction of best answers on CQA sites. We
started by defining a general discussion model for CQA. Then, we proposed the
features that rely on the network information of the questioner and answerer,
the background information of the questioner, the sentence features, and the
user information of the answerer. We evaluated the performance of the proposed
method on a Stack Exchange dataset with a focus on huge groups of CQA websites
on diverse topics. We used the AUC value as evaluation index to remove the
dependence on thresholds. The experimental results showed that adding the
proposed features is superior to the existing method and emphasises the
importance of the information about the questioners and answerers, especially
their participation in the community.

As a future direction, we will first look at the rating scores. Rating score
is an important feature but it is sensitive to change once the best answer is
selected. Furthermore, such a score depends highly on other users’ valuations.
There is therefore a difficulty when using the rating score in real time
predictions of best answers. Since the rating score relies on human
comprehension, we hypothesise that such a rating could be replaced by a vector
space semantic method. Another feature that is worth investigating is the use
of emotion tags in CQA and their predictive effect on the correct answers. To
account for emotions in CQA, we could either use the existing emotion tags, or
classify the emotions directly from the textual content [37]. Finally, we are
considering incorporating the proposed metrics in conversational agents that
could firstly assess the questions and answers in real time and then react to
the users through means of automated facilitation [38, 39].

## References

  * Mamykina et al. [2011] Mamykina, L.; Manoim, B.; Mittal, M.; Hripcsak, G.; Hartmann, B.  Design lessons from the fastest q&a site in the west.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2011; pp. 2857–2866.
  * Gkotsis et al. [2015] Gkotsis, G.; Liakata, M.; Pedrinaci, C.; Stepanyan, K.; Domingue, J.  ACQUA: automated community-based question answering through the discretisation of shallow linguistic features.  J. Web Sci. 2015, 1, 1–15.
  * Rajpurkar et al. [2016] Rajpurkar, P.; Zhang, J.; Lopyrev, K.; Liang, P.  Squad: 100,000+ questions for machine comprehension of text.  arXiv 2016, arXiv:1606.05250.
  * Cui et al. [2019] Cui, W.; Xiao, Y.; Wang, H.; Song, Y.; Hwang, S.w.; Wang, W.  KBQA: learning question answering over QA corpora and knowledge bases.  arXiv 2019, arXiv:1903.02419.
  * Zhou et al. [2020] Zhou, M.; Shi, Z.; Huang, M.; Zhu, X.  Knowledge-Aided Open-Domain Question Answering.  arXiv 2020, arXiv:2006.05244.
  * Zheng and Casari [2018] Zheng, A.; Casari, A.  Feature Engineering for Machine Learning: Principles and Techniques for Data Scientists; O’Reilly Media, Inc.: 2018.
  * Shah and Pomerantz [2010] Shah, C.; Pomerantz, J.  Evaluating and predicting answer quality in community QA.  In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2010; pp. 411–418.
  * Nie et al. [2017] Nie, L.; Wei, X.; Zhang, D.; Wang, X.; Gao, Z.; Yang, Y.  Data-driven answer selection in community QA systems.  IEEE Trans. Knowl. Data Eng. 2017, 29, 1186–1198.
  * Bian et al. [2008] Bian, J.; Liu, Y.; Agichtein, E.; Zha, H.  Finding the right facts in the crowd: factoid question answering over social media.  In Proceedings of the 17th International Conference on World Wide Web, 2008; pp. 467–476.
  * Xu and Li [2007] Xu, J.; Li, H.  Adarank: a boosting algorithm for information retrieval.  In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2007; pp. 391–398.
  * Chen et al. [2019] Chen, Q.; Wang, J.; Lan, X.; Zheng, N.  Preference Relationship-Based CrossCMN Scheme for Answer Ranking in Community QA.  In Proceedings of the 2019 IEEE International Conference on Data Mining (ICDM), Beijing, China, 8–11 November 2019; pp. 81–90.
  * Tan et al. [2015] Tan, M.; Santos, C.d.; Xiang, B.; Zhou, B.  Lstm-based deep learning models for non-factoid answer selection.  arXiv 2015, arXiv:1511.04108.
  * Santos et al. [2016] Santos, C.d.; Tan, M.; Xiang, B.; Zhou, B.  Attentive pooling networks.  arXiv 2016, arXiv:1602.03609.
  * Bian et al. [2017] Bian, W.; Li, S.; Yang, Z.; Chen, G.; Lin, Z.  A compare-aggregate model with dynamic-clip attention for answer selection.  In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, 2017; pp. 1987–1990.
  * Shen et al. [2017] Shen, G.; Yang, Y.; Deng, Z.H.  Inter-weighted alignment network for sentence pair modeling.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017; pp. 1179–1189.
  * Tran et al. [2018] Tran, Q.H.; Lai, T.; Haffari, G.; Zukerman, I.; Bui, T.; Bui, H.  The context-dependent additive recurrent neural net.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018; pp. 1274–1283.
  * Tay et al. [2018] Tay, Y.; Tuan, L.A.; Hui, S.C.  Multi-cast attention networks.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018; pp. 2299–2308.
  * Lai et al. [2018] Lai, T.; Bui, T.; Li, S.  A review on deep learning techniques applied to answer selection.  In Proceedings of the 27th International Conference on Computational Linguistics, 2018; pp. 2132–2144.
  * Yang and Manandhar [2014] Yang, B.; Manandhar, S.  Tag-based expert recommendation in community question answering.  In Proceedings of the 2014 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2014), Beijing, China, 17–20 August 2014; pp. 960–963.
  * Liu et al. [2008] Liu, Y.; Bian, J.; Agichtein, E.  Predicting information seeker satisfaction in community question answering.  In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2008; pp. 483–490.
  * Fang et al. [2016] Fang, H.; Wu, F.; Zhao, Z.; Duan, X.; Zhuang, Y.; Ester, M.  Community-based question answering via heterogeneous social network learning.  In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 2016.
  * Zhao et al. [2017] Zhao, Z.; Lu, H.; Zheng, V.W.; Cai, D.; He, X.; Zhuang, Y.  Community-based question answering via asymmetric multi-faceted ranking network learning.  In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017.
  * Calefato et al. [2019] Calefato, F.; Lanubile, F.; Novielli, N.  An empirical assessment of best-answer prediction models in technical Q&A sites.  Empir. Softw. Eng. 2019, 24, 854–901.
  * Kunz and Rittel [1970] Kunz, W.; Rittel, H.W.  Issues as Elements of Information Systems; Volume 131, Citeseer: University Park, PA, USA, 1970\.
  * Gkotsis et al. [2014] Gkotsis, G.; Stepanyan, K.; Pedrinaci, C.; Domingue, J.; Liakata, M.  It’s all in the content: state of the art best answer prediction based on discretisation of shallow linguistic features.  In Proceedings of the 2014 ACM Conference on Web Science, 2014; pp. 202–210.
  * Feng et al. [2010] Feng, L.; Jansche, M.; Huenerfauth, M.; Elhadad, N.  _A Comparison of Features for Automatic Readability Assessment_ ; 2010.
  * Piantadosi et al. [2011] Piantadosi, S.T.; Tily, H.; Gibson, E.  Word lengths are optimized for efficient communication.  Proc. Natl. Acad. Sci. USA 2011, 108, 3526–3529.
  * Kincaid et al. [1975] Kincaid, J.P.; Fishburne, R.P., Jr.; Rogers, R.L.; Chissom, B.S.  Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel, 1975.
  * Pitler and Nenkova [2008] Pitler, E.; Nenkova, A.  Revisiting Readability: A Unified Framework for Predicting Text Quality.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ ; Association for Computational Linguistics: Stroudsburg, PA, USA, 2008; pp. 186–195.
  * Zhou et al. [2015] Zhou, X.; Hu, B.; Chen, Q.; Tang, B.; Wang, X.  Answer sequence learning with neural networks for answer selection in community question answering.  arXiv 2015, arXiv:1506.06490.
  * Zhang et al. [2014] Zhang, W.N.; Liu, T.; Yang, Y.; Cao, L.; Zhang, Y.; Ji, R.  A topic clustering approach to finding similar questions from large question and answer archives.  PloS ONE 2014, 9, e71511.
  * Blei et al. [2003] Blei, D.M.; Ng, A.Y.; Jordan, M.I.  Latent dirichlet allocation.  J. Mach. Learn. Res. 2003, 3, 993–1022.
  * Mimno et al. [2011] Mimno, D.; Wallach, H.M.; Talley, E.; Leenders, M.; McCallum, A.  Optimizing semantic coherence in topic models.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ ; Association for Computational Linguistics: Stroudsburg, PA, USA, 2011; pp. 262–272.
  * ste [2019] Stack Exchange API.  2019\.
  * Cai and Chakravarthy [2011] Cai, Y.; Chakravarthy, S.  Predicting answer quality in q/a social networks: Using temporal features.  In Arlington: Department of Computer Science and Engineering, University of Texas at Arlingtong; 2011.
  * Molino et al. [2016] Molino, P.; Aiello, L.M.; Lops, P.  Social question answering: Textual, user, and network features for best answer prediction.  ACM Trans. Inf. Syst. (TOIS) 2016, 35, 1–40.
  * Ekman [1993] Ekman, P.  Facial expression and emotion.  Am. Psychol. 1993, 48, 384.
  * Hadfi et al. [2021] Hadfi, R.; Haqbeen, J.; Sahab, S.; Ito, T.  Argumentative Conversational Agents for Online Discussions.  J. Syst. Sci. Syst. Eng. 2021, pp. 1–15.
  * Ito et al. [2020] Ito, T.; Hadfi, R.; Haqbeen, J.; Suzuki, S.; Sakai, A.; Kawamura, N.; Yamaguchi, N.  Agent-Based Crowd Discussion Support System and Its Societal Experiments.  In _International Conference on Practical Applications of Agents and Multi-Agent Systems_ ; Springer: New York, NY, USA, 2020; pp. 430–433.