# Query-Variant Advertisement Text Generation
with Association Knowledge

Siyu Duan  Research Center for Digital Humanities& Department of Information
Management,Peking UniversityBeijingChina<EMAIL_ADDRESS>,  Wei Li  School
of Information Science,Beijing Language and Culture UniversityBeijingChina
<EMAIL_ADDRESS>,  Jing Cai  ,  Yancheng He  Platform and Content
Group,TencentShenzhenGuangzhouChina<EMAIL_ADDRESS>,  Yunfang Wu
and  Xu Sun  MOE Key Lab of Computational Linguistics,School of EECS,Peking
UniversityBeijingChina<EMAIL_ADDRESS>
(2021)

###### Abstract.

Online advertising is an important revenue source for many IT companies. In
the search advertising scenario, advertisement text that meets the need of the
search query would be more attractive to the user. However, the manual
creation of query-variant advertisement texts for massive items is expensive.
Traditional text generation methods tend to focus on the general searching
needs with high frequency while ignoring the diverse personalized searching
needs with low frequency. In this paper, we propose the query-variant
advertisement text generation task that aims to generate candidate
advertisement texts for different web search queries with various needs based
on queries and item keywords. To solve the problem of ignoring low-frequency
needs, we propose a dynamic association mechanism to expand the receptive
field based on external knowledge, which can obtain associated words to be
added to the input. These associated words can serve as bridges to transfer
the ability of the model from the familiar high-frequency words to the
unfamiliar low-frequency words. With association, the model can make use of
various personalized needs in queries and generate query-variant advertisement
texts. Both automatic and human evaluations show that our model can generate
more attractive advertisement text than baselines. 111Code and data set are
available at https://github.com/CissyDuan/Query_Variant_AD.

advertisement; text generation; knowledge; graph; search

††journalyear: 2021††copyright: acmlicensed††conference: Proceedings of the
30th ACM International Conference on Information and Knowledge Management;
November 1–5, 2021; Virtual Event, QLD, Australia††booktitle: Proceedings of
the 30th ACM International Conference on Information and Knowledge Management
(CIKM ’21), November 1–5, 2021, Virtual Event, QLD, Australia††price:
15.00††doi: 10.1145/3459637.3482290††isbn: 978-1-4503-8446-9/21/11††ccs:
Computing methodologies Natural language generation††ccs: Computing
methodologies Knowledge representation and reasoning

##  1\. Introduction

Figure 1. An example of the proposed query-variant advertisement text
generation task. User Queries are natural language words given by users with
various needs. Item Keywords are provided by ad sponsors, contains the core
information of an item that usually reflects general needs.

The revenue of many Internet companies relies on advertising. In a typical
online web search advertising system, each time a user inputs a search query
in natural language words, the advertising system would respond with a list of
advertisements of different items. However, one item involves different
aspects. One predefined general purposed advertisement text can not address
the focus of diverse personalized queries. If the advertising system can
provide advertisement texts that are closer to user needs considering the
search queries, the ads will be more attractive to the users.

An example of query variant advertisement text is given in Figure 1. In this
example, query 1 and query 2 pay attention to the camera and CPU performance
separately, which are different selling points for the same item. The general
purposed advertisement text fails to address these personalized needs for the
two queries, while the ideal query-variant advertisement texts would emphasize
the diverse needs that users are interested in. Although being a favourable
feature for the advertising system, the manual creation of all the
advertisement texts targeting different queries with different needs for
massive items is too expensive.

In this paper, we propose the query-variant advertisement text generation
task, which aims to automatically generate different advertisement texts
considering the needs of various queries. This task serves the query-variant
search ad system. An illustration of query-variant search ad system is shown
in Figure 2.

Figure 2. The process of search ad system. “K” is item keywords, “Q” is user
query. Query-variant search ad system would respond the user query with more
focused ad text for each item, which arouses a demand for the proposed query-
variant ad text generation.

Although query-variant advertisement text generation is a very practicable
task, it faces a serious challenge that traditional text generation models
tend to ignore the personalized needs with low frequency, while
overemphasizing general purpose needs with high frequency.

To improve the performance of the model on the low-frequency needs, we propose
a dynamic association method to expand the receptive field of the model based
on external knowledge. The mechanism of association is shown in Figure 3. The
proposed association module can explore and select the suitable associated
words based on both global structural information from external knowledge and
local semantic information within the query-item pair. These selected
associated words, which serve as the bridge between high-frequency keywords
and low-frequency keywords, will be added to the original input words. When
encountering low-frequency words, the associated words(the nodes with “A”
inside in Figure 3) shared with high-frequency words could be activated. The
ability of the model to deal with the familiar high-frequency words can be
passed to the encountered low-frequency words through these activated
associated words. With association, the model can make better use of the
personalized needs contained in a specific query.

For example, the cellphone brand “Apple” is much more popular than
“Blackberry”, and they share the associated word “cellphone”. When
encountering the low-frequency word “Blackberry”, the ability to deal with the
high-frequency word “Apple” can be transferred to “Blackberry” through
“cellphone”.

To test the effectiveness of our method, we collect over 2,140,000 queries
with the retrieved advertisement texts. We also build an association knowledge
word graph based on 17,000,000 advertisement texts. Extensive experiments and
human evaluations show that our model outperforms all the baselines, including
the retrieved general purposed human written advertisement text.

Figure 3. Association and Activation. The red circle“L” represents the low-
frequency words encountered, and the blue circles “H” represents the high-
frequency words that the model dealt with in the training. The orange circles
represent the associated words, which expands the receptive field of the
model. The orange circles with “A” represent the shared words that are
activated when being associated by our model. These associated words allow the
modeling ability to transfer from familiar high-frequency words to unfamiliar
low-frequency words. Figure 4. A sketch of the proposed model. Blue circles
are item keywords, red circles are query words, orange circles are the
associated words retrieved by the Association module from the Association
Knowledge Word Graph. The Generation module is designed to generate
advertisement text based on the extended sub-graph.

Our contributions are summarized as follows:

  * •

We apply query-variant advertisement text generation task to search
advertisement scenario. This task aims to generate different advertisement
texts that satisfy the needs for various queries.

  * •

We propose a query-variant advertisement text generation model with a novel
association method to transfer the modeling ability from familiar high-
frequency words to unfamiliar low-frequency words, so as to better deal with
the diverse and low-frequency personalized needs.

  * •

We conduct extensive experiments and analyses. Both automatic and human
evaluation results show that our model can generate attractive advertisement
texts for various queries.

Table 1. Statistics of the dataset. q/k indicates the number of user queries
and item keywords.

Data | Train | Dev ( q / k ) | Test ( q / k ) | Max Len | Vocab | AKWG-node | AKWG-edge
---|---|---|---|---|---|---|---
Number | 2,126,938 | 4,516 / 1,458 | 4,174 / 1,321 | 20 | 50,000 | 49,460 | 869,371

##  2\. Task and Data

In this section, we introduce our proposed task and the collection of
experimental data.

###  2.1. Task

Before introducing our task formulation, we first describe the definitions of
some phrases in this paper.
Query and Item Keywords: Queries are the natural language search content of
users in the website. Item keywords are isolated words containing the core
information of the item provided by advertisement sponsors.
General and Personalized Needs: In online search advertising, for multiple
user queries matching the same item, we treat the overlap words of different
queries as general needs and the others as personalized needs. General needs
are often included in item keywords, while personalized needs are distributed
in different search queries.
High-Frequency and Low-Frequency Words: We calculated the average Document
Frequency and vocabulary size of general needs words and personalized needs
words in the test set with multiple queries. The average Document Frequency of
general needs words is 1.88 times that of personalized needs words, while the
vocabulary size of personalized needs words is 1.52 times that of general
needs words. It reveals that general needs words are high-frequency and
centralized words, while personalized needs words are low-frequency and
diverse words. Concretely, in the example shown in Figure 1, “phone, new,
sale” rank at 44, 149, 761 respectively out of 50,000 words, while “camera,
CPU” rank at 2070, 4921 respectively. The general needs (phone, new, sale)
appear much more often than the personalized needs (camera, CPU).
Task Formulation:
Given the user query $q$, the item keywords $k$ and the original advertisement
text $t$ of the item. The objective of the task is to generate advertisement
texts $ads$ that are coherent to the item information while being closer to
the focus of the given user queries(query-variant ads).

###  2.2. Data Collection

We collect the data from the search advertising system of the Tencent QQ
Browser. Some statistics of the dataset are given in Table 1.
Query-Ad Pair Data
We use the query-ad pairs that are retrieved by online search advertising.
Each pair of data consists of three parts: query q from the user, human
written advertisement text t retrieved by the system and keywords k of this
item. The keywords are provided by the advertisement sponsors. The maximum
length of the advertisement text is limited to 20 words. The number of item
keywords and the length of user query in each case are limited to 10 words.
This length constraint setting balances the coverage of data (95%) and
calculation consumption. Furthermore, to ensure that the item keywords can
reflect core item information, we set the number of the item keywords to be at
least 3. There are 2.14 million different advertisement texts in our data, of
which 2.12 million are used as the training set. For the test set and the
validation set, the advertisement text that matches at least 2 queries in the
system are selected from 10,000 advertisement texts. The test set is built in
this way to evaluate the ability to satisfy the needs of different user
queries.
Association Knowledge Data
We propose to construct the association knowledge word graph based on the co-
occurrence information between words in the original advertisement text. The
co-occurrence word graph extracted from the advertising text corpus with the
same distribution brings the historical co-occurrence information of words in
advertising text. We collect 17 million advertisement texts to build an
Association Knowledge Word Graph (AKWG). Formally, we use point-wise mutual
information (PMI) to calculate the correlation between words. In the $AKWG$
$(\mathcal{N},\mathcal{E})$, the nodes $\mathcal{N}$ are the words, and an
edge $e(i,j)\in\mathcal{E}$ is built between $w_{i}$ and $w_{j}$ when the
$PMI(w_{i},w_{j})$ reaches a threshold $\xi$. In our construction setting,
edges with PMI scores bigger than 8 are built in the $AKWG$, and the maximum
number of neighbors for a word is limited to 20. Edge weight is normalized by
dividing PMI by 8.

##  3\. Approach

In this section, we formally introduce our proposed query-variant
advertisement text generation method. Our model mainly consists of two
modules, the Association module and the Generation module. A sketch of our
model is given in Figure 4.

The association module is designed to expand the receptive field of the model
based on external knowledge, which can help improve the ability of the model
in dealing with diverse personalized needs.

The generation module is designed to generate advertisement text based on the
original input and the associated knowledge retrieved by the association
module.

For each item, the forward process is as follows:

  1. (1)

Sub-graph Construction: We first construct the sub-graph of input words
$\mathcal{A}$ based on the item keywords $k$ and user queries $q$, which are
connected by referring to the AKWG.

  2. (2)

Knowledge Association: We associate external knowledge to $\mathcal{A}$ by
referring to AKWG with the association module and obtain the extended sub-
graph $\mathcal{A}^{\prime}$. This step follows a coarse-to-fine paradigm by
considering both statistical and semantic information.

  3. (3)

Ads Generation: Finally, we generate the advertisement text based on the
extended sub-graph $\mathcal{A}^{\prime}$.

###  3.1. Sub-graph Construction

User queries are natural language text with many expressions, while item
keywords are discrete words. To combine the two types of input, we choose to
construct user query and discrete keywords into a sub-graph through the graph
structure in $AKWG$, which provides historical information reflecting the co-
occurrence between words in user query and item keywords.

Assume the bag-of-words containing item keywords $k$ and words in the user
query $q$ as $\alpha$, we use the edges in AKWG$(\mathcal{N},\mathcal{E})$ to
link the original input words. Formally, if $w_{i}\in\alpha$ ,
$w_{j}\in\alpha$ and $e(i,j)\in\mathcal{E}$, we build an edge between $w_{i}$
and $w_{j}$ in the sub-graph $\mathcal{A}$ .

Figure 5. A sketch of association process. Blue circles are item keywords, red
circles are query words, orange circles are the associated words retrieved by
the Association Module from the AKWG.

###  3.2. Association Module

The Association Module is designed to expand the receptive field of the model
based on external knowledge, which can help improve the ability of the model
in dealing with diverse personalized needs. It can retrieve the relevant
external knowledge and combine it into the input words. The input words can be
linked and expanded according to the graph structure in the $AKWG$. When the
model encounters diverse and low-frequency input words, the associated words
shared with some high-frequency words could be activated. The activated
associated words can be regarded as a bridge to help the model transfer its
ability in dealing with familiar high-frequency words to unfamiliar low-
frequency words. In this way, the ability of the model to deal with low-
frequency words can benefit from it. Furthermore, only applying statistical
information would bring too much noise. Therefore, we propose to take
advantage of the local semantic information within each case to select
appropriate associated words.

To expand the receptive field and select the expanded information, we design
two steps in the association module, extension and denoising. An illustration
is shown in Figure 5.

Association: Extension
We take the one-hop neighboring nodes of the input words($k$,$q$) in $AKWG$ as
the candidate nodes to be added to the sub-graph $\mathcal{A}$. The edges in
$AKWG$ reflect the historical co-occurrence relation between words, which can
help the model make reasonable associations. These candidate words that
frequently co-occur with the keywords in the advertisement text data have
similar semantic information with original input words, which allows the
candidate words to be relevant. The candidate nodes introduced here can be
noisy and needed to be filtered before being added to the graph.

Association: Denoising
In sub-graph extension, we roughly select candidate neighboring nodes by
retrieving all the one-hop neighboring nodes of the input words in AKWG. This
rough selection procedure only makes use of the historical co-occurrence
information and does not consider the semantic connection of different input
words in each case, which means these candidate additional words can be noisy.
Therefore, to combine the semantic information in each case, we propose a
reinforcement learning (RL) based method to score (select) the suitable
candidates based on the semantic information and the graph structure within
the sub-graph $\mathcal{A}$ of input words.

We propose to use a graph encoder and a score predictor to score the
candidates. The graph encoder is designed for encoding the sub-graph
$\mathcal{A}$ into dense vectors, which contains the semantic information and
structure information of the nodes. The score predictor is to predict the
probability of the candidate nodes from AKWG to be extended into the sub-graph
$\mathcal{A}$ based on the graph representation learned by the graph encoder
and the AKWG edges.

Graph Encoder: We propose a Gated Graph convolutional neural networks
(GatedGCN) with gated attentive pooling as the graph encoder, which is
calculated as follows:

(1) $\displaystyle\tilde{H}^{l}$ $\displaystyle=GCN(H^{l-1},adj)$ (2)
$\displaystyle H^{l}$ $\displaystyle=LSTM(\tilde{H}^{l},H^{l-1})$ (3)
$\displaystyle\tilde{g}^{l}$ $\displaystyle=AttnPooling(H^{l})$ (4)
$\displaystyle g^{l}$ $\displaystyle=LSTM(g^{l-1},\tilde{g}^{l})$

where $H^{l}$ is the hidden states of the $l$-th layer in the graph, $adj$ is
the normalized adjacency matrix following the setting of GCN (Kipf and
Welling, 2017), $g$ is the global representation of the graph learned with
attentive pooling in (Li et al., 2016d). Attentive pooling allows the model to
get a graph representation with calculated importance weights.

This encoder applies GCN to aggregate the neighboring information and uses the
gate mechanism in LSTM (Hochreiter and Schmidhuber, 1997) to decide which part
of the aggregated information should be transmitted into the next layer in the
update of both node and global representation. GCN can combine semantic
information and topological structure information to encode features. The gate
mechanism of LSTM can select out important information between the two layers
of GCN.

Score Predictor: After we get the global graph representations, we score the
candidates with the score predictor:

(5) $score=w([g^{L};v_{c}])$

where $g^{L}$ is the global representation of the last layer, $v_{c}$ is the
embedding of the candidate word $w_{c}$, $[;]$ means concatenation, and $w\in
R^{h}$ is a learnable parameter matrix. The candidates with top $\phi$ scores
are added to the original sub-graph $\mathcal{A}$, which form the new sub-
graph $\mathcal{A}^{\prime}$.

###  3.3. Generation Module

After association, the original input words and the associated words are
connected through the graph structure. We treat the original input words and
the associated words as a kind of heterogeneous graph to make use of the
associated words without damaging the original input information. The
generation module is designed to generate advertisement text based on the
extended sub-graph $\mathcal{A}^{\prime}$ obtained in the association. This
module is developed based on Transformer (Vaswani et al., 2017). Although
Transformer has proved to be successful in modeling linear structure data, it
is not suitable for the graph structure information in $\mathcal{A}^{\prime}$.

The graph structure of $AKWG$ contains prior information of massive
advertising text, which can help our text generation task. To make use of the
graph structure that links the input words, we propose to add a GatedGCN layer
used in the association module before the Transformer encoder. The decoder
follows the same architecture of the original Transformer. To distinguish the
role between the original input words and the associated knowledge words, we
add type embedding $emb_{t}$ to the word embedding $emb_{w}$,
$x=emb_{w}+emb_{t}$, which is similar to the positional encoding. In this way,
the original input words and associated words can be seen as one kind of
heterogeneous graph. The model can make use of the associated words without
damaging the original input information.

###  3.4. Training and Inference

In the search advertisement system shown in figure 2, the ad of the same item
can be recalled by multiple queries focusing on different aspects of the item.
Item keywords usually indicate the general type of needs for the item, which
is tightly coupled with the item. On the contrary, different user queries
contain different personalized needs that are loosely coupled with the item.

Figure 6. Different input settings between training and inference. “K” is item
keywords, “Q” is user query. In the training phase, item keywords are used as
input to fit the ad text. In the inference phase, item keywords and query are
both used as input to generate query-variant advertisement text.

If simply treating both keywords and queries as the input, the model tends to
treat the loosely coupled parts of queries as noise and ignore them. As a
consequence, the model would ignore the personalized needs and fail to
generate query-variant advertisement text.

Therefore, we use different input setting strategies in training and
inference. An illustration is shown in Figure 6. In the training phase, we use
item keywords as input and advertisement text as generation target. In the
inference phase, words in the query are treated as discrete words like item
keywords. Item keywords and query are both used as input to generate query-
variant advertisement text. In this way, the model in the training phase will
not be confused by the multiple loosely-coupled queries. It can learn the
ability to generate general purpose advertisement text from tightly-coupled
item keywords during training, and generate advertisement text for each query
in the inference.

Table 2. Automatic evaluation results. Original means the human written
advertisement text retrieved by the advertising system. We show the change of
Recall(q) and Recall(q+k) compared with the Original ads in parentheses. In
Train and Test columns we show the input settings. k is Item Keywords, q is
User Query. Our model beats all the baseline models in all the metrics except
Dist-1. Although the diversity promoting model Transformer+itf gets a higher
Dist-1 score, its Recall (q) is very low, which means the generated text is
not targeted on the query.

Model | BLEU | Recall(k) | Recall(q) | Recall(q+k) | Dist-1 | Dist-2 | Train | Test
---|---|---|---|---|---|---|---|---
Original | —— | 95.45 | 65.54 | 74.51 | 9.28 | 21.73 | —— | ——
Pointer Network | 21.58 | 84.43 | 75.91 (+10.37) | 77.70 (+3.19) | 11.11 | 36.65 | k | k+q
Transformer | 23.65 | 90.77 | 88.06 (+22.52) | 87.66 (+13.15) | 13.07 | 38.85 | k | k+q
GPMM | 20.69 | 90.10 | 63.45 (-2.09) | 71.04 (-3.47) | 7.96 | 20.20 | k+q | k+q
ExpansionNet | 21.40 | 89.68 | 66.26 (+0.72) | 71.87 (-2.64) | 8.88 | 25.80 | k+q | k+q
Transformer+itf | 10.79 | 89.77 | 64.40 (-1.14) | 71.13 (-3.38) | 14.84 | 25.81 | k | k
Proposal | 27.12 | 92.36 |  91.03 (+25.49) |  90.28 (+15.77) | 13.53 | 41.36 | k | k+q

Training: We propose a three-stage training process that facilitates RL
training with supervised training.

In the first stage, we omit the score predictor in the forward pass. The
extended keywords are randomly selected from the candidates (words connected
with $\mathcal{A}$ in the AKWG). This first supervised training process gives
the model the clue of the extended graph with new words from the AKWG, which
can be used to roughly train the parameters (e.g. embeddings) of both the
association module and the generation module. The generation module is trained
for 15 epoch with cross-entropy weighted by the sentence length.

In the second stage, we use policy gradient (Williams, 1992) based RL to train
the association module. This stage aims to train the module to select the
associated words by referring to the supervised objective of the task. The
reward is calculated by comparing the advertisement text generated by the
model in the first stage with the original human-written one. Formally, the
reward is calculated as follows:

(6) $Reward=1-tanh(-\frac{1}{n}\times\sum_{j=1}^{n}{t_{j}log(p_{j})})$

where $t$ is the gold label, $p$ is the prediction probability. For different
inputs, the value of the sampling probability is affected by the total number
of sampled words, thus affecting the RL loss. Therefore the probability for
each sample is normalized by the total number of words. In this stage, the
embedding of the generated model learned in the first stage is used as the
embedding of the association module. Fine-tuning for reinforcement learning
costs 5 epochs.

In the third stage, we also apply supervised training after the reinforcement
learning process when the parameters of the association module are settled.

Inference: In inference, we treat the query as discrete words like keywords.
These words, together with item keywords, are used as input during inference.
We split the query of the natural sentence into a bag of words and removed the
stop words, and merged with the item keywords. The sub-graph is constructed
with the merged bag of words, which forms the basis of association and
generation.

##  4\. Experiment

In this section, we give the experiment results. Extensive analysis is
conducted to demonstrate the effectiveness of our model.

###  4.1. Setting

The embedding size is 128. The hidden size is 256. For a fair comparison, the
generation module of our model is obtained by replacing the first layer in the
Transformer encoder with the mentioned GatedGCN, which has parameters of the
same scale as Transformer. Both the GatedGCN and the Transformer encoder have
two layers, while the decoder has three layers. We use Adam optimizer (Kingma
and Ba, 2015) to train the model. Considering the memory consumption and model
performance, we set the value of $\phi$ to 10 in our experiment.

###  4.2. Baseline

In this work, we compare our model with five strong baselines in addition to
the original human-written one. The numbers of parameters are all of the
comparable scales to our model.

General End-to-End Baselines:
To compare the modeling capabilities of the proposed models, we introduced two
end-to-end text generation models with different architectures.

  * •

Pointer network (See et al., 2017): A Seq2Seq model with copy mechanism. The
attention mechanism is used to calculate the additional probability of the
input words to be combined with the original output probability. This baseline
is applied because the advertisement text usually shares many words in the
item keywords and user query.

  * •

Transformer (Vaswani et al., 2017): Powerful text generation model. Our
generation module is a graph suitable improvement over it.

Personalized Text Generation Baselines:
The task proposed in this paper is close to the personalized text generation
task that uses search queries as personalized information, so we introduce two
personalized text generation baselines.

  * •

GPMM (Generative Profile Memory Network) (Zhang et al., 2018): A Seq2Seq based
method aiming at personalized text generation. We adapt the model to our task
by treating each word of user query as individual memory representations in
the memory network.

  * •

ExpansionNet (Ni and McAuley, 2018): Another Seq2Seq based method aiming at
personalized text generation, which calculates two attention scores between
decoder hidden state and the word embedding in the user query.

Diversity Promoting Baseline:
The proposed task hopes to obtain diversified texts corresponding to different
search queries, so we add a pure diversity promoting baseline.

  * •

Transformer + itf (Nakamura et al., 2018): A diversity-promoting method that
weights the loss function by inverse token frequency. The network is the same
as Transformer.

The Seq2Seq models in GPMM, ExpansionNet and Pointer Network are all LSTM-
based (Hochreiter and Schmidhuber, 1997) Seq2Seq model with attention
mechanism (Luong et al., 2015). Both encoder and decoder have three layers.

###  4.3. Automatic Evaluation

Metrics:
we choose three widely applied automatic metrics to evaluate the quality of
the generated text regarding the expression ability, query coverage, and
diversity.

BLEU (Papineni et al., 2002): BLEU is calculated between the original human
written advertisement text and the one generated under the same input setting
during training. This metric is to measure the expression ability of the model
from item keywords to advertisement text. If the model can fit general-purpose
advertisement text well when using keywords as input, then when the query is
added as input, the model can also generate smooth advertisement text
corresponding to specific query.

Recall: Recall(k), Recall(q), Recall(q+k) calculate the recall between the
generated advertisement text and item keywords (k), user query (q), the union
of item keywords and user query (q+k) respectively in inference. We also
present the recall improvement over the original advertisement text in
parentheses to measure the ability to absorb the query information. We
calculate the recall by averaging over the results of different queries of the
item, which reflects the ability to generate query-variant advertisement texts
when facing different queries for the same item.

Dist-1, Dist-2 (Li et al., 2016a): These two metrics count the number of
distinct unigrams and bigrams divided by the total number of unigrams and
bigrams in the generated text. These two metrics reflect informativeness and
diversity.

Table 3. Human evaluation results. “win” indicates the ratio of the generated
advertisement text that our proposed model is better than the counter model.
“win>lose” means our proposal is better than the other one. The Pearson’s r
for each metric is shown in the header within the braces.

Versus | Attractiveness (p=0.44) | Informativeness (p=0.46) | Fluency (p=0.22)
---|---|---|---
| win | lose | tie | win | lose | tie | win | lose | tie
Proposal-Original | 57.27 | 8.47 | 34.27 | 60.53 | 21.73 | 17.73 | 25.40 | 19.53 | 55.07
Proposal-Transformer | 38.00 | 14.87 | 47.13 | 48.07 | 22.33 | 29.60 | 21.67 | 10.93 | 67.40
Proposal-NoQuery | 58.67 | 4.13 | 37.20 | 69.07 | 9.07 | 21.87 | 21.73 | 18.93 | 59.33

Result:
In Table 2 we show the automatic evaluation results of our model compared with
the baselines and the original advertisement text. We can observe that the
Recall(q) of the original advertisement text is significantly lower than
Recall(k), showing that the original coverage of query words is not
satisfactory, which means that the general purposed advertisement text can not
meet the diverse personalized needs of different queries focusing on different
aspects of the item. This phenomenon testifies the necessity of our proposed
task, which aims to generate query-variant advertisement text.

From the results, we can also see that traditional personalized text
generation baseline methods (GPMM, ExpansionNet) do not perform well. Due to
the loose-coupling problem of item-queries that one item can be matched with
different queries, the model tends to treat the queries as noise and ignore
them in the training, which makes the model unable to generate various
advertisement texts corresponding to different queries in the inference. As a
result, these common methods that model query information in the training
phase are not suitable for search advertisement text generation task. It also
proves that our different input setting strategies in the training and
inference phase are necessary.

Note that although the pure diversity promotion method (Transformer + itf)
gets a higher ratio in Dist-1, the Recall(q) is too low to be applicable,
which means the generated text ignores the query. This indicates that general-
purpose diversity promotion method is not suitable for the proposed task where
the generated advertisement texts should answer the needs of different user
queries.

Compared with the Transformer and the Pointer Network that take the same input
setting in training and inference with our model, our model achieves much
better results for all metrics. This is because that the associated external
knowledge not only brings more fruitful information but also enhances the
ability of the model to deal with words with different exposure frequency, so
that the model can better fit the inputs in the training phase, and achieve
better generalization effects after adding query in the inference phase. The
proposed model gains the highest Recall(q) improvement among different user
queries, which means that our proposed method can be adapted to variant user
queries and generate query-variant advertisement texts.

Figure 7. Concrete example of advertisement text generation. Blue words
represent general needs information, which is usually included in the item
keywords that match the item. Red words represent diverse personalized needs
contained in different user queries. Table 4. Ablation study on the effect of
association module and graph structure in generation module. -Association
means that the association module is removed. -Graph Structure means to
replace GatedGCN in the generation module with normal Transformer encode
layer.

Model | BLEU | Recall(k) | Recall(q) | Recall(q+k) | Dist-1 | Dist-2
---|---|---|---|---|---|---
Proposal | 27.12 | 92.36 | 91.03 (+25.49) |  90.28 (+15.77) | 13.53 | 41.36
-Association | 23.91 | 89.39 | 86.54 (+21.00) | 86.16 (+11.65) | 13.18 | 39.05
-Graph Structure | 25.77 | 91.89 | 90.05 (+24.51) | 89.97 (+15.16) | 13.39 | 40.66

Table 5. Effect of different retrieving methods. “Random” means randomly
selecting $\phi$ neighbors from AKWG. “PMI” means selecting neighbors with
top-$\phi$ PMI scores. “no-filter” means selecting all the neighbors in AKWG.
“Proposal” means selecting with our proposed association module.

Model | BLEU | Recall(k) | Recall(q) | Recall(q+k) | Dist-1 | Dist-2
---|---|---|---|---|---|---
random | 24.52 | 91.60 | 89.52 (+23.98) | 89.09 (+14.58) | 13.48 | 41.13
pmi | 24.77 | 91.65 | 89.66 (+24.12) | 89.14 (+14.63) | 13.39 | 40.46
no-filter | 25.20 | 90.63 | 88.41 (+22.87) | 87.82 (+13.31) | 13.14 | 39.19
Proposal | 27.12 | 92.36 |  91.03 (+25.49) |  90.28 (+15.77) | 13.53 | 41.36

###  4.4. Human Evaluation

Evaluation Data:
In the human evaluation, we focus on the cases where the default retrieved
advertisement text does not meet the needs of the queries. If more than 50% of
the characters in the query without stop words do not appear in the
advertisement text, we assume that there exists a major divergence between the
query and the retrieved advertisement text, which are chosen as the candidates
for human evaluation. After sensitive data violating privacy issues is
automatically filtered out, we randomly sample 500 cases as human evaluation
data and invite 3 human annotators to do the evaluation. Evaluators are
employees hired by IT companies to review various online text, which means
they have expertise in reviewing advertisement text.

Evaluation Guideline:
In human evaluation, the user query and two advertisement texts options are
provided to the evaluators. The evaluators are asked to choose the better one
out of the two optional advertisement texts in terms of three aspects:

  * •

Attractiveness is a comprehensive and the most important indicator,
representing which advertisement text the users prefer given the query.

  * •

Informativeness represents which advertisement text provides more useful
information regarding the user query.

  * •

Fluency represents the language smoothness of the advertisement text.

If the two advertisements are equally good (bad), they are given a tie. In the
final statistical results, if win $>$ lose, it means that our proposed model
beats the baseline.

Result:
In Table 3 we show the human evaluation results. We compare our model with
three baselines, namely, the original human written advertisement text,
Transformer and $NoQuery$. $NoQuery$ means the result is generated by the
proposed model without query during inference, which means that the query
information is totally ignored. Results show that our proposed model beats all
the baselines, especially in attractiveness and informativeness. The results
on attractiveness and informativeness show good correlation among different
evaluators, while rather low correlation on fluency. This is expected because
the concept of being fluent is hard to define, especially in the advertisement
text area.

We provide concrete advertisement text generation cases in Figure 7. The
advertisement generated by our method takes into account both general needs
contained in item keywords and query preference, which is not available in
general-purpose human written advertisement and advertisement generated
without query.

###  4.5. Ablation Study

Effect of Association and Graph:
Table 4 shows the ablation study results on association and graph structure.
The removal of associative thinking (-Association) causes the decrease of the
BLEU score and the Recall (q+k) by 3.21 and 4.12. We think this is because the
ability of the model to deal with different types of words is benefited from
the association module. This allows the model to achieve better fitting
results in training. In the inference phase, the model can better absorb the
query information, which makes the model less likely to generate irrelevant
advertisement text.

As for the graph structure in the generation module, after replacing the
GatedGCN with the normal Transformer encode layer(-Graph Structure) in the
generation module, the BLEU score drop by 1.35. This shows that the graph
structure with prior information can help to organize discrete input
information into a fluent natural language sentence.

Effect of Different Retrieval Methods:
In Table 5 we show the results of different retrieving methods. Among all the
retrieving methods, our proposed method achieves the best on all the metrics.

Although the “no-filter” method adds all the one-hop neighbors on $AKWG$ to
the input, its performance on all metrics is still weaker than our proposed
model. We think that this is because there is too much noise brought by adding
all the neighbors in AKWG to the input sub-graph. This testifies that the
denoising procedure of the association module is necessary.

“PMI” signifies the correlation of the candidate knowledge and gains higher
recall improvement (+14.63) than “random” and “no-filter”. However, the PMI
score only concerns the global historical co-occurrence knowledge, while our
method not only makes use of the PMI score via the edge weight in the GCN
component in the association module but also considers the semantic
information of the keywords encoded in the graph representations. This design
helps our method achieve the best BLEU score (27.12) and the highest recall
improvement (+15.77) among all the retrieval methods.

Figure 8. Results of different training stages. Stage 2 is RL based. Stage 1
and stage 3 are supervised.

Effect of RL:
In Figure 8 we show the performance trend during the RL training stages
(dictated in section 3.4). From the figure, we can see that all three stages
are necessary to train the model. In the reinforcement learning stage, the
improvement proves that the association module can denoise the associated
candidate words after reinforcement learning and obtain association words that
dynamically consider the semantic information in the sub-graph of a specific
case. With the help of the two supervised stages, RL training can indeed
improve the performance of the model.

##  5\. Related Work

Product Information Generation: Product text information (e.g., advertisement,
product description) is widely displayed in various Internet scenarios.
Previous researchers have conducted many studies for different application
scenarios. Wang et al. (2017) propose a statistical framework that generates
product descriptions from product attributes. (Chen et al., 2019; Wang et al.,
2020) generate personalized product descriptions on e-commerce platforms. Chan
et al. (2019a) utilize the entity information for fidelity-oriented product
descriptions. (Hughes et al., 2019; Wang et al., 2019) use user clicks
feedback information to guide the generation of advertisement text. Wang et
al. (2018) apply a multi-task learning approach for improving product title
compression with user search log data. (Chan et al., 2020) propose a new task,
i.e., how to generate a multi-product advertisement post. The task we proposed
is applied to search advertising, which is to generate advertisement text for
a specific query. Our method is to generate query-variant ad text offline,
which are then recalled online. The existing methods cannot completely solve
the problems in our task scenarios.

Personalized and Diverse Text Generation: Many previous works have been
proposed to use personalized information for text generation. (Li et al.,
2016c; Zhang et al., 2018; Chan et al., 2019b) make use of speaker information
and capture characteristics of speakers in neural conversation. Xing et al.
(2017) incorporate topic information into the Seq2Seq framework to generate
informative and interesting responses for chatbots. Ni and McAuley (2018) aim
at generating personalized reviews by expanding short phrases with aspect-
level information. These works provide good views of personalized generation.
However, there is one big obstacle making their methods not suitable for our
proposed task, that is, there is no strict coupled personalized query-
advertisement pair in our task. In our experiment, we compared several
personalized text generation methods and revealed their inapplicability to the
proposed task.

There are also works focusing on generating general purposed diverse text.
With the idea of reducing the frequency of high-frequency text and increasing
the frequency of low-frequency text, various methods (Li et al., 2016b; Xu et
al., 2018; Liu et al., 2018; Nakamura et al., 2018) have been used to increase
the diversity of text generation. Vijayakumar et al. (2016) propose to
diversify the outputs by optimizing diversity-augmented objective in beam
search. Unlike the diversity-promoting works above, our proposed task aims to
generate query-variant advertisement text that can diversify the advertisement
text in the context of meeting the purpose of different query needs.
Therefore, those methods that do not consider the queries are not applicable.

External Knowledge: External knowledge is an important information source for
natural language understanding. Earlier attempts using the knowledge graph
mainly focus on answering questions with a single clue retrieved from the
knowledge graph (Hao et al., 2017; Bordes et al., 2014a, b; Dong et al.,
2015). Although these works achieve big success, the applications are limited
because of the idealized task setting. Other works propose to supplement the
state-of-the-art model with external knowledge (Chen et al., 2016; Chen et
al., 2018; Parthasarathi and Pineau, 2018). The knowledge usage in these works
remains latent, and it is hard to tell the effect of the knowledge. Zhou et
al. (2018) use large-scale commonsense knowledge in conversation generation.
Mihaylov and Frank (2018) integrate external knowledge in a cloze-style
setting for the reading comprehension task. These works try to reason over the
graph, which is a good way to use external knowledge. However, there is no
pre-defined knowledge in our task. Therefore, we propose to first construct an
association knowledge word graph based on the co-occurrence information in the
advertisement text.

##  6\. Ethical Issues

Industrial Application: For the industrial application, the generated ad
candidates will be further checked by human annotators before added to the
online advertising system to prevent fake and misleading advertisement text
from being shown to users. In the manual review phase, 85.80% of generated ads
are accepted. This acceptance rate can meet the efficiency and accuracy
requirements of industrial applications. In the industrial application, item
keywords and possible queries are provided by the advertising sponsor, which
means that our method will not use user information in the training and
inference phase.

Dataset Open-Sourced: The data set contains search query content of website
users and brand information of advertising sponsors. To protect the their
privacy, the data set will be open-sourced after ID projection, which means
that the mapping between word ID and word text will not be open-sourced. This
approach allows peers to reproduce our automatic evaluation results and
conduct computational experiments, but cannot get the original text
information. Although this will affect the reproducibility of human
evaluation, we believe that the protection of users and customers privacy is
the highest priority.

##  7\. Conclusion and Future Work

In this paper, we propose the query-variant advertisement text generation task
that aims to generate candidate advertisement texts for different search
queries with various needs. An association method based on external knowledge
is proposed to improve the ability to deal with diverse personalized needs of
search queries. Both automatic and human evaluation results show that our
model can generate attractive and query-variant advertisement text. Our work
provides a new perspective on the industrial application and research of
search advertisement text generation.

Pre-training is an important method of using historical text knowledge. In
future work, we want to explore how to incorporate pre-training into existing
methods. Furthermore, our data and experiments are conducted on the Chinese
data set. Because the quality of the Chinese open lexicon is not ideal, the
lexicon we currently use only includes $AKWG$ constructed using historical
advertising text. In the future, we will explore using lexicon from different
information sources.

## References

  * (1)
  * Bordes et al. (2014a) Antoine Bordes, Sumit Chopra, and Jason Weston. 2014a.  Question Answering with Subgraph Embeddings. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Doha, Qatar, 615–620.  https://doi.org/10.3115/v1/D14-1067
  * Bordes et al. (2014b) Antoine Bordes, Jason Weston, and Nicolas Usunier. 2014b.  Open question answering with weakly supervised embedding models. In _Joint European conference on machine learning and knowledge discovery in databases_. Springer, 165–180.
  * Chan et al. (2019a) Zhangming Chan, Xiuying Chen, Yongliang Wang, Juntao Li, Zhiqiang Zhang, Kun Gai, Dongyan Zhao, and Rui Yan. 2019a.  Stick to the facts: Learning towards a fidelity-oriented e-commerce product description generation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_. 4959–4968.
  * Chan et al. (2019b) Zhangming Chan, Juntao Li, Xiaopeng Yang, Xiuying Chen, Wenpeng Hu, Dongyan Zhao, and Rui Yan. 2019b.  Modeling personalization in continuous space for response generation via augmented wasserstein autoencoders. In _Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing_. 1931–1940.
  * Chan et al. (2020) Zhangming Chan, Yuchi Zhang, Xiuying Chen, Shen Gao, Zhiqiang Zhang, Dongyan Zhao, and Rui Yan. 2020.  Selection and Generation: Learning towards Multi-Product Advertisement Post Generation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 3818–3829.
  * Chen et al. (2019) Qibin Chen, Junyang Lin, Yichang Zhang, Hongxia Yang, Jingren Zhou, and Jie Tang. 2019\.  Towards Knowledge-Based Personalized Product Description Generation in E-commerce. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019_. 3040–3050.  https://doi.org/10.1145/3292500.3330725
  * Chen et al. (2018) Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Diana Inkpen, and Si Wei. 2018.  Neural Natural Language Inference Models Enhanced with External Knowledge. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_. 2406–2417.
  * Chen et al. (2016) Yun-Nung Chen, Dilek Hakkani-Tur, Gokhan Tur, Asli Celikyilmaz, Jianfeng Gao, and Li Deng. 2016.  Knowledge as a teacher: Knowledge-guided structural attention networks.  _arXiv preprint arXiv:1609.03286_ (2016).
  * Dong et al. (2015) Li Dong, Furu Wei, Ming Zhou, and Ke Xu. 2015\.  Question answering over freebase with multi-column convolutional neural networks. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 260–269.
  * Hao et al. (2017) Yanchao Hao, Yuanzhe Zhang, Kang Liu, Shizhu He, Zhanyi Liu, Hua Wu, and Jun Zhao. 2017.  An End-to-End Model for Question Answering over Knowledge Base with Cross-Attention Combining Global Knowledge. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Vancouver, Canada, 221–231.  https://doi.org/10.18653/v1/P17-1021
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Hughes et al. (2019) J. Weston Hughes, Keng-hao Chang, and Ruofei Zhang. 2019\.  Generating Better Search Engine Text Advertisements with Deep Reinforcement Learning. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery I & Data Mining_ (Anchorage, AK, USA) _(KDD ’19)_. Association for Computing Machinery, New York, NY, USA, 2269–2277.  https://doi.org/10.1145/3292500.3330754
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kipf and Welling (2017) Thomas N Kipf and Max Welling. 2017.  Semi-supervised classification with graph convolutional networks.  _ICLR_ (2017).
  * Li et al. (2016a) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a.  A Diversity-Promoting Objective Function for Neural Conversation Models. In _NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016_. 110–119.  https://doi.org/10.18653/v1/n16-1014
  * Li et al. (2016b) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016b.  A Diversity-Promoting Objective Function for Neural Conversation Models. In _NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016_. 110–119.
  * Li et al. (2016c) Jiwei Li, Michel Galley, Chris Brockett, Georgios P. Spithourakis, Jianfeng Gao, and William B. Dolan. 2016c.  A Persona-Based Neural Conversation Model. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_.  https://doi.org/10.18653/v1/p16-1094
  * Li et al. (2016d) Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard S. Zemel. 2016d.  Gated Graph Sequence Neural Networks. In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.  http://arxiv.org/abs/1511.05493
  * Liu et al. (2018) Yahui Liu, Wei Bi, Jun Gao, Xiaojiang Liu, Jian Yao, and Shuming Shi. 2018\.  Towards less generic responses in neural conversation models: A statistical re-weighting method. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. 2769–2774.
  * Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015\.  Effective Approaches to Attention-based Neural Machine Translation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015_. 1412–1421.
  * Mihaylov and Frank (2018) Todor Mihaylov and Anette Frank. 2018.  Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_. 821–832.
  * Nakamura et al. (2018) Ryo Nakamura, Katsuhito Sudoh, Koichiro Yoshino, and Satoshi Nakamura. 2018.  Another Diversity-Promoting Objective Function for Neural Dialogue Generation.  _arXiv preprint arXiv:1811.08100_ (2018).
  * Ni and McAuley (2018) Jianmo Ni and Julian J. McAuley. 2018.  Personalized Review Generation By Expanding Phrases and Attending on Aspect-Aware Representations. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers_. 706–711.  https://doi.org/10.18653/v1/P18-2112
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002\.  BLEU: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting on association for computational linguistics_. Association for Computational Linguistics, 311–318.
  * Parthasarathi and Pineau (2018) Prasanna Parthasarathi and Joelle Pineau. 2018.  Extending Neural Generative Conversational Model using External Knowledge Sources. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018_. 690–695.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017\.  Get To The Point: Summarization with Pointer-Generator Networks. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers_. 1073–1083.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Advances in neural information processing systems_. 5998–6008.
  * Vijayakumar et al. (2016) Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2016\.  Diverse beam search: Decoding diverse solutions from neural sequence models.  _arXiv preprint arXiv:1610.02424_ (2016).
  * Wang et al. (2017) Jinpeng Wang, Yutai Hou, Jing Liu, Yunbo Cao, and Chin-Yew Lin. 2017.  A statistical framework for product description generation. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_. 187–192.
  * Wang et al. (2018) Jingang Wang, Junfeng Tian, Long Qiu, Sheng Li, Jun Lang, Luo Si, and Man Lan. 2018.  A Multi-Task Learning Approach for Improving Product Title Compression with User Search Log Data.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 32, 1 (Apr. 2018).  https://ojs.aaai.org/index.php/AAAI/article/view/11264
  * Wang et al. (2019) Yongzhen Wang, Heng Huang, Yuliang Yan, and Xiaozhong Liu. 2019.  Quality-Sensitive Training! Social Advertisement Generation by Leveraging User Click Behavior. In _The World Wide Web Conference_ (San Francisco, CA, USA) _(WWW ’19)_. Association for Computing Machinery, New York, NY, USA, 2045–2055.  https://doi.org/10.1145/3308558.3313536
  * Wang et al. (2020) Yongzhen Wang, Jian Wang, Heng Huang, Hongsong Li, and Xiaozhong Liu. 2020.  _Evolutionary Product Description Generation: A Dynamic Fine-Tuning Approach Leveraging User Click Behavior_.  Association for Computing Machinery, New York, NY, USA, 119–128.  https://doi.org/10.1145/3397271.3401140
  * Williams (1992) Ronald J Williams. 1992\.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Machine learning_ 8, 3-4 (1992), 229–256.
  * Xing et al. (2017) Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. 2017\.  Topic Aware Neural Response Generation. In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA_. 3351–3357.
  * Xu et al. (2018) Jingjing Xu, Xuancheng Ren, Junyang Lin, and Xu Sun. 2018\.  Diversity-promoting gan: A cross-entropy based generative adversarial network for diversified text generation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. 3940–3949.
  * Zhang et al. (2018) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018\.  Personalizing Dialogue Agents: I have a dog, do you have pets too?. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_. 2204–2213.  https://doi.org/10.18653/v1/P18-1205
  * Zhou et al. (2018) Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018\.  Commonsense Knowledge Aware Conversation Generation with Graph Attention.. In _IJCAI_. 4623–4629.


# Authorship Identification of Source Code Segments Written by Multiple
Authors Using Stacking Ensemble Method

Parvez Mahbub  CSE Discipline
Khulna University
Khulna, Bangladesh
<EMAIL_ADDRESS>Naz Zarreen Oishie  CSE Discipline
Khulna University
Khulna, Bangladesh
<EMAIL_ADDRESS>S.M. Rafizul Haque  CSE Discipline
Khulna University
Khulna, Bangladesh
<EMAIL_ADDRESS>
###### Abstract

Source code segment authorship identification is the task of identifying the
author of a source code segment through supervised learning. It has vast
importance in plagiarism detection, digital forensics, and several other law
enforcement issues. However, when a source code segment is written by multiple
authors, typical author identification methods no longer work. Here, an author
identification technique, capable of predicting the authorship of source code
segments, even in case of multiple authors, has been proposed which uses
stacking ensemble classifier. This proposed technique is built upon several
deep neural networks, random forests and support vector machine classifiers.
It has been shown that for identifying the author-group, a single
classification technique is no longer sufficient and using a deep neural
network based stacking ensemble method can enhance the accuracy significantly.
Performance of the proposed technique has been compared with some existing
methods which only deal with the source code segments written exactly by a
single author. Despite the harder task of authorship identification for source
code segments written by multiple authors, our proposed technique has achieved
promising results evident by the identification accuracy, compared to the
related works which only deal with code segments written by a single author.

###### Index Terms:

Source Code Authorship Identification, Multiple Author, Deep Neural Network,
Random Forest, Support Vector Machine, Stacking Ensemble

##  I Introduction

Source code segment author identification is a major research topic in the
field of software forensics. It has many uses such as plagiarism detection,
law enforcement, copyright infringement etc.[8, 12]. Frantzeskou[7] mentioned
that source code author identification is useful against cyber attacks in the
form of viruses, trojan horses, logic bombs, fraud, credit card cloning, and
authorship disputes or proof of authorship in court. There are certain
patterns that developers sub-consciously reflect in their codes based on their
particular coding style while still following the guidelines, standards,
rules, and grammars of a language or framework[12]. These pieces of
information can be used to identify the author of the source code segment.

In recent years, open source software development has entered a new era. A lot
of big companies like Google, Microsoft, and many others are maintaining their
projects open source. Alongside, small and mid-level projects are being
written by a group of authors. In these cases, trivial author identification
schemes no longer work. When someone contributes to an open source project,
the writing style of the original author of the source code segment is no
longer unique and it makes the author identification task harder. Even worse
case is when a project is equally contributed by a number of authors. The
writing style is then the aggregation of all the authors. We aimed to solve
this problem and proposed an approach to identify the author of a source code
even when it is contributed by more than one author. In this paper, we have
proposed an author identification technique using a stacking ensemble method
composed of several deep neural networks(DNN), random forests and support
vector machines(SVM).

###  I-A Problem Definition

Authorship identification is the task of having some samples of code for
several programmers and determining the likelihood of a new piece of code
having been written by each programmer[9]. As the name suggests, authorship
identification of source code segments written by multiple authors is
identifying the author-label when the number of authors of source code segment
is more than one. This can be of two types. One is the source code segment can
be written by mostly one author and have small contributions from several
other authors. Another is a source code segment can be directly written by a
group of authors and have a roughly equal contribution from each of them. Both
of these happen in open-source software and projects which are very popular
nowadays.

###  I-B Motivation

Authorship identification of source code segment has a vast application area
including plagiarism detection, authorship dispute, software forensics,
malicious code tracking, criminal prosecution, software intellectual property
infringement, corporate litigation, and software maintenance[20, 19, 8, 13,
21, 16]. In the case of authorship dispute, authorship identification can be a
solution. Given the source code segment and the candidate owners, the
likelihood of each candidate of being the author of the source code can be
determined[12]. Again, Kothari[12] identified that author identification is
useful for detecting the author of the malicious code. Software companies can
also use authorship identification system to keep track of programs and
modules for better maintenance[21]. Though source code segments are much more
restrictive and formal than spoken or written language, they inhibit a large
degree of flexibility[7]. According to Shevertalov[18], using differences in
the way programmers express their idea, their programming style can be
captured. This programming style, in turn, can be used for author
identification. Although, a large number of works already done regarding
source code segment author identification, according to Frantzeskou[8], the
future of source code segment author identification is in collaborative
projects to which we aimed at.

The remaining sections are organized as follows. Section II contains a
briefing on background topics regarding this work. Section III contains a
summary of the related works. In section IV, we discuss our author
identification technique for multiple authors. In section V, the experimental
results of our proposed technique are analyzed and compared with that of some
related works. Finally, in section VI, the conclusion is stated with possible
future direction of this work.

##  II Background

###  II-A Ensemble Method

By combining several methods, ensembling method helps to improve the results
of machine learning. An ensemble is often more accurate than any of the single
classifiers in the ensemble. According to Maclin[14], an ensemble consists of
a set of individually trained classifiers whose predictions are combined,
while classifying instances, by the ensemble method. These meta-algorithm
combines several machine learning techniques into one predictive model. In our
work, we used stacking ensemble in order to improve our prediction
performance.

###  II-B Random Forests

Random forest is an ensemble learning method where each classifier in the
ensemble is a decision tree classifier. This collection of classifiers is
called a forest. During classification, each of the decision trees gives its
vote and the result is based on the majority of the votes.

##  III Related Works

Numerous works are available on source code segment author identification
using a variety of features and classifiers. However, very few of them use
machine learning techniques to identify the author of source code segments.

According to Ďuračík[6], there are several approaches to identify the author
of source code segment. The first one is text-based and uses plain text as an
input. The second level is token or metric based.

###  III-A Text Based Approaches

The first approach, which treats source code segment as plain text, is a form
of natural language processing. This approach cannot make use of the
programmatic structure of source code segment.

Frantzeskou et al.[8] proposed a technique called Source Code Author
Profiles(SCAP) for author identification. They generated byte level n-gram
author profile and compared with previously calculated author profiles.
Burrows[4] mentioned, the SCAP method truncates the author profiles that are
greater than the maximum profile length causing a bias towards the truncated
profiles.

Burrows et al.[3] proposed an approach using information retrieval. They
generated n-gram tokens from the source code segments and indexed them in a
search engine to query the author of source code and return a ranking list of
authors which matched the n-gram token of the source code segment with 67%
accuracy.

###  III-B Metric Based Approaches

Frantzeskou[8] pointed out that metric-based author identification is divided
into two steps. The first step is extracting the code metrics that represent
the author’s style and the second part is using those metrics to generate a
model that is capable of labeling a source code segment by corresponding
author name. However, a large amount of time is required to gather all
possible metrics and examine to choose only the metrics responsible for
differing the authors’ style.

Lange and Spiros[13] assumed that the code metrics histogram should vary from
author to author as of their coding style. From a number of source code
metrics, an optimum set was selected using genetic algorithms(GA) and used as
input for the nearest neighbor(NN) classifier. This method achieved 55%
accuracy. According to Yang[20], some of the features of this paper are
unbounded. For example, the indentation category.

Shevertalov el al.[18] proposed a technique based on GA. The metrics are
extracted from the source code segment to make a histogram which is sampled
using GA. The author profile is produced using categorized histogram samples.
For files, they achieved 54% accuracy and for projects, they achieved 75%
accuracy. Yang[20] mentioned that the details of the final feature set are not
mentioned in this paper. So, the feature set is non-reproducible.

Bandara and Wijayarathna[1] used the deep Neural Network for source code
segment author identification. The converted source code metrics they used to
feed a neural network are identical to that of Lange et al.[13]. Their deep
neural network consisted of three restricted Boltzmann machine (RBM) layers
and one output layer. They achieved 93% accuracy.

Zhang et al.[21] used SVM to identify the author of source code segment. They
categorized their feature into four groups namely – programming layout
feature, programming style feature, programming structure feature and
programming logic feature. They used sequential minimal optimization(SMO) as
the classifier for SVM and achieved 98% and 80% accuracy for two different
datasets.

##  IV Author Identification of Source Codes Written by Multiple Authors

Our developed author identification approach consists of four phases. Firstly,
source code metrics are extracted from the source code segments in the
training set. These extracted metrics are then converted to feature vectors.

Secondly, these feature vectors are fed to five individual base classifiers
along with corresponding class-labels to train the author signatures to the
base classifiers. In the case of open source contribution, class-label means
the owner of the source code segment and in the case of a group of authors,
the whole group is considered as the class label. By author signature, the
coding style of a particular class-label is meant. Caruana[5] showed that, in
general, for classification problem, random forest, DNN, decision tree, and
SVM are the top four algorithms. Hence, our chosen classifiers are DNN, random
forest with CART decision trees[2], random forest with C4.5 decision
trees[10], $C$-SVM and $\nu$-SVM.

Thirdly, each of the classifiers generates the posterior class-probability
according to their predictions. These outputs are called meta-features. Meta-
features are used as the input for a meta-classifier. Then the meta-classifier
is trained based on the meta-features and output. This approach is known as
stacking ensemble. Another deep neural network is used as the meta-classifier.
Figure 1 shows a block diagram of the architecture of the stacking ensemble
method we have designed.

Figure 1: Block diagram of the architecture of the stacking ensemble method

Finally, to identify the author of a new source code segment, that is from the
test set, the same metrics are extracted from the test source code segments
and are converted to feature vectors. These feature vectors are fed to the
meta-classifier via the base classifiers. Using the experience from the
training, the meta-classifier along with the base classifiers predict the
class labels of the test source code segments. Figure 2 shows the block
diagram of the proposed approach for author identification of source codes
written by multiple authors.

Figure 2: Block diagram of proposed author identification approach

In the following sub-sections, the building blocks of the author
identification approach are described.

###  IV-A Dataset

Some careful considerations are needed while choosing the dataset. Data must
be collected from a diverse population of programmers and should provide
enough information about the authors so that a clear distinction can be
computed from author to author and valid comparison of their programming style
can be made. In addition, the dataset must be close to real-world data as well
as open for academic study[13].

In our study, we have generated our dataset based on open source codes from
github.com. All the source codes have a permissive license like MIT or BSD.
The dataset contains 6063 python source code segments from 8 authors/ author
groups which are considered as individual classes. Each source code segment
contains roughly 226 lines on average. Source code segments of each author are
roughly split into 2:1 ratio to make the training and testing set.

Each class label consists of authors and contributors. By author, we mean the
true owner of the projects. This could be a single author or a group of
authors. By contributors, we mean a group of people who are not the owner of
the project but willingly contribute to the project by writing or editing a
segment of it. The number of authors and the number of contributors per class-
label is listed in table I.

TABLE I: Number of authors and contributors for each class Class Label |  Number of
Authors |  Number of
Contributors
---|---|---
Azure |  3 |  136
GoogleCloudPlatform |  33 |  820
StackStorm |  2 |  147
dimagi |  2 |  101
enthought |  9 |  224
fp7-ofelia |  1 |  4
freenas |  2 |  126
sympy |  2 |  712

###  IV-B Metric Extraction

Previously, Shevertalov, Lange, Bandara, and Zhang[18, 13, 1, 21] used source
code metrics for author identification. From a set of probable code metrics,
Lange selected the optimal set of code metrics using the genetic algorithm.
Bandara used almost the same set of source code metrics. We have used the same
set of metrics for our author identification approach only except the access
modifier metric. The access modifier feature is present only in a limited
number of programming languages and makes the whole system language dependent.
Table II shows the set of metrics to be used and corresponding descriptions.

TABLE II: Set of code metrics and descriptions Metric Name |  Metric Description
---|---
Line Length |  This metric measures the number of characters in one source code line.
Line Words |  This metric measures the number of words in one source code line.
Comments Frequency |  This metric calculates the relative frequency of line comment, block comment and optionally doc-comment used by the programmers.
Identifiers Length |  This metric calculates the length of each identifier of programs.
Inline
Space-Tab |  This metric calculates the whitespaces that occur on the interior areas of non-whitespace lines.
Trail
Space-Tab |  This metric measures the whitespace and tab occurrence at the end of each non-whitespace line.
Indent
Space-Tab |  This metric calculates the indentation whitespaces used at the beginning of each non-whitespace line.
Underscores |  This metric measures the number of underscore characters used in identifiers.

After extracting the metrics, we have counted the number of occurrences for
each possible values for each of the metrics. For example, for underscore
metrics, we have counted the number of words with no underscore, one
underscore, two underscores etc. These counts have been fed to the base
classifiers.

###  IV-C Base Classifiers

There are total five base classifiers in our author identification system.
They are – DNN, random forest based on CART, random forest based on C4.5,
$C$-SVM and $\nu$-SVM. Each of the base classifiers is described below:

####  IV-C1 Deep Neural Network

The DNN model used as the base classifier consists of 14 layers. Data are fed
to the DNN as batches of 32 entries. They are one input layer, followed by
eight fully connected layers, a dropout layer, a fully connected layer, a
dropout layer, a fully connected layer and finally the output layer.

In the fully connected layers, _ReLU_ activation function and in the output
layer _softmax_ activation function are used. _Categorical cross-entropy_ is
chosen as the loss function. _Adam_[11] optimizer is used to optimize the
network.

####  IV-C2 Random Forest

The second base classifier is a random forest with one hundred decision trees.
Classification and Regression Tree(CART)[2] algorithm is used to build the
trees which selects the split node based on Gini impurity.

The third base classifier is another random forest with one hundred decision
trees. Decision trees in the third base classifier are built with the C4.5[10]
algorithm. This algorithm chooses the split node based on the entropy ratio.

####  IV-C3 Support Vector Machine

The fourth base classifier is a $C$-support vector classifier. It is a support
vector machine where $C$ is a penalty parameter for the error term.

The fifth base classifier is a $\nu$-support vector classifier. It is a
support vector machine where $\nu$ is the upper bound of training error and
the lower bound of the number of support vectors.

###  IV-D Meta Classifier

We have used another deep neural network as the meta-classifier. The outputs
of the base classifiers (meta-features) are fed to the meta-classifier to
learn the mapping from the meta-features to the actual output.

The neural network consists of 19 layers. They are one input layer, followed
by eight fully connected layers, a dropout layer, two fully connected layers,
a dropout layer, a fully connected layer, a dropout layer, a fully connected
layer, a dropout layer, a fully connected layer, a dropout layer, and finally
the output layer. The output from this output layer is the final output of our
author identification system for source code segment written by multiple
authors.

The activation functions of the network are _ReLU_ for fully connected layers
and _softmax_ for the output layer. The loss function used in the meta-
classifier is _categorical cross-entropy_. _Stochastic Gradient Descent(SGD)_
is used as the optimizer of the meta-classifier.

Figure 3: Steps for training the stacking ensemble system TABLE III: Parameter values of the classifiers Classifier |  Parameter |  Value
---|---|---
$C$-SVM |  $C$ |  $1.0$
$\nu$-SVM |  $\nu$ |  $0.15$
Base DNN |  learning rate |  $0.01$
Adam optimizer |  $\beta_{1}$ |  $0.9$
$\beta_{2}$ |  $0.999$
Meta DNN |  learning rate |  $0.001$
SGD optimizer |  momentum |  $0$
TABLE IV: Accuracy of the base classifiers Classifier Name |  Accuracy
---|---
Deep Neural Network |  82%
CART Based Random Forest |  83%
Random Forest |  83%
$C$-Support Vector Machine |  79%
$\nu$-Support Vector Machine |  79%
TABLE V: Comparison among the methods for source code segment author identification Method Name |  Features |  Language independent features |  Multiple author |  Number of classes |  Total source code segment |  Accuracy
---|---|---|---|---|---|---
Information retrieval
approach[3] |  Character level
n-gram |  Yes |  No |  100 |  1640 |  67%
Code metric histogram[13] |  7 code metrics |  Yes |  No |  20 |  4068 |  55%
Genetic algorithm[18] |  4 code metrics |  Yes |  No |  20 |  N\A |  75%
Deep neural network[1] |  9 code metrics |  No |  No |  10, 10, 8, 5, 9 |  1644, 780, 475, 131, 520 |  93%, 93%, 93%, 78%, 89%
Support vector machine[21] |  46 code metrics |  No |  No |  8, 53 |  8000, 502 |  98%, 80%
_Stacking ensemble method_ |  8 code metrics |  Yes |  Yes |  8 (group of authors) |  6063 |  87%

###  IV-E Training

We have implemented our author identification system for source code segment
written by multiple authors in multi-class classification category. Here, a
unique list of authors(or groups of authors) of the source code segments in
the training set is treated as classes. The author identification system
produces its confidence for each class of being the actual class of given
source code. The actual author is expected to have the highest confidence.

Roughly, 67% source code segments from each class formed the training dataset
and rest are used for testing. The training set contains 4034 files and the
test set contains 2039 source code segments.

The training stage of our system is divided into three phases – feature
extraction from the source code segments, training the base classifiers and
training the meta-classifier. Figure 3 shows the steps followed in our author
identification system for source code segment written by multiple authors.

First of all, the source code metrics mentioned in table II are extracted from
source code segments. Then the extracted metrics are converted to feature
vectors as mentioned in section IV-B. These feature vectors are fed to each of
the base classifiers as input.

The base classifiers run according to their own learning algorithm to learn to
identify the writing style of each class. During this training phase, several
configurations of each of the base classifiers, specially DNN, are used to
find out which configuration works best for the training set.

After completing the training of each of the base models, the posterior
probability for each input in the training set is generated. This produced a
$5\times|classes|$ sized feature vector for each of the input feature vectors
where $|classes|$ is the number of classes. These feature vectors are known as
meta-features. Meta features are fed to the meta-classifier along with the
class labels through which the meta-classifier learned to predict the actual
class from the meta-features.

##  V Experimental Results

###  V-A Experimental Setup

While implementing our author identification system for source code segment
written by multiple contributors, we have used keras as the framework for deep
neural networks and Scikit Learn[17] as the library for general purpose
machine learning. For data pre-processing and visualization, we have used
numpy and pandas[15] library. We have developed a feature extractor that
extracts the features mentioned in table II from the source codes.

For $C$-SVM, the parameter $C$ is a penalty for the error term. For $\nu$-SVM,
the parameter $\nu$ is an upper bound to the training error and lower bound to
the number of support vectors. During the experiment, we found that for both
the random forests, a hundred trees were sufficient to converge to the highest
accuracy. After numerous iterations, we reached to a decision that the set of
values stated in table III classifies the source code segments most
accurately.

_Accuracy_ and _f1-score_ were used to evaluate the accuracy of our method.
_Accuracy_ is the ratio between the number of correctly identified samples and
the number of total samples. _F1-score_ is the harmonic mean of _precision_
and _recall_. _Micro averaging_ was used to compute the _f1-score_.

###  V-B Results of The Base Classifiers

Table IV contains the accuracies for the five base models of our stacking
ensemble method.

###  V-C Results of The Meta Classifier

After training the meta-classifier by the meta-features, we have achieved 87%
accuracy with f1-score $0.86$. Identifying the authorship of source codes is
more difficult when the number of authors is more than one as the writing
style of the source code is then inconsistent from segment to segment. Table V
shows a comparison between the type of features, language independence, the
capability of handling multiple authorship, number of classes and the total
number of source code segments used in training and testing. From that table,
we can see that even after dealing with source code segments written by
multiple authors, our method has achieved an accuracy that is pretty close to
that of the methods that deal with single authors. Our chosen set of metrics
is compact and is still able to achieve a satisfactory accuracy. Alongside a
number of works suffer from choosing a set of metrics that are not language
independent. So, the main contribution of this work is the identification of
multiple authors using language independent set of metrics.

##  VI Conclusion

Here, we have proposed a new approach for identifying the author of source
code segment where the number of authors of the source code segment is more
than one. The main challenge of this work is to select the base estimators
from a large number of possible combinations. Again, as several classifiers
need to be trained, each classifier needs to be fine-tuned individually to
produce a good final result. On the other hand, the problem of identifying the
authorship of source code segments is harder when the number of authors is
more than one.

We have developed a stacking ensemble classifier that consists of five base
classifiers and a meta-classifier which uses a relatively small set of code
metrics that are relatively easy to compute and language independent as well.

In spite of the fact that our stacking ensemble method achieved a satisfactory
accuracy, this still can be improved. Even though our code metrics are
language independent, we only tested with python source code segments. Future
works may test on other languages and check how the set of metrics works for
other languages. Other sets of metrics can also be examined to see how they
contribute to the writing style of source code segments.

## References

  * [1] Bandara, U., Wijayarathna, G., “Deep neural networks for source code author identification,” In: M. Lee, A. Hirose, Z.G. Hou, R.M. Kil (eds.) Neural Information Processing, pp. 368–375, Springer Berlin Heidelberg, Berlin, Heidelberg (2013)
  * [2] Breiman, L., Friedman, J., Stone, C.J. Olshen, R., “Classification and Regression Trees,” CRC Press (1984)
  * [3] Burrows, S., Tahaghoghi, S.,“Source code authorship attribution using n-grams,” In: A.T. Amanda Spink, M. Wu (eds.) Proceedings of the Twelfth Australasian Document Computing Symposium, pp. 32–40, School of Computer Science and Information Technology, RMIT University (2007)
  * [4] Burrows, S., Uitdenbogerd, A., Turpin, A.,“Comparing techniques for authorship attribution of source code,” Software: Practice and Experience 44 (2014)
  * [5] Caruana, R., Karampatziakis, N., Yessenalina, A., “An empirical evaluation of supervised learning in high dimensions,” In: Proceedings of the 25th International Conference on Machine Learning, ICML ’08, pp. 96–103, ACM, New York, NY, USA (2008). DOI 10.1145/1390156.1390169. URL http://doi.acm.org/10.1145/1390156.1390169
  * [6] Duracik, M., Krsak, E., Hrkut, P., “Current trends in source code analysis, plagiarism detection and issues of analysis big datasets,” In: TRANSCOM 2017: International scientific conference on sustainable, modern and safe transport, Elsevier: Procedia Engineering, vol. 192, pp. 136–141 (2017)
  * [7] Frantzeskou, G., Stamatatos, E., Gritzalis, S., “ Supporting the cybercrime investigation process: Effective discrimination of source code authors based on byte-level information,” In: J. Filipe, H. Coelhas, M. Saramago (eds.) E-business and Telecommunication Networks, pp. 163–173, Springer Berlin Heidelberg, Berlin, Heidelberg(2007)
  * [8] Frantzeskou, G., Stamatatos, E., Gritzalis, S., Katsikas, S., “Source code author identification based on n-gram author profiles,” In: I. Maglogiannis, K. Karpouzis, M. Bramer (eds.) Artificial Intelligence Applications and Innovations, pp. 508–515. Springer US, Boston, MA (2006)
  * [9] Gray, A., Sallis, P., MacDonell, S., “Identified: A dictionary-based system for extracting source code metrics for software forensics,” In: Procceedings of SE: EI&P, pp. 252–259, IEEE Computer Society Press, Washington, DC (1998). DOI 10.1109/SEEP.1998.707658
  * [10] JR, Q., “C4.5: Programs for Machine Learning,” Morgan Kaufmann, San Mateo (1993)
  * [11] Kingma, D., Ba, J., “Adam: A method for stochastic optimization,” In: Procceedings of 3rd International Conference for Learning Representations, San Diego (2015)
  * [12] Kothari, J., Shevertalov, M., Stehle, E., Mancoridis, S., “A probabilistic approach to source code authorship identification,” In: Proceedings of International Conference on Information Technology: New Generations, IEEE (2007)
  * [13] Lange, R.C., Mancoridis, S., “Using code metric histograms and genetic algorithms to perform author identification for software forensics,” In: Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation, GECCO ’07, pp. 2082–2089. ACM, New York, NY, USA (2007). DOI 10.1145/1276958.1277364. URL http://doi.acm.org/10.1145/1276958.1277364
  * [14] Maclin, R., Opitz, D., “Popular ensemble methods: An empirical study,” Journal Of Artificial Intelligence Research 11, 169–198 (1999)
  * [15] McKinney, W., “Data structures for statistical computing in python,” In: S. van der Walt, J. Millman (eds.) Proceedings of the 9th Python in Science Conference, pp. 51–56 (2010)
  * [16] Mirza, O., Joy, M., “Style analysis for source code plagiarism detection,” In: Proceedings of International Conference on Plagiarism across Europe and Beyond, pp. 53–61. Brno, Czech Republic (2015)
  * [17] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E., “Scikitlearn: Machine learning in Python. Journal of Machine Learning Research 12, 2825–2830 (2011)
  * [18] Shevertalov, M., Kothari, J., Stehle, E., Mancoridis, S., “On the use of discretized source code metrics for author identification,” In: M.D. Penta, S. Poulding (eds.) Proceedings of 1st International Symposium on Search Based Software Engineering, pp. 69–78, IEEE Computer Society, Cumberland Lodge, Windsor, UK (2009)
  * [19] Tennyson, M.F., Mitropoulos, F.J. “A bayesian ensemble classifier for source code authorship attribution,” In: A.M.T. et al. (ed.) SISAP, LNCS, vol. 8821, p. 265276. Springer International Publishing, Switzerland (2014). DOI 10.1007/978-3-319-11988-525
  * [20] Yang, X., Xu, G., Li, Q., Guo, Y., Zhang, M., “Authorship attribution of source code by using back propagation neural network based on particle swarm optimization,” PLOS ONE 12(11), 1–18 (2017). DOI 10.1371/journal.pone.0187204. URL https://doi.org/10.1371/journal.pone.0187204
  * [21] Zhang, C., Wang, S., Wu, J., Niu, Z., “Authorship identification of source codes,” In: C. L., J. C., S. C., Y. X., L. X. (eds.) Proceedings of Asia-Pacific Web (APWeb) and Web-Age Information Management (WAIM) Joint Conference on Web and Big Data, Lecture Notes in Computer Science, vol. 10366, pp. 282–296, Springer, Cham, Switzerland (2017)