citnum

# A Survey on Artificial Intelligence for Source Code: A Dialogue Systems
Perspective

Erfan Al-Hossami  Samira Shaikh
Department of Computer Science
University of North Carolina at Charlotte
Charlotte, NC 28223
{ealhossa<EMAIL_ADDRESS>

###### Abstract

In this survey paper, we overview major deep learning methods used in Natural
Language Processing (NLP) and source code over the last 35 years. Next, we
present a survey of the applications of Artificial Intelligence (AI) for
source code, also known as Code Intelligence (CI) and Programming Language
Processing (PLP). We survey over citnum publications and present a software-
engineering centered taxonomy for CI placing each of the works into one
category describing how it best assists the software development cycle. Then,
we overview the field of conversational assistants and their applications in
software engineering and education. Lastly, we highlight research
opportunities at the intersection of AI for code and conversational assistants
and provide future directions for researching conversational assistants with
CI capabilities.

##  1 Introduction & Motivation

Conversational Assistants, also known as task-oriented dialogue systems, are
very widely used and accessible such as Siri and Alexa. These assistants have
been increasingly used to assist human users in a variety of tasks such as
reserving hotels, booking flights, or forecasting the weather. In recent
years, we have also seen advancements in the field of Artificial Intelligence
(AI) applied to source code also known as Code Intelligence (CI) and
Programming Language Processing (PLP). Github
Copilot111https://copilot.github.com/ powered by GPT-Codex Chen et al. (2021a)
is currently in beta as an Integrated Development Environment plugin that is
able to assist software developers as a pair-programmer by suggesting code
snippets or writing source code on its own. Given these advancements, perhaps
task-oriented bots can be equipped with more capabilities to assist humans in
cognitively demanding tasks, including programming by professionals or
novices. By building models and tools that can generate both language and
code, we could potentially better understand the cognitive basis of
programming which can have key impacts on computer science education practices
Fedorenko et al. (2019). This survey is structured as follows: Section §2
explores general deep learning techniques that have been used to model
language and source code over the last 35 years. Section §3 surveys the field
of CI with a systemic review (a) on datasets containing natural language and
executable code (§3.8.2), and (b) of all methods used to generate source code
from natural language on the CoNaLa dataset, a popular python code generation
benchmark (§3.8.4). Section §4 overviews the field of conversational
artificial intelligence and its applications in software engineering and
education. Finally, Section §5 highlights research opportunities at the
intersection of CI and conversational assistants to provide future directions
for research in this new area.

##  2 Deep Learning Methods

This section overviews major developments over the last 35 years in deep
learning neural architectures that are used to generate and understand both
natural language and source code. These deep learning methods can be used to
generate natural language (e.g. by chatbots Zhang et al. (2020)). In the
context of code generation, deep learning methods have been used to generate
code snippets (e.g. Liguori et al. (2021b, a); Frempong et al. (2021); Yin and
Neubig (2018)). The need for neural architectures arose as deep learning was
tasked to solve specific problems such as machine translation, question-
answering, and code generation. Neural architectures describe the general
structure of the neural network(s), including how many layers it has and how
units in these layers are connected to each other Sarkar et al. (2019). Neural
models can solve problems specific to the target domain through abstract
modeling. For instance, in natural language, often, word order matters to the
semantics of a sentence Payne (1992). Furthermore, word order differences
between languages can be problematic when translating from one language to
another Jurafsky and Martin (2000). How can neural architectures consider the
order of the input sequence? We explore relevant neural architectures further
in the following subsections.

###  2.1 Multilayered Perceptrons

Here we introduce the multi-layered perceptron (MLP) as well as common terms
associated with neural networks, including activation functions, loss
functions, and gradient descent, and backpropagation.

A multi-layered perceptron (MLP), commonly known as an artificial neural
network, is a variant of the Perceptron model Rosenblatt (1961), composed of
an input layer, multiple hidden layers fully connected with each other, and an
output layer (c.f. Fig. 1). MLPs excel at learning the input representations
and mapping them to the output variable. The input given to neural networks is
numerical vectors. Often, categorical inputs are one-hot encoded.

Figure 1: The Multilayer Perceptron architecture is composed of an input
layer, an output layer and multiple hidden layers. Figure from Yuen et al.
(2013).

Each layer is made up of several neurons. A neuron is a unit that takes in
weighted inputs, a bias (an input of 1.0 with a computed weight), and computes
an output using an activation function. Weights can be initialized to a
constant number, a random number, or through more nuanced means such as the
initialization proposed by He et al. (2015). Each neuron computes the dot
product of its input vector $Z=(z_{1},z_{2},...z_{n})$ and weight vector
$W=(w_{1},w_{2},w_{n})$ through $x=Z\cdot W$ . Once the dot product is
computed it is passed to an activation function (see table 1) before being
passed on as the output of the neuron. The process of neurons passing values
forward in the network using their activation functions until an output is
produced is called forward passing.

Activation Function | Definition
---|---
Binary Threshold | $f(x)=\left\\{\begin{matrix}1\;,if\;x>0\\\ \;0\;,otherwise\end{matrix}\right.$
Sigmoid | $f(x)=\frac{1}{1+e^{-x}}$
Tanh | $f(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}$
ReLu | $f(x)=\left\\{\begin{matrix}x\;,if\;x>0\\\ \;0\;,otherwise\;OR\;max\\{0,x\\}\end{matrix}\right.$
Table 1: Example activation functions used in neural networks

Lastly, another activation function called softmax is commonly used in the
output layer. Given a vector $z$ from the last hidden layer, the softmax
function produces a vector $\sigma(z)$ with length $K$. The numbers in
$\sigma(z)$ are values between 0 and 1. Furthermore, the sum of all elements
in $\sigma(z)$ equal to 1, $\sum_{j=1}^{K}\sigma(z)=1$. This property of
softmax is often useful in classification tasks. The softmax equation is
illustrated below:

$\sigma(z_{j})=\frac{e^{z_{j}}}{\sum_{k}e^{z_{k}}}$ (1)

where $j=1,2,...,K$.

Once the MLP network produces an output (prediction), it is compared with the
target values in the dataset (ground truth). This comparison yields a
measurement of how well the output matches the expected values. There are
several methods, called loss functions, to measure how well the predicted
output minimizes the error. Here are some examples of loss functions:

  * •

Binary cross-entropy: It is also called the log-loss applied for a two class
classification problem.

  * •

Categorical cross-entropy: log-loss applied for a classification task of N
classes

  * •

Mean Squared Error: Mean of the squared sum of error, often used in regression
tasks.

  * •

Mean Absolute Error: Average magnitude of the errors given a set of
predictions. It gives higher weights to larger errors since it squares the
errors.

Figure 2: The MLP network has different and independent weights at each layer.
$W_{h1}$ and $W_{h2}$ are different. Source: Medium post authored by Mady.

In the MLP, the computed error is propagated back through each of the layers.
This is done so that the weights for each neuron can be updated empirically.
This process is called back-propagation Rumelhart et al. (1986). The weights
can be updated after each training example, in a process called online
learning. This process can result in big changes in the network from example
to example and is largely unstable. Alternatively, computed errors can be
saved for a set of training examples and the MLP network weights are updated
after training on the set. This is called batch learning. The learning rate
$\alpha$ controls the size of a change made to a weight, so the weights do not
change dramatically and learning can be more stable.

MLP networks are trained using a training algorithm such as stochastic
gradient descent (SGD) Bottou (2010).

SGD is a simplified version of gradient descent that computes the gradient
descent for only a small sample of training data to estimate the gradient
descent given a loss function for the entire training dataset. These
optimization algorithms are responsible for updating weights in a model given
a training set sample and a loss function. Other algorithms are commonly used
to further optimize SGD such as Adaptive Moment Estimation (Adam) Kingma and
Ba (2014a). Adam further optimizes SGD by computing adaptive learning rates
for each parameter. MLPs excel at mapping inputs to outputs, however, these
inputs are assumed to be independent of one another. What if the inputs are
dependent? What if their order matters such as in word sequences forming
sentences? Next, we introduce another neural network architecture designed
specifically to address this problem, the Recurrent Neural Networks.

###  2.2 Recurrent Neural Networks

Figure 3: The Recurrent Neural Network architecture is composed of an input
layer $x$, an output layer $o$, a hidden layer $h$, and a feedback loop $v$
for each time step $t$. The weights $v,u,w$ are not different for each layer.
The weights are instead shared across all the time steps. Source: Wikipedia.

Recurrent Neural Networks (RNNs) are specialized network architectures for
processing a sequence of input values that are dependent upon one another.
RNNs are often applied in predicting the next word/token in a sequence of
words, or in translating from one language to another. RNNs follow an
architecture distinct from that of MLPs. RNNs are equipped with a feedback
loop. This feedback loop enables the RNN to share weights $v,u,w$ across
different time steps, meanwhile, MLPs use a different set of parameters and
weights each at each hidden layer. This is illustrated in Figure 2 where
$W_{h1}$ and $W_{h2}$ are different. We can imagine that the RNN is performing
the same computations in each time step, but with different inputs and updated
weights. This setup greatly reduces the number of parameters required to learn
a task. Figure 3 showcases the RNN architecture with a single hidden unit.
When unfolding the RNN, we are left with multiple feedforward MLP networks, a
network at each time step $t$. The total number of time steps is equivalent to
the length of the input sequence.

While MLPs compute the hidden layer values from the input values and weights
exclusively, the hidden layers in RNNs are computed from the input value
$x_{t}$ at time step $t$, weights, and the previous hidden layer value. This
is illustrated below:

$\begin{matrix}h_{t}=\sigma(u*x_{t}+v*h_{t-1})\\\
o_{t}=softmax(w*h_{t})\end{matrix}$ (2)

where $v$ is a weight vector for the different time steps, $\sigma$ as an
activation function as described in Table 1, $x_{t}$ is an input value at time
step $t$, $u$ is a weight vector for the hidden layer, $w$ is a weight vector
for the output layer, and $o_{t}$ is the output value at time $t$. Softmax is
described in Equation 1 and it is used on the product of the hidden state
$h_{t}$ and the weight vector $w$. The error is computed using a loss function
$E$ (loss functions are described in section 2.1) for output $o_{t}$ and
target sequence $s_{t}$. This yields an error $E_{t}$. As for the total error
across the different timestamps it is computed through summing all of $E_{t}$
for an input length of $n$ $\sum_{t=0}^{n}E_{t}$. The weights are updated
using back-propagation Rumelhart et al. (1986) as described in section 2.1,
with a small tweak, the current time step is dependent on the previous time
step, so the back-propagation traverses back to the first time step. u,v, and
w are updated using SGD Bottou (2010). There is a big drawback for RNNs
however, as we go back to adjust the weights during back-propagation, the
signal gets close to zero this is called vanishing gradient, or it grows
exponentially, this is called exploding gradient. This is an issue in
particular when we expect RNNs to ’remember’ and keep track of long-term
dependencies. For that researchers have developed a special type of RNNs, the
Long-Short Term Memory (LSTM).

###  2.3 Long-Short Term Memory

Recurrent Neural Networks (RNNs) lose historical context and dependencies over
longer periods. To address the problem of memory in RNNs Hochreiter et al.
(1997) proposed a novel RNN architecture called the Long-Short Term Memory
(LSTM) in 1997. It excels at maintaining long-term dependencies and preventing
problems such as the exploding and vanishing gradient issues. LSTMs are a
variant, or a type, of RNNs. Their similarity is denoted in Figure 4.

Figure 4: The Long-Short Term Memory (LSTM) network is similar to the
Recurrent Neural Network (RNN) except the hidden state is replaced with an
LSTM cell. Similarly to Figure 3 $n$ represents a timestep, $x^{(n)}$
represents the input at time $n$, $U$ represents a weight vector for the LSTM
cell, $W$ represents the weight vector in the feedback loop, $C$ represents
the LSTM cell state, $V$ represents the weight vector for the output layer,
and $o^{(n)}$ represents the output at time $n$. Source: CodeEmporium on
Youtube. Figure 5: The LSTM is composed of 3 different gates: 1) an input gate
controlling what is updated in the cell state, 2) a forget gate controlling
what is omitted from the previous cell states (memory), and 3) an output gate
controls what is passed on as the final output of the cell.

LSTMs contain gates which enable the LSTM to add or remove information to a
cell state. Each gate is like a one-layered neural network with each with its
weight vector and biases. These networks then learn what information is
relevant and what is not throughout training. LSTMs consist of three gates:
The input, output, and forget gates. As shown in Figure 5, an LSTM cell is
composed of 3 different gates. At a given time step, the input gate allows or
denies incoming signals from the input layer. The output gate controls what
gets passed on to other cells or layers as the cell’s final output. The forget
gate administers the self-recurrent connection to forget or remember previous
cell states as deemed fit.

Figure 6: The Long-Short Term Memory (LSTM) cell architecture. At teach time
step $t$, an LSTM cell takes in the previous cell state and the previous
hidden state as inputs. It contains multiple sigmoid and tanh functions to
compute on those inputs for each of the 3 gates (input, forget, and output).
The LSTM cell outputs, a cell state $C_{t}$, and a hidden state $h_{t}$ that
is passed onto other layers to produce an output $o_{t}$, and to LSTM cell’s
future self at time step $t+1$. Source: Colah’s blog.

The LSTM cell architecture is displayed in Figure 6. Let $C$ indicate a cell
state, $h$ the hidden state for each time step $t$. Let $I$, $F$, and $O$,
represent the input, forget, and output gates respectively. All gates are
regulated by a sigmoid $\sigma$ function, which as mentioned earlier outputs a
number between 0 and 1, controlling how much of the outputs of each gate
should get passed along to the cell state. The processes of LSTM computations
can be illustrated in four steps:

  1. 1.

Deciding what to forget and what to remember: This decision is made by the
forget gate at each time step. The forget gate utilizes information from the
previous hidden state $h_{t-1}$ , the current input $x_{t}$, its own weight
vector $W_{F}$, and its own bias $b_{F}$. $h_{t-1}$ and $x_{t}$ are
concatenated and they are multiplied in a dot product with $W_{F}$ and summed
with $b_{F}$ . The result of that is then squashed through the sigmoid
function $\sigma$. The result is $F_{t}$ a vector of numbers between 0 and 1,
corresponding to each number in the previous hidden state and current input
vectors, indicating how much should be remembered (if the value is greater
than 0) and forgotten (if the value is 0). The process is described in
mathematical notation in equation 3.

  2. 2.

Deciding what to write to memory: This decision is handled by the input layer.
The input layer utilizes information from the previous hidden state $h_{t-1}$
, the current input $x_{t}$, its own weight vector $W_{I}$, and its own bias
$b_{I}$. $h_{t-1}$ and $x_{t}$ are concatenated and then are multiplied in a
dot product with $W_{I}$ and summed with $b_{I}$ . The result of that is then
squashed through the sigmoid function $\sigma$. The result is $I_{t}$ a vector
of numbers between 0 and 1, corresponding to each number in the previous
hidden state and current input vectors, indicating where should new
information should be written to the current cell state (memory). Next,
another layer $\tilde{C_{t}}$, decides what new information are good
candidates to be written to the current cell state using tanh as its
activation function. The process is described in mathematical notation in
equation 4.

  3. 3.

Updating the memory: The current cell state is decided by utilizing the forget
gate vector output $F_{t}$, erasing things that are determined to be
forgettable from the previous cell state $C_{t-1}$, and the new information
from step 2 is added after scaling by the input gate vector $I_{t}$. The
process is described in mathematical notation in equation 5.

  4. 4.

Output: The final output for the LSTM cell is decided by the current input
$x_{t}$, and the current cell state $C_{t}$. The output gate vector $O_{t}$
decides which parts of the cell state are going to be outputted and is
computed using the previous hidden state $h_{t-1}$ and the current input
$x_{t}$. Then, the current cell state $C_{t}$ is passed through a tanh
function and multiplied by the output gate vector giving us $h_{t}$ which is
passed on as the output of the LSTM to other cells and a copy to itself. The
process is described in mathematical notation in equation 6.

$F_{t}=\sigma(W_{F}\dot{[}h_{t-1},x_{t}]+b_{F})$ (3)

$\begin{matrix}I_{t}=\sigma(W_{I}\dot{[}h_{t-1},x_{t}]+b_{I})\\\
\tilde{C_{t}}=tanh(W_{C}\dot{[}h_{t-1},x_{t}]+b_{C})\end{matrix}$ (4)

$C_{t}=F_{t}*C_{t-1}+I_{t}*\tilde{C_{t}}\\\ $ (5)

$\begin{matrix}O_{t}=\sigma(W_{O}\dot{[}h_{t-1},x_{t}]+b_{O})\\\
h_{t}=O_{t}*tanh(C_{t})\end{matrix}$ (6)

LSTM hidden states (as seen in figure 6) can be connected to a softmax layer
for text classification problems or it can be connected to another LSTM cell
composing several LSTM layers. These LSTM layers would output a sequence of
hidden state vectors, a vector for each input in a timestamp. This hierarchy
of LSTMs layers enables a more complex representation of sequential data and
it is often referred to as stacked LSTMs. In the next section, we discuss an
architecture that often utilizes stacked LSTMs to generate language.

###  2.4 Sequence-to-Sequence

Figure 7: A Sequence to Sequence architecture is composed of an encoder and
decoder LSTM networks. Source: Jalammar’s Blog.

In this subsection, we describe the neural sequence-to-sequence (Seq2Seq)
architecture. It is used to generate a sequence of tokens in a target language
given a sequence of tokens from a source language. Seq2Seq learning maximizes
the likelihood (or potentially other evaluation metrics Shen et al. (2016)) of
the target sequence (conversational response) given an input source sequence
(user utterances).

Typically, Encoder-Decoder architectures using RNNs and applied to solve
Seq2Seq tasks were first proposed by Cho et al. (2014) and Sutskever et al.
(2014). In practice, gated RNNs such as LSTMs Hochreiter and Schmidhuber
(1997) tend to perform better over vanilla RNNs Gu et al. (2016a).

Notably, Bahadanau et al. (2014) propose an encoder-decoder architecture with
attention, using a bi-directional LSTM as the encoder to transform an embedded
source sequence $E=|e_{1},...,e_{T_{S}}|$ into a vector $c$ of hidden states
with equal length. This vector is known as the context vector or the thought
vector.

Each hidden state $h_{t}$ corresponds to an embedded token $e_{t}$ Each hidden
state is computed by concatenating the hidden states of the forward and
backward orders as follows:

$\begin{matrix}\textbf{h}_{t}=[f(e_{t},\textbf{h}_{t-1});f(e_{t},\textbf{h}_{t+1})];\\\
\\\ \textbf{c}=\phi(\\{\textbf{h}_{1},...,\textbf{h}_{T_{S}}\\})\end{matrix}$
(7)

where $\\{h_{t}\\}$ are hidden LSTM states, $f$ is a standard LSTM update
function, and $\phi$ summarizes the hidden states. The decoder LSTM (recurrent
neural network (RNN) architecture) Luong et al. (2015a) generates the target
token $a$ using the conditional probability defined in Eq. 8 which takes in
the context vector $c$ and generates target token $a$ at time $t$ using the
following conditional probability:

$\begin{matrix}\textbf{s}_{t}=f(a_{t-1},\textbf{s}_{t-1},\textbf{c})\\\ \\\
\>\>\>\>\>\>p(a_{t}|a_{<t},E)=g(a_{t-1},\textbf{s}_{t},\textbf{c}))\end{matrix}$
(8)

where $s_{t}$ is the decoder LSTM state at time $t$, $a_{t}$ is the
conversational response token at $t$ using function $g$. $a_{<t}$ denotes
previous predicted tokens $\\{a_{1},...,a_{t-1}\\}$.

Figure 8: The sequence to sequence attention mechanism computing attention
weights for each of the hidden states from the encoder network forming the
context vector $c_{t}$ at time $t$ that is used by the decoder LSTMs. Figure
adapted from Luong et al. (2015b). Adapted by: SyncedReview on Medium.

The context vector by itself is of a fixed length and hence it struggles with
long sequences or sentences. To address this, the Seq2Seq model often comes
with an attention mechanism. A noteable attention mechanism is the Bahdanau-
style attention mechanism described in Bahdanau et al. (2014), in which it
considers a dynamic context $\textbf{c}_{t}$ in the decoding process. This
attender attends to the entire input space (i.e. soft attention) by
representing $\textbf{c}_{t}$ as the weighted sum of the source hidden states
as follows:

$\begin{matrix}\mathbf{c}_{t}=\sum_{i=1}^{{T_{S}}}\alpha_{t,i}\>\textbf{h}_{i}\\\
\\\ \alpha_{t,i}=softmax(score(\textbf{s}_{t-1},\textbf{h}_{i}))\\\ \\\
score(\textbf{s}_{t},\textbf{h}_{i})=v_{z}^{\top}tanh(\textbf{W}_{z}[\textbf{s}_{t};\textbf{h}_{i}])\end{matrix}$
(9)

where $\alpha_{t,i}$ represents how well a target token $a_{t}$ and source
token $e_{i}$ align, $T_{S}$ represents the last hidden state in $c$, $s_{t}$
is the decoder LSTM state at time $t$, and lastly, $v_{z}$ and $W_{z}$ are
weight matrices to be learned by the alignment model. The score $\alpha_{t,i}$
is parametrized by a feed-forward multi-layer perceptron neural network. Since
the encoder uses a bi-directional LSTM each hidden state $\textbf{h}_{i}$ is
aware of the context on both ends.

In this section, we looked at the Seq2Seq model that uses an encoder LSTM
network that processes input into a context vector and a decoder LSTM network
that produces the output one token at a time. We noted how attention
mechanisms weigh tokens based on how important they are at the current time
step. In the next section, we discuss a novel architecture, that still manages
to perform well on very long sequences and is not as vulnerable to over-
fitting as LSTMs.

###  2.5 Transformers & Transfer Learning

In this section, we discuss the Transformer architecture proposed by Vaswani
et al. (2017). In the previous section, we’ve touched on the attention
mechanisms and how they supplemented RNNs in Seq2Seq models to model longer
sequences. The Transformer architecture takes attention a step further to
improve state of the art. In the infamous paper Attention is All you Need,
Vaswani et al. (2017) propose a new encoder-decoder architecture with a novel
attention mechanism. The novel attention mechanism called the multi-head
attention not only enables models to model longer sequences of text more
effectively. The Transformer model also opens the door for transfer learning
in natural language processing and generation.

Figure 9: Overview of the Transformer architecture proposed in Vaswani et al.
(2017). It is composed of embeddings, positional encoders, stacked encoders
(left), and stacked decoders (right),

Transfer learning is a situation where what has been learned in one setting is
exploited to improve generalization in another setting. Pan et al. (2010)
define transfer learning as follows, consider a domain $D$ consisting of a
feature space $\chi$ and a marginal probability $P(X)$ and
$X=\\{x_{1},...,x_{n}\\}$. Here, $x_{i}$ is a specific vector and $X\in\chi$ .
Consider task $T$, consisting of a label space $\gamma$ and an objective
function $P(\gamma|X)$. Transfer learning is the process aimed at improving
target task $T_{T}$ in the target domain $D_{T}$ using knowledge from the
source task $T_{S}$ in the domain $D_{S}$. Inductive learning is inferring
mapping from a set of training data samples. The trained model then works with
inductive bias, a set of assumptions related to the training data used to
generalize on unseen data. Deep learning algorithms use inductive transfer
techniques that utilize inductive biases of the source task to assist in the
target task. To apply transfer learning, several methods are used: 1) Feature
extraction, using pre-trained networks without its final layer as a feature
extractor for other tasks. 2) Fine-tuning, where selected layers are re-
trained from a pre-trained model, while other layers are frozen (weights are
fixed). 3) Pretrained models, are models that are trained and perform well on
a source task $T_{S}$, and the model weights and parameters are saved and then
re-trained to perform a target task $T_{T}$. There are several types of
transfer learning: 1) Domain adaptation, noted when there is a data shift
between the source and target domains, usually the marginal probabilities
$P(X_{S})\neq P(X_{T})$. An example of domain adaptation can be training a
pre-trained sentiment classifier for movie reviews to classify product review
sentiments. 2) Domain confusion aims to make the domain representations as
similar as possible by applying pre-processing steps on the representations.
The basic idea is to add another objective to the source model to encourage
similarity between the domains by confusing the domain itself Ganin et al.
(2016). 3) Multitask learning, where the model receives information about
multiple tasks simultaneously. 4) One and few-shot learning, where the model
infers the required output based on one or a few training examples
respectively Fei-Fei et al. (2006). 5) Zero-shot learning, where the model
relies on no examples to learn a task.

The Transformer architecture is composed of 6 stacked encoders and 6 stacked
decoders along with embeddings for both the inputs and outputs and positional
encoders. Figure 9 overviews the proposed architecture.

We will first start by describing the encoders and decoders. The encoder
receives inputs and passes them through the multi-headed attention. Multi-
headed attention in each encoder is a type of self-attention. Self-attention
helps the encoder map relationships between the current input token and other
words in the input sequence as it encodes a specific word or a token Cheng et
al. (2016). Then the output of the attention is fed into a feedforward (MLP)
neural network. The output of each encoder is fed into another encoder until
it reaches the last encoder, the sixth encoder. Each decoder receives the
output of the sixth encoder and the target language input. It has both a feed-
forward network and multi-headed attention but it also has another attention
mechanism, the masked multi-headed attention. The masked multi-headed
attention mechanism is also a self-attention mechanism, which takes in the
output sequence as its input. It is modified to prevent attending to
subsequent tokens/words so that predictions for the current timestep are only
dependent on previously known output tokens. All the produced matrices are
added with either the previous layer or the embedding and normalized according
to Ba et al. (2016) in between as indicated in Figure 9.

The multi-headed attention has an overall algorithmic complexity of $O(N^{2})$
and is made up of 8 heads $h=8$ running in parallel. Each token in a sequence
has an embedding vector. An embedding is a numerical form of a word that
preserves its meaning by using dimensionality reduction and maintaining
context. The paper uses 512 dimensions for the embedding size
($d_{model}=512$). Each attention head takes in 3 input vectors for each input
token: the query vector $Q$, the key vector $K$, and the value vector $V$.
Each of these vectors has a dimension of 64. These vectors are computed by
multiplying the embedded input token with 3 different weight matrices:
$W^{Q}$, $W^{K}$, and $W^{V}$ respectively. The output of the attention head
is computed as a weighted sum of the values vector. $d_{k}$ is the dimension
of the key vectors which is 64.

$Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V$ (10)

To compute the attention for each head we start by computing the dot product
of the query and the transposition of the key vector. The output is then
scaled by $\sqrt{d_{k}}$ which is 8. Next, the softmax function is applied so
that all the values are between 0 and 1. Lastly, we compute the dot product of
the softmax output with the values vector. The process is called the Scaled
Dot-Product Attention and is described in Equation 10.

To compute the multi-head attention we take the output from each attention
head, concatenate them, and multiply them with a bigger weight matrix $W^{O}$
as described in Equation 11.

$MultiHead(Q,K,V)=Concat(head_{1},...,head_{h})W^{O}$

(11)

The weight matrices $W^{Q}$, $W^{K}$, $W^{V}$, and $W^{O}$ are all learned and
updated throughout the training process which greatly assists in the process
of transfer learning. Each of the 8 attention heads can focus on a different
relationship between the input words (e.g., one for pronoun-noun, one for
gender, etc.)

Another component is positional encoding applied to both the input and output
sequences. The positional encoding has a dimension of 512 which is the same as
the embeddings. Positional encodings provide a vector to be added to the input
embedding. This provides fixed positioning of tokens during training which
allows further extrapolation of longer sequences. The paper uses sinusoidal
version of positional embeddings described in Equation 12, where $pos$ is the
position and $i$ is the dimension. Each dimension of a positional encoding
corresponds to a sinusoid which goes from $2\pi$ to 10000.

$\begin{matrix}PE_{(pos,2i)}=sin(pos/10000^{2i/512})\\\
PE_{(pos,2i+1)}=cos(pos/10000^{2i/512})\end{matrix}$ (12)

There are many applications of the Transformer model and it is not restricted
to the field of NLP. We note in particular BERT Devlin et al. (2018) which
advanced state of the art for language modeling and GPT-3 Brown et al. (2020)
which advanced language generation. In the next section, we will discuss the
GPT models.

Figure 10: The GPT-1 architecture proposed in Radford et al. (a). It is
composed of 12 stacked decoders. Each decoder containing self-attention,
followed by a position-wise feed-forward network with normalization in
between.

###  2.6 GPT Models

In the section, we describe a specific transformer architecture, the
Generative Pretrained Transformer family of models.

####  2.6.1 GPT-1

Prior to GPT-1 most NLP models required annotated corpora for natural language
understanding. Annotated data for specific natural language understanding
(NLU) tasks are scarce while large unlabeled corpora are abundant. GPT-1, or
Generative Pre-trained Transformer 1, introduces the concept of generative
pre-training on large unlabeled text and then fine-tunes for specific NLU
tasks Radford et al. (a). GPT-1 demonstrates significant gains in several NLU
tasks and marks a significant step towards the trend of task-agnostic NLP
models. Next, we describe the semi-supervised learning process (pre-training
then fine-tuning) more in detail.

The unsupervised pre-training is a language model is tasked with maximizing
the following objective for a corpus of tokens $U={u_{1},...,u_{n}}$:

$L_{1}(U)=\sum_{i}logP(u_{i}|u_{i-k},...,u_{i-1};\Theta)$ (13)

where $k$ is the size of the context window, P is a conditional probability
modeled by a neural network with parameters $\Theta$. The parameters are
trained using stochastic gradient descent (SGD) described earlier in 2.1.

The fine-tuning training (supervised training) takes in a labeled dataset $C$
with a sequence of tokens ${x_{1},...,x_{m}}$, and a label $y$ for the token
sequence and aims to maximize the objective given the pre-training objective
$L_{1}$, and a weight $\lambda$ set to 0.5 in the paper:

$\begin{matrix}L_{2}(C)=\sum_{(x,y)}logP(y|x_{1},...,x_{m})\\\
L_{3}(C)=L_{2}(C)+\lambda L_{1}(C)\end{matrix}$ (14)

Using the language modeling (pre-training) objective helps with improving the
supervised model’s generalization and accelerating convergence Radford et al.
(a).
The GPT Architecture  is composed of 12 stacked transformer decoder layers
(see the decoder component in Figure 9), with multi-headed masked self-
attention. The decoder layer had 12 attention heads, the embedding size is 768
($d_{model}=768$), an Adam optimizer Kingma and Ba (2014b) was used with a
learning rate of $\alpha=2.5\times 10^{-4}$, and a drop out rate of 0.1. As
shown in Figure 10, the model applies its multi-headed masked self-attention
on the input tokens. Following attention, are position-wise feed-forward
layers generate a distribution over the target tokens. The dimension of the
position-wise feed-forward layer is 3072. During pre-training, the output
distribution $P(u)$ for a token $u$ is computed through the following:

$\begin{matrix}h_{0}=UW_{e}+W_{p}\\\
h_{i}=\textrm{decoder}(h_{i-1})\forall{i}\in{[1,n]}\\\
P(u)=\textrm{softmax}(h_{n}W^{T}_{e})\end{matrix}$ (15)

Here, $U={u_{k},...,u_{1}}$ is the context vector of tokens starting from $k$
the size of the context window, $n$ is the number of layers (12 layers), and
$W_{e}$ is the token embedding matrix, and $W_{p}$ is the position embedding
matrix. We observe that the first decoder layer $h_{0}$ takes in the embedding
matrix and position embedding matrix. Subsequent decoder layers take in the
previous layer, and softmax is applied at the last transformer layer and
maximizes for Equation 13. When fine-tuning GPT-1, input tokens
$x^{1},...,x^{m}$ which come along with label $y$ are passed through the pre-
trained model to obtain the last decoder activation $h^{m}_{l}$. It is then
passed to the linear layer (shown in Figure 9) with parameters $W_{y}$ to
predict $y$ and maximizes for Equation 14 :

$P(y|x^{1},...,x^{m})=\textrm{softmax}(h^{m}_{l}W_{y})$ (16)

The model was trained for 100 epochs with mini-batches of 64 and a sequence
length (context window) of 512 and used GELU (Gaussian Error Linear Unit)
Hendrycks and Gimpel (2016) as its activation function. Byte Pair Encoding
(BPE) Sennrich et al. (2016) with a vocabulary size of 40,000 was used. GPT-1
had 117 million parameters in total. The Bookcorpus dataset Zhu et al. (2015)
was used for the pre-training stage. GELU is approximated by the following
using $tanh$ described in section 2.1:

$\begin{split}GELU(x)=0.5x(1+tanh[\\\
\sqrt{2/\pi}(x+0.044715x^{3})])\end{split}$ (17)

Figure 11: Figure from Brown et al. (2020). Shows GPT-3 with different number
of parameters making more use of context (K) on a simple task requiring the
model to remove random symbols from a word.

####  2.6.2 GPT-2

The Generative Pre-trained Transformer 2 (GPT-2) serves as the first attempt
at creating a pre-trained language model that can perform downstream tasks
without any fine-tuning or architectural change. This is also known as the
zero-shot setting. To achieve this goal, GPT-2 updates the objective function
from the pre-training objective $P(output|input)$ to include a particular task
$P(output|input,task)$ Radford et al. (b). This is called task conditioning.
With task conditioning GPT-2 can produce different outputs for the same input
given a different task. The task is passed as a natural language description
(sequence of tokens describing the task). GPT-2 is trained using a dataset
called WebText222https://github.com/jcpeterson/openwebtext which is around 40
GB in size Radford et al. (b). GPT-2 experiments highlighted that the
perplexity (intrinsic evaluation) of language models on the same dataset
decreases the bigger the language model in terms of parameters.
The GPT-2 Architecture  follows the architecture of GPT Radford et al. (a)
with a few notable differences. GPT-2 is made up of 48 stacked transformer
decoder layers. The normalization layers were moved to the input of the self-
attention and feed-forward sub-blocks. There is also an additional
normalization layer was added after the final self-attention block. Weight
initialization was also modified by a factor of $1/\sqrt{N}$ where $N$ is the
number of residual layers. The word embedding dimension was increased to 1600
($d_{model}=1600$). Byte Pair Encoding (BPE) Sennrich et al. (2016) with an
increased vocabulary size of 50,257 was used. The model was trained with a
larger batch size of 512 and a larger context window of 1024. This resulted in
a total of 1.5B parameters for GPT-2 which is almost 10 times bigger than
GPT-1.

####  2.6.3 GPT-3

Large transformers such as BERT Devlin et al. (2018), and GPT-2 Radford et al.
(b) are task agnostic in architecture, however, they require fine-tuning on
specific tasks using thousands of examples. In a new transformer-based
architecture, researchers at OpenAI explore greatly scaling up transformer-
based language models to explore its task-agnostic performance using few-shot
and zero-shot learning. GPT-3, or Generative Pre-trained Transformer 3, is a
massive auto regressive transformer-based model that underwent unsupervised
pre-training on large corpora (over 300 billion tokens of text or 45 TB). The
data sources include 3 years of filtered CommonCrawl Wenzek et al. (2020),
Wikipedia, and WebText. GPT-3’s pre-training objective is to predict the next
word given a sequence of words. GPT-3 is massive, containing about 175 billion
parameters, making it one of the world’s largest pre-trained transformer
models for language generation in the world. GPT-3 is used without gradient
updates nor fine-tuning but rather with few-short demonstrative interactions
with the model. Researchers have tested the model on various NLP tasks
including question answering, translation, and cloze tasks competing with
state-of-the-art models without fine-tuning. GPT-3 generates language one
token at a time, just like previously mentioned models, given a sequence of
tokens as input in many different domains. Upon its release to the public
enthusiasts have used GPT-3 for many different applications. These
applications include mainstream tasks such as translating into different
languages, writing google ads and has been particularly used to generate code
from natural language such as SQL, Bash, and Python. For a thorough list of
GPT-3 applications please refer to this333https://github.com/elyase/awesome-
gpt3.
The GPT-3 Architecture is the same as that of a GPT-2 model Radford et al.
(b). Similar to the GPT-2 Radford et al. (b), the GPT-3 model is composed of
stacked transfomer Decoder layers (see the decoder component in Figure 9) and
it includes the modified initialization, pre-normalization, and reversible
tokenization. GPT-3 is different from GPT-2, in that it uses alternating dense
and banded sparse attention patterns in the transformer layers proposed in the
Sparse Transformer paper Child et al. (2019). Open AI researchers have
experimented with different GPT-3 configurations, sizes, and parameters. An
example experiment is shown in Figure 11. The final GPT-3 model uses 96
transformer decoder layers, 96 attention heads, the dimension of the model
($d_{model}=12888$), dimension of each attention head ($d_{head}=128$), batch
size is 3.2M, and $\alpha=0.6\times 10^{-4}$.

##  3 Code Intelligence (CI)

With the availability of big code from public sources like GitHub and
StackOverflow, an opportunity to use data-driven tools and algorithms comes to
light. This opens the door for machine learning, and NLP techniques, in
particular, to be used for various applications on source code. Such
applications tend to improve the software development process and are gaining
attention from both AI and software engineering academics. Applications of
machine learning and NLP techniques on source code includes applying
statistical models Oda et al. (2015a), neural networks such as LSTMs
Rabinovich et al. (2017), and more recently pre-trained transformers including
CodeBERT Feng et al. (2020), CodeGPT Svyatkovskiy et al. (2020), Codex Chen et
al. (2021a) which are respectively variants of BERT Devlin et al. (2018),
GPT-2 Radford et al. (b), and GPT-3 Brown et al. (2020) pretrained on source
code . Recently, researchers at Microsoft have published an international
benchmark CodeXGLUE Lu et al. (2021a) for Code Intelligence, similar to
existing NLP benchmarks such as GLUE Wang et al. (2018) and SQuaD Rajpurkar et
al. (2018, 2016). Another benchmark worth mentioning is GLUECode Karmakar et
al. (2020). GLUECode evaluates machine learning models on local and global
understanding evaluation of source code. Different from CodeXGLUE, GLUECode
contains two tasks for “local reasoning" in which models are tasked with
showing understanding of codes on a local level. The first task is operator
prediction, in which a model is tasked with predicting a masked operator
including boolean comparison operators and arithmetic operators. The second
task is NPath prediction, where models are tasked with counting the number of
distinct paths control flow can take in a method. We categorize works in NLP
on source code fall into a set of areas. The areas include Program Synthesis,
Program Analysis, Program Repair, Clone Detection, Defect Detection, Cloze
Testing, Code Translation, Code Refinement, Code Completion, Code Search, Code
Documentation Generation, Documentation Translation, Semantic Parsing, and
Code Generation. This section will overview each of these areas and highlight
notable works. We focus on Code Generation and semantic parsing more in
particular for this survey.

###  3.1 Program Synthesis

Artificial intelligence research has long aimed at having computers synthesize
their programs Manna and Waldinger (1971); Waldinger and Lee (1969). Program
Synthesis involves constructing full or partial programs that complete a
specified task Allamanis et al. (2018); Gulwani et al. (2017). Program
specifications are taken by the program synthesizer model as input and an
executable program is synthesized by the model as the output. Program
specifications are traditionally expressed with formal logic statements. Other
program specifications include providing the model with example input/output
pairs and more recently natural language descriptions of what the program
needs to accomplish. In program synthesis, if the program specifications
include natural language descriptions it is accompanied by another
specification type such as input/output pairs Austin et al. (2021); Shi et al.
(2020); Ye et al. (2020b). If the program specification is in natural language
only then it would be considered a semantic parsing task which is described in
§12. Program synthesis works usually synthesis programming using restricted
domain-specific languages (DSLs) Gulwani (2011) or languages that have more
features but are still designed with program synthesis in mind Odena and
Sutton (2020); Ellis et al. (2021). More recently, program synthesizers have
started to synthesize programs in general-purpose languages such as Python Shi
et al. (2020); Austin et al. (2021).

Program synthesis approaches include applying both symbolic Manna and
Waldinger (1975, 1980) and neuro-symbolic Parisotto et al. (2016) techniques
Balog et al. (2016); Odena and Sutton (2020); Devlin et al. (2017). Symbolic
approaches are also known as rule-based program synthesis Manna and Waldinger
(1980) focus on deriving a program from a well-defined specification using a
set of formal grammar rules. A drawback of the symbolic approach is it often
relies on using formal logic to describe the program specification, however
writing the formal logic to describe the program is often more difficult than
writing the program itself. To remedy this issue, modern program synthesis
models often learn from input/output examples. A recent notable work utilizing
input/output examples in various domains is DreamCoder Ellis et al. (2021).
DreamCoder learns to synthesize programs in a domain-specific language using
input and output examples to solve inductive programming tasks such as list
processing and text editing, creative tasks such as drawing pictures and
building scenes and to solve classical physics problems. DreamCoder uses a
wake-sleep learning algorithm. In its, sleep DreamCoder goes through to learn
concepts from combinations of DSL primitive operations improving its library
of operations in the process. When DreamCoder is awake it uses operations from
its library to solve problems given input and output examples. There are also
notable works that synthesize programs in Python, a general-purpose
programming language, which has the potential to power tools to further enable
programmers. Austin et al. (2021) explore the limits of using a large
pretrained transformer-based language model to synthesize basic python
programs to solve entry-level programmer tasks and math problems given
input/output examples and a natural language description. The authors note
that large language models enable program synthesis models to consider various
types of program specifications such as natural language. Furthermore, the
authors demonstrate that such language models can reduce the error rate of
synthesized programs by up to 50% when engaging in dialogue with a human and
incorporating human feedback given on the synthesized code. Shi et al. (2020)
propose TF-Coder, a program synthesizer for TensorFlow, a well-known deep
learning library by Google. TF-Coder uses input/output examples and natural
language description as program specifications and generates Python code in
the TensorFlow library to solve real-world tensor manipulation problems
achieving superhuman performance.

Program synthesis models are generally evaluated based on how many compilable
(executable) programs are synthesized. They are also evaluated based on how
many programs are logically correct, i.e. match the specification constraints
or solve the programming problem by passing pre-programmed unit tests Austin
et al. (2021). Other metrics also include how long it takes for the model to
synthesize the program, and duration speedup compared to human programmers.
Recent work by Schuster et al. (2021) proposes a dataset, programming puzzles
(P3), to evaluate program synthesis models. Program synthesis models are
tasked with predicting an input $x$ to a short Python program $f$ with makes
$f$ output True.

Although program synthesis generally assumes that the program compiles with
some specification, some notable works utilize program synthesizers to
generate random but functioning programs for benchmarks and compiler fuzzing.
Cummins et al. (2017a) synthesize a large number of programs used as
benchmarks in OpenCL code. Fuzzing is a popular method that creates structured
data samples that are used as inputs to test complex software such as
compilers and web applications. Patra and Pradel (2016) synthesize programs
using TreeFuzz in JavaScript using a corpus of example structured data (in
this case example program outputs). These programs generate random structured
data for fuzz-testing JavaScript interpreters. Program Synthesis has been
applied to build usable products in fields including data science Drosos et
al. (2020), to assist data scientists in wrangling data using Python, and
general software engineering to generate code edit suggestions based on
repetitive changes Miltner et al. (2019).

###  3.2 Program Analysis

Computers can execute programs but they do not necessarily understand what the
programs do or mean. When analyzing program source code, we can
computationally estimate a program’s behavior, functionality, complexity,
meaning, etc. Program analysis focuses on extracting semantic properties from
programs. In a sense, it is similar to natural language understanding (NLU) in
natural language processing where both fields focus on comprehending a snippet
of. Probabilistic models of source code compose a notable family of models to
analyze programs and extracting properties. Sometimes codebases contain
uninformative variable names such as e and which makes it difficult for humans
to comprehend code. This can be remedied by refactoring codebases to include
descriptive comments and variable names. Raychev et al. (2015) propose a
system JSNice that models the statistical patterns of source code through an
AST-based dependency network to predict variable names and types in JavaScript
programs. A notable issue in program analysis has been scalable but imprecise
models. Models that tended to scale on large datasets tended to have higher
numbers of false positives. To remedy this, Oh et al. (2015) proposes using
Bayesian Optimization to optimize models for static program analysis and
report achieving high precision while lowering false positives. Mangal et al.
(2015) take a more user-centered approach and utilize online learning to tune
their model. The user would provide feedback to the program analysis system as
to whether the user likes or dislike the program analysis output. Mapping
comments onto code tokens can be a useful feature in helping programmers debug
and understand code. In an attempt to model this, Panthaplackel et al. (2020)
work on associating entities mentioned in Javadoc comments with source code
tokens using a feedforward neural network trained as a classifier trained on
noisy data from Github. Managing large software repositories can be gruesome.
Source code classification and tagging are crucial in helping to organize,
search, and reuse codebases. In an attempt to automate source code tagging and
classification, Mou et al. (2016a) propose a novel architecture of Tree-based
convolutional neural networks (TBCNNs). TBCNNs combine structured information
from Abstract-Syntax Trees (ASTs), which are tree-based meaning
representations of code snippets, with Convolutional Neural Networks (CNNs).
CNN’s are a deep learning model derived from MLPs. Their work is then applied
to classify code bases by functionality with over 104 unique functionalities.
TBCNNs were reported to outperform other classification machine learning
models including SVM and RNNs for this task. Program analysis has been also
useful in software reverse engineering, in particular, recovering variable
names when decompiling C code Lacomis et al. (2019).

###  3.3 Clone Detection

Code Clone Detection is another area of research that focuses on detecting
portions of code that mean or do the same thing. Program analysis (see §3.2)
focuses on analyzing semantic properties of programs. Clone detection is
closely related to program analysis and can be considered a sub-field of
program analysis that is particularly focused on estimating semantic
similarity between two codes. Early work on code clone detection by Chilowicz
and Duris (2009) focused on indexing and retrieving abstract syntax trees.
Clone detection has been approached as a machine learning problem of binary
classification of two codebases (e.g. labeling a code pair with 1 to indicate
that the pair is a clone and labeling with 0 to indicate otherwise) Svajlenko
et al. (2014); Wei and Li (2017); Wang et al. (2020b, 2021a). Code clone
detection has also been approached as an information retrieval problem, where
the model is tasked with retrieving semantically similar codes to a given code
input Ganguly et al. (2018); Mou et al. (2016b); Lu et al. (2021a). Code clone
detection is included as a task on the CodeXGLUE benchmark Lu et al. (2021a).
The task evaluates models using the F1-score for the binary classification
subtask and Mean Average Precision (MAP) Beitzel et al. (2009); Lu et al.
(2021a) for the code retrieval task. Mean Average Precision (MAP) is the
arithmetic mean of the average precision values ($AP$) for a system over a set
of $n$ documents. MAP is described in equation 18.

$MAP=\frac{1}{n}\sum_{n}AP_{n}$ (18)

Notable datasets for clone detection include the POJ-104 dataset Mou et al.
(2016b) for clone code retrieval. The POJ-104 dataset is collected from a
programming open judge (OJ) system where students submit their source code for
104 problems. The dataset contains about 500 student submissions in the C/C++
programming language for each problem. The task is to retrieve other codes
that solve the same problem as the input example. Another notable dataset is
the BigCloneBench dataset Svajlenko et al. (2014) which focuses on the binary
classification of clone codes. After filtering the dataset contains about
$1,731,860$ Java code pairs.

Currently, to the best of our knowledge, CodeBERT Feng et al. (2020)
outperforms prior models on both the POJ-104 and the BigCloneBench datasets.
Prior works in clone detection have used AST-based RNNs with pretrained word
embeddings like word2vec Büch and Andrzejak (2019). Other notable approaches
for clone detection include Clone Detection with Learning to Hash (CDLH) Wei
and Li (2017) which learns hash functions and representations of code snippets
via an AST-based LSTM. Another approach is Flow-augmented abstract syntax tree
(FA-AST), which augments the original AST representation of source code with
data flow edges and uses graph neural networks Wang et al. (2020b). MulCode
Wang et al. (2021a) integrates a pre-trained transformer model BERT and an
AST-based LSTM to encode a code sequence and structure and is evaluated on
three classification tasks including clone detection.

Code clone detection has become of increasing importance since the rise of
machine learning on “big code". Lopes et al. (2017) discovered that most of
the code on Github is near-duplicate. A study followed by Allamanis (2019)
explores the effects of code duplication on machine learning models for source
code. Allamanis observes big inflation in machine learning metrics when
testing on duplicated code corpora. Code clone detection is crucial in
creating higher quality code datasets for machine learning and can also have
other uses such as detecting code plagiarism in academic contexts Ganguly et
al. (2018).

###  3.4 Code Search

Model | Description | Best NDCG | Baseline | GitHub
---|---|---|---|---
Neural Bag of Words (BoW) |  Encodes the codebases and English queries by encoding them into a standard token embedding with masking. The model optimizes to minimize the cosine distance between the code and the query. | 0.384 | 0.340 | Wu (2020)
Self-Attention |  Transformer masked multi-headed attention Vaswani et al. (2017) is used to compute the token embeddings. | 0.240 | 0.240 | -
One Dimensional Convolutional Network (CNN) |  CNN is used over query and code embeddings. | 0.165 | 0.165 | -
Bidirectional RNN |  Employing gated recurrent network (GRU) to summarize the query. | 0.046 | 0.046 | -
Table 2: Overview of top scoring CodeSearchNet models sorted by the best score
for the particular model across all leaderboard submissions. The score is the
average Normalized Discounted Cumulative Gain (NDCG) score across all six
programming languages.

Searching for code snippets online is a common activity for programmers and
software engineers. Semantic code search is the task of retrieving
semantically equivalent code to a natural language query Husain et al. (2019).
Husain et al. (2019) release a corpus specific for the task of code search
called CodeSearchNet444https://github.com/github/CodeSearchNet. The corpus is
collected from public, licensed, and non-forked GitHub repositories. Functions
and methods are parsing using TreeSitter and corresponding documentation is
parsing using regular expressions. The training set of the CodeSearchNet
corpus includes only code functions that have an associated documentation
string with them. Other inclusion and exclusion criterion include: the
documentation is shortened to only the first paragraph, samples that include 3
tokens or less of documentation or 3 lines or less in its function body are
removed. Additionally, methods with the string test in its function name,
constructors and standard extension methods such as __str__ are removed, and
lastly duplicate functions are removed from the dataset using clone detection
methods described in Lopes et al. (2017); Allamanis
(2019)555https://github.com/Microsoft/near-duplicate-code-detector. It
contains around 6 million functions in six programming languages: Java, Go,
JavaScript, Python, Ruby, and PHP along with corresponding 2 million natural
language queries in English. While the dataset is relatively big, it suffers
from being quite noisy. Scraped function documentation is far from a query,
hence the authors reported difficulty in identifying how accurately a
documentation sample describes its corresponding code function. The code
search task is evaluated using Mean Reciprocal Rank (MRR) and Normalized
Discounted Cumulative Gain (NCDG) a commonly used metric in information
retrieval described in Vechtomova (2009). Top scoring attempts of each unique
method are listed in Table 2 along with the GitHub reference, and the baseline
score for that approach. Submissions without publications or with private code
bases were excluded. A notable attempt not mentioned in the table by Arumugam
Arumugam (2020) who uses code2vec Alon et al. (2019), a code embedding of
abstract syntax tree paths for semantic code search. He uses code2vec to embed
the code functions and combined it with another neural model such as Neural
Bag of Words (NBoW) which are table-look up embeddings or embeddings with
self-attention for the query and scores 0.148 average NDCG using NBoW for the
query and code2vec for code.

There are additional works that tackled the problem of code search that do not
use the CodeSearchNet dataset exclusively. Reid et al. (2020) propose
NLP2TestableCode an Eclipse IDE plugin that retrieves code snippets from the
StackOverflow database using natural language queries and integrates the
retrieved code into a Java codebase. Li et al. (2019) propose a dataset
composed of natural language queries and code snippets for evaluating code
search models. The dataset was constructed using 287 Stackoverflow question
and answer pairs as well as from method code snippets from the most popular
Android repositories on Github that correctly answer the Stackoverflow
question. The Github corpora contains about 24,549 repositories and 4,716,814
methods.

###  3.5 Code Documentation Generation & Summarization

The goal of code documentation generation is to generate a comment in natural
language that accurately describes a given code snippet. This research area
intersects heavily in terms of datasets and models with Code Generation (§3.8)
since both areas heavily utilize corpora of source code and natural language
and utilize machine translation approaches. Notable work by Oda et al. (2015b)
propose two novel dataset of Python and natural language comments called the
Django and Euler datasets. The authors then statistical machine translation to
generate comments in natural language given a Python source code snippet. The
system was evaluated using BLEU Papineni et al. (2002) and human evaluation of
the generated comments. Fudaba et al. (2015) build a tool called Pseudogen
that also generates pseudocode in both Japanese and English from Python code
using statistical machine translation. Iyer et al. (2016a) propose a dataset
and an LSTM-encoder decoder model to produce sentences in natural language
that describe C# code snippets. Gros et al. (2020) examine the task of
generating comments in 4 code-natural language datasets using a sample
information-retrieval (IR) model using the BM25 scoring function Robertson and
Walker (1994); Amati (2009), LSTM-based sequence-to-sequence, and a
transformer-based sequence-to-sequence model. The authors find that the simple
IR model provides competitive performance on these datasets. Notable work by
Gao et al. (2020b) propose generating Stack Overflow questions given source
code which is similar to generating comments. Code Documentation Generation is
also included as a task in the CodeXGLUE Feng et al. (2020) which utilizes the
CodeSearchNet corpus Husain et al. (2019) for training and testing machine
learning models. More recent approaches to code documentation utilize
pretrained transformers. Currently, CoTexT Phan et al. (2021) outperforms
CodeBERT Feng et al. (2020), PLBART Ahmad et al. (2021), and ProphetNET-X Qi
et al. (2021) in this task. The CodeSearchNet corpus is further described in
§3.4 and the models are described later on in the code generation subsection
§3.8. Code summarization focuses on generating method names in natural
language given a method’s body of code snippets.A notable work on representing
codes as vectors is code2vec Alon et al. (2019) which focuses on embedding
abstract syntax tree paths into a fixed-length vector. Code2seq Alon et al.
(2018) is an LSTM-based sequence to sequence model that encodes the raw source
code tokens along with the AST paths in its encoder and attends to the
relevant paths while decoding. Code2seq outperforms code2vec on code
summarization and outperformed prior state of the art in code documentation
generation CodeNN Iyer et al. (2016a) by 2% BLEU.

###  3.6 Documentation Translation

Code Documentation Translation focuses on translating web-pages containing
code-related documentation such as library docs from a source language into a
target language. CodeXGLUE proposes this task and curates a dataset by
crawling the Microsoft Documentation website666https://docs.microsoft.com/en-
us/. and it includes software and code description documents in various
languages Lu et al. (2021a). The task focuses on translation between English
and other languages such as Danish, Latvian, Norwegian, and Chinese. The
multi-lingual zero-shot transformer model proposed by Johnson et al. (2017)
and a variant of the model with the encoder initialized with XLM-R Conneau et
al. (2020) pretrained weights were used as models for this task. The
pretrained transformer was reported to outperform the baseline on all
translation tasks Lu et al. (2021a) using the token overlap metric BLEU
Papineni et al. (2002). The pretrained transformer scored 66.16 while the
transformer baseline score 52.67 BLEU.

###  3.7 Semantic Parsing

(a) Sample task of parsing natural language interactions into a domain
specific logical form Chen Jr and Bunescu (2019).

(b) Abstract Meaning Representation of two sentences that are semantically
equivalent but have different grammar structures Banarescu et al. (2013).

(c) Sample task of parsing natural language into an SQL query Zhong et al.
(2017).

Figure 12: Example semantic parsing tasks with various meaning
representations.

When interfacing with machines via natural language, a key component is the
machine’s understanding of the natural language intent. The intersection
between natural language processing, information retrieval, and interaction
with humans is called natural language understanding (NLU). Semantic parsing
converts natural language utterances to executable meaning representations,
such as lambda calculus or database queries, that can be executed on a
knowledge-base Kamath and Das (2019). Semantic parsing can also be known as
auto-formalization, which is defined as the task of turning informal
descriptions to formal and automatically verifiable formats Szegedy (2020).
Semantic parsing has become increasingly prominent in applications focused on
interfacing via natural language with computer systems Yin (2020). Semantic
Parsing enables machines to identify roles and objects in a sentence
Fernández-González and Gómez-Rodríguez (2020), converting natural language to
database queries Zhong et al. (2017) that can then be plugged into
conversational assistants like Alexa, interfacing with robots using natural
language commands Matuszek et al. (2012); Artzi and Zettlemoyer (2013),
generating mathematical statements Wang et al. (2020a); Sun et al. (2019a),
and interacting with time series medical data Chen Jr and Bunescu (2019).

Semantic parsing works typically generate or map a natural language utterance
into an unambiguous meaning representation (also referred to as logical forms
or semantic representations). The representations need to be executable in an
environment or a specific context. The representations typically follow a
specified grammar (also referred to as a logical formalism). Grammar is used
to identify valid meaning representations. Grammar can interact with a model
which produces a distribution over meaning representations, to ensure that all
meaning representations in the distribution are valid.

Meaning Representations of natural language utterances for semantic parsing
tend to often fall into one of three categories: first order logic (FOL),
graphs, and code snippets. First order logic often expresses unambiguous
logical expressions. It consists of variables and functions that only take
objects as arguments. First order logic can be augmented with lambda calculus
to increase its expressiveness Carpenter (1997). An illustrative example from
Singh et al. (2020), “a man eating" can be represented as
$\exists{A}(\exists{B}(man(A)\land eat(B)\land agent(B,A)))$ in first order
logic. First order logic has been used in various semantic parsing works to
query databases Zettlemoyer and Collins (2005), knowledge bases Berant et al.
(2013), Wikipedia tables Pasupat and Liang (2015), parsing conversations and
implementing conversational agents Artzi and Zettlemoyer (2011), and mapping
linguistic instructions onto robotic actions Artzi and Zettlemoyer (2013).
Graph-based meaning representations often denote a labeled graph where nodes
represents entities or events and edges represent semantic relationships
between the nodes. Graphs tend to be easy to understand compared to other
meaning representations like code snippets and first order logic. Graph-based
meaning representations also have the advantage in that they also provide
abstraction and can pivot away from any general syntax. Notable examples of
graph-based meaning representations include Semantic Dependency Parsing Oepen
et al. (2015); Fernández-González and Gómez-Rodríguez (2020), Abstract Meaning
Representation (AMR) Banarescu et al. (2013), and Universal Conceptual
Cognitive Annotation Abend and Rappoport (2013). Code snippets in general
purpose programming languages provide a good medium for representing meaning
since they are domain specific and easily executable. More recently there have
been various works that focused on converting natural language into equivalent
executable code snippet representations in languages such as SQL Zhong et al.
(2017), Python Yin and Neubig (2017a), and Bash Lin et al. (2018). These works
will be discussed later on in §3.8. Figure 12 showcases example semantic
parsing tasks with an AMR, a logical form, and SQL, a database specific
programming language.

Grammar of meaning representations defines rules to determine whether a
representation is valid or not. Grammar also plays a role in determining how a
semantic parser expresses its meaning representations and how computationally
complex building the meaning representation is. A notable example of strict
grammar for meaning representations is Combinatory Categorical Grammar
(CCG)777https://yoavartzi.com/tutorial/Steedman (2000); Artzi et al. (2014).
It is efficiently parsable, but still expressive. CCG has a lot of rules and
hence it makes it easier to parse since it lessens the amount of possible
meaning representations for a given input Zettlemoyer and Collins (2007).
Kwiatkowski et al. (2011) consider learning a probabilistic model of CCG
grammar for semantic parsing. Recent works on code generation from natural
language leverage the programming language grammar itself by using an abstract
syntax tree associated with the programming language.

Context of semantic parsing tasks form the bases of mapping natural language
to its meaning representation. Often these contexts or environments are
executable. Example semantic parsing contexts include knowledge bases Berant
et al. (2013), Wikipedia tables Pasupat and Liang (2015), geography queries
Zelle and Mooney (1996), SQL databases Zhong et al. (2017), spreadsheets
Gulwani et al. (2012), and programming languages Yin and Neubig (2018, 2017b).

Models for semantic parsing relied on rule-based techniques (e.g. Johnson
(1984); Woods (1973); Hendrix et al. (1978)), symbolic artificial intelligence
(e.g. Zelle and Mooney (1996, 1993)), statistical-based techniques (e.g.
Zettlemoyer and Collins (2005); Tang and Mooney (2001)), RNN-based sequence-
to-sequence (e.g. Chen Jr and Bunescu (2019); Jia and Liang (2016); Singh et
al. (2020); Yin and Neubig (2017a); Rabinovich et al. (2017); Ling et al.
(2016); Sun et al. (2019a)), and transformer-based architectures (e.g. Kacupaj
et al. (2021); Ferraro and Suominen (2020); Shen et al. (2019); Sun et al.
(2019b); Gemmell et al. (2020); Svyatkovskiy et al. (2020); Kusupati and
Ailavarapu ). In the late 1970’s, Hendrix et al. (1978) pioneer the task of
interfacing with databases using natural language. The rule-based system
proposed called LADDER which takes in a restricted set of natural language
questions. LADDER accepts questions if they match a preset template, extract
field and file names, and then produces a query or a sequence of queries in a
language Data Language by inserting the relevant file and field names into
preset templates. In 1993, Zelle et al. (1993) introduce a first-order
induction algorithm that utilizes symbolic artificial intelligence to learn
word classes an semantic relationships between the words to support parsing of
sentences and conduct semantic analyses. The algorithm takes in a set of
training samples containing sentences paired with ’case representations’ (e.g.
parts of speech tags). The algorithm utilizes shift-reduce parsing where in
each time step either a new word from the sentence is added (shifted) onto a
stack or the top two elements on the stack are popped from the stack then
merged to form a new element and pushed back into the stack. After the
training is complete the shift-reduce parser is introduced to search control
heuristics to help the parser become more specialized. In 2005, Zettlemoyer
and Collins (2005) propose a statistical probabilistic model that learns given
a training set to induce grammar for mapping natural language sentences to
lambda calculus. The probabilistic model learns given a sentence in lambda
calculus $L$ and a parse tree $T$. The parse tree $T$ is defined as the
sequence of transformations needed to derive $L$ from the given natural
language sentence $S$ within the constrains of the grammar CCG. The
conditional distribution would be $P(L,T|S)$. In 2020, Singh et al. (2020)
propose using an LSTM-based sequence-to-sequence with Bahdanau attention
Bahdanau et al. (2014) to convert natural language utterances into first order
logic. The authors report on improving upon the sequence-to-sequence by
introducing a mechanism that aligns variables across predicted predicates in
first order logic. This mechanism utilizes a classifier at each decoding step
of a variable token to predict whether the variable is aligned with any
previously decoded tokens or whether it’s a new variable. Following the
classifier is self-attention-like mechanism where the decoder hidden states
are used to estimate alignment between the variable and other previous decoded
tokens. Chen and Bunescu (2019) design an LSTM-based encoder decoder model to
parse natural language or GUI interactions into a database-executable lambda-
calculus logical form. The database contains time series data from sensors
monitoring type I diabetes patients along with patient-reported information
about discrete life events (e.g. stress, sleep, meals, etc. The data is
temporal and user interactions are sequential, where the current interaction
may dependent on a prior interaction with the system. This is illustrated in
the Figure 12(a) example. Hence, it is crucial for the model to understand
coreferential time expressions such as “then" and temporal relations between
entities such as “after". To enhance the model with such understanding
capabilities, the encoder decoder architecture works with a copying mechanism
to model context dependency and utilizes 3 attention mechanisms: the first
attention mechanism attends over the previous input, the second attends over
the previous logical form, and the third attends over the current input. The
proposed approach scores 66.9 in exact match accuracy significantly
outperforms the baseline of a standard LSTM-based seq2seq model Bahdanau et
al. (2017); Cho et al. (2014) which scored 22.2 accuracy on a real-world
dataset. Kacupaj et al. (2021) employ a transformer architecture to generate
logical forms in in a question-answering conversational assistant capable of
answering complex questions by utilizing a pre-filled knowledge graph. Ferraro
et al. (2020) compare the transformer architecture with other statistical and
neural semantic parsing systems on the ATIS Dahl et al. (1994) and Geo Cai and
Yates (2013) semantic parsing datasets and find that the transformer
architecture outperforms prior strong models in certain settings and achieve
competitive results across all experimental settings.

(a) An assembly code generation task. The task is to generate the assembly
code that is then compiled into shellcode (small pieces of code used as a
payload to exploit software vulnerabilities) using the natural language
descriptions on the right. The dataset contains multi-line snippets mapping
onto one intent. Lines 4-5, 6-7-8, 9-10, 11-12 are multi-line snippets Liguori
et al. (2021a).

(b) A Python code generation task where models are tasked with generating the
blue code cell given the natural language utterance Create and train the model
along with the previous natural language and code cells in the Jupyter
notebook Agashe et al. (2019).

Figure 13: Example code generation parsing tasks with various programming
languages.

Semantic parsing is a unique area with the aim of improving natural language
understanding (NLU). It is distinguished from natural language generation and
machine translation. Both natural language generation and some semantic
parsing works use similar deep learning models and techniques such as an LSTM
based sequence-to-sequence architecture in the works of Yin and Neubig Yin and
Neubig (2017a). The objective of semantic parsing involves the prediction or
generation of inherently structured representations (not only utterances).
These representations need to be executable against a context or an
environment. Semantic parsing is related to Code Generation from Natural
Language, since both predict structured representations of natural language,
in this case these structure representations would be code snippets which are
executable in an environment and can also be represented as abstract syntax
trees. However, semantic parsing also includes other executable
representations that are not limited to code such as first order logic and
graphs. For other survey papers on semantic parsing c.f. Kamath and Das
(2019); Lee et al. (2021).

###  3.8 Code Generation

This section overviews generating a snippet of code given a natural language
description.

####  3.8.1 Overview

Code generation is the task of generating an executable snippet given a
description in natural language. The executable snippet can be in a particular
programming language or domain specific language (DSL) that can be trans-
compiled into a programming language. To illustrate consider the following
example generating a Python snippet from an English intent: check if all
elements in list ‘mylist’ are the same $\rightarrow$ len(set(mylist)) == 1.
Code generation from natural language focuses on linking natural language with
source code which is something it has in common with other aforementioned code
intelligence tasks code search and semantic parsing. Code generation is
distinct from code search. Code search focuses on ranking code snippets and
codebases given a natural language query, whilst code generation attempts to
generate a meaning representation, an executable code snippet that aligns with
a natural language description. With that being said, code generation is
closely related to the area of semantic parsing. It can be considered a
subfield of semantic parsing that specializes in generating executable code
snippets in programming languages an excludes any other meaning representation
of natural language. Code generation has been inspired by Natural Language
Generation methods including statistical machine translation, RNN-based
sequence-to-sequence, and generative transformer models. Linking codebases to
natural language intents or other forms of unstructured input compose face a
series of difficult challenges. Firstly, the compositional and abstract nature
of code can make pairing text with code difficult. Secondly, natural language
is ambiguous in nature while source code is not ambiguous. Code generation can
enable users to interface with machines using natural language. For instance,
Kuhlmann et al. (2004) enable interfacing with robots using natural language.
Furthermore, code generation enables users to build code more effectively and
efficiently and enhance the overall software engineering process. Code
software engineering Miltner et al. (2019); Xu et al. (2021), robotics
Kuhlmann et al. (2004), and cyber-security You et al. (2017); Liguori et al.
(2021a); Frempong et al. (2021); Liguori et al. (2021b, 2022). Figure 13
showcases two examples of code generation tasks.

Recent works in code generation have been used to assist data scientists to
perform data visualization and file manipulation Xu et al. (2021), generating
bash commands Lin et al. (2018), generate exploits Liguori et al. (2021b, a);
Frempong et al. (2021); Liguori et al. (2022), solve interview-level
programming questions Hendrycks et al. (2021), manipulate data Zhong et al.
(2018), and generate code snippets from natural language descriptions in many
programming languages. These programming languages include but are not limited
to Python Xu et al. (2020a); Ling et al. (2016); Liguori et al. (2021b), Java
Ling et al. (2016), SQL Zhong et al. (2017), Excel macro commands Gulwani and
Marron (2014), Assembly Liguori et al. (2021a, b, 2022), and JavaScript
Frempong et al. (2021).

####  3.8.2 Datasets

This section introduces a comprehensive collection of supervised datasets
containing natural language (NL) mapped onto programming language snippets
(PL). The snippets can be in general purpose programming languages (e.g.
Python, Java) or in a Domain Specific Language (DSL). In table
LABEL:tab:code_datasets, we curate a systematic list (to the best of our
knowledge) of English NL-PL datasets.
Survey Method. The survey process started by searching through publications on
GoogleScholar888https://scholar.google.com/ with keywords such as “Code
generation", “Program synthesis from natural language", “natural language to
code", and “Conversational Semantic Parsing". We collected highly cited papers
and carefully reviewed all of the works that cite them for potentially
proposed datasets. For each paper we reviewed, we carefully checked the
references for any papers proposing datasets as well. We also searched through
PapersWithCode999https://paperswithcode.com/datasets and Huggingface Datasets
101010https://huggingface.co/datasets for datasets with the tags “Code
Generation", “Code Search", and “Code". Datasets that map other forms of input
like images of Graphical User Interfaces Beltramelli (2018), input-output
examples (without natural language descriptions) Drosos et al. (2020),
questions in natural language not descriptions of non-query code (e.g. the
neural code search dataset Li et al. (2019)), and compiled code Lacomis et al.
(2019) to source code are excluded from the table. We’ve also excluded large
mined datasets that are not processed into NL-PL pairs like the Code-Clippy
dataset Cooper et al. (2021), CodeNet Puri et al. (2021), and The Pile Gao et
al. (2020a). Multi-lingual datasets were not systematically surveyed.
Automatic Exploit Generation. Automatic exploit generation is defined as an
offensive security technique in which software exploits are automatically
generated to explore and test critical vulnerabilities before malicious
attackers discover such vulnerabilities Avgerinos et al. (2011). With the goal
of offensive security in mind, Liguori et al. Liguori et al. (2021a) share a
dataset, Shellcode_IA32, for shellcode generation from natural language
descriptions. Shellcodes are compiled from assembly programs for the 32-bit
version of the x86 Intel Architecture and contain a payload that is used in
exploiting software vulnerabilities. The Shellcode_IA32 dataset is later on
extended in the EVIL-Decoder Liguori et al. (2021b) dataset. Shellcodes are
often encoded using Python software to evade detection by antivirus programs
and then decoded using an assembly program to the victim machine. In an
attempt to automate the whole pipeline Liguori et al. (2021b) propose the EVIL
dataset composed of two different languages. EVIL-Decoder dataset contains
full assembly programs along with their NL descriptions that decode a
shellcode on a host machine. The EVIL-Encoder dataset extends the general
purpose Python Django dataset Oda et al. (2015a) with 1,114 original exploit-
oriented snippets. Frempong et al. (2021) curate a synthesized dataset of
JavaScript snippets used in cross-site scripting (XSS) which are a common web-
based attack that target websites.
Bash. In an effort to make bash terminal (a Linux operating system command
line interface) interactions more accessible, Lin et al. (2018) curate a
dataset of bash commands along with their corresponding natural language
descriptions. This work helps lay down a foundational dataset and a baseline
model in building natural language interfaces to terminals.
General Purpose Programming. There has been extensive work in curating
datasets for general purpose programming languages such as Java, JavaScript,
and Python, since general purpose programming languages are abundant in open
source code -bases Husain et al. (2019). While some datasets were expert
curated Oda et al. (2015a); Chen et al. (2021a), and crowd sourced Long et al.
(2016); Zavershynskyi et al. (2018); Kulal et al. (2019); Austin et al.
(2021); Huang et al. (2021), extensive datasets with NL-PL pairs were mined
from open source code sources such as Github Allamanis et al. (2016); Barone
and Sennrich (2017); Iyer et al. (2018); Alon et al. (2018); Hu et al. (2018);
Husain et al. (2019); Agashe et al. (2019); Clement et al. (2020); Hu et al.
(2020); Bahrami et al. (2021), coding competition websites Caballero et al.
(2016); Li et al. (2022), Sourcerer Lopes et al. (2010); LeClair et al.
(2019), IFTTT Quirk et al. (2015) (a website that allows users to create
simple programs using triggers and actions), and StackOverflow Iyer et al.
(2016b); Yin et al. (2018); Yao et al. (2018); Orlanski and Gittens (2021).
CoDesc Hasan et al. (2021b) combines multiple mined Java datasets into one
bigger dataset after removing data-related noise. Hendrycks et al. (2021)
propose the Automated Programming Progress Standard, a dataset curated from
specifially mining websites where programmers share programming problems with
each other such as CodeForces, Codewars, AtCoder, and Kattis. APPS contains
Python programming problems in various difficulties (Introductory, Interview,
and Competition levels), a natural language description, and unit tests for
each problem. Some datasets contain synthetically generated NL descriptions
such as the Hearthstone and Magic The Gathering trading card game datasets,
were the NL-descriptions were automatically populated fields describing the
card Ling et al. (2016). Akinbou et al. (2021) propose using back-translation
on Python code collected from Aizu Online Judge
(AOJ)111111https://judge.u-aizu.ac.jp/onlinejudge/ using a trans-compiler. The
trans-compiler outputs natural language descriptions, in Japanese, from a
Python code input. The Python snippet is converted into an abstract syntax
tree and then using pre-defined transformation rules, the abstract syntax tree
is transformed into a natural language description. Hasan et al. (2021a)
propose text2app, a dataset containing descriptions of Android app
specifications along with a Simplified App Representation (SAR) codebase. SAR
is a domain specific language that is then compiled into a Java Android app.
Text2App used data BERT-masking for data augmentation to curate NL
descriptions of apps from a small amount of crowd-sourced app descriptions.
Turducken-style Code Generation. Turducken-style code snippets include one
language embedded in another. Often-times a general purpose programming
language like Python being embedded with another server-related programming
language like SQL. In an effort to benchmark code generation systems on
Turducken-style code, Liang et al. (2021) propose the Lyra dataset which
contains natural language intents in both English and Chinese languages mapped
onto Python code with embedded SQL snippets.
Conversational Semantic Parsing. Semantic parsing is a task in which an
utterance is converted into an executable form (§12). Semantic Parsing works
often focus on isolated utterances. Conversational Semantic Parsing takes that
a step further, where the context is a conversation. A crucial part of a
dialogue system is being able to track the user’s goal through out the
conversation, this task is called Dialogue State Tracking (DST). In
conversational semantic parsing, researchers have formulated the user’s goal
into an SQL query Yu et al. (2019a), a tree-like representation using a domain
specific language Gupta et al. (2018); Cheng et al. (2020); Aghajanyan et al.
(2020), and a program in a domain specific language that extends a dataflow
graph Andreas et al. (2020). These representations are executable by a
dialogue system to interface with a data source. People tend to explore
databases by asking multiple related questions Iyyer et al. (2017), which
require systems to be able to process conversational data requests, clarify
ambiguous questions, and process user utterances that can not be mapped onto
queries Yu et al. (2019a). To address this, Yu et al. (2019a) release the
Conversational Text-to-SQL (CoSQL) corpus, to build database querying dialogue
systems. Human utterances in dialogue sometimes contain nested requests. An
example from Gupta et al. (2018) illustrates this well: “Get me driving
directions to the Eagles game" it is composed of two requests GET_DIRECTIONS
and GET_EVENT. Linear representations do not allow for such compositionality.
To tackle this phenomena, both the TreeDST Cheng et al. (2020) and the TOP
datasets Gupta et al. (2018), use a compositional (tree-like) forms. SB-TOP
builds on the TOP dataset to include a decoupled representation that can
represent co-reference and context carry over Aghajanyan et al. (2020).
Database Querying. Creating natural language interfaces to interacting with
data sources such as databases has been a long standing research Price (1990);
Dahl et al. (1994). To this end, various hand-curated datasets containing NL-
descriptions and data-related questions mapping onto a domain specific
language (such as a logical form) trans-compilable into a query in various
domains such as Airline Travel in English Price (1990); Dahl et al. (1994) and
other various languages Upadhyay et al. (2018); Xu et al. (2020b), Job posts
Tang and Mooney (2001), open-domain question answering Cai and Yates (2013);
Berant et al. (2013); Wang et al. (2015); Yih et al. (2016). We also observe
an array of that focus on generating SQL queries from natural language. Some
of these datasets are synthetic Zhong et al. (2017), mined from StackOverflow
Yao et al. (2018); Hazoom et al. (2021) and Github Yao et al. (2018), and
human-curated Tang and Mooney (2000); Popescu et al. (2003); Giordani and
Moschitti (2012); Li and Jagadish (2014); Iyer et al. (2017); Yu et al.
(2018); Yaghmazadeh et al. (2017); Finegan-Dollak et al. (2018); Yu et al.
(2019b).
Map Question-Answering. Maps (e.g. Google Maps) contain data on entities
restaurants, landmarks, geographic landmarks like rivers and mountains. This
data is often stored in a database. A specific-type of text-to-query task
deals with querying map databases. In an effort to mainstream interfacing
through natural language with geographic data in map-related databases, Zelle
and Mooney (1996) formulated the dataset GeoQuery contain questions about the
United States geography in natural language and corresponding them to Prolog
programs to answer them. GeoQuery was later adapted to SQL Iyer et al. (2017);
Finegan-Dollak et al. (2018). Another notable dataset that contains more
complex natural language questions and entities such as restaurants, museums,
and hotels is NLMAPS Haas and Riezler (2016); Lawrence and Riezler (2016).
NLMAPS contain manually curated human questions that can be run against a
worldwide map data from OpenStreetMap. Natural language intents were later on
expanded using synthetic methods in NLMAPS-V2 Lawrence and Riezler (2018).
Data Manipulation. Datases were curated to with the aim of automatically
generated codes that manipulate data. Gulwani et al (2014) curated a dataset
that maps English utterances onto Excel macro-commands. Other datasets focus
on generating a regular expression (Regex) Kushman and Barzilay (2013);
Locascio et al. (2016); Zhong et al. (2018) given a natural language
description.

Table 3: Survey of datasets containing natural-language-programming language (NL-PL) pairs that are usable for the tasks of code generation and semantic parsing. We also include domain specific languages (DSL) if they can be trans-compiled into a programming language. EC stands for expert curated, CC, stands for crowdsourced, Synth stands for synthesized, Synth-BT sands for synthesized with back-translation. The authors will add footnote to table for datasets reporting on total number of tokens vs total number of unique tokens. Dataset Name |  | Programming
---
Language
(PL)
#NL-PL pairs | #NL tokens | #PL tokens |  | #Avg tokens
---
per NL intent
| #Avg tokens per
---
code snippet
| Data
---
Collection
(NL)
Public |  Reference
Automatic Exploit Generation
Shellcode_IA32 | Assembly | 3,202.00 | 1,490.00 | 1,238.00 | 9.24 | 4.40 | EC |  |  Liguori et al. (2021a)
EVIL-Encoder | Python | 15,540 | 10,605 | 9,511 | 14.90 | 11.90 | EC |  |  Liguori et al. (2021b)
EVIL-Decoder | Assembly | 3,715 | 1,924 | 1,657 | 9.53 | 4.75 | EC |  |  Liguori et al. (2021b)
HIJAX | JavaScript | 100,000 | - | - | - | - | Synth | $\times$ |  Frempong et al. (2021)
Bash
NL2Bash | Bash | 9,305 | 7,790 | 6,234 | 11.70 | 7.70 | EC |  |  Lin et al. (2018)
General Purpose Programming
IFTTT | DSL | 86,960 | 593,123 | 1,881,451 | 6.82 | 21.64 | Mined | $\times$ |  Quirk et al. (2015)
| Code-Docstring Corpus
---
Python | 150,370 | 5,789,741 | 12,601,929 | 38.50 | 83.81 | Mined |  |  Barone and Sennrich (2017)
C#2NL | C# | 66,015 | 24,857 | 91,156 | 12.00 | 38.00 | Mined |  |  Iyer et al. (2016b)
Django | Python | 18,805 | - | - | 14.30 | - | EC |  |  Oda et al. (2015a)
CoNaLa | Python | 2,879* | - | - | - | - | EC |  |  Yin et al. (2018)
CoNaLa Mined | Python | 593,837 | - | - | 11.41 | 28.70 | Mined |  |  Yin et al. (2018)
CoNaLa-Ext | Python | 596,711 | - | - | 11.41 | 28.70 | Mined |  |  Orlanski and Gittens (2021)
| Magic The Gathering
---
Java | 13,297 | - | - | 21.00 | 1,080.00 | Synth |  |  Ling et al. (2016)
CONCODE | Java | 104,000 |  |  | 13.73 | 26.30 | Mined |  |  Iyer et al. (2018)
HearthStone | Python | 665 | - | - | 7.00 | 352.00 | Synth |  |  Ling et al. (2016)
JuICe | Python (Jupyter) | 1,521,774 | 862,269 | 1,006,402 | 39.78 | 38.75 | Mined |  |  Agashe et al. (2019)
Text2App | SAR (DSL)* | 50,000 | - | - | - | - | Synth |  |  Hasan et al. (2021a)
DeepCom | Java | 588,108 | - | 794,711 | 8.86 | 99.94 | Mined |  |  Hu et al. (2018, 2020)
FunCom | Java | 2,100,000 | - | - | - | - | Mined |  |  Lopes et al. (2010); LeClair et al. (2019)
PyMT5 | Python | 7,700,000 | - | - | - | - | Mined | $\times$ |  Clement et al. (2020)
CodeSearchNet |  | Python, Java, Go,
---
JavaScript,
Ruby, and PHP
2,326,976 | - | - | - | - | Mined |  |  Husain et al. (2019)
Java-small | Java | 665,115 | - | - | 3.00 | 60.00 | Mined |  |  Allamanis et al. (2016); Alon et al. (2018)
Java-med | Java | 3,004,536 | - | - | 3.00 | 63.00 | Mined |  |  Alon et al. (2018)
Java-large | Java | 15,344,512 | - | - | 3.00 | 65.00 | Mined |  |  Alon et al. (2018)
StaQC-Python | Python | 147,546 | 17,635 | 137,123 | 9.00 | 86.00 | Mined |  |  Yao et al. (2018)
Euler | Python | 589 | - | - | - | - | EC |  |  Oda et al. (2015a)
AOJ | Python | 89,862 | - | - | - | - | Synth-BT | $\times$ |  Akinobu et al. (2021)
NAPS | DSL | 2,716 | - | - | - | - | CC |  |  Zavershynskyi et al. (2018)
APPS | Python | 10,000 | - | - | - | - | Mined+revision |  |  Hendrycks et al. (2021)
CodeContests | Python, C++, Java | 13,610 | - | - | - | - | Mined |  |  Li et al. (2022)
Description2Code | Python, C++ | 7,764 | - | - | - | - | Mined |  |  Caballero et al. (2016)
SCONE-Scene | DSL | 4,402 | - | - | 56.20 | - | CC |  |  Long et al. (2016)
SCONE-Alchemy | DSL | 4,560 | - | - | 39.90 | - | CC |  |  Long et al. (2016)
SCONE-Tangrams | DSL | 4,989 | - | - | 27.20 | - | CC |  |  Long et al. (2016)
CoDesc | Java | 4,200,000 | 813,078 | 1,128,909 | 21.04 | 77.97 | Mined |  |  Hasan et al. (2021b)
HumanEval | Python | 164 | - | - | - | - | EC |  |  Chen et al. (2021a)
MBPPS | Python | 974 | - | - | - | - | CC |  |  Austin et al. (2021)
Math-QA | Python | 23,914 | - | - | - | - | CC |  |  Amini et al. (2019); Austin et al. (2021)
PyTorrent | Python | 13,825,647 | - | - | - | - | Mined |  |  Bahrami et al. (2021)
SPoC | C++ | 18,356 | - | - | - | - | CC |  |  Kulal et al. (2019)
CoSQA | Python | 20,604 | 6,784 | 28,254 | 6.60 | 71.51 | CC |  |  Huang et al. (2021)
Turducken-style Code Generation
Lyra-English | Python + SQL | 2,000 | - | - | 57.71 | 44.24 | EC |  |  Liang et al. (2021)
Lyra-Chinese | Python + SQL | 2,000 | - | - | 70.46 | 44.24 | EC |  |  Liang et al. (2021)
Conversational Semantic Parsing
CoSQL | SQL | 11,039 | 9,585 | - | 11.21 | - | CC |  |  Yu et al. (2019a)
TOP | DSL | 44,000 | - | - | 8.93 | - | CC |  |  Gupta et al. (2018)
SB-TOP | DSL | 60,000 | - | - | - | - | CC |  |  Aghajanyan et al. (2020)
TreeDST | DSL | 167,507 | - | - | 7.59 | - | CC |  |  Cheng et al. (2020)
SMCalFlow | DSL | 155,923 | 17,397 | 338 | 8 | 40 | CC |  |  Andreas et al. (2020)
Database Querying
ATIS | DSL | 5,410 | 936 | 176 | 11.10 | 28.10 | CC |  |  Price (1990); Dahl et al. (1994)
Multi-ATIS | DSL | 3,846 | - | - | - | - | CC |  |  Upadhyay et al. (2018)
Multi-ATIS++ | DSL | 44,943 | - | - | - | - | CC |  |  Xu et al. (2020b)
Freebase917 | DSL | 917 | - | - | - | - | EC |  |  Cai and Yates (2013); Berant et al. (2013)
Jobs640 | DSL | 640 | 391 | 58 | 9.80 | 22.90 | EC |  |  Tang and Mooney (2001)
WebQSP | DSL | 4,737 | - | - | - | - | EC |  |  Yih et al. (2016)
WikiSQL | SQL | 80,654 | - | - | - | - | Synth |  |  Zhong et al. (2017)
StaQC-SQL | SQL | 119,519 | 9,920 | 21,413 | 9.00 | 60.00 | Mined |  |  Yao et al. (2018)
SQL2NL | SQL | 32,337 | 10,086 | 1,287 | 9.00 | 46.00 | Mined |  |  Iyer et al. (2016b)
SParC | SQL | 12,726 | 3,794 | 3,794 | 8.10 | - | EC |  |  Yu et al. (2019b)
SPIDER | SQL | 10,181 | - | - | 13.00 | 21 | EC |  |  Yu et al. (2018)
Restaurants | SQL | 378 | - | - | - | - | EC |  |  Tang and Mooney (2000); Popescu et al. (2003); Giordani and Moschitti (2012)
Scholar | SQL | 817 | - | - | - | - | EC |  |  Iyer et al. (2017)
Yelp | SQL | 128 | - | - | - | - | EC |  |  Yaghmazadeh et al. (2017)
IMDB | SQL | 131 | - | - | - | - | EC |  |  Yaghmazadeh et al. (2017)
Advising | SQL | 4,570 | - | - | - | - | EC |  |  Finegan-Dollak et al. (2018)
Academic | SQL | 196 | - | - | - | - | EC |  |  Li and Jagadish (2014)
Overnight | DSL | 12,602 | - | - | - | - | CC |  |  Wang et al. (2015)
SEDE | SQL | 12,023 | - | - | - | - | Mined |  |  Hazoom et al. (2021)
Map Question-Answering
NLMAPS-V1 | DSL | 2,380 | 1,014 | - | 10.90 | 16.00 | EC |  |  Haas and Riezler (2016); Lawrence and Riezler (2016)
NLMAPS-V2 | DSL | 202,088 | 8,710 | - | 7.06 | - | Synth |  |  Lawrence and Riezler (2018)
GeoQuery | Prolog, SQL | 880 | 284 | 60 | 7.60 | 19.10 | EC |  |  Zelle and Mooney (1996); Iyer et al. (2017); Finegan-Dollak et al. (2018)
Data Manipulation
NLyze-Data | DSL (Excel) | 3,570 | - | - | - | - | CC | $\times$ |  Gulwani and Marron (2014)
RegexLib | Regex | 3,619 | 13,491 | 179 | 36.40 | 58.80 | Mined | $\times$ |  Zhong et al. (2018)
NL2RX | Regex | 10,000 | 560 | 45 | 10.60 | 26.00 | Synth |  |  Locascio et al. (2016)
NL2RX-KB13 | Regex | 824 | 715 | 85 | 7.10 | 19.00 | CC |  |  Kushman and Barzilay (2013)

####  3.8.3 Evaluation

Code Generation systems are generally evaluated using exact match accuracy.
Exact match accuracy refers to the ratio of model-generates snippets that
exactly match the ground-truth snippets. Another common metric used for
evaluating code generation systems is token level BLEU Papineni et al. (2002).
It is commonly used to evaluate machine translation systems. BLEU is used to
evaluate code generation systems since many prior works in code generation
formulated the problem as a machine translation problem of translating English
to code snippets (e.g. Liguori et al. (2021a)). Both exact match and averaged
token level BLEU scores have been extensively used in evaluating code
generation models Liguori et al. (2021a, b); Oda et al. (2015b); Ling et al.
(2016); Gemmell et al. (2020). It is becoming increasingly important to note
the drawback of using BLEU to evaluate code generation systems. Recent studies
have shown low correlation between BLEU score and program correctness Austin
et al. (2021); Chen et al. (2021a); Hendrycks et al. (2021), i.e. where the
generated program logically matches the natural language description and
passes all the unit tests. BLEU score increases when there is significant
token overlap between the generated code and the ground-truth. The low
correlation can be attributed to the abundance of variable and method
identifiers in programming languages, which if the model predicts the
identifier tokens correctly, it would result in a higher BLEU score. However,
even though there may be signification token overlap the generated code may
not compile (be syntactically incorrect), token differences between the ground
truth and generated code may cause the generate code to follow a distinct
logic. Furthermore, there may be more than one correct code snippet that
accomplishes the task described in natural language. Both token-level BLEU and
exact match accuracy penalize models for generating working code snippets that
are different from the ground truth, even if they are correct. To remedy this,
recent works in code generation such as APPS Hendrycks et al. (2021),
HumanEval Chen et al. (2021a), and EVIL Liguori et al. (2021b) focus on
evaluating code generation systems by formulating metrics focused on
evaluating the semantics of the generated snippets. Liguori et al. (2021b)
define two new metrics of syntactic and semantic correctness. Syntactic
correctness is the ratio of snippets that are compilable. This is achieved by
compiling the generated snippets with a compiler and recording the ratio of
generated snippets that do compile. For semantic correctness, domain experts
were asked to evaluate whether generated code snippets semantically express
the natural language description. The APPS dataset evaluates generated
snippets based on the percentage of unit tests the generated snippets pass,
this metric is called the Test Case Average Hendrycks et al. (2021).
Additionally, APPS evaluate generated codes on their ability to pass all unit
tests for a particular problem, this metric is called Strict Accuracy.
Similarly in the HumanEval dataset, the authors introduce a $pass@k$ metric,
in which $k$ code snippets are sampled from the model. $pass@k$ would be the
ratio of problems in which any of the $k$ samples pass the problem’s
corresponding unit tests. Another notable metric proposed for evaluating code
generation systems is CodeBLEU Ren et al. (2020).
CodeBLEU121212https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/code-
to-code-trans/evaluator/CodeBLEU, combines the n-gram match inspired from the
BLEU score, along with weighed N-gram match, abstract syntax tree match, and
semantic data-flow match into one metric. The drawback of CodeBLEU is it needs
to have language specific AST-parsers implemented which may not be available
to low-resource programming languages like Haskell and Assembly yet. Current
CodeBLEU supports 6 programming languages: Java, JavaScript, C#, PHP, GO,
Python, and Ruby.

####  3.8.4 Methods

Here we introduce different deep learning architectures used for the task of
code generation. We categorize the approaches into LSTM-based sequence to
sequence, LSTM-based sequence to sequence with abstract syntax trees,
transformers, pretrained transformers, and transformers with abstract syntax
trees. We survey works in each of these approaches and describe them in
detail. We focus on the CoNaLa dataset Yin et al. (2018) as a case study in
Figure 14 and overview all the works that have benchmarked their systems on
this dataset.

Figure 14: Comprehensive review of all code generation models evaluated on the
CoNaLa dataset Yin et al. (2018) across time. Models are scored using BLEU-4
Papineni et al. (2002) and use one of the following deep learning
architectures: LSTM-based sequence to sequence, LSTM-based sequence to
sequence models that encode Abstract Syntax Trees (ASTs), Transformer-based
models, Transformer models loaded with pretrained weights, and Transformer
models that encode Abstract Syntax Trees. We note that the LSTM-Seq2Seq with
Bahdanau attention Bahdanau et al. (2014) and CodeBERT Feng et al. (2020)
experiments were conducted by the authors of this paper. The method of
variable standardization is described in Liguori et al. (2021b, 2022).

LSTM-based Seq2Seq. Ling et al. (2016) LSTM based sequence-to-sequence model
called the Latent Predictor Network (LPN) has been proposed for general-
purpose programming languages (Python and Java) it includes a copying
mechanism similar to Gu et al. (2016b) to copy over identifier names. A
weakness in that approach is that it does not consider code syntax in its
processing and output. Dong et al. (2018) developed Coarse-to-Fine Decoding, a
methodology for semantic parsing and code generation simultaneously. Coarse-
to-Fine Decoding essentially uses two neural networks. The first neural
network is used to parse the input into a rough sketch. The rough sketch
preserves the original meaning of the input while removing lower-level
details. This sketch and the original input is fed into the second neural
network which then produces the output. This method is better than simply
parsing the input through a single neural network. The results show that this
method increases performance when compared to using one neural network to
parse the input to code. Additionally, the Coarse-to-Fine Decoding can be used
for a variety of parsing tasks making it a fairly flexible method. Lin et al.
(2018) use an LSTM-based Seq2Seq and CopyNet Gu et al. (2016b) to generate
linux bash commands from natural language descriptions. Zhong et al. (2017)
develop an LSTM-based sequence to sequence model adapted from Dong and Lapta
(2016) with a pointer network Vinyals et al. (2015) containing augmented
inputs specific to the SQL language. Chen et al. (2021b) propose PlotCoder, an
LSTM-based Seq2Seq with a copying mechanism trained on a portion on the JuICe
dataset Agashe et al. (2019) trained to generate plot code snippets in Python
Liguori et al. (2021a) use a standard LSTM-based sequence to sequence model
Bahdanau et al. (2014) to generate assembly snippets from natural language
intents that are then compiled into shellcodes used in exploiting
vulnerabilities. Frempong et al. (2021) generate JavaScript XSS exploits from
natural language intents. Weaknesses of LSTM-based models include generating
syntactically incorrect code, however given enough training samples or on
syntactically simple languages like Assembly, LSTM-based models tend to
generate syntactically correct code with good accuracy Liguori et al. (2021b).

LSTM-based Seq2Seq with Abstract Syntax Trees. To remedy generated
syntactically incorrect codes, Yin and Neubig (2017a) propose a novel LSTM-
based architecture that generates an Abstract Syntax Tree (AST) given a
natural language description and a grammar model. Their system includes a set
of production rules for python’s abstract grammar, a probabilistic grammar
model, and finally a Recurrent Neural Network with the dimensions of 256 by 50
with a beam size of 15 for the decoder. The results in the model mentioned
above scored 16.2 exact match accuracy and a 75.8 BLEU-4 on the Hearthstone
dataset Ling et al. (2016), scoring 11.7 accuracy above the state-of-the-art
at the time the Latent Predictor Network model. The proposed system’s
weaknesses include mismatching parameter names when defining a function,
omitting or adding default values of parameters when defining a function,
failing to copy a variable name into the correct position, and lastly the
generated code partially implemented the required functionality. Rabinovich et
al. (2017) leveraged the Abstract Syntax Description Language (ASDL) framework
for semantic parsing and a Long Short-term Memory network (LSTM) for code
generation. These trees include two types of modules–composite types (function
and class definitions, return statements, etc) and primitive types (integers,
identifiers). Composite modules use LSTM to determine the appropriate
constructor module which details how the node should expand. These constructor
modules use LSTM to determine what constructor field modules are needed. TranX
Yin and Neubig (2018) uses Abstract Syntax Trees as an intermediary
representation (interlingua) to generate a programming language snippet from a
natural language statement. TranX achieved state of the art on the CoNaLa
dataset Yin et al. (2018) at the time scoring 24.3 BLEU-4 Yin and Neubig
(2019). Yin and Neubig (2019) then incorporate a hypothesis reranker to rerank
the outputs of TranX and see a 5.7% increase in BLEU on the CoNaLa dataset
setting state of the art at 30.11 BLEU-4. Xu et al. (2020a) then explore
incorporating external knowledge from API documentation and mined NL-PL pairs
from StackOverflow to the TranX model Yin and Neubig (2018) with the
hypothesis reranking proposed in Yin and Neubig (2019). This approach improves
upon the reranker model by 2.2% scoring 32.26 BLEU. We note that LSTM methods
with ASTs significantly outperform the standard LSTM sequence to sequence
scoring around 14.72 BLEU on the CoNaLa dataset. Another notable work,
although not used in code generation to our knowledge, is Code2Seq Alon et al.
(2018). Code2seq encodes the source code tokens along with the AST paths
simultaneously and has seen good success in the area of Code Documentation
Generation & Summarization (c.f.§3.5).

Incorporating structural knowledge from ASTs into a neural deep learning model
ensures correct syntax of generated code all the time, it also improves model
generalizability to NL intents Yin and Neubig (2017a). The drawback is
incorporating ASTs will take time since grammar rules normally need to be
specified.

Transformer Models. Bonthu et al. (2021) use a standard transformer
architecture Vaswani et al. (2017) to generate Python source code from natural
language intents and report BLEU score of 32.4 and Rouge-L of 82.1 on a custom
curated dataset. Liang et al. (2021) set a baseline for a turducken-style code
generation task where SQL is embedded with Python. The baseline used is a
transformer model and it achieves 48-49 BLEU and 18%-21% executable codes when
generating codes from the English and Chinese natural language intents
respectively. Akinbou et al. (2021) use a standard transformer architecture
and achieve 8.97 BLEU on the CoNaLa dataset with a 42% syntax error rate.
Gemmell et al. (2020) utilize a transformer architecture to translate natural
language intents to executable snippet representations in Python. The authors
further improve upon the baseline transformer model by introducing pseudo-
relevance feedback during the decoding process. During decoding process top
$k$ documents relevant to the input are retrieved and a set of common tokens
from the documents are emphasized. The authors additionally use copying
mechanisms inspired by prior work in pointer networks See et al. (2017) to
assist with tokens common between intents and snippets like method names and
identifiers. This method achieves 22.3 BLEU on the CoNaLa dataset. We observe
that it is outperformed by a standard LSTM seq2seq which scores around 24.34
BLEU when using variable standardization using an intent parser as proposed in
Liguori et al. (2021b, 2022). A higher BLEU score using the standardization
method is likely due to higher likelihood of generalization and more token
overlap when standardizing tokens referring to variable names and values.

Pretrained Transformer Models.  Mastropaolo et al. (2021) propose pretraining
and fine-tuning a T5 model Raffel et al. (2019a) on a corpora of English text
and source code and then fine-tune on several software engineering tasks
including to generate assert statements (test methods) in Java. CodeBERT Feng
et al. (2020) is a pretrained transformer language model based on RoBERTa Liu
et al. (2019) (a variant of the BERT model Devlin et al. (2018)) pretrained on
the CodeSearchNet corpus Husain et al. (2019). CodeBERT has been used to
generate exploit code snippets from natural language intents Liguori et al.
(2021b). Orlanski and Gittens (2021) finetune the pretrained transformer BART
Lewis et al. (2020) on the CoNaLa dataset and achieve a performance of 26.24
BLEU which is comparable to that of CodeBERT scoring around 22.51 BLEU.
Furthermore, BART when finetuned with both the CoNaLa annotated and mined
datasets it achieved a BLEU score of 30.55. The authors also explore adding
the question body along with the natural language intent as input to BART and
find an increase in performance. BART achieves 34.35 when finetuned with
question bodies and 35.32 when finetuned with questioned bodies on the both
the annotated CoNaLa dataset and the mined CoNaLa datasets. Norouzi et al.
(2021) find that a BERT-based encoder Devlin et al. (2018) and decoder model
with a copying mechanism Gu et al. (2016b) can achieve better performance by
utilizing mine-able monolingual code data. The encoder is frozen on the
monolingual code data and the decoder gets an additional objective during
training of autoencoding the monolingual source code that corresponds to the
intent that is fed into the encoder. This process is named Target Auto
Encoding (TAE). Using the approach the decoder gets to also learn the source
code meaning representation along with the encoder representation on the
natural language intent. This approach achieves 32.57 BLEU on the CoNaLa
dataset compared to the same set up without TAE which scores 30.98 BLEU.
Scholak et al. (2021) propose PICARD a simple and effective decoder-constraint
algorithm that works with pre-trained encoder-decoder models. Using PICARD
with a T5-3B model Raffel et al. (2019b) achieves state of the art on two SQL
generation tasks from NL Spider Yu et al. (2018) and CoSQL Yu et al. (2019a).
GraphCodeBERT Guo et al. (2020) is a pretrained model for programming language
that is pretrained using dataflow which encompasses the semantic structure of
the code. The pretraining objectives used include masked language modeling,
code structure edges, and representation alignment between source code and
code structure. Other pretrained transformers used on source code include
CodeT5 Wang et al. (2021b). CodeTrans Elnaggar et al. (2021), PyMT5 Clement et
al. (2020), CuBERT Kanade et al. (2020), PLBART Ahmad et al. (2021),
ProphetNet-X Qi et al. (2021), CoTexT Phan et al. (2021), T5-Code Mastropaolo
et al. (2021), GraphCodeBERT Guo et al. (2020), and AlphaCode Li et al.
(2022). Pretrained GPT-style Models for source code generation include CodeGPT
Svyatkovskiy et al. (2020), and GPT-Codex Chen et al. (2021a).

Area | Description | Evaluation | References
---|---|---|---
Program Synthesis |  The task of synthesizing complete or partial programs given a specification, often structured input. |  % compilable programs, % logically correct programs, synthesis duration, problems solved Schuster et al. (2021) & other application specific metrics |  Shi et al. (2020); Patra and Pradel (2016); Gulwani et al. (2017); Cummins et al. (2017a); Ellis et al. (2021); Austin et al. (2021)
Program Analysis |  Focuses on extracting semantic properties from programs and often classifying codebases using extracted properties. |  Accuracy, Precision, Recall, & F1 |  Raychev et al. (2015); Mangal et al. (2015); Oh et al. (2015); Panthaplackel et al. (2020); Mou et al. (2016a)
Program Repair |  Focuses on building models to repair codebases given a compiler error message and a corresponding codebase. |  full repair, partial repair, resolved error messages Hajipour et al. (2019). |  Gupta et al. (2017); Hajipour et al. (2019); Yasunaga and Liang (2020); Chirkova and Troshin (2020); Lutellier et al. (2020); Lu et al. (2021b); Jiang et al. (2021)
Clone Detection∗ |  Focuses on measuring the semantic similarity between two codes. There are variations of this task binary classification and retrieving code of semantic equivalence. |  Precision, Recall, F1, MAP Beitzel et al. (2009); Lu et al. (2021a) |  Svajlenko et al. (2014); Wang et al. (2020b); Büch and Andrzejak (2019); Wei and Li (2017); Chilowicz et al. (2009)
Defect Detection∗ |  Focuses on detecting insecure code that can be exploited in attacks such as DoS. The task is a binary classification problem. |  Accuracy, F1, Precision, Recall |  Zhou et al. (2019); Li et al. (2018)
Cloze Test∗ |  Focuses on predicting the correct token in a code function given a natural language description and a code function with a masked token. |  Accuracy |  Husain et al. (2019); Lu et al. (2021a); Feng et al. (2020)
Code Translation∗ |  Focuses on using machine learning to translate source code from a particular language (e.g. Python) to another language (e.g. JavaScript). Also known as Code Transpiling. |  Exactness, BLEU Papineni et al. (2002), & CodeBLEU Ren et al. (2020) |  Lachaux et al. (2020); Feng et al. (2020)
Code Refinement∗ |  Focuses on automatically rewriting a codebase which is either buggy or convoluted. |  Exactness, BLEU Papineni et al. (2002), & CodeBLEU Ren et al. (2020) |  Hata et al. (2018); Tufano et al. (2019)
Code Completion∗ |  Focuses on generating a code snippet that best completes a given program. |  ROUGE Lin (2004), exactness, edit similarity |  Svyatkovskiy et al. (2019, 2020); Chirkova (2020)
Code Search and Retrieval∗ (Information Retrieval) |  Retrieve a code snippet given a natural language query. |  Mean Reciprocal Rank (MRR) & NDCG Vechtomova (2009); Husain et al. (2019) |  Feng et al. (2020); Husain et al. (2019); Gu et al. (2018)
Code Documentation Generation∗ |  Generating natural language descriptions of code snippets. Includes generating a function name for a function in code. Also known as Code summarization. |  BLEU Papineni et al. (2002) and human evaluation |  Oda et al. (2015a); Feng et al. (2020)
Documentation Translation∗ |  Focuses on translating online code doc pages from one human language to another. |  BLEU Papineni et al. (2002) |  Lu et al. (2021a)
Code Generation∗ |  Generating code snippets given natural language intents. |  Exactness, BLEU Papineni et al. (2002), semantic correctness Liguori et al. (2021b), pass@k Chen et al. (2021a), CodeBLEU Ren et al. (2020) |  Yin and Neubig (2018); Xu et al. (2020a); Lin et al. (2018)
Semantic Parsing |  Converting natural language to unambiguous executable logical forms. |  Exactness |  Chen Jr and Bunescu (2019); Carpenter (1997); Berant and Liang (2014)
∗ In CodeXGLUE. |  |  |
Table 4: Overview of the areas in Code Intelligence (CI) , typical evaluation
methods, and notable works in each area.

Transformer Models with Abstract Syntax Trees. Dahal et al. (2021) explore
using tree representations of natural language utterances in code generation.
Tree representations of natural language utterances are derived using
constituency trees, which aim to describe the syntactic (grammatical)
structure of an uttered sentence by dividing the sentence into sub-phrases.
The authors run a series of experiments generating the AST representation of
code using text-to-AST as a baseline, linearized tree-to-tree which uses the
constituency tree of NL as input encoded using a standard transformer encoder,
and structured tree-to-tree model. For the structured tree-to-tree model the
authors utilize a structure aware Tree Transformer architecture Nguyen et al.
(2020) with a pointer network Vinyals et al. (2015) to copy tree leaves from
the input to the output AST. The authors report improvements when using
structure aware encodings of NL in their structured tree-to-tree model and
achieve 30.30 BLEU on the CoNaLa dataset. Sun et al. (2019b) propose TreeGen a
neural architecture based on the Transformer model that incorporates AST
grammar rules into the network. Incorporating ASTs into Transformers is
observed to alleviate long-range dependencies between source code tokens Sun
et al. (2019b); Dahal et al. (2021) and better models input with multiple
variables Dahal et al. (2021). Multi-task Learning for Code Generation. Multi-
task learning in machine learning is training the model to perform multiple
tasks simultaneously. Wei et al. (2020a) propose using multi-task learning of
code generation and summarization and report improvements in code
summarization and retrieval benchmarks. Wei et al. (2019) report improvements
when dual training a model to generate code and to summarize code
simultaneously. Parvez et al. (2021) propose REDCODER, a code-search augmented
model that retrieves code relevant to the inputted natural language
description from an open source database to supplement the code generator
model.

###  3.9 Other Code Intelligence Areas

Other areas of Code Intelligence (CI) include program repair, which focuses on
building systems that automatically fix bugs in codebases using a compiler
error message and a codebase as input. Notable program repair works include
DeepFix Gupta et al. (2017) and DrRepair Yasunaga and Liang (2020). Defect
detection is another area of CI focused on detecting insecure code primarily
through binary classification of codebases e.g. Li et al. (2018); Zhou et al.
(2019)). Cloze Test is coding multiple choice problem where a model is tasked
with predicting a masked code token. Code translation focuses on translating a
codebase from one language (e.g. Java) to another programming language (e.g
Python). There are various works in code translation Lachaux et al. (2020);
Feng et al. (2020). Code refinement focuses on rewriting a codebase with the
aim of simplification of removing bugs e.g. Hata et al. (2018); Tufano et al.
(2019). Lastly, code completion focuses on generating a code snippet that best
completes a given codebase e.g. Svyatkovskiy et al. (2019, 2020); Chirkova
(2020). See Table 4 for an overview of all the CI fields, metrics used in each
field, and example works.

###  3.10 Summary

Application areas of NLP on source code such as program synthesis and program
repair focus on automating redundant processes of software development.
Program repair tends to focus on repairing programs given a compiler error
message and is evaluated by percent of compilable programs, completely
repaired (no logical errors) and partially repaired programs (some logical
errors). Program synthesis tends to focus on generating full or partial
programs often given structured input such as program examples, program
description and output Shi et al. (2020). Synthesized programs are often
evaluated using percentages (%) of semantic and compilable programs. However,
not all programming synthesis evaluate their programs, some works such as
Cummins et al. (2017b), just focus on their application domain (program run-
time speed-up) and discard generated programs that do not work. On the other
hand, Program analysis is analogous to understanding language as opposed to
generating it. It tends to focus on extracting features from developed code
and using classification methods. CodeXGLUE lists several tasks of code
intelligence and understanding. Those tasks are further listed along with all
other known applications of NLP on source code in Table 4. The table also
lists a description of the task, how the task is evaluated, and works that
have participated in the task domain.

For an extensive survey on machine learning for big code c.f. Allamanis _et
al._ (2018), code generation and semantic parsing Lee et al. (2021), and a
short survey specifically on code generation from natural language Shin and
Nam (2021).

Code generation has promise in being integrated into conversational interfaces
to enhance language understanding (e.g. Artzi and Zettlemoyer (2011)). If code
generation is integrated with conversational assistants we can see a rise of
programming assistants which could be very helpful for programmers. Code
generation may also be helpful for novice programmers and can have
applications in fields such as computer science education. In the next section
we will overview Conversational Artificial Intelligence (A.I.) and application
of conversational artificial intelligence in education and software
engineering.

##  4 Conversational Artificial Intelligence

Figure 15: Example output from a Natural Language Understanding (NLU) module
by Ni et al. (2021). The utterance “Recommend a movie at Golden Village
tonight" is within the “movie" domain and the user’s goal is to “find_movie"
at the destination “Golden Village" with the time “tonight". Here desti and
time are slots and are accompanied by their corresponding slot-values “Golden
Village" and “tonight" respectively.

In this section, we will focus on overviewing deep-learning-based automated
conversational agents, also known as, dialogue systems. Conversational agents
are popular and are widely accessible, from virtual sales agents to personal
assistants like Google, Alex, or Siri, and have been applied in big domains
such as general healthcare Montenegro et al. (2019) and mental healthcare
Weizenbaum et al. (1966); D’Alfonso et al. (2017). There are generally two
types of deep learning dialogue systems: Open-domain dialogue systems (also
known as chit-chat chatbots) and task-oriented dialogue systems (also known as
close-domain dialogue systems) Santhanam and Shaikh (2019); Ni et al. (2021).
Task-oriented systems tend to be oriented towards a specific goal or a task
such as booking a hotel, reserving a restaurant table, etc. Task-Oriented
dialogue systems such as SimpleTOD Hosseini-Asl et al. (2020) are often
entirely data-driven and are proficient in a certain set of domains given
enough training examples within that domain. Task-oriented systems are often
made up of 4 modules: Natural Language Understanding (NLU), Dialogue State
Tracking (DST), Dialogue Policy Learning, and Natural Language Generation
(NLG). NLU focuses on classifying what the user’s goal is within a domain and
parses out task-relevant words from the user utterance into slots c.f. Figure
15. DST focuses on calibrating the dialogue state based on the current user
utterance and the conversational history, it includes the user’s goal and
slot-value pairs. Dialogue Policy Learning is a module that decides the next
action the system should take based on the dialogue state. NLG focuses on
generating natural language from the selected dialogue action. Open-domain
dialogue systems such as MILABOT Serban et al. (2017) focus on chatting with a
user without any domain restrictions and can cover a wide variety of topics
Ram et al. (2018). For a survey on dialogue systems c.f. Santhanam and Shaikh
(2019); Ni et al. (2021). Next, we will overview works that apply task-
oriented dialogue systems in software engineering and computer science
education.

###  4.1 Conversational Assistants for Software Engineering

This section focuses on overviewing some of the uses of conversational systems
in the field of software engineering. We observe conversational assistants
assisting programming in performing general programming tasks and assisting
with specific software engineering workflows (e.g. Github version control
actions).
General Programming Assistants. Chaurisa and Mooney (2017) propose a system
that can engage a human user in dialogue for IFTTT code generation Quirk et
al. (2015). However, the dialogue system mainly engages with the user to
clarify their intent until the correct code is produced. Austin et al. (2021)
run controlled experiments with a simple dialogue system that generates Python
code. The system mainly focuses on collaborating with a human to solve a
particular programming task primarily through asking for clarifications and
modifying the generated code as illustrated in Figure 16. These dialogue
systems are simple and are mainly focused on solving one particular program,
they neither include modules for dialogue state tracking nor code
understanding. The dialogue systems also do not seek to collaborate with the