We study the problem of semantic code repair, which can be broadly defined as automatically fixing non-syntactic bugs in source code. The majority of past work in semantic code repair assumed access to unit tests against which candidate repairs could be validated. In contrast, the goal here is to develop a strong statistical model to accurately predict both bug locations and exact fixes without access to information about the intended correct behavior of the program. Achieving such a goal requires a robust contextual repair model, which we train on a large corpus of real-world source code that has been augmented with synthetically injected bugs. Our framework adopts a two-stage approach where first a large set of repair candidates are generated by rule-based processors, and then these candidates are scored by a statistical model using a novel neural network architecture which we refer to as Share, Specialize, and Compete. Specifically, the architecture (1) generates a  shared encoding of the source code using an RNN over the abstract syntax tree, (2) scores each candidate repair using specialized network modules, and (3) then normalizes these scores together so they can compete against one another in comparable probability space. We evaluate our model on a real-world test set gathered from GitHub containing four common categories of bugs. Our model is able to predict the exact correct repair 41% of the time with a single guess, compared to 13% accuracy for an attentional sequence-to-sequence model. The term automatic code repair is typically used to describe two overarching tasks: The first involves fixing syntactic errors, which are malformations that cause the code to not adhere to some language specification BID9 BID5 . The second, which is the focus of this work, involves fixing semantic bugs, which refer to any case where the actual program behavior is not the same as the behavior the programmer intended. Clearly, this covers an extremely wide range of code issues, so this work is limited to a class of semantic bugs, which we roughly define as: "Bugs that can be identified and fixed by an experienced human programmer, without running the code or having deep contextual knowledge of the program." This does not imply that the bugs are trivially fixable, as they often require time-consuming analysis of the code, rich background knowledge of the language and APIs, and complex logical reasoning about the original programmer's intent.Unlike previous work, we do not assume access to unit tests at training or test time. This requirement is important because it forces development of models which can infer intended semantic purpose from source code before proposing repairs, as a human programmer might. Most previous work relies on unit tests -a common theme is combining coarse-grained repair models with search algorithms to find some repair that satisfies unit tests BID10 BID18 . In contrast, our proposed task requires models to deeply understand the code in order to propose a single set of repairs. Thus, semantic code repair without unit tests presents a concrete, real-world test bed for the more general task of understanding and modifying source code.Our semantic repair model was trained on a large corpus of open-source Python projects with synthetically injected bugs. We test on both real-bug and synthetic-bug test sets. 1 To train the repair model, we first evaluated an attentional sequence-to-sequence architecture. Although this model was able to achieve non-trivial results, we believe it to be an unsuitable solution in a number of ways, such as the lack of direct competition between repair candidates at different locations. Instead, we use an alternative approach which decouples the non-statistical process of generating and applying repair proposal from the statistical process of scoring and ranking repairs.This two-stage process itself is not new, but the core novelty in this work is the specific neural framework we propose for scoring repair candidates. We refer to our architecture as a Share, Specialize, and Compete (SSC) network:• SHARE: The input code snippet is encoded with a neural network. This is a shared representation used by all repair types.• SPECIALIZE: Each repair type is associated with its own specialized neural module BID2 , which emits a score for every repair candidate of that type.• COMPETE: The raw scores from the specialized modules are normalized to compete in comparable probability space.Our model can also be thought of as an evolution of work on neural code completion and summarization BID1 BID6 . Like those systems, our SHARE network is used to learn a rich semantic understanding of the code snippet. Our SPECIALIZE modules then build on top of this representation to learn how to identify and fix specific bug types.Although we have described our framework in relation to the problem of code repair, it has a number of other possible applications in sequence transformation scenarios where the input and output sequences have high overlap. For example, it could be applied to natural language grammar correction BID17 , machine translation post editing BID11 , source code refactoring BID0 , or program optimization BID7 . Our first goal is to conceptually understand at what "level" the model was able to generalize to new snippets. Although the hidden activations of the neural network model are not directly interpretable, we can attempt to interpret the latent model space using nearest neighbor retrieval on the hidden vectors h i . The goal is to determine if the model is simply memorizing common n-grams, or if it is actually learning high-level repair concepts. Nearest neighbor retrieval for several test snippets are presented here:In Example 1, we see the model is able to learn a high-level pattern "y.x = x". In Example 2 we see the pattern "if (x c 1 y...) elif (x c 2 y...)". In Example 3 we see the pattern "Strings usually use the equality (or inequality) operator." In all cases, the surface form of the training nearest neighbor is very different from the test snippet. From this, it appears that the SSC model is able to learn a number of interesting, high-level patterns which it uses to generalize to new data.We next examined failure cases of the SSC model which a human evaluator was able to repair correctly. Here, the primary weakness of the model was that humans were able to better infer program intent by using variable names, function names, and string literals. One major fault in the current implementation is a lack of sub-word representation. For example, consider a repair of the expression "dtypes.append(x )" where x could be dtype or syncnode. It is easy for a human to infer that dtype is the more sensible choice even without deeper understand of the code. In future work we plan to explore character-level encoding of value strings so that lexical similarity can be modeled latently by the network.We finally examined cases where the SSC model succeeded but the human evaluator failed. Generally, we conclude that the model's primary advantage was the sheer amount of data it was able to learn from. For example, consider the expression "if (db.version_info <= 3)". This may not be immediately suspicious to a human, but if we analyze the reference training data we can measure that the pattern "if (x.version_info <= y )" is 10 times less frequent than the pattern "if (x.version_info < y)". Intuitively, this makes sense because if a feature is added in version y, it is not useful to check <= y. However, the neural model is able to easily learn such probabilistic distributions even without deeper understanding of why they are true. We presented a novel neural network architecture that allows specialized network modules to explicitly model different transformation types based on a shared input representation. When applied to the domain of semantic code repair, our model achieves high accuracy relative to a seq2seq baseline and an expert human evaluation. In our analysis of the results, we find that our system is able to learn fairly sophisticated repair patterns from the training data. In future work we plan to expand our model to cover a larger set of bug types, and ideally these bug types would be learned automatically from a corpus of real-world bugs. We also plan to apply the SSC model to other tasks. The application of a pooled pointer module at a single time step, to predict the variable replacement scores for each potential replacement of the token fname. The input here is the per-token representation computed by the SHARE module. Representations for variable names are passed through a pooling module which outputs per-variable pooled representations. These representations are then passed through a similarity module, as in standard pointer networks, to yield a (dynamically-sized) output dictionary containing one score for each unique variable.
A neural architecture for scoring and ranking program repair candidates to perform semantic program repair statically without access to unit tests. Presents a neural network architecture consisting of the share, specialize and compete parts for repairing code in four cases.