Debugging with Open-Source Large Language Models: An Evaluation
Yacine Majdoub Eya Ben Charrada






Debugging with Open-Source Large Language Models: An Evaluation]Debugging with Open-Source Large Language Models:
 An Evaluation










  University of Gabes, Tunisia
  

yacinemajdoub@fsg.u-gabes.tn



  University of Gabes, Tunisia
  

eya.bencharrada@fsg.rnu.tn














§ ABSTRACT 

Large language models have shown good potential in supporting software development tasks. This is why more and more developers turn to LLMs (e.g. ChatGPT) to support them in fixing their buggy code. While this can save time and effort, many companies prohibit it due to strict code sharing policies. To address this, companies can run open-source LLMs locally. But until now there is not much research evaluating the performance of open-source large language models in debugging. This work is a preliminary evaluation of the capabilities of open-source LLMs in fixing buggy code. The evaluation covers five open-source large language models and uses the benchmark DebugBench which includes more than 4000 buggy code instances written in Python, Java and C++. Open-source LLMs achieved scores ranging from 43.9% to 66.6% with DeepSeek-Coder achieving the best score for all three programming languages.  



























  
    < g r a p h i c s >

  Seattle Mariners at Spring Training, 2010.
  Enjoying the baseball game from the third-base
  seats. Ichiro Suzuki preparing to bat.
  


20 February 2007
[revised]12 March 2009
[accepted]5 June 2009





[
    Eya Ben Charrada
    
====================





§ INTRODUCTION


 


 "I'd spend an hour figuring out what exactly goes wrong, then five minutes writing the code to fix it, and then half an hour testing the whole thing. That's just over 5% coding vs. almost 95% non-coding."[text taken from an answer on stackoverflow regarding the time spent debugging https://softwareengineering.stackexchange.com/a/93323] 

 

 
Debugging is known to be time consuming and frustrating. Therefore it is not surprising to find out that developers are turning to large language models to help them solve their problems. In a study with practitioners, Khojah et al. <cit.> found that software engineers were found to turn often to chatGPT for assistance in various software engineering tasks.

Recent research showed promising results in using LLMs for software Engineering tasks in general and for debugging in particular.  For example, LLMs were able to perform well in bug reproduction <cit.>, fault localisation <cit.> and program repair <cit.>. 

Despite these advantages, using current state of the art LLMs such as 
ChatGPT can be inappropriate for practitioners due to code sharing policies. In fact, most companies consider their code to be private and don't want it to be sent to LLMs run by third parties.
A solution to this problem would be to run an open source LLM locally. 
So far, there has been very limited assessments of the debugging capabilities of open-source large language models. In fact, earlier works mostly focus on evaluating code generation capabilities, for which many benchmarks exist such as the famous OpenAI's HumanEval <cit.> and its descendants (e.g. HumanEval+ <cit.> and  Multilingual HumanEval <cit.>) or the Google's MBPP <cit.>. 

The goal of this work is to evaluate and compare the capabilities of  open-source  large language models in performing debugging tasks. We would like to answer the following two research questions:

    
  * RQ1: How do open source LLMs perform in debugging? To answer this question, we use benchmarking to evaluate five open-source LLMs. The benchmark we used includes more than 4000 buggy code instances in Python, C++ and Java.
    
  * RQ2: How does the performance of open-source LLMs in code generation impact their performance in debugging? We compare the scores that the LLMs obtained for debugging with the scores that they achieved for coding as evaluated by the HumanEval Benchmark.


Our evaluation suggests that although less capable than the most advanced closed source models (e.g. GPT-4), some open source models were able to achieve decent results compared to their relatively small size. For instance, DeepSeek-coder-instruct, which has only 34B parameters, achieved a score above 63% in all three programming languages. We also found that except for DeepSeek-coder, all models that achieved a higher scores in HumanEval also got better scores in debugging. 

The contributions of this work are:

    
  * We conduct an empirical study that evaluates the debugging capabilities of  open source Large Language Models using a large benchmark that includes a few thousands of buggy code instances
    
  * We compare the debugging capabilities of the open source LLMs to their coding capabilities as evaluated by the HumanEval benchmark
    
  * We provide an extensive discussion of the strengths and limitations of current debugging and coding benchmarks








§ OPEN SOURCE LARGE LANGUAGE MODELS
 
There are many open-source LLMs available in the market. Although nearly[All models we found used the transformer architecture.] all models use the transformer architecture, they differ in their capabilities due to various factors such as model size, quality and volume of training data, and fine-tuning methods.

For this evaluation, we selected five reputed models. Four of them are code models, while the last one is a general-purpose model.



 §.§ Code models

We selected the coding models that  achieved the best results on the HumanEval benchmark <cit.>. HumanEval is a code generation benchmark released by OpenAI that includes 146 coding tasks.
We present each of the coding models in the following paragraphs.



  §.§.§ Code Llama
 Code Llama <cit.>
is a  family of large language models that is specialised for code, based on Llama2. Code Llama models have been created by fine-tuning the general language model Llama2 using code specific datasets. The developers of Codellama found that for a given budget, fine-tuning the generic Llama2 to generate code outperforms the same architecture trained on code only.
The training was done with publicly available code (mostly near-deduplicated dataset), which includes 8% of natural language text related to code such as discussions or questions and answers including code snippets. In addition to supporting several natural languages, the Code Llama models are trained to handle long contexts of up to 100K tokens. 
Meta AI released Codellama in three main variants namely (1) Code Llama, which is the foundation model (2) Code Llama - Python, which is specialized for python code generation and Code Llama - Instruct, which is fine-tuned to follow human instructions. All models are available in four sizes: 7B, 13B, 34B and 70B.

For this evaluation, we use the Code Llama - Instruct 70B variant. This variant was trained using 1 trillion tokens and achieved the best performance on HumanEval with a 67.8% pass@1.




  §.§.§ Phind-Codellama

Phind-Codellama <cit.> is a fine-tuned version of Code Llama 34B. The first version of Phind-Codellama was fine-tuned on a dataset of nearly 80,000 programming problems and their corresponding solutions. The second version is Phind-CodeLlama-34-v2, which was initialised from the first version, was trained on 1.5B additional tokens. Although Phind-Codellama has smaller number of parameters  compared to the larger Code Llama 70B, it was able to achive relatively high results on HumanEval. For instance Phind-CodeLlama-34B-v2 achieved 73.8% pass@1 on HumanEval.








  §.§.§ WizardCoder
 WizardCoder <cit.> is a family of LLMs that use the Evol-Instruct method <cit.>, an instruction fine tuning method that  makes the code instructions more complex and which enhances the performance of  coding models. 
Wizardcoder is available in five different sizes ranging from 1B to 33B parameters. The 15B version of WizardCoder <cit.>, the results of a collaboration between researchers from Microsoft and researchers from Haong Kong Baptist University, is a fine-tuned version of StarCoder <cit.> and it achieved 57.3 %  pass@1 on HumanEval. The 33B version is  trained from the DeepSeek-Coder-base model and achieved 79.9%  pass@1 on HumanEval <cit.>.   In this evaluation we use the WizardCoder-33B-V1.1.



  §.§.§ Deepseek-Coder
 DeepSeek-Coder <cit.> is a series of code models trained on a dataset comprising 2 trillion tokens from 87 programming languages. The dataset  is composed of  87% code and 13% natural language in English and Chinese. The model is available in various sizes, from 1.3B to 33B parameters.


The DeepSeek-Coder-Instruct variant is an enhancement of the base model that was fine-tuned with an additional 2 billion tokens of instruction data. This improved the model's ability to execute coding tasks given using human instructions. DeepSeek-Coder-Base 33B achieved 50.3%  pass@1 on HumanEval, while  DeepSeek-Coder-Instruct-33B achieved 69.2%  pass@1 on HumanEval. We used DeepSeek-Coder-Instruct-33B in our evaluation.  





 §.§ General-Purpose model: Llama 3





The last model we chose is Llama3, a general purpose LLM. We selected it because it is the best open source LLM available for now, and we wanted to compare its capabilities to the code-specialized large language models.

LLama3, which is developed by Meta AI,  was released in two sizes: 8B and 70B each with a pre-trained and instruction finetuned version. 
Data quality was a major focus for LLama 3, the model has been pre-trained on over 15 trillion high-quality tokens from publicly available sources, seven times more than LLama 2. The training data incorporates four times more coding data to boost capabilities in that domain and over 5% of the data covers 30+ languages beyond English. The dataset was filtered using a serie of filtering pipelines, heuristic filtering, NSFW detection, deduplication, and quality classifiers.
The model also utilizes a more efficient tokenizer compared to the previous models of Meta AI, and it uses grouped query attention (GQA) to improve inference efficiency and to handle sequences of up to 8,192 tokens.

Llama3 8B achieved 62.2% pass@1 on HumanEval while Llama3 70B  achieved 81.7% pass@1 <cit.>. For this evaluation, we used Llama3 70B. 

