Paper Title: Can Developers Prompt? A Controlled Experiment for Code Documentation Generation.














Can Developers Prompt? A Controlled Experiment for Code Documentation Generation
    

Hans-Alexander Kruse 0009-0006-5568-7841,
Tim Puhlfürß 0000-0001-8421-8071,
Walid Maalej 0000-0002-6899-4393


Universität Hamburg
  
Hamburg, Germany

hans-alexander.kruse@studium.uni-hamburg.de,
tim.puhlfuerss@uni-hamburg.de,
walid.maalej@uni-hamburg.de

    August 5, 2024
======================================================================================================================================================================================================================================================================



§ ABSTRACT 

llm bear great potential for automating tedious development tasks such as creating and maintaining code documentation.

However, it is unclear to what extent developers can effectively prompt llm to create concise and useful documentation.

We report on a controlled experiment with 20 professionals and 30 computer science students tasked with code documentation generation for two Python functions.
The experimental group freely entered ad-hoc prompts in a ChatGPT-like extension of Visual Studio Code, while the control group executed a predefined few-shot prompt.

Our results reveal that professionals and students were unaware of or unable to apply prompt engineering techniques.
Especially students perceived the documentation produced from ad-hoc prompts as significantly less readable, less concise, and less helpful than documentation from prepared prompts.
Some professionals produced higher quality documentation by just including the keyword Docstring in their ad-hoc prompts.
While students desired more support in formulating prompts, professionals appreciated the flexibility of ad-hoc prompting.
Participants in both groups rarely assessed the output as perfect.
Instead, they understood the tools as support to iteratively refine the documentation.

Further research is needed to understand which prompting skills and preferences developers have and which support they need for certain tasks.



Software Documentation,
Large Language Model,
Program Comprehension,
Developer Study,
AI4SE


Can Developers Prompt? A Controlled Experiment for Code Documentation Generation
    

Hans-Alexander Kruse 0009-0006-5568-7841,
Tim Puhlfürß 0000-0001-8421-8071,
Walid Maalej 0000-0002-6899-4393


Universität Hamburg
  
Hamburg, Germany

hans-alexander.kruse@studium.uni-hamburg.de,
tim.puhlfuerss@uni-hamburg.de,
walid.maalej@uni-hamburg.de

    August 5, 2024
======================================================================================================================================================================================================================================================================










§ INTRODUCTION




Developers often overlook or ignore software documentation and generally assign it a low priority <cit.>. 
Yet, carefully documenting code is an essential task in software engineering. 
Up-to-date and high-quality documentation facilitates program comprehension <cit.>, accelerates developer onboarding <cit.>, and mitigates technical debt <cit.>. 
Software documentation is also central to software maintenance, as documentation often requires updates and should evolve with software  <cit.>.


Researchers and tool vendors have thus investigated different ways to automate documentation <cit.>. 
In particular, llm designed to model and generate human language <cit.> hold great potential in automating documentation tasks <cit.>.
Among popular llm use cases, code summarization has recently gained much attention <cit.>.
However, the output of llm usually depends on the user input, called prompt. 
Slightly changing the prompt can lead to different results regarding conciseness, language style, and content of generated text <cit.>. 


Several prompt engineering techniques have emerged to optimize interactions with llm <cit.>.
Few-shot prompting is one of the most prominent techniques and comprises adding example outputs to a predefined prompt to specify output requirements <cit.>.
This technique allows users to optimize llm responses while minimizing the number of input messages sent to the model <cit.>.


Previous research has primarily focused on benchmarking the performance of various models and prompt engineering techniques for generating source code documentation <cit.>.
Recently, Ahmed and Devanbu concluded that common prompt engineering techniques perform better than ad-hoc prompting according to popular benchmark metrics <cit.>.
However, studies focusing on the perspectives of developers when using llm, e.g., to generate documentation, are still sparse.
In fact, a recent study showed that common metrics to evaluate llm do not align with human evaluation to assess the quality of generated documentation <cit.>.
Hence, it remains uncertain whether prompt engineering techniques meet developers' requirements for generating and using code documentation.
Moreover, it is unclear whether developers prefer a flexible, iterative interaction using chatbot-like ad-hoc prompting or a rather transaction-like execution of predefined prompts.

This study takes a first step towards filling this gap, focusing on two research questions:



  RQ1: How well can developers prompt llm to generate code documentation compared to a predefined prompt?

  RQ2: How is the developer experience for ad-hoc prompting compared to executing predefined prompts?



To answer these questions, we conducted a randomized controlled experiment with 20 professional developers and 30 computer science students.
The experiment involved generating code documentation using an llm-powered ide.
The first group utilized a vscode extension based on GPT-4, enabling participants to enter ad-hoc prompts for selected code.
The second group used a similar vscode extension that executed a predefined few-shot prompt to generate code documentation via GPT-4 with a single click.



Overall, we observed that developers with less experience require more guidance in phrasing an ad-hoc prompt, whereas more experienced developers can generate fairly good documentation by including specific keywords like Docstring in the prompts.
Overall, participants preferred transactional or guided llm interactions to create code documentation while enjoying the flexibility of ad-hoc and iterative prompting.
Our results show that predefined prompts deliver code documentation of significantly higher quality than ad-hoc prompts, especially due to a consistent documentation format.

We present the design of our study in Section <ref> and report on the results in Section <ref> (RQ1) and Section <ref> (RQ2). 
We then discuss the implication of our findings in Section <ref> and potential threats to validity in Section <ref>. 
Finally, Section <ref> summarizes related work, and Section <ref> concludes the paper.
We share our experiment results, including the ide extension, in our replication package <cit.>.







§ METHODOLOGY





 §.§ Experimental Design




We designed a controlled experiment in a laboratory setting to compare developer usage of and experience with two vscode extensions employing different prompts for code documentation generation.
We followed a between-subject design, with one participant testing only one of both tools to avoid learning effects <cit.>.
The ad-hoc prompt group created prompts in an already available GPT-4-powered ide extension, while the predefined prompt group interacted with our extension.
We randomly assigned participants to the groups, with adjustments made to balance the occupations. 
Participants were unaware of their group.


We offered offline and online versions of the experiment.
In the offline version, participants used our computer with vscode and the digital questionnaire displayed on two screens.
In the online version, we utilized Zoom for video conferencing, sharing a questionnaire hyperlink, and screen-sharing our vscode window with external input control.
Participants employed Thinking Aloud to express thoughts during the task <cit.>.
To ensure a calm environment, we put a participant in a room empty of people.
The questionnaire included questions about the code and the generated documentation.
Additionally, we collected the ad-hoc prompts for a later comparison with the predefined prompt.


The experiment comprised three parts, as shown in Figure <ref>.
In the first part, participants answered questions about occupation, as well as Python and vscode experience to assess their ability to understand complex functions and use the ide.
Furthermore, we could check whether these variables have any effect on the observed behavior.

The second part focused on code comprehension and documentation generation.
It consisted of two rounds, each involving a pre-selected Python function (Listing <ref>).
Per round, participants (1) first attempted to comprehend the code without any documentation.
This enabled us to analyze the effect of the llm-generated documentation on understanding the code functions.
Subsequently, (2) participants rated their comprehension on a scale from 1 (very low) to 5 (very high), and answered True/False questions about the code to further check their understanding (Table <ref>).
Participants could also state that they had insufficient information to answer a question.
Furthermore, (3) participants manually created a function comment to externalize their understanding and better assess the quality of the comment subsequently generated by the tool.
To guide the participants and shorten the completion time, we provided the participants with a comment template adhering to guidelines for clean Python code <cit.> and instructed them to add only the function description without the explanations of parameters.
Participants then used the respective llm-powered ide extension to generate code documentation.
Afterward, (4) as documentation should serve to facilitate code comprehension, they studied the generated comment and revisited the True/False questions.
This step provided insights if the documentation supported or even hindered the comprehension.
Finally, (5) they rated the generated documentation on six quality dimensions based on the human evaluation metrics by Hu et al. <cit.>.

The documentation quality dimensions each provided a question and a five-point answer scale, addressing grammatical correctness, readability, missing relevant information, unnecessary information presence, usefulness for developers, and helpfulness for code comprehension. 
We made minor adjustments to the original answers of two dimensions as we found them overloaded for our goal.
In particular, the answers for grammatical correctness originally also included a rating of fluency, and the original readability answers also concerned the comment's understandability.
The lowest point of each scale represents the negative pole, like very low readability, while the highest point represents the positive pole, like very low amount of unnecessary information.

In the third part of the experiment, participants assessed the usability of the respective ide extension.
They completed the standardized ueq with 26 items, each associated with a seven-point answer scale and one of six usability categories <cit.>.
Finally, participants provided comments on tool strengths and areas for improvement in two free-text fields.



 §.§ Ethical Considerations



We followed the standard procedure of the ethics committee of our department for assessing and mitigating risks regarding ethics and data processing.
Upon welcoming participants, we explained the privacy policy, study purpose, upcoming tasks, and data processing procedures.
We designed the study to last for approx. 30 minutes per participant to prevent exhaustion and instructed participants not to seek perfect solutions to reduce stress. 
We offered clarifications on tasks and questionnaire content without providing solutions.
We minimized observational bias by facing away while participants answered the questionnaire and also encouraged negative feedback <cit.>.
To express our gratitude for the participation, we raffled vouchers among participants.
We pseudonymized all published data as far as possible while maintaining our study objective.



 §.§ Technical Setup



[float, caption=Python functions that participants had to comprehend and for which they generated documentation., label=lst:exp_fct]list=no
Function 1:
def string_from_vector_bool(data):
    return ",".join(str(int(i)) for i in data)
—————————————————-
Function 2:
def _parse_date(date):
    if date is None:
        date = Timestamp()
    if isinstance(date, Timestamp):
        date = date.toLocal()
    d = dateutil.parser.parse(date)
    if d.tzinfo is None:
        d = d.replace(tzinfo=dateutil.tz.tzlocal())
    return d.astimezone(dateutil.tz.tzutc())
    .replace(tzinfo=None).isoformat()


[float, caption=Predefined few-shot prompt. Few-shots 2 and 3 are comparable to 1 and included in the replication package <cit.>., label=lst:const_prompt, xleftmargin=0pt]list=no
For the following prompt, take into account these 
three input/output pairs of functions and 
corresponding appropriate comments:

Function 1:
def elapsed_tid(cls, reference, new):
    time_difference = new.toTimestamp() - reference.toTimestamp()
    return np.int64(time_difference * 1.0e6 // cls._period)
Comment 1:
"""
    Calculate the elapsed trainId between reference and newest timestamp

    :param reference: the reference timestamp
    :param new: the new timestamp

    :type reference: Timestamp
    :type new: Timestamp

    :returns: elapsed trainId's between reference and new timestamp
"""
Function 2:
...

Generate a comment for the following function: 
FUNCTION_CONTENT. Fill in this template: 
TEMPLATE. Adhere to the appropriate comment 
syntax for multi-line comments.




  §.§.§ Selection of Experiment Tasks



We selected the Python source code for the experiment tasks from the open-source control system Karabo <cit.>.
Karabo is maintained by the scientific facility euxfel, whose developers partially participated in the experiment.
By choosing this project and involving project insiders, we could assess how their programming and domain experience supports their code comprehension and documentation rating, especially in comparison to outsiders.

Two authors conducted a thorough screening of available Karabo functions and identified six candidates.
All six were utility functions, which we deemed comprehensible for experiment participants not associated with euxfel.
Our selection criteria also included code documentation quality aspects such as conciseness, completeness, and usefulness especially relevant for the few-shot examples <cit.>.
We independently analyzed the candidates and categorized them into the complexity levels easy, medium, and hard.
Afterward, we resolved categorization conflicts by discussion.

We selected one easy-, and one medium-rated function for the tasks of the experiment (Listing <ref>).
The first function converts a boolean vector into a comma-separated string.
We considered the code easy to understand as only standard Python operations were used within this short function.
The second function converts a given date to the ISO format.
The complexity of this function lies in the usage of multiple classes and functions with abbreviated identifiers, and the lack of inline comments.
Hence, participants must make assumptions based on identifier names and usage.

Initially, we also selected a third function rated hard but removed it after a pilot study with three software engineering researchers, who reported difficulties with this function and far exceeded the time limit.
While we acknowledge that llm might require more sophisticated prompts and context information to generate appropriate documentation for more complex code, our user study focused on how differently skilled developers can and prefer to interact with an llm for the creative process of documentation generation.
Therefore, within the experiment's time limit, all participants should be able to phrase ad-hoc prompts that generate appropriate comments for these functions.



  §.§.§ LLM Selection and Prompt Construction



Research and industry have introduced multiple llm with individual strengths and limitations. 
For our study, we focused solely on the GPT-4 model by OpenAI (version gpt-4-0613). 
We did not aim to benchmark different models but rather to study developers prompting skills particularly for code documentation generation.
OpenAI's models have been prominent in recent research on documentation generation <cit.>. 
Furthermore, during our study period, GPT-4 demonstrated general superiority over other models <cit.>.

We defined the predefined few-shot prompt (Listing <ref>) based on prompt engineering guidelines <cit.> and tested it in a pilot study.
Given GPT-4's limited context window at the time of the study, we applied a three-shot approach to specify the required output format.
We chose three distinct function-comment pairs from the remaining previously selected Karabo code candidates.
Hence, these functions were from the same domain and of similar complexity as the two used in the experiment.
They adhered to a structured documentation format with precise and concise content.
We instructed GPT-4 to generate a comment for the selected source code that we passed as input (FUNCTION_CONTENT).
To further enforce the clean code format <cit.>, we explicitly defined an output template within the prompt (TEMPLATE) and requested adherence to Python's syntax for multi-line comments (Docstrings).
Listing <ref> only contains example 1 of the few-shot prompt. 
Examples 2 and 3 are available in our replication package <cit.>.
We acknowledge that the prompt can be further optimized and generalized to multiple programming languages and application domains.
We considered its performance and complexity sufficient for our research goal.



  §.§.§ Tool of the Ad-hoc Prompt Group





We focused on ide extensions as developers usually use ide for generating code documentation.
We chose vscode for its extensive customization capabilities through various extensions maintained by an open-source community. 
During our study, the extension marketplace already offered multiple ChatGPT-like tools that we examined.
Additionally, vscode supports multiple programming languages, including Python.

We selected the ad-hoc prompt tool based on two criteria.
First, it needed to provide a user interface similar to the ChatGPT interface, allowing users to engage in a conversation with the software. 
This approach involves at least one self-written message to the model and copying and pasting the generated comment from the chat interface to the code.
This criterion was crucial as we aimed to compare the user experience for this classic Q&A-based interaction with the one for the predefined prompt tool, which generates and inserts code documentation with a button click.
Second, the tool had to support and behave like the original GPT-4 api to handle any confounding factors related to the use of different llm models.

After a thorough testing of available extensions, we opted for ChatGPT by zhang-renyang <cit.> in version 1.6.62. 
This extension met all specified requirements.
It offers a small prompt field (Figure <ref>) and a larger ChatGPT-like chat window within the ide.
Our test showed that it mimics GPT-4.
It was one of the most popular GPT extensions in the marketplace during our study period.



  §.§.§ Tool of the Predefined Prompt Group




Our design objective for the predefined prompt group was to minimize the steps required for developers to generate a high-quality Docstring comment for a specific code section. 
We determined that marking the code within the ide's code editor and selecting the Generate a comment action from the context menu should suffice, requiring three clicks to generate documentation (Figure <ref>).
Alternatively, the developer can choose this action through vscode's command field. 
Hence, inexperienced and experienced vscode users can use the primary feature of the tool with their preferred workflow.
A secondary feature allows developers to customize the predefined prompt via the Edit default prompt action, enabling them to tailor the tool output to their preferred format.


We followed the vscode guidelines to create the TypeScript-based extension <cit.> and utilized the axios package to implement calls to the OpenAI api.
Based on the input prompt, the api returns a code comment generated by GPT-4.
The extension automatically inserts this documentation above the previously selected code.
To set up the extension, developers must enter their OpenAI api token into a configuration field.
This token is stored locally on the developer's machine.
For the experiment, we pre-configured the tool with our token.


We used GPT-4 version gpt-4-0613 for both tools. 
Generating code documentation with this model took, on average, four seconds.
The vscode version was 1.85.
The predefined prompt tool is included in our replication package <cit.> and also available in the vscode marketplace <cit.>.



 §.§ Experiment Execution and Participants




The experiment took place in October and November 2023, involving 50 participants: 25 per ad-hoc and predefined prompt group. 
20 participants were professionals, including 12 software engineers, four data scientists, and four with other software development roles. 17 worked in the domains of scientific computing and three in finance.
These participants took the expert perspective due to their professional experience with Python programming and, eventually, the application domain.
The remaining 30 participants were students in computer science or related subjects, taking the newcomer perspective.
We recruited participants through personal contacts in academia and industry to ensure that the participants had at least some experience with Python and vscode. 
We conducted 33 experiments online and 17 offline at the euxfel campus in Schenefeld, Germany, with the first author leading the experiment and the second author assisting.

On a scale from 1 (very low) to 5 (very high), the mean value and standard deviation of the Python experience among professionals were 3.7 (1.06) in the ad-hoc and 4.5 (0.71) in the predefined prompt group.
The values among students were notably lower, with 2.53 (0.83) in the ad-hoc and 2.47 (1.19) in the predefined prompt group.
This difference confirms the expert/newcomer setting.

All participants were familiar with vscode, which was important to mitigate biases due to the tool setup.
Mean values and standard deviations in the ad-hoc/predefined prompt group among professionals were 2.9 (0.99) and 3.5 (1.18); among students they were 3.07 (1.03) and 3.07 (1.28).

At the beginning of each session, we trained each participant in using the respective tool to mitigate learning bias.
We introduced them to the tool features, generated documentation for an example function, and answered tool-related questions.
Afterward, we started the experiment.
Due to manually writing prompts, ad-hoc prompt group participants completed the experiment after 23:53 minutes on average, while predefined prompt group participants required only 19:58 minutes.



 §.§ Data Analysis




Our experiment comprised quantitative and qualitative data analyses.
We report totals, mean values, and standard deviations where applicable.
Furthermore, we employed inferential statistics to test for significant differences between the groups' ratings of the six documentation quality dimensions. 
Depending on the statistical assumptions fulfilled by the respective variables, we utilized the parametric Welch's t test or the non-parametric mwu test <cit.>.

The assumptions for both tests were that the collected data points were quantitative and independent.
The first assumption was fulfilled as we provided a five-point answer scale for each question related to a quality dimension.
The second assumption was fulfilled as we conducted the experiment with each participant individually, and we asked participants to not share information with others to avoid external influences.
The third assumption of the Welch's t test is a normal distribution of data points, which we tested with the Shapiro-Wilk test <cit.>.
If a normal distribution was not met for a specific quality dimension, we applied the mwu test instead.
Besides reporting the p-values for statistical significance, we also report the effect sizes to indicate practical significance.
We particularly applied Cliff's δ <cit.>, which describes the overlap of two groups of ordinal numbers.
Values between 0.0 and 0.8 indicate a low to medium effect size, whereas values from 0.8 represent a large effect size, meaning a large difference between both groups.


To quantify the experience of developers using the experiment tools, we utilized the 26 seven-point scales of the ueq <cit.> and its official data analysis tool <cit.>.
This tool provides descriptive (mean values and standard deviations) and inferential statistics (t test) to compare user experience ratings for the two studied ide extensions.

Each of the 26 scales is related to one of six quality categories.
The first category, Attractiveness, represents the pure valence of and overall satisfaction with a tool.
Efficiency, Perspicuity, and Dependability are the pragmatic, goal-directed categories, and Stimulation and Novelty are hedonistic, not goal-directed categories.
We chose Efficiency, Perspicuity, Dependability, and Novelty to measure tool-related experience, as these categories focus on tool interactions.

To evaluate participants' assessment of the quality of generated documentation, we used the Stimulation category as it focuses on the output of a tool.
We enhanced this analysis with qualitative insights made by manually analyzing participants' free-text answers to the questions What did you like most about the tool? and  What would you change about the tool?, placed at the end of the questionnaire. 
Two authors independently conducted Open Coding for all answers and discussed their codes to achieve consensus <cit.>.
Furthermore, we report relevant remarks expressed by participants during the experiment.










§ PROMPTING FOR CODE DOCUMENTATION (RQ1)





 §.§ Patterns in Ad-hoc Prompts




The prompts entered by the ad-hoc prompt group varied in detail and keywords. 
About half of the participants had to re-enter a different prompt after their first attempt to generate a code comment.
Among the failed attempts were prompts such as Explain the function and Describe what this function does, which summarized the code in longer prose text.


We categorized the 50 prompts that led to a successful generation of code comments for both code functions into five prompt patterns, listed in Table <ref>.
These patterns commonly included documentation-specific keywords, i.e., comment (34x), Docstring (12), and Python-conform description (1).
Also, slightly longer prompts that included the comprehension-focused terms explain (2) and summarize (1) led to appropriate function comments.
Professionals used the Docstring term more often (9x) than students (3x), which is in line with the higher Python experience of most participating professionals.


Successful prompts were generally short, with a mean length and standard deviation of 7.1 (2.09) words among professionals and 6.7 (2.62) words among students.
The shortest prompt was Generate docstring.
None of the participants enhanced their prompts with examples or templates.



 §.§ Perceived Quality of Generated Documentation




Participants rated the documentation generated by the ad-hoc and predefined prompt tools based on six five-point-scale questions.
All participants were aware of the expected format of a Python Docstring, as they were familiar with Python. 
Moreover, before using the tool, they manually created a function comment based on a Docstring template.
Table <ref> displays the ratings per group, subgroup (student vs. professional), code function, and quality dimension, along with the p-values <cit.> and effect sizes <cit.>.


We performed several trials to determine the appropriate statistical test (see section <ref> for details). 
We rejected the null hypothesis of the Shapiro-Wilk test for all groups and dimensions, indicating a non-normal data distribution, likely due to small sample sizes. 
We chose the mwu test for all comparisons due to its suitability for non-normally distributed data <cit.>.
The null hypothesis of this test indicates no statistically significant difference between the mean values of both groups for a specific dimension. 
We rejected this null hypothesis if the p-value was lower than 0.05.
We noted that in all cases when the null hypothesis was rejected, the effect size δ was large (0.8 or higher), indicating practical significance <cit.>.




Comparing all participants in the ad-hoc and predefined prompt groups, we observed several statistically significant differences.
For function 1, we found significant differences in the dimensions of Readability (4.12/4.72) and Unnecessary Information (3.28/4.32).
The predefined prompt tool consistently provided concise Docstrings that participants perceived as more readable and contained less non-informative content compared to the output of the ad-hoc prompt tool.
For function 2, significant differences were found in four dimensions, with the predefined prompt tool consistently receiving higher ratings than the ad-hoc prompt tool.
The differences in Readability (3.52/4.68) and Unnecessary Information (2.84/4.68) reinforced the conclusions drawn from function 1.

We attribute the ratings of Usefulness (3.52/4.6) to the often excessive output length of the ad-hoc prompt tool.
Participants noted that longer, prose documentation could impede development workflows, as these comments require more time to read while providing little additional information.
Helpfulness was rated high in both groups (4.0/4.6), with some participants suggesting that the explanations provided by both tools were either too complicated or lacked the required level of detail.
Across both groups, participants rated Grammatical Correctness and Missing Information very high.


When we compared documentation ratings between students and professionals, we observed similar statistically significant differences in the ad-hoc and predefined prompt groups for students, while professionals did not exhibit significant differences.
Among students, we observed significant differences for function 1 in the dimensions Readability (4.0/4.66), Unnecessary Information (3.33/4.86), and Usefulness (3.93/4.6).
The different expectations regarding the documentation content contributed to the disparity in Usefulness, with some students preferring the longer summarizations by the ad-hoc prompt tool, while most others expected a structured comment as taught in university.

For function 2, students in the predefined prompt group consistently provided significantly better ratings for Readability (3.2/4.6), Unnecessary Information (2.66/4.86), Usefulness (3.13/4.73), and Helpfulness (3.66/4.6) compared to students in the ad-hoc prompt group.
Unnecessary Information had the highest differences, likely due to the unstructured and lengthy comments generated via the ad-hoc prompts.

Among the professionals, we found no statistically significant differences between the ad-hoc and predefined prompt groups.
However, three dimensions showed large effect sizes (≥ 0.8), indicating a practical significance.
These differences concerned Readability for functions 1 (4.3/4.8) and 2 (4.0/4.8), and Unnecessary Information for function 2 (3.1/4.4).
This indicates that professionals also found the comments of the ad-hoc prompt tool to contain non-informative content, impairing readability.
Conversely, the ad-hoc prompt group provided a more positive rating for the dimension Missing Information (4.7/4.0) for function 1.
Professionals expected more detailed descriptions, which the concise comments by the predefined prompt tool did not provide.
Furthermore, as professionals used the Docstring keyword more often than students in their prompts (Table <ref>), the ad-hoc prompts in these cases resulted in similar documentation to the predefined prompt tool.



 §.§ Code Comprehension




We analyzed how the llm-generated documentation influenced the comprehension of the code.
Initially, participants rated their perceived comprehension of the respective code function without available documentation, on a scale from 1 (not at all) to 5 (very well).
For function 1, the mean and standard deviation for these ratings were 3.6 (0.96) by the ad-hoc and 3.84 (1.11) by the predefined prompt group, indicating a similar good understanding.
The assessment for function 2 was lower, with 2.4 (0.96) by the ad-hoc and 2.92 (0.81) by the predefined prompt group.
Hence, the predefined prompt group expressed slightly higher confidence in the comprehension than the ad-hoc prompt group, despite similar stated Python experience.
Across both groups and functions, professionals stated a higher comprehension than students, which aligns with their higher Python experience.


We assessed participants' actual comprehension through True/False questions about the code (Table <ref>).
We counted the number of participants who correctly answered these questions before and after generating the documentation. 
Participants could also choose the option Not enough information available to answer the question, which we treated as an incorrect answer.

Before using the tools, most participants in both groups answered the two questions for function 1 correctly (Q1: 19+23; Q2: 23+24), as expected for this short and easy-categorized function.
After using the tool, the correct answers for question 1 increased (Q1: 22+24), especially among students, while the ones for question 2 slightly decreased (Q2: 21+20) for both students and professionals.
Thinking-aloud observations revealed that many participants were confused by the less technical documentation generated by both tools regarding this question, as it did not include the term iterable.

The number of correct answers to function 2 before using the tool was lower (Q1: 10+14; Q2: 15+13), reflecting its higher complexity. 
In both groups, correct answers increased after using the tools (Q1: 22+18; Q2: 21+21). 
Students had the highest increases: in the ad-hoc prompt group for Q1 from 5 to 13, and in the predefined prompt group for Q2 from 5 to 14. 
The increase in comprehension among professionals was marginal, except for the correct Q1 answers in the ad-hoc prompt group, which increased from 5 to 9.
This data shows that the availability of documentation helped inexperienced developers to better comprehend the code, while the effect on professionals was mixed.


Answering RQ1, we conclude that developers were able to generate code documentation using ad-hoc prompts without prompt engineering techniques, often by providing relevant documentation keywords.
However, particularly less experienced students initially did not use such keywords and generated lengthy, less readable, and less useful code explanations instead.

This is reflected in participants rating the quality of generated documentation.
We observed statistically significant differences exclusively among students, who rated the readability, conciseness, usefulness, and helpfulness of comments generated via the predefined few-shot prompt higher than those created via ad-hoc prompts.
Although we observed large effect sizes for some quality dimensions among professionals, this subgroup was generally less critical as they often generated documentation of the required quality and format by using specific keywords.

We also conclude that documentation generated by both tools helped (particularly students) comprehend more complex code.
The generated documentation could also lead to confusion, even for professionals.









§ DEVELOPER EXPERIENCES WITH PROMPTING (RQ2)





 §.§ Overall Experience



Participants rated both tools positively in all six categories of the standardized ueq <cit.>, with values ranging from -3 to +3 (Table <ref>).
However, students found the predefined prompt tool more positive in all user experience categories than the ad-hoc prompt tool, with statistically significant differences and large effect sizes in five categories.
Professionals also rated the predefined prompt tool more positively in five categories, without significant differences but a large effect size in one category.

For the pure valence category Attractiveness (1.01 ad-hoc / 1.99 predefined prompt group), the predefined prompt tool received higher ratings than the ad-hoc prompt tool, with a high difference among students (0.64/1.94). 
This indicates an overall better developer experience with the predefined prompt tool.



 §.§ Tool-Related Experience



The most significant difference between the groups was in the category Efficiency (0.7/1.98), with the predefined prompt tool receiving higher ratings, especially among students (0.42/2.12).
Participants praised the ad-hoc prompt tool for its practicality of not having to open ChatGPT in the browser (two students, one professional), flexibility in selecting relevant code (one professional), and custom prompts (one professional).
They noted issues with copy-pasting the tool output to the code panel (six students, four professionals), multiple tries sometimes required for sufficient results (one student), the need to phrase a prompt (one professional), missing keyboard shortcuts (one professional), and the slow output generation, which is primarily caused by the performance of GPT-4 (one student).
Participants who interacted with the predefined prompt tool also noted the slow response (one student, two professionals), but expressed no further positive or negative comments.


The ratings for Perspicuity (1.25/1.87) were higher for the predefined prompt tool, with large effect sizes for students and professionals.
While some participants of the ad-hoc prompt group participants found their tool easy to use (four students, one professional), others reported that they would find it easier to enter the prompt in a separate ide panel instead of the pop-up at the top of the screen (one professional) and that the right-click context menu, which listed the tool features besides non-related ide features, was obscure to them (one professional).
Multiple predefined prompt group participants also found their tool easy to use (eight students, four professionals), highlighting the ease of generating comments without the need for further prompt input besides the code and a short command (one professional).
Nevertheless, they assessed that the selection of relevant code could be facilitated (one professional).
Overall, this assessment aligns with our observation that participants quickly understood both ide extensions but tended to prefer the simplicity of the predefined prompt tool without having to create an effective prompt.


Ratings for Dependability (1.05/1.84) showed strongly diverging views among students (0.85/1.95).
Participants praised the ad-hoc prompt tool for supporting the comment creation workflow (six students, two professionals) and code comprehension (three students, four professionals).
They criticized the lack of prompt templates (three students), that the tool did not always generate code comments (two students, one professional), and the missing opportunity for iterative improvements caused by the limitations of the OpenAI api at the time of our study (one student, one professional).
The predefined prompt tool was also noted for facilitating the workflow (three students, one professional) and code comprehension (four students), as well as for its reliability (two professionals).
However, participants suggested adding a progress bar (one professional), previewing generated comments within the function (one professional), enabling the configuration of the required comment content (one student), adding a disclaimer that the output might be incorrect (one student), implementing a linter that alerts when generated comments became outdated (one professional), and using a locally deployed api for data security (one professional).
Besides these improvement comments, the predefined prompt group was overall satisfied regarding the Dependability aspects, whereas the ad-hoc prompt group, and especially the students, required more enhanced tool support.


Both tools achieved similar positive results concerning Novelty (1.1/1.18), without significant differences.
Interestingly, the ad-hoc prompt tool received a higher rating from professionals (1.38/1.18) for its chat-like interaction within an IDE, which they considered more novel than clicking a button for documentation generation.
Participants expressed no further comments regarding this ueq category.



 §.§ Documentation-Related Experience



For Stimulation (0.82/1.62), predefined prompt tool ratings were higher, especially among students (0.6/1.6).
Concerning the output of the ad-hoc prompt tool, participants noted the detailed explanations (two students, two professionals), the mitigation of personal biases (one student), inspirational value (one professional), and overall high quality (three professionals).
Other participants criticized the output for containing unnecessary information (five students, three professionals), lack of detail regarding complex code (three students), and inconsistent format due to using prompts of varying qualities (two students, one professional).
The predefined prompt tool output received positive ratings for its conciseness (four students, one professional), overall high quality (three professionals), and similar structure across all generated comments (one student, one professional).
Improvements were deemed necessary for the parameter descriptions (one student, one professional), line breaks (one professional), consistent punctuation (one student), and lack of detail for complex code (one student, one professional).
These insights show that students were especially unsatisfied with the ad-hoc prompt tool output, while professionals were overall satisfied with both tools.


Answering RQ2, we conclude that the participants overall preferred a tool that automates code documentation generation with a few clicks while offering options to configure this process.
On average, professionals and students preferred executing predefined prompts over creating ad-hoc prompts.
The simplicity and efficiency of a single button click to receive consistently high-quality documentation were key factors for this assessment.
While the predefined prompt group praised their tool, some missed the flexibility to adjust the documentation depth.
Users of the ad-hoc prompt tool appreciated this flexibility, which often unintentionally resulted in longer and more explanatory comments that may assist code comprehension.








§ DISCUSSION



We summarize the results of our experiment and discuss potential implications for research and practice.



 §.§ Developers are not Necessarily Prompting Experts


Our study indicates that developers are generally neither skilled (RQ1) nor inclined (RQ2) to effectively prompt an llm in a way that it generates concise, readable code documentation. 
This partly aligns with previous studies showing that optimized, predefined prompts outperform ad-hoc prompts <cit.>. 
But it also partly contradicts the underlying assumption that effective optimized prompts, such as few-shot prompts, are trivial and can intuitively be created by developers in their daily work. 
We observed in our experiment that participants interacted with the provided tools intuitively via natural language queries.
However, they were often disappointed by the results as the tools provided outputs that were aligned with their unspecific prompts.
In fact, we even observed that the generated documentation can partly be misleading during comprehending code. 

The lack of prompting skills among developers is unsurprising, as this is not a standard skill taught in universities or commonly practiced in software development.
Therefore, researchers should explore ways to learn effective prompting and to intuitively support conversations with llm, particularly in domains like software engineering where the llm outcome has an impact on other users.
One approach is to compile a catalog of evaluated prompt templates for different well-defined tasks (e.g., documentation of Python functions).
Recently, Torricelli et al. also found such templates beneficial for llm users <cit.>.
Tool vendors could integrate the templates into their tools.
Each template could include instructions for the llm to ask clarifying and contextual questions, e.g., related to the expected documentation style <cit.>.
Additionally, educators could use these prompt templates, together with underlying ideas and hints, to teach prompting skills in schools and universities.
Such templates should only provide guidance and should not hinder human creativity in asking out-of-the-box and follow-up questions to an llm-powered tool.



 §.§ Generating Documentation With LLMs is an Iterative Task


Our results show that the participants rated the quality of documentation generated via the predefined prompt as high.
However, many were not completely satisfied with the documentation quality after this initial iteration.
The first iteration helped to comprehend the code, formulate the initial comment version, and gather ideas to alter and extend the documentation during further iterations.
This indicates that generating code documentation is rather an iterative task, not to be fully automated without the feedback of developers.

To better understand and assist this task, additional research should be conducted to identify and satisfy the preferences of code documentation providers and consumers <cit.>. 
Personal aspects include the depth of explanations, tone of the text, and lyrical style. 
Project aspects include documentation style used <cit.> and knowledge needs of code users.
This research can help vendors of ai-powered code documentation tools align the generated comments with the requirements of their users.
Thus, vendors can focus on applying and optimizing predefined prompts, which participants in our experiment preferred, while also offering additional iterations to customize the output based on preferences and needs.
Furthermore, generating multiple documentation versions at once might also stimulate developer creativity in exploring possibilities to document code <cit.>.



 §.§ Evaluating Code Documentation Quality Requires Human-Centered Quality Metrics


As Hu et al. <cit.> pointed out, current metrics used to assess the quality of ai-generated content do not align with human evaluation regarding code documentation generation.
For instance, the usefulness of generated content <cit.> should be crucial in practice but is barely evaluated in common llm benchmarks. 
Our results suggest that the assessment of code documentation quality depends on multiple factors <cit.>, such as the preferences of documentation consumers, the ratio between documentation conciseness and code complexity, and the availability of certain information and style that developers expect to be included in the documentation.
Current metrics for automatically assessing the quality of generated texts do not incorporate these aspects.
Hence, while automatically generating code documentation is fairly straightforward with llm, an expert assessment is still required to ensure good quality, e.g., during a code review.
It remains unclear how far this assessment can be fully automated, e.g., by other agents or other llm.
Future research should focus on creating and testing quality metrics for generated documentation through documentation analyses, expert surveys, and developer and benchmark studies.



 §.§ Better Code Comprehension Requires Support by Personalized LLM-Generated Explanations On The Fly


The results of our code comprehension tasks show that llm can support developers in understanding code.
Especially, the longer explanations provided by the ad-hoc prompt tool in response to vague prompts helped participants understand the given code function.
However, such explanations can also mislead developers if the generated output does not consider the information needs <cit.> or provide the particular knowledge nuggets needed for the comprehension task <cit.>.

Therefore, similar to the work of Nam et al. <cit.>, we propose that vendors of ide-integrated code comprehension tools should offer multiple code-related aspects that tool users can explore during the comprehension process.
Examples include the concepts mentioned in the code or the typical code usage <cit.>, but could also extend to explaining the code in a personalized way.
Researchers should survey developers to identify additional aspects and test their relevance in practice.



 §.§ Practitioners May Have Specific LLM Requirements


Participants in our study expressed multiple requirements for an llm-powered documentation generation tool.
For example, participants expressed concerns regarding security risks associated with using closed-source, externally hosted models and highlighted the extended time required for generating comments via the llm model we used.

To address these issues, tool vendors should support a variety of llm within a single tool, including smaller, open-source models, to better meet the needs of organizations operating with sensitive data and under specific constraints.
One of the research challenges is that different models may respond variably to the same prompt, impacting the consistency and reliability of generated outputs.






§ THREATS TO VALIDITY





 §.§ Construct Validity





(T01) Our study exclusively employed human evaluation techniques to assess code documentation quality, introducing a mono-method bias <cit.>.
Although automated metrics are used in other studies, we opted for human evaluation due to its limited correlation with automated metrics <cit.>, and considered it beneficial for a more comprehensive understanding.



 §.§ Internal Validity



(T02) We unintentionally excluded the Understandable scale from our ueq, which may compromise the Perspicuity category's accuracy. 
Remedying this, we calculated Perspicuity scores with and without the scale's median value, finding only marginal differences. 
Despite this omission, participant feedback indicated an adequate understanding of the tool.
(T03) Customizability of the predefined prompt could have positively influenced participant satisfaction and tool assessment. 
However, allowing customization could have affected the comparability of assessments among the predefined prompt group. 
(T04) Lack of significant differences in professional ratings may be attributed to the small sample size of ten professionals per group. 
Nevertheless, descriptive statistics reveal fewer differences among the professionals' ratings compared to students, likely due to their more targeted use of keywords in ad-hoc prompts.
(T05) A potential language barrier existed given the non-native English-speaking background of most participants. 
However, this was deemed a minor threat as all were able to produce coherent prompts in English.



 §.§ External Validity




(T06) Our controlled lab setup, where developers documented unfamiliar code, may not accurately represent typical documentation practices. 
However, this setting was intended to eliminate variability and enable statistical analysis. 
(T07) The professionals in our study predominantly came from euxfel, with specific expertise in Python, which might influence the results due to familiarity with the experimental code. 
This was seen as potentially beneficial, providing insight into how domain knowledge of one experiment group might improve their prompt creation.
(T08) The findings' generalizability is limited, yet they corroborate and broaden the insights from similar studies that involved different methodologies.



 §.§ Conclusion Validity




(T09) The selection of utility functions as code candidates might limit the applicability of our results to more complex coding tasks.
We chose these functions based on a pilot study confirming the difficulty of comprehending challenging code under time constraints.
(T10) Variations in llm outputs due to inherent randomness <cit.> may have influenced assessments in the predefined prompt group, though differences in outputs were marginal.
(T11) Dividing the experimental and control groups into professionals (experts) and students (newcomers) reduced overall sample sizes, weakening statistical power but allowing for a more detailed analysis of differing perceptions between these groups.
The separation between experts and newcomers was confirmed by the participants' self-perceived Python experience.
Hence, we believe our findings can be generalized to a broader population.
Nevertheless, developers with different expertise in conversational agents might behave differently.







§ RELATED WORK





 §.§ Prompting Skills of Practitioners



Prompt engineering has been pivotal since the introduction of few-shot prompting <cit.>. 
Ahmed et al. demonstrated its effectiveness in code summarization <cit.>.
They highlighted that applying a few-shot prompt can outperform traditional models like CodeBERT <cit.>, while zero- and one-shot prompts were less effective, aligning with findings of Geng et al. <cit.>. 
Meanwhile, subsequent techniques, like chain-of-thought prompting <cit.> and active prompts <cit.>, have emerged.

Prompt engineering remains a skill requiring practice <cit.>.
llm providers and researchers have published guidelines and prompt pattern catalogs for effective llm interaction across various domains, including healthcare <cit.>.
Our study is among the first to examine if and how well developers utilize prompt-related knowledge in software engineering tasks.



 §.§ LLM-Powered Developer Assistants



Recent studies on ai programming assistants show that tools like GitHub Copilot <cit.> boost productivity, reduce keystrokes, and assist in syntax recall <cit.>.
However, users desire tool enhancements for direct feedback, personalized output, and improved code context understanding <cit.>.
Currently, they often avoid these tools due to unmet requirements and lack of control.
For example, Copilot often fails to grasp instructions for code adjustments unless precisely specified <cit.>.
However, tool shortcuts to optimize an initial prompt can also nudge users to explore fewer prompts and lead to less diverse output <cit.>.
We can confirm these findings with our study.
While our participants overall preferred the interaction with and the output of the predefined prompt tool, the more exploratory ad-hoc prompts often led to more extensive output that supported code comprehension.



 §.§ Code Documentation Generation in the IDE




In the last decade, researchers have developed various methods for generating code documentation <cit.>, including template-based <cit.>, information retrieval <cit.>, and, more recently, deep learning techniques <cit.>.
Especially deep learning approaches trained on large datasets have shown high performance in code documentation generation and enabled better scalability and more complete results than the other techniques <cit.>.

We found many available ide extensions that use these ai models to facilitate developer workflows.
However, they were conceptualized as general-purpose tools and lacked code documentation features.
In the academic realm, the vscode extension Divinator was the most relevant tool to our project <cit.>. 
It provides short code summaries in multiple programming languages but has limited performance and interactivity.
With our study, we aimed to counter this lack of ide features related to code documentation generation.



 §.§ Evaluating Documentation Quality



Empirical research on software documentation quality is an active field that focuses on various artifacts, like api reference documentation <cit.> or README files <cit.>, and the perspectives of documentation writers <cit.>.
Studies on the evaluation of ai-generated documentation usually focus on automated metrics like BLEU, ROUGE, and METEOR <cit.>.
Hu et al. compared such automated metrics with human evaluations on six documentation quality dimensions <cit.> and found that automated metrics often misalign with human judgment <cit.>. 
Consequently, our study benefited from their human-centered metric by capturing nuanced aspects in the generated code documentation that automated approaches may have missed.







§ CONCLUSION




Recently, many llm have emerged, offering the potential to automate developer tasks, including creating code documentation.
However, it remained unclear how effectively developers could prompt llm to generate useful documentation.

We studied the interactions of developers with and their perception of llm-powered ide extensions during a controlled experiment with professionals and computer science students.
To generate documentation for two Python functions, the experimental group freely prompted an llm and the control group applied a predefined few-shot prompt.

Our results revealed that students, who had relatively low Python experience, preferred the guidance of the few-shot prompt over ad-hoc prompting.
They rated the documentation generated by predefined prompts significantly higher in quality, particularly regarding readability, conciseness, usefulness, and helpfulness.
Professionals were more adept than students at including Python-specific keywords in their ad-hoc prompts, resulting in the generation of higher-quality documentation.
Consequently, they enjoyed the flexibility of ad-hoc prompting more than students, even if they did not apply prompt engineering techniques.
Overall, both types of llm interactions improved the code comprehension of the study participants, but the participants often viewed the generated documentation as an intermediate result that needed iterative improvement.

We hope our findings encourage researchers to replicate this study <cit.> in diverse settings, aiming to improve developers' prompting skills and ai-powered tool support.




§ ACKNOWLEDGEMENT

We thank all participants of our experiment.
We acknowledge the support by DASHH (Data Science in Hamburg - Helmholtz Graduate School for the Structure of Matter) with the Grant-No. HIDSS-0002.

IEEEtran




 Paper Title: WIP: An Engaging Undergraduate Intro to Model Checking in Software Engineering Using TLA+.









WIP: An Engaging Undergraduate Intro to Model Checking in Software Engineering Using 
    
  
    Konstantin Läufer1,
    Gunda Mertin2,
    and
    George K. Thiruvathukal1
  
  
    1Software and Systems Laboratory, 
      Department of Computer Science,
      Loyola University Chicago

      {laufer,gkt}@cs.luc.edu
    

    2Institute for Software Engineering and Programming Languages,
      University of Lübeck

      gunda.mertin@student.uni-luebeck.de
  

    August 5, 2024
============================================================================================================================================================================================================================================================================================================================================================================================




grammar check by exporting plain text
attributions for microwave images
























plain
plain










































§ ABSTRACT 
Background 
In this Innovative Practice Work in Progress, we present our initial efforts to integrate formal methods, with a focus on model-checking specifications written in Temporal Logic of Actions (), into computer science education, targeting undergraduate juniors/seniors and graduate students. Many safety-critical systems and services crucially depend on correct and reliable behavior. Formal methods can play a key role in ensuring correct and safe system behavior, yet remain underutilized in educational and industry contexts. Aims
We aim to (1) qualitatively assess the state of formal methods in computer science programs, (2) construct level-appropriate examples that could be included midway into one’s undergraduate studies, (3) demonstrate how to address successive “failures” through progressively stringent safety and liveness requirements, and (4) establish an ongoing framework for assessing interest and relevance among students. 
Methods
We detail our pedagogical strategy for embedding into an intermediate course on formal methods at our institution. After starting with a refresher on mathematical logic, students specify the rules of simple puzzles in and use its included model checker (known as TLC) to find a solution. We gradually escalate to more complex, dynamic, event-driven systems, such as the control logic of a microwave oven, where students will study safety and liveness requirements. We subsequently discuss explicit concurrency, along with thread safety and deadlock avoidance, by modeling bounded counters and buffers. 
Results
Our initial findings suggest that through careful curricular design and choice of examples and tools, it is possible to inspire and cultivate a new generation of software engineers proficient in formal methods. Conclusions Our initial efforts suggest that 84% of our students had a positive experience in our formal methods course. Our future plans include a longitudinal analysis within our own institution and proposals to partner with other institutions to explore the effectiveness of our open-source and open-access modules.






Computer Science Education,
Formal Methods,
Model Checking,
Software Testing,




Safety-Critical Systems 
















§ INTRODUCTION


In the evolving landscape of computer science education, methodologies that help to enhance the correctness and reliability of safety-critical systems are of vital importance <cit.>.
Examples include those that George and Konstantin will discuss. I still remember Jeanette Wing coming to ANL to discuss Larch [theorem prover -> specification language tied to C++ https://www.cs.ucf.edu/ leavens/larchc++.html] Many examples are connected to hardware with embedded software and cyber-physical systems, e.g. drones, sensor-based systems [garage door?], failure-sensitive applications (distributed/cloud services), others?
Recent advances in formal methods, particularly model checking of specifications written in  <cit.> have emerged as a powerful tool in bridging the gap between theoretical computer science and practical software engineering.
Although formal methods—rooted in automata theory, formal logic and, more specifically, Hoare logic <cit.>—
have a long history and continue to evolve, they have only modestly impacted CS education compared to many topics (e.g. machine learning, cybersecurity, Internet of Things, among others).
Although generative AI shows promise to reduce the error-prone nature of human coding, we remain convinced that humans—and non-humans—can employ formal methods to provide critical correctness and reliability checks on solutions, whether crafted by hand or generated by (statistical) AI and large-language ML models.OK to abbreviate here to save a line?







In this paper, we describe our initial efforts to introduce students (undergraduate juniors/seniors and graduate students) to formal methods in software engineering with a focus on model checking using , backed by preliminary results from ongoing anonymous input from our own students.
We note that the selection of is intentional, as it is one of the most actively-maintained model checking systems (evidence provided in related work). 
We also incorporate other strategies and systems in our teaching: Students give presentations and demos on a broad landscape of tool- and language-based formal methods, including many of those shown in <ref>.

KL explain that this is a relevant embedded system with firmware
By applying to practical, familiar, and compelling examples, such as the microwave oven shown in <ref>, we aim to demystify formal methods and make them more relevant to students.

More to be added here including a transition to the rest of the paper.

Our contributions include:

    
  * Performing an initial analysis of academic programs offering formal methods courses, organized by course level, tools and techniques covered, and when last taught.
    
  * Creating a set of curricular materials (i.e., exemplars) to motivate student interest in formal methods. We emphasize reproducible behavior using modern software engineering principles (i.e., version control and continuous integration) as our own materials and both continue to evolve.
    
  * Demonstrating how to address successive “failures” in model checking through a set of progressively stringent safety and liveness requirements.
    
  * Conducting an initial evaluation of the effectiveness of formal methods among our current students. This effort will inform ongoing study on the long-term effectiveness of our curricular approach.




§ BACKGROUND AND RELATED WORK






The private computing industry and high-level government agencies have long recognized the value of formal methods for building safety-critical and other highly reliable systems <cit.>.
A major example is the National Aeronautics and Space Administration (NASA), which “develops formal methods technology for the development of mission-critical and safety-critical digital systems of interest to NASA <cit.>.”
Another safety-critical sector is ground transportation; a comprehensive survey of the use of formal methods in the railroad sector found that model checking is the most commonly adopted technique (47%), followed by simulation (27%) and theorem proving (19.5%) <cit.>. 
Specifically, model checking arose in the early 1980s in response to the challenges of concurrent program verification <cit.> and is now well established and mature <cit.>.
Among various specification languages that support model checking, has emerged as a practical and effective option <cit.>.




The need to recruit qualified professionals in these areas became apparent several decades ago and is now receiving renewed attention <cit.>.




Accordingly, academic educators have increasingly focused on formal methods teaching since the 1990s <cit.> and have been teaching model checking more broadly since the early 2000s <cit.>.
Several academic institutions have reported on their use of in their courses <cit.>.

Nevertheless, multiple educators have recognized the challenges of motivating the need for formal methods topics and teaching them effectively in the face of some resistance <cit.>.
Key challenges include insufficient mathematical background from high school and university-level prerequisite courses, which may not focus sufficiently on discrete math and proof techniques; a lack of practical, engaging case studies that motivate the use of formal methods; insufficient documentation of some of the tools for students to gain confidence in using them; and learning styles of today's students (millennials and beyond), such as active, discovery-driven learning, an emphasis on solutions over theory, and a desire for immediate feedback <cit.>.

Which others to cite?
+reed_motivating_2004 - mental resistance, need small, engaging examples
zingaro_another_2008 - emphasize coupling of spec and impl in earlier courses
zamansky_mathematical_2016 - need to explain connection between logic and SE
korner_increasing_2021 - overwhelmed with transfer tasks
yatapanage_introducing_2021 - hate math, struggle with programming
+noble_more_2022 - perceived difficulty, excessive math content, practical irrelevance -> programming intensive approach based on Dafny





While some educators actually propose starting in the first year of the post-secondary curriculum <cit.> or even earlier <cit.>, we are concerned that these efforts will reach a relatively small number of students mostly at selective institutions. 
Therefore, we target juniors and seniors (at undergraduate level) and graduate students, and start with a three-week review of fundamentals studied earlier.










Our work builds on Askarpour and Bersani's study <cit.>, which not only reports on specific experiences at their institution, but also includes a general analysis of the Formal Methods Education Database (FMEDB) <cit.> and a review of related work on the main challenges students face in formal methods courses.

We analyzed FMEDB for the most commonly used tools in 34 courses focused on model checking (out of 88 courses total) and found that 
7 use NuSMV/NuXMV, 
2 use Dafny, and 
2 use (one in industry and one in academia).

these are the results of our own web-search-based analysis of 17 courses   
   
   
   
   
   
Previously, we had this; however, I simplified to just focus on the results. We could still include this sentence but I wanted to make sure the numbers are right as they are not mentioned in Figure 2: The entire FME database includes 88 courses, 34 of which include model checking as a topic).  
You lost the separation between FME and our own results - please do not edit without checking with me
Because these entries are self-submitted and the database doesn't say when a course was last taught, we also conducted our own preliminary survey of relevant courses taught during the last five years, summarized in <ref>.


As practicing software engineers, there are other reasons to use it: It is mature and well-maintained (by Leslie Lamport, a Turing Award winner) and has a strong community and ecosystem.
In sum, the overall related work strongly supports the decision to emphasize in our own course. 









§ CURRICULAR GOALS


We target primarily undergraduates majoring in computer science or software engineering who have completed at least five foundational courses over three semesters: Introduction to Programming (CS1), Data Structures I (CS2), Introduction to Computer Systems (CS3), Discrete Structures, and a one-semester, hands-on introduction to the Linux command line.

In our semester-long, 15-week, three-credit course, we focus on the most relevant learning outcomes from the ACM/IEEE Computer Society Software Engineering 2014 Curriculum Guidelines <cit.> in the order listed in <ref>.











For many of our students, the most recent logic-based mathematics course is Discrete Structures, which most students take in their first year.
To aid in the transition to our course on Formal Methods, we reinforce the essential mathematical foundations. (FND/MAA - elaborate)
This material, however, is targeted toward what we need here, using unit testing as a vehicle to connect these foundations with the programming practice they have undergone in their CS1 and CS2 courses.

Is there a week or so where you do any sort of refresher on the ideas of mathematical logic? I think we want to include something about this, based on the other related work that informs the idea. For many students, I'm guessing COMP 163 is the last time they saw any math related to this course.
Yes, MAA.md.2 covers Hoare logic, or we could insert 3h of FND.mf.1/2: functions, relations, sets, predicate logic, and reduce testing from 12 to 9.









We first build on students' understanding of correctness of a system under test w.r.t. a set of functional requirements and their ability to write automated, non-exhaustive unit tests that reflect these requirements. We also leverage students' understanding of discrete structures, especially predicate logic, to progress from example-based testing to universally quantified properties that more generally reflect the functional requirements, using libraries that automatically test these properties on a range of pseudo-randomly generated arguments. 

We then use to progressively introduce students to the formal specification and verification of dynamic and concurrent software systems. 
In , a model has a finite set of variables, and each state maps these variables to their current values.
allows us to model specific system behaviors that start in an initial state and undergo a sequence of states by performing transitions available in the next-state relation.
Beginning with intuitive examples, such as puzzles, we specify the rules in and use its included model checker (known as TLC) to falsify the absence of a solution, thereby finding a counterexample that constitutes a solution. 
We gradually escalate to more complex, dynamic systems, such as the control logic of a microwave oven, where students will study safety and liveness requirements; for greater engagement, we emphasize event-driven, interactive systems.
We subsequently study explicit concurrency, along with thread safety and deadlock avoidance, by modeling bounded counters and buffers. 
By proceeding gradually, we scaffold student understanding and application of formal methods (see <ref>).












§ COURSE ACTIVITIES



Here we present a series of activities for students based on a simple microwave oven as a running example.



 §.§ Running Example: A Simple Microwave Oven




We represent the state of our microwave as two booleans, door (open or closed) and radiation (on or off), along with a natural number, timeRemaining. 
The system also supports various actions, mostly corresponding to user controls:


  * The user can open or close the door.

  * The user can increment the remaining time.

  * The user can start or cancel the heating process.

  * An internal timer decrements the remaining time to zero.


As seen in <ref>, radiation is initially off with no time remaining, while the door may be open or closed.
The model's next-state relation Next is defined as the logical disjunction (choice) of the user-triggered actions OpenDoor, CloseDoor, IncTime, Start, Cancel, as well as the internal Tick action. 



Activity 1 Starting with a skeletal model where each action manually controls the corresponding variable, develop appropriate state-dependent behavior, such as the timer counting down and automatically shutting off radiation when reaching zero. 
<ref> defines the resulting Tick action, and <ref> shows a step in a normal behavior.







 §.§ Invariants and Safety


To study system safety, we will initially allow the door and radiation to operate independently.
The TLC model checker finds nothing wrong with this model because we have not yet defined any invariants it can attempt to falsify.

Exercise 2a Define an invariant DoorSafety to guarantee that the microwave will never be radiating with the door open.
What happens after you add this invariant?


pvspace4.0pt
xs8.2 DoorSafety  door =̇ OPEN  radiation =̇ OFF
pvspace4.0pt


Once we add this invariant to our model, TLC considers unsafe any state in which the invariant is false and proves that there is at least one behavior leading to such a state (see <ref>).



Exercise 2b Refine the microwave model to satisfy the DoorSafety invariant, making the smallest possible changes.

This requires introducing more state dependency within the actions constituting the next-state relation.
To prevent the scenario from <ref>, we can require the door to be closed by adding a precondition to the Start action:


pvspace4.0pt
xs8.2  door =̇ CLOSED
pvspace4.0pt


But TLC quickly finds another unsafe behavior, as shown in <ref>.
To prevent this scenario, we will add to the OpenDoor action the effect of immediately shutting off radiation.


pvspace4.0pt
xs8.2  radiation '̇ =̇ OFF
pvspace4.0pt






 §.§ Temporal Properties and Liveness


Our microwave model is now safe in terms of preventing any behaviors that allow radiation to occur when the door is open.
But is this notion of safety enough? How do we know it won't radiate indefinitely, overheat, and catch on fire?

includes temporal logic operators for predicates on a sequence of steps, e.g., some false condition eventually becoming true.
Temporal logic predicates are often used to express liveness requirements, such as a radiating microwave oven eventually reaching zero remaining time and shutting off.

Exercise 3a Define a temporal property HeatLiveness requiring a radiating microwave to eventually turn off.
What happens after you add this property?


pvspace4.0pt
xs8.2 HeatLiveness  radiation =̇ ON  radiation =̇ OFF
pvspace4.0pt


The “leads to” operator in p  q indicates that if p is currently true, then q must become true in a future step.

Once we add this liveness property to our model, TLC will indicate that our current model does allow the scenario from <ref>.
Even though this behavior involves only valid states, it is undesirable because it allows radiation to continue indefinitely and the food inside the microwave to catch on fire.

In general, does not require the model to choose an action even if one is available. 
Instead, the model is allowed to perform a transition where it stays in its current state; this is called a stuttering step.
This can model scenarios where a system loses power or, as in <ref>, fails to make progress. 



Exercise 3b Refine the model to satisfy the HeatLiveness temporal property, making the smallest possible changes.

To break the observed stutter-invariance, we need to use a mechanism called (weak) fairness.
We can enable this for the Tick action by adding _vars(Tick) to our top-level specification.
The resulting model satisfies our previously established safety and liveness requirements.



§ PRELIMINARY EVALUATION
 




We offered the course for the first time in the Fall 2022 semester to 15 students, and again in the Spring 2024 semester to 22 students. 
Our new course-specific 15-question survey, given as a pretest-posttest in Spring 2024, indicates a marked improvement in student receptiveness to formal methods, with enhanced ability to conceptualize and apply these techniques in real-world scenarios; 
the cohort average of equally weighted composite scores increased from 3.0 to 4.1 (n = 19), with 84% reporting a positive experience.
Notably, the use of has not only facilitated a deeper understanding of software correctness but also stimulated student interest in the subject matter.
In addition, preliminary results from standardized teacher-course evaluations are highly encouraging, with average scores significantly above departmental averages and relatively high consensus among respondents. 
For later? In addition, standardized teacher-course evaluations for the Spring 2024 offering of this course were very positive, with average scores significantly above departmental averages and relatively high consensus among respondents; while acknowledging the complexity of the material, students' written comments were also overwhelmingly positive.











§ CONCLUSION


In summary, this work in progress demonstrates the successful integration of model checking using into our intermediate undergraduate computer science curriculum.
With just three semesters (five courses) of prerequisites, students can learn about formal methods and model checking as early as their second semester of sophomore year and most certainly by the first semester of junior year, grounded in the ACM/IEEE Computing Curricula, which strongly suggest what students should learn and when they should learn it, including all appropriate foundational preparation.
With our open source and open access materials (code and lecture) notes continuously updated on GitHub and rebuilt and tested using continuous integration, anyone at any institution can use and contribute to our work.
Our initial findings strongly suggest that this approach, including the use of effective, well-documented tools such as , is engaging to students.

Our future plans include reaching out to other universities, starting with our own region, which has dozens of universities, and potentially offering a workshop aimed at workforce development for those who may have missed the opportunity to learn this material while studying computer science or a closely-related discipline.







§ ACKNOWLEDGMENTS



This material is based upon work supported by the National Science Foundation under Grant No. DGE-1919004. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
The authors thank Ryan Hasler for helpful suggestions for improving the included examples.








 Paper Title: Bug Analysis Towards Bug Resolution Time Prediction.












Bug Analysis Towards Bug Resolution Time Prediction
    

Hasan Yagiz Özkan1,
Poul E. Heegaard2,
Wolfgang Kellerer1,
Carmen Mas-Machuca3,



1Technical University of Munich, Chair of Communication Networks, Germany 

2NTNU - Norwegian University of Science and Technology, Department of Information Security and Communication Technology,



3 University of the Bundeswehr Munich, Chair of Communication Networks, Germany



    August 5, 2024
===================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 

Bugs are inevitable in software development, and their reporting in open repositories can enhance software transparency and reliability assessment. This study aims to extract information from the issue tracking system Jira and proposes a methodology to estimate resolution time for new bugs. The methodology is applied to network project ONAP, addressing concerns of network operators and manufacturers. This research provides insights into bug resolution times and related aspects in network softwarization projects.







 Data Mining, Bug Fixing Time, Softwarized Networks, Machine Learning







§ INTRODUCTION





Network softwarization entails replacing dedicated hardware for specific applications with programmable components, leading to advantages like easy maintenance, flexibility, scalability and decreased costs associated with development and operations<cit.>. Software Defined Networking (SDN) and Network Function Virtualization (NFV) are examples of these softwarization efforts. In this context, manufacturers rely on open-source solutions to provide extra functionalities while complying with the standards.

However, despite the advantages of network softwarization, operators are reluctant to use these solutions as the software is more complexity, incurring to more bugs, as well as the code reliability is uncertain as it is open to any software developer(s) differing in terms of programming skills, experience, trustiness, etc.   


In order to increase the transparency of software reliability, any bugs which are encountered during the operation of a software, are reported in open repositories. These repositories allow knowing the reported bugs, their status, etc., which are useful to evaluate the software reliability. Each software project has defined a bug resolution process, which should be followed by any software developer (i.e., debugger) of that project. 

The goal of this study is first of all, give an overview of the information that can be extracted from these repositories. Some information can be directly obtained from the gathered data, but some other need some processing. Secondly, we propose a methodology that can be used to estimate the resolution time of new bugs. The proposed methodology has been applied to several network related projects such as ONOS, ONAP, etc. We believe that network operators and manufacturers using an open source project like those, are concerned about the bug resolution times and other aspects addressed in this paper.  







The contributions of this paper are:

    
  * Collection strategy of reported bugs and the required filtering.
    
  * Basic data analysis such as identification of the most common paths in the workflow, the transition distribution between different states, bug resolution time distribution, etc.
    
  * Advance data analysis e.g., impact of aspects such as priority, reporter to the debugging time. 

    
  * Investigating state probabilities of the bugs in time.
    
  * Comparison of the bug resolution time prediction with different strategies
    
  * Predicting the exact bug resolution time for different bugs with neural network.

The paper is organized as follows: Section II gives an overview of the state of the art on bug data analysis and bug resolution time models. Section III introduces the bug repositories and the data that can be extracted from them. Section IV presents the proposed bug collection and the data analysis used in this work. Section V proposes a machine learning model to estimate the bug resolution time of new bugs. Finally, Section VI concludes the paper.













§ RELATED WORK



This section will discuss previous works on bug data analysis and bug resolution processes. 

Issue tracking systems are mined, and researchers of <cit.> use this process to offer the dataset of all the issues from different projects. Additionally, the authors of <cit.> conduct various analyses of the collected data, which comes from 55 different Apache projects. These 55 projects are classified into 9 categories, such as big data, security, and libraries. The authors examine the percentage of fields that change the reporting of the bugs. They define bug resolution duration time from the creation and closure of a bug as a bug-fix effort and compare the bug resolution times across different project categories. Additionally, they investigate when the priority and assignee are defined and analyze the number of bugs in which these values change after the initial bug report. Furthermore, the study delves into the changes in bug status, analyzing the percentage of bugs transitioning between states. This dataset is used in this work. As the primary contribution of this work is the provided dataset, it does not delve into a comprehensive analysis or propose a methodology to estimate bug resolution duration. In addition to the self-collected data, the bug reports from <cit.> are also examined in this study.

The authors investigate the impact of bug attributes such as reporter and priority on the time it takes to resolve them in the study <cit.>. The used data in this work is not publicly accessible. They employed the K-nearest neighbour algorithm to predict the resolution time of bugs. Based on these attributes from the issue-tracking system, they predicted whether a bug would be resolved quickly or slowly. The authors of <cit.> replicated the study presented in <cit.>, but they utilized open-source data sourced from Bugzilla Firefox. Both studies concluded that the identity of the reporter or assignee is more crucial than the bug's priority. However, a weakness of these predictions is that they are binary (quick or slow) rather than numerical. 

Furthermore, their proposal incorporates a Markov-based approach to estimate the number of bugs that can be resolved within a specific timeframe. They also employ a Monte Carlo-based method to predict the overall time required to fix a given quantity of bugs. The time is divided into months, and they estimate the percentage of bugs that will be resolved within a particular time period. It is important to note that the Markov-based model they used does not focus on examining the solution process of individual bugs as separate entities but rather on determining the percentage of bugs in different states after a certain period of time, measured in months.
























§ DATA REPOSITORIES




Issue tracking systems are software tools used to manage software projects' development, maintenance and debugging processes <cit.>. These systems use issues which can describe bugs, feature requests, milestones and any other tasks, depending on the requirements of the considered project. Moreover, issue tracking systems also serve as a repository for the history of reported and addressed issues. Thus, they offer an excellent opportunity to understand the software project's development and debugging process. Most open-source software use publicly available issue-tracking systems because it facilitates collaboration among software developers and debuggers by giving access to the status, process, and resolution of the reported issues.

Jira is one of the most popular issue-tracking systems used by many different open-source projects<cit.>. As an issue tracking system, the issue types in Jira extend beyond 'Bug,' including 'Task,' 'Story,' and more. Every issue is kept in the Jira repository of the respective project. Therefore, Jira repositories become a valuable information source for the project's development history. Jira defines a workflow, which comprises of a sequence of stages that issues may undergo, from their creation (referred as "Open") and ending with their resolution (referred as "Closed"). The workflow is depicted in Fig. <ref> in blue color. 
Although Jira provides this initial workflow, it can be customized to meet the specific requirements of a project. 

This study considers two open-source projects as examples: Open Network Operation System (ONOS) and Open Network Automation Platform (ONAP). ONOS, developed by the Open Network Foundation, is an operating system designed for software-defined networking (SDN) and follows a standard workflow. ONAP is a Network Function Virtualization (NFV) Management and Orchestration (MANO) framework developed by the Linux Network Foundation <cit.>. ONAP involves collaboration among various working groups. ONAP community modified the standard workflow of Atlassian to fit their bug-solving strategy by adding a state referred as "Submitted". Standard Atlassian workflow and modifications of ONAP is shown in Fig. <ref>.

Furthermore, the dataset provided by <cit.> is also analized. This dataset includes bug reports from 55 Apache projects. These Apache projects also use a modified version of the standard Atlassian workflow, which can be seen in Fig. <ref>. This modified workflow has an additional state called 'Patch Available', similar to the 'Submitted' state of ONAP.



















 §.§ Bug related data



In this work, we are interested in the bug resolution time and hence, all issues other than 'Bug' are filtered out. Jira provides a collection of reported bugs. This collection contains details about the bug and its resolution process. Certain information is provided when a bug is reported, while others change throughout the resolution process. The following information is utilized in our analysis:


    
  * Priority indicates the importance of the bug. The priority levels vary between projects. Hence, in this work, priorities 1 and 2 refer to the highest and second highest priority of any project.
    
  * Reporter is the person reporting the bug.  
    
  * Assignee is the person who is responsible for fixing the bug.
    
  * Creation Time is assigned to the date and time when the bug was reported.
    
  * State corresponds to one of the different states of the workflow. The initial state when reporting a new bug is "Open". The state is updated during the resolution process, according to the state in the workflow.
    
  * Resolution Status reflects the progression of the resolution process, initially set as "Unresolved" and subsequently updated based on outcomes such as "Fixed" or "Cannot Reproduce" upon completion of bug resolution.
    
  * Update Time is the time of the last edit
    
  * State Transition includes the previous and next states and the time of the transition.




Additional attributes that can be found in Jira are listed below:


    
  * Summary is the brief description of the bug.
    
  * Sub-Project is the sub-project to which the bug is associated.
    
  * Description is the detailed explanation of the bug.
    
  * Labels  are issue-related labels.
    
  * Affects Version is the version of the project in which the issue is occurring.
    
  * Fix Version is the version of the project in which the issue was resolved.
    
  * Watches Count is the number of people watching the issue.
    
  * Comments are the comments during the bug resolution process.

    




Any modification to bug-related information in Jira (such as status, states, or additional comments) is collected, along with the timestamp and the person responsible for the update. This allows us to determine the number of people involved in the resolution process or the calculation of the time between commits and state transitions. The collected state transitions are employed to calculate the duration in each state until the bug is resolved.
















 §.§ Bug resolution process
 

As bugs in Jira are a specific type of issue, they also follow the workflow for the issues. While other transitions from Fig. <ref> are also utilized, a primary path within the workflow is typically followed for the bug resolution process. A bug begins in the 'Open' state when it is initially reported. Once a developer starts working on the bug resolution, the state is changed to 'In Progress'. The state of the bug changes to the 'Resolved' state, once the bug is resolved, where it awaits approval by another team member. After approval, the bug is closed by moving to the 'Closed' state. However, a bug may reappear so that it can be reopened and returned to the 'Reopen' state.


 


§ BUG COLLECTION AND ANALYSIS



A developed Python script has been deployed for the proposed bug collection and analysis. It is important to mention that the bugs for ONAP and ONOS were collected directly from Jira. However, the bugs for Apache were collected from the dataset at <cit.>. The data analysis included in this work focuses on the two top-priority bugs (i.e., priority 1 and 2).



 §.§ Bug collection and filtering





The collected data in this study includes the reported bugs from the start of each project to the collection date (in this case, 05.04.2023). The collected data encompasses all bug related details (refer to Section <ref>) as well as all related information during the bug resolution process such as transition states and times, authors involved in each change, etc.


In this study, we investigate the bug resolution process. However, the dataset does not only include the data relevant to our study, but it also includes exceptional/unwilling data, which must be filtered.


Let us introduce the proposed data filtering on the dataset:

    
  * Not Resolved Bugs: 
   Not all reported bugs are necessarily resolved; some of them may be closed for alternative reasons. The resolution status is determined as a result of the debugging process. In the case of ONAP and ONOS bugs, the resolution status begins as 'Unresolved' and subsequently changes to one of the multiple resolution statuses, which are listed in Table <ref>. The table also presents the respective percentages of each status. As this study targets the resolution time of bugs, only the bugs with the status 'Done' and 'Fixed' are considered, which account for more than 80% of the priority 1 and 2 bugs for both ONAP and ONOS projects. Regarding the Apache dataset, no filtering is necessary as the considered dataset contains only bugs with a resolution status of 'Fixed'.
    
  * Transient State: As depicted in Fig. <ref>, the resolution process consists of a set of transitions between different states, starting at "Open" and ending at "Closed". For the bug resolution time study, it is relevant to know the time at each state before any transition. However, in certain cases, bugs may spend only a few minutes or even seconds in a particular state. This short duration might not provide accurate information about the actual effort expended in that state. Transient states of this nature can arise due to automated tasks or the practice of updating Jira after completing multiple states. Consequently, states with a duration shorter than a few minutes 
    are removed as a state, and the time is added to the original state. 
    
  * Not Defined States: Observations of the dataset revealed that a few bugs have states that are not defined within the workflow of the project. The number of such states is significantly lower than those within the Jira workflow. Since the states are not in the workflow, we filter them out. The duration spent in these states is accumulated with the previous state, ensuring that the resolution duration remains unaffected.
    
  * Loop to the same State: There is a possibility that a bug has a transition from a state to the same state (referred as loop). Some of these transitions are defined as such in Jira, while others may arise due to the removal of transient states or undefined states. In the case of a loop, no additional transition is considered, and the time spent in these states is merged. 
    
  * Transitions from "Close" State: The goal of finding the bug resolution time implies that bugs eventually reach the "Closed" state. While some bugs may occasionally be reopened after a certain period, it is often observed that reopening occurs after a substantial amount of time, which significantly inflates the resolution time. To address this issue, the "Closed" state is treated as an absorbing state, meaning that once a bug reaches this state, any subsequent transitions from the "Closed" state are excluded.




 

















 §.§ Bug resolution time distribution and common flows in transition workflow






The resolution time is defined as the time elapsed from the reporting of a bug (i.e., in the "Open" state) to the time when it reaches the "Closed" state. The resolution time is the sum of the time spent on various states, but it does not provide information about the duration spent on each individual state.


In this section, we introduce the resolution time analysis for the particular case of the priority 1 bugs of the ONAP project.  These priority 1 bugs amount to a total of 906 in the project. 
The most common resolution paths for all the ONAP priority 1 bugs, can be found, as summarized in Table <ref>.

It can be observed in this case, that almost 91% of the bugs follow the seven main paths. Furthermore, 45% of the bugs follow the path  Open-In Progress-Closed or  Open-Closed.











The time spent at each state before any possible transition can also be found. These time distributions are crucial for bug resolution time estimation from a specific state to the 'Closed' state. For this purpose, Table <ref> provides the mean and median time spent in a state before a particular transition to a different state, as well as the number of such transitions. 
Upon close examination of the transitions from the 'Open' state, it becomes apparent that the duration is longer when a bug transitions directly to the 'Closed' or 'Delivered' states compared to transitioning to the 'In Progress' or 'Submitted' states. One possible explanation for this phenomenon is that bugs typically progress through these states, even though they may not be explicitly recorded in Jira. 

The significant difference between the mean and median time for most of the states is primarily attributed to the fact that most of the bugs have a relatively short duration in each state. However, some bugs are more challenging to handle or the assignee starts later to  work on them, which causes outliers and significantly increases the mean values.






 §.§ Impact of different bug aspects



This section investigates the impact of different properties of the bugs on the bug resolution time. Properties like priority level, assignee and reporter of a particular bug may impact its resolution duration. The Apache dataset encompasses various projects, but we have excluded this data to eliminate variations caused by project differences. The number of bugs in ONOS should be higher to ensure a meaningful comparison. Thus, ONAP is the preferred option for investigating the assignee's and reporter's influence. Similar to the previous case, the bugs are separated based on priority.


Let us first analyse the impact of the reporter on the bug resolution time. The reporter of the bug may impact the time given his/her experience by providing an accurate and clear bug description as well as listing all the essential information to reproduce the bug accurately.






Fig. <ref> depicts the impact of the top 10 reporters who reported the highest number of bugs, on the bug resolution time for priorities 1 and 2. The ordering of reporters is determined by the median value of the priority 2 bugs, taking into consideration that some reporters have a limited number of priority 1 bugs, with half of them reporting fewer than 20 bugs. The total number of reported bugs is shown under the x-axis of the graph, whereas the median resolution time for each case is given at the top of the graph. The decision to use the median value instead of the mean value is made in order to prioritize typical values over outliers, as the mean value can be heavily influenced by the outliers. Firstly, it is noteworthy that the resolution time for priority 1 bugs is generally shorter than that of priority 2 bugs for most reporters. Secondly, the impact of reporters on the bug resolution process is significant. In comparison to the reporter with the slowest bug resolution times, the one with the fastest experiences a median value that is 15 times quicker for priority 1 bugs and 35 times quicker for priority 2 bugs.




Another important factor is the assignee of a bug, who is responsible for resolving it. Assignees possess different expertise, availability, and familiarity with the project. The impact of various assignees is presented in Fig. <ref>. Similar to reporters, the top 10 assignees, assigned to the most bugs, are selected and ordered based on the median values of priority 2 bugs. It is evident that the assignee significantly affects the bug resolution time. Moreover, the median values on top of the figure indicate that nine out of ten authors tend to resolve higher priority bugs faster.

















The previous observations in this section indicate that the reporter and assignee play an important role in the bug resolution process. However, the reporter and assignee can be the same person, which may occur when the reporter wants to take responsibility for its resolution, e.g. when the reporter is familiar with that part of the code. In this case, it can be assumed that reproducing the bug can be easier, and there is no time required to communicate the assignee with the reporter to comprehend the bug. Thus, we further explore the impact of assigning the reporter to him/herself than to a different assignee. Fig. <ref> shows that assigning the reporter to the issue has a significant impact on bug solving process for both priority 1 and 2. 



 





 §.§ Most relevant workflow paths and study of state times









 §.§ Bug Distribution across States over Time



In this section, the distribution of bug states over time is presented. It presents the probabilities of being in specific states at given times for the ONAP, ONOS, and Apache projects.


The time when a bug is reported (i.e., the bug is at 'Open' state), is considered to be the time 0. Hence, at this time, the initial percentage of bugs in the 'Open' state is 100% (1.0) and zero at the other states. As time progresses, the bugs change to different states, leading to different distributions across the various states. The percentage of bugs in different states over time is provided for ONAP, ONOS and Apache projects in Fig. <ref> for priorities 1 and 2. In all cases, it can be easily observed that since the 'Closed' state is absorbing, there is an always increasing percentage of bugs in that state. 

The analysis of Fig. <ref> reveals a notable distinction in bug resolution times among different projects. Specifically, in ONAP, bugs consistently tend to reach the 'Closed' state within approximately 8,000 hours, independently of the priority. Conversely, such a trend is not observed for bugs in ONOS and Apache projects, as after this time, still 10-20% of the bugs have not reached the "Closed" state. This divergence stems from the approach adopted by Apache projects, where the emphasis is not solely on expeditiously closing bugs, but rather on considering them solved once they reach the 'Resolved' state. Additionally, in the ONOS project, some bugs remained in the 'Delivered' state for an extended period before eventually being closed.
 







Examining the percentage of bugs that achieve 'Closed' status, it can be observed that for priority 1 bugs, 50% is reached after around 1,200 hours, while for priority 2 bugs, it takes approximately 2,000 hours. Comparatively, in ONAP and ONOS projects, more than half of the bugs are closed within 300 hours for priority 1 bugs and within 500 hours for priority 2 bugs. Analyzing the graph depicting bugs in the 'Closed' state, it becomes apparent that the curve for ONAP bugs exhibits a steeper ascent compared to that of ONOS and Apache projects. Additionally, it is noticeable that the number of bugs in the 'Open' state decreases more rapidly in the ONAP project. This observation suggests that the debugging teams in ONAP work on the bugs at an earlier stage in comparison to the other projects. 

However, it is important to note that making a direct comparison is challenging due to variations in the strategies employed by different working groups when transitioning between states in their respective workflows. Some projects may strictly update the bug's state as soon as it is reached in the actual handling process, while others may implement such changes after a certain amount of time has passed. This discrepancy becomes visible in the utilization of the 'Delivered' state in Fig. <ref>. In Apache projects, bugs in the 'Delivered' state can be considered solved, whereas the same claim may not hold true for ONAP and ONOS projects.






























 
 
 
 
 

































§ PREDICTING RESOLUTION TIME WITH NEURAL NETWORK



Even after doing filterings, which are explained in Section <ref>, there are still bugs that can be filtered. One of the main reason to filter is the long living bugs, which are discussed in Subsection <ref>. Possible reasons of the short and long living bugs and different possibilities to filtering those bugs are discussed in <cit.>. It has been explained that some bugs are forgotten, wrongly labeled or reported after its resolution. Therefore removing short living bugs or long living bugs might improve prediction quality. In results we compared different filtering strategies for the bugs. In <cit.>, authors discuss removing long living bugs with the help of inner fence and outer fence. We also used the same strategy if all the bugs outside of inner fence is removed it is called removing mild outliers in case of all bugs above outer fence is removed is called removing extreme outliers. As an extend to these outlier removal we also proposed to remove the bugs without any update (comment, state change etc.) for more than 30 days. We compared these filtering methods and investigate their impact on the results.

We used neural networks (NN) to predict the resolution time of the bugs. The results are compared with Naive Bayesian (NB) and k Nearest Neighbour (kNN) algorithms. These two algorithms are chosen because they are widely used for resolution time prediction <cit.>. Since the NB and kNN cannot be applied to get numerical results but only used for classification, the results which used for comparison takes two classes into account. These two classes divided depending on the resolution time. If the resolution time is shorter than median resolution time of the training dataset, the bug considered as 'fast' bug otherwise it is classed as 'slow' bug. 

The input of the NN is the information related to the priority, subproject, assignee, reporter and either the assignee is same person of the reporter or not. Two different NN model compared for the classification one of them is trained for binary classification and returns either the bug is belongs to 'fast' or 'slow' group and the other one predicts the exact resolution time. The size and input of those models are same, where the difference lies on the loss, activation functions and the output values. For binary predicted output values are either 0 or 1 and for exact value prediction values are time to solve as hours. Since the exact time prediction is not same as binary classification, the time prediction values should be converted to two class depending on either it is predicted lower than median time or higher than median time to resolve. Both two NN consist of 5 layers, where the first one is dedicated to normalization of the input data. 

The ONAP project is selected for the model training, because it has more bug reports than the ONOS project and insufficient amount of data might impact the performance of the NN. In order to make prediction we divided the dataset into two group, training, which corresponds to 80% of all dataset and test which is the rest of the dataset. In order to be able to generalize the results we cross validate our results with randomly dividing the dataset 10 times. As it can be seen from the Fig.<ref>, even though NN does not make binary estimation of 'slow' and 'fast' its accuracy is higher than kNN estimation for any type of filtering. NN results for binary classification provides either best or second best results in any case. The filtering does not help to improve the results, one reason for that can be the reduction of the training dataset. Another possible reason is the filtered bugs mostly have similar information like assignee or subproject and classifying them as 'slow' increase the accuracy. 


The results for exact time prediction is given in Figure <ref>. It can be seen that the median normalized prediction error is 20% for all bugs and 30% for filtered bugs. One of the reason for this high error can be the holiday periods and weekends. The bug report day actually plays an important role on its resolution time. Even though there is an error around 20% between predicted and real time to solve, predicted value can be more helpful than two classes.














§ CONCLUSION


In this work the importance of different aspects of the bugs on their resolution process has been shown. The time to solve a bug decrease, when the reporter is assigned to the issue. Also the impact of different resolution process or strategy to report this process has been shown. It has been seen that in Apache projects, reaching to the 'Closed' state is not forced like ONAP, instead the bugs stays in 'Resolved' state for long time. It has been shown that bug resolution time can be estimated with neural networks and the predicted value can be used by the developer teams to distribute their workforce according to the estimation. This work can be extended with using different machine learning techniques like pretraining and data augmentation to make a valid prediction for new and smaller projects.




§ ACKNOWLEDGMENT


This work has received funding from the "DFG" under grant numbers MA 6529/4-1 and KE 1863/10-1.









1
macedo2015programmable
D. F. Macedo, D. Guedes, L. F. Vieira, M. A. Vieira, and M. Nogueira, Programmable networks—From software-defined radio to software-defined networking,1em plus
 0.5em minus 0.4emIEEE communications surveys & tutorials, vol. 17, no. 2, pp. 1102–1125, 2015.

 ortu2015jira
M. Ortu, G. Destefanis, B. Adams, A. Murgia, M. Marchesi, and R. Tonelli, The jira repository dataset: Understanding social aspects of software development,1em plus
 0.5em minus 0.4emProceedings of the 11th international confer- ence on predictive models and data analytics in software engineering, 2015.

 vieira2019reports
R. Vieira, A. da Silva, L. Rocha, and J. P. Gomes Fromreportstobug-fix commits: A 10 years dataset of bug-fixing activity from 55 apache’s open source projects,1em plus
 0.5em minus 0.4emProceedings of the Fifteenth International Conference on Predictive Models and Data Analytics in Software Engineering, 2019.
 
 zhang2013predicting
H. Zhang, L. Gong, and S. Versteeg Predicting bug-fixing time: an empirical study of commercial software projects,1em plus 0.5em minus 0.4em35th International Conference on Software Engineering (ICSE), 2013.

 akbarinasaji2018predicting
S. Akbarinasaji, B. Caglayan, and A. Bener Predicting bug-fixing time: A replication study using an open source software project,,1em plus
 0.5em minus 0.4emJournal of Systems and Software, vol. 136, pp. 173–186, 2018.

 fisher2013utilizing
 J. Fisher, D. Koning, and A .Ludwigsen Utilizinga tlassian jira for largescale software development management,1em plus
 0.5em minus 0.4emLawrence Livermore National Lab.(LLNL), Livermore, CA (United States), Tech. Rep., 2013.

 yilma2020benchmarking
G. M. Yilma, Z. F. Yousaf, V. Sciancalepore, and X. Costa-Perez, Benchmarking open source nfv mano systems: Osm and onap,1em plus
 0.5em minus 0.4emComputer communications, vol. 161, pp. 86–98, 2020.
 
 abdelmoez2013improving
W. AbdelMoez, M. Kholief, and F. M. Elsalmy, Improving bug fix-time prediction model by filtering out outliers,1em plus
 0.5em minus 0.4emThe International Conference on Technological Advances in Electrical, Electronics and Computer Engineering (TAEECE). IEEE, 2013.

 lamkanfi2012filtering
A. Lamkanfi and S. Demeyer Filtering bug reports for fix-time analysis,,1em plus
 0.5em minus 0.4em16th European Conference on Software Maintenance and Reengineering, 2012.





 Paper Title: An LLM-based Readability Measurement for Unit Tests' Context-aware Inputs.

























































Journal of  Class Files, Vol. 14, No. 8, August 2015
Shell et al.: Bare Advanced Demo of IEEEtran.cls for IEEE Computer Society Journals





































§ ABSTRACT 
Automated test techniques usually generate unit tests with higher code coverage than manual tests. However, the readability of automated tests is crucial for code comprehension and maintenance. The readability of unit tests involves many aspects. In this paper, we focus on test inputs. The central limitation of existing studies on input readability is that they focus on test codes alone without taking the tested source codes into consideration, making them either ignore different source codes' different readability requirements or require manual efforts to write readable inputs. However, we observe that the source codes specify the contexts that test inputs must satisfy. Based on such observation, we introduce the Context Consistency Criterion (a.k.a, C3), which is a readability measurement tool that leverages Large Language Models to extract primitive-type (including string-type) parameters' readability contexts from the source codes and checks whether test inputs are consistent with those contexts. We have also proposed EvoSuiteC3. It leverages C3's extracted contexts to help EvoSuite generate readable test inputs. We have evaluated C3's performance on 409  classes and compared manual and automated tests' readability under C3 measurement. The results are two-fold. First, The Precision, Recall, and F1-Score of C3's mined readability contexts are , , and , respectively. Second, under C3's measurement, the string-type input readability scores of EvoSuiteC3, ChatUniTest (an LLM-based test generation tool), manual tests, and two traditional tools (EvoSuite and Randoop) are 90%, 83%, 68%, 8%, and 8%, showing the traditional tools' inability in generating readable string-type inputs. We have conducted a survey based on the questionnaires collected from 30 programmers with varied backgrounds. The results reveal that when C3 identifies readable differences between tests, programmers tend to give similar opinions of the test's readability of C3.




  readability, test generation, large language models.




An LLM-based Readability Measurement for Unit Tests' Context-aware Inputs
    Zhichao Zhou,
        Yutian Tang,
        Yun Lin,
        and Jingzhu He
Z. Zhou and J. He are with School of Information Science and Technology, ShanghaiTech University, China
Y. Lin is with the Shanghai Jiao Tong University, China
Y. Tang is with University of Glasgow, United Kingdom
J. He (hejzh1@shanghaitech.edu.cn) is the corresponding author.
    August 5, 2024
====================================================================================================================================================================================================================================================================================================================================================================































§ INTRODUCTION
 

Unit testing is a popular testing technique among developers. A unit test checks whether a method inside the source code behaves functionally correctly. Existing unit test generation tools (e.g., search-based tool EvoSuite  <cit.> and feedback-directed random tool Randoop <cit.>) can generate unit tests that are close to and even outperform the manually written ones in many coverage criteria <cit.>, such as branch coverage.

We observe that automated tests' readability is crucial for programmers. For example, in a scenario where programmers use tools to obtain regression tests for two code versions <cit.>, they must review a test failing at an assertion to locate and fix new bugs. It is time-consuming and takes lots of manual effort <cit.>.

The readability of unit tests involves many aspects, including but not limited to test method names <cit.>, comments <cit.>, documents <cit.>, identifiers, and assertions <cit.>. In this paper, we focus on test inputs. Many studies <cit.> and our survey (see Sec. <ref>) show that test inputs are crucial in readability. However, the random inputs generated by existing tools (EvoSuite and Randoop) cost understanding time. A former survey <cit.> showed that compared to strings occurring in natural languages (e.g., "mail"), random strings (e.g., "Qu5ua") require people at most 1.6 times more comprehension time.
A constructor from Airsonic <cit.> with three tests differing in inputscode:airsonic_user
// method under test
public User(String username, String password, String email) 
    this.username = username; this.password = password;
    this.email = email;

public void testCreateUser()  // manual test
    User user = new User("Simon", "secret", "simon@example.org");

public void testConstructor()  // automated test by ChatUniTest
    User user = new User("testUser", "testPassword", "test@example.com");

public void test05() throws Throwable  // automated test by EvoSuite
    User user0 = new User("|x45e*3q4+", " [stream]", "");

public void test001() throws Throwable  // automated test by Randoop
    User user7 = new User("hi!", "", "");

public void test25() throws Throwable  // automated test by EvoSuiteC3
    User user0 = new User("Enrico Fermi", "guest", "careers@jobs.com");


To deal with this issue, Afshan et al. <cit.> used a statistical model to predict the likelihood of the test input occurring in the real world. They formulated the likelihood as a secondary objective to guide test generation. The main issue with this approach is that its model ignores the source code when judging the input readability. However, different source codes have different requirements for input readability. Deljouyi et al. <cit.> proposed to extract understandable inputs from end-to-end (E2E) tests' execution trace logs since an E2E test aims to simulate a real user to execute an application’s entire workflow <cit.> and may contain understandable inputs. However, this approach is semi-automated since it requires human effort to write E2E tests.


 §.§ A Motivating Example

We leverage an example to demonstrate the lack of readability of automated tests and the challenges of existing studies. This example is a simplified constructor method under test (MUT) with a manual and two automated tests on Code <ref>. The MUT comes from an open-source project, Airsonic <cit.>. It has three string-type parameters, which are the user's name, password, and email, respectively. The manual test assigns the values "Simon", "secret", and "simon@example.org" to them, matching the contexts of the three parameters. The input values generated by an LLM-based test generation tool (ChatUniTest <cit.>) are "testUser", "testPassword", and "test@example.com". They also match the contexts. Conversely, the input values from  <cit.> are "|x45e*3q4+", " [stream]", and "". These values make the automated test less readable than the manual one. In our survey (see Sec. <ref>), 22 out of 30 respondents prefer the manual test to EvoSuite's test regarding readability. The corresponding values from Randoop <cit.> are "hi!", "", and "", posing the same issue. Such deficiency prevails in tests generated by traditional tools. Fig. <ref> aggregates the inputs from manual tests and tests generated by various testing tools for 30 classes (see Sec. <ref>). It shows that manual tests contain diversified real-world objects (e.g., person names, emails, and locations), while  and Randoop <cit.> generate a large number of random inputs. It could be worse for methods with multiple parameters. One interviewee gave us the following feedback:


“When there are many parameters, readers can not easily match parameters with values if a test uses a bunch of random inputs.”


The secondary objective complementing branch objectives proposed by Afshan et al. <cit.> guides genetic algorithms <cit.> to generate inputs that are more likely to occur in the real language. However, this objective ignores the source code, thus ignoring the different input readability requirements of username and email. Hence, the optimized inputs may also not satisfy these readability requirements hidden in the source code. The approach of Deljouyi et al. <cit.> to extract readable test inputs from manual-written E2E tests can not applied in this  class under test without extra manual efforts since the Airsonic project does not have E2E tests.  

To cope with this readability issue, we have to solve two challenges:

∙Challenge 1 is how to extract the readability requirements for the parameters from the source code.  Due to their varied occurrence (e.g., comments, variable names, and their operations), it is challenging to identify such requirements using traditional techniques like data-flow analysis <cit.>.

∙Challenge 2 is how to improve existing traditional tools (e.g., EvoSuite) to fulfill such requirements after they are extracted. 


 §.§ Contributions
 
To solve the challenges shown in the motivating example, this paper proposes a novel concept called readability context, which refers to the requirements for test readability described in the source code (e.g., valid person names for the username parameter in Code <ref>). Based on the readability context, this paper proposes the Context Consistency Criterion (C3). First, C3 passes detailed information (including the source code and the comments) to a Large Language Model (LLM) and requests it to mine the readability contexts for the primitive-type (including string-type) parameters. Besides, we also pass few-shots <cit.> (i.e., pre-defined similar questions and their answers for LLM to quickly adapt to this mine-context task) to LLM. Second, C3 checks how well the inputs of the tests are consistent with readability contexts based on LLMs, NLP tools, and regex matching.

To sum up, this paper's contributions include:

∙ We observe that test inputs highly depend on the source codes. Based on the observation, we propose a novel concept, readability context, to represent the test readability requirements for the primitive-type variables of source codes. We design and implement an automatic tool, i.e., C3, to mine the readability contexts and leverage these contexts to measure the consistency of the tests' inputs with readability contexts (Sec. <ref>).

∙ We propose a variant of EvoSuite <cit.> (a search-based test generation tool), called EvoSuiteC3. EvoSuiteC3 leverages the readability contexts mined by C3 to help EvoSuite generate test inputs better in readability (Sec. <ref>). The EvoSuiteC3 test in Code <ref> exemplifies readability improvements brought by EvoSuiteC3.

∙ We evaluate C3 on 107  projects, including 409 classes. C3 detects 1024 ones with readability contexts from 4485 primitive-type parameters. After a human review process, we confirm that C3's Precision, Recall, and F1-Score are , , and , respectively (Sec. <ref>).

∙ We compare manual tests, two traditional test generation tools (EvoSuite <cit.> and Randoop <cit.>), EvoSuiteC3, and an LLM-based tool (ChatUniTest <cit.>) on 30  classes with 115 parameters that have readability contexts. Fig. <ref> summarizes test inputs for these parameters and reveals that EvoSuiteC3 and ChatUniTest's tests perform closely to manual ones in input readability. Meanwhile, EvoSuite and Randoop largely fail to generate readable string-type inputs (Sec. <ref>). Furthermore, we invite 30 programmers with varied backgrounds to quantify the readability of manual and automated tests. The results show that programmers reach a consensus with C3 (Sec. <ref>).

The rest of this paper is organized as follows. Sec. <ref> discusses the related work. Sec. <ref> and <ref> describe C3 and EvoSuiteC3's system design, respectively. Sec. <ref> illustrates our evaluation. Sec. <ref> presents the threats to validity. Sec. <ref> concludes the whole paper. Sec. <ref> provides the data availability.



§ RELATED WORK

In this section, we discuss related studies.

Traditional test generation.
This paper uses two traditional test generation tools, Randoop <cit.> and EvoSuite <cit.>, to represent two methods. The first, random testing <cit.>, generates random test inputs to satisfy coverage criteria or expose faults in the software under test. Its lightweight nature allows for easy testing of large-scale systems, providing a solution to the scalability issues of static methods like model checking and symbolic execution <cit.>. However, random testing has limitations. While it can cover source lines, the random inputs may lack readability. Our evaluation (Sec. <ref>) reveals that most string-type inputs generated by Randoop are meaningless (e.g., "hi!"), which can confuse developers. Additionally, random testing without explicit guidance is inefficient in generating high-coverage tests. The second method, search-based testing <cit.>, addresses this by using coverage criteria as fitness functions to guide genetic algorithms in generating test inputs. This approach may produce readable inputs if the optimization of a fitness function requires it (e.g., name == "Peter" in a condition statement). However, most readability requirements are not explicitly defined. Consequently, our experiments indicate that EvoSuite's readability is close to Randoop's.

Readability of traditional test generation.
Existing studies on the readability of traditional test generation can be divided into two categories. The first focuses on generating test summaries. Daka et al. <cit.> proposed summarizing a test's covered goals as its name. Panichella et al. <cit.> suggested a more thorough summary incorporating covered goals, the  class being tested, and additional details to create comments. Roy et al. <cit.> employed deep learning to name tests and internal variables and create comments. Li et al. <cit.> proposed an approach to generate test documentation. These summary-based approaches ignore the concrete inputs, which are also regarded as necessary for readability (see Sec. <ref>). The second category applies code readability studies to test readability. Based on Buse and Weimer's research on code readability <cit.>, Daka et al. <cit.> gathered a dataset of tests along with their human-rated readability, used common code metrics as features (including identifier length, parentheses, test length, and casts), and added extra features for tests (e.g., assertions). They trained a linear regression model to improve the readability of automated unit tests. Grano et al. <cit.> leveraged the existing code readability model <cit.> to compare the readability of manual and automated tests. Gregory Gay <cit.> proposed improving tests' readability by leveraging LLMs to refactor tests, such as dividing a test into small ones and formatting test code to satisfy common code style standards. This paper closely aligns with the studies of Afshan et al. <cit.> and Deljouyi et al. <cit.>. Afshan et al. <cit.> incorporated a statistical language model into search-based test generation to predict the likelihood of the test input occurring in the real world and guide test generation as a secondary fitness function. Deljouyi et al. <cit.> proposed to leverage end-to-end (E2E) tests to generate understandable unit test inputs since an E2E test simulates a real user to execute an application’s entire workflow and may contain understandable inputs. However, this approach requires human effort to write E2E tests. All of the above-mentioned studies ignore the source code, which plays a crucial role in determining the readability of the test code. C3 complements these studies. It mines the readability contexts of the primitive-type parameter from the source code and judges whether a test input is consistent with the context as a readability indicator. Our experiments show that C3 identifies the shortcomings of EvoSuite and Randoop in readability, and EvoSuiteC3 improves EvoSuite's readability.



LLM-based test generation.
Large language models (e.g., the GPT series <cit.>) have demonstrated capabilities across diverse domains, including unit test generation. Tang et al. <cit.> compared test suites generated by both LLM and EvoSuite in correctness, code coverage, complexity metrics <cit.>, and bug detection. Lemieux et al. <cit.> suggested using LLMs to overcome plateaus in fitness functions during search-based test generation. AthenaTest <cit.> employs a pre-trained BART model <cit.> for test generation, while ChatUniTest <cit.> is an automated tool that guides LLMs in generating and repairing tests. More studies can be found in the survey paper <cit.>. Given its ready-to-use nature and ability to continuously guide LLM in generating syntactically correct tests, this paper uses ChatUniTest as a baseline for comparison with EvoSuite and Randoop. Many LLM studies <cit.> assert that tests generated by LLMs are more readable than those produced by traditional tools, a conclusion reached through manual review or developer interviews. Our approach, C3, offers a quantitative metric for measuring test readability automatically. Our experimental results validate their findings that LLMs surpass EvoSuite and Randoop in generating readable tests. We also find that LLMs outperform developers as their tests are readable and achieve higher code coverage than manual tests.


§ CONTEXT CONSISTENCY CRITERION (C3)

This section describes the system design of Context Consistency Criterion (C3). Fig. <ref> shows its overview. Without losing any generality, we use  as the target language to introduce C3, while C3 is not tied to any specific language. C3 takes the tested source code (e.g., a  class or a method) and the test code as the input. C3's framework comprises two components, i.e., the Mining Component and the Judging Component. The Mining Component mines the readability contexts from the tested source code (Sec. <ref>). C3 extracts the primitive-type parameters from the source code. Then, for each parameter, C3 passes its text information (i.e., the source code and comments) to a Large Language Model (LLM) and lets LLM choose a context from a prepared context collection (shown in Table <ref>). The Judging Component measures the consistency of test inputs with the readability contexts (Sec. <ref>). C3 extracts the primitive-type inputs from the test. Then, it uses various tools (i.e., LLM, NLP tools, and regex matching) to check whether an input satisfies its context as the readability result.


 §.§ Mining Component

The first step is to mine the contexts from the code under test. C3 extracts the primitive-type parameters; then, C3 uses each parameter's text information and the prepared contexts to ask an LLM which context it matches.

Primitive-type parameter extractor.
We leverage an open-source  parser <cit.> to analyze the code and extract its parameters of primitive types, including string, byte, short, int, long, float, and double. Note that (1) the primitive types in this paper include string although  regards it as an object type since it is common in nearly all real-world projects and can be initialized by literal values like other primitive types, and (2) we omit char and boolean since their range of values is small and finite, posing no issues with readability. Besides, for a parameter, we extract its method's source and comments used to construct LLM's prompt. Note that we only use the method source code instead of the whole class code, thus significantly reducing the prompt size and decreasing the model's financial costs.

Prepared contexts.
We divide the concrete primitive types into two abstract types, i.e., String (including string) and Number (others), and introduce their contexts, respectively.

(1) String.
We use the named entity (a concept from NLP) <cit.> to prepare contexts for String. Named entity refers to real-world context-aware objects, such as person names and emails. Referencing previous studies <cit.>, we prepare 23 contexts shown in Table <ref>, which can be grouped into six context categories: Cyberspace (7), Geographic and Abstract Location (6), Temporal (4), Number (not Number type, but Number as String contexts) (3), Personal Identifier (2), and Finance (1). Note that LOCATION and TIME are two generic contexts. We use them to capture potential contexts in the source code that belong to Geo-Abstract Location and Temporal but do not match any other contexts in these two categories. For example, “restaurant” does not match any specific contexts  (e.g., CITY and GPE) of Geo-Abstract Location but can be captured by LOCATION.

(2) Number.
While named entities serve as String's contexts, we employ literal formats for Number's contexts, including base formats (e.g., 0b100, 0xfe) and Scientific Notation (e.g., 1.2e20f). We add two more contexts, FIXEDLENGTH and LONGNUMBER. The former targets numbers of fixed lengths, such as credit card numbers, while the latter is designed for potentially overly long numbers. Code <ref> from our survey questions (Sec. <ref>) provides an example.


Primitive parameter context miner.
Based on a primitive parameter's text information (the source code and comments) and the prepared contexts, we construct a prompt to ask an LLM which context the parameter matches. The prompt is created by following the few-shot technique <cit.>. Few-shot means that before asking a question, pass some similar questions and the prepared answers to an LLM in advance. This technique is proven to work well in many fields <cit.>. Fig. <ref> shows our prompt example with one shot for one context category. The first part of Fig. <ref> is to set the model role and control its behaviors. The second part is one shot, i.e., a question example with an answer. The third part is the question of interest. The first three placeholders store the parameter name, the method name, and the source code with comments. The following placeholders store each context and its instance (e.g., PERSON and "Simon").

We start illustrating the prompt-constructing process by introducing two parameters for controlling the prompt size. In LLM, a token is a basic unit of text <cit.>, and an LLM has a token limit for a conversation. We use MaxToken to represent the token size limit. Then, we use Remain to keep the token space for the target question (i.e., the third part of Fig. <ref>) and LLM's response. First, we calculate the RestToken by (MaxToken- (Remain+strToken(targetQuestion))), where strToken computes the number of tokens within a string. Then, we add the role setting message to Messages and update RestToken. Next, we iterate to add a shot and update RestToken if there is enough token space. Finally, we add the targeting question. For the Number type, we have an extra operation. We observe that an important factor determining a Number parameter's context is the operators it uses. The or method on Code <ref> presents an example. The | operator denotes that the values of a and b should be in binary format, i.e., their contexts are BINARY. Therefore, we emphasize this factor by appending a sentence to the end of the targeting question, i.e., . Meanwhile, two tests identical for compilers, test1 and test2, vary in readability: The second is more readable since its inputs match the contexts.
An or method with two tests differing in number formatscode:calc
// method under test
int or(int a, int b)  return a | b; 
// a test satisfying traditional criteria like line coverage
void test1()  assert 7 == or(4, 3); 
// a more readable test
void test2()  assert 0b111 == or(0b100, 0b011); 


After generating messages, we pass them to LLM to get the result, which could be a concrete context or MISC. MISC means that LLM regards that no context matches this parameter.




 §.§ Judging Component

The Judging Component accepts parameters with contexts and a test as the inputs. For each parameter, it judges whether the test has an input passed to it and matches its context. We utilize various tools to judge the context matching, including  LLM, arbitrary NLP tool (this paper uses CoreNLP <cit.>), and regex matching. We first introduce each of them. Then, we describe combining them to complete the judging process.

Tools.

(1) LLM.
Fig. <ref> shows an example of asking LLM to check whether an input matches the context. We prepare a shot for each context category like in the Mining Component (see Sec. <ref>). The prompt-constructing process is similar to the process of constructing mining prompts, too.

(2) CoreNLP.
Traditional NLP tools (e.g., CoreNLP) can verify the consistency of a character string to the context. CoreNLP supports all context categories of String <cit.>, except Cyberspace (see Table <ref>). Its advantage over LLM is that it is financially cost-free.

(3) Regex matching.
Regex matching works for String's Cyberspace and Number's all categories. For example, we can use "ˆ0[bB][01]+$" to check whether a number is binary.


Primitive-type input extractor.
We use a  parser <cit.> to extract the test's inputs whose parameter has a context. Then, we run the judger on each input. Note that we only take three input types into account: the literal values, the primitive-type variables defined with a literal in the test method, and the test's final static fields. This choice ignores many input sources, such as another class's field. However, according to our survey (see Sec. <ref>), programmers prefer straightforward test statements. For example, pass-by-value is preferable to other ways when calling the method under test. Hence, this choice has a tiny impact on readability judgment.

Primitive-type input judger.
We use different tool combinations for context categories (shown in Table <ref>'s last column).

(1) LLM + Regex matching.
We use this combination to check String's Cyberspace category (see Table <ref>). Regex matching is enough to check whether an input is valid for these contexts (e.g., EMAIL). However, in realistic development, programmers may use an invalid but readable input, such as "testEmail" for EMAIL. Hence, we complement LLM to make the judge results more elastic than they are judged by regex matching only. We regard an input as readable as long as one tool gives a positive result.

(2) LLM + CoreNLP.
This combination checks all other String's categories. Like the previous combination, an input is readable if at least one tool says yes, reducing the risk of ignored readable inputs.

(3) Regex matching.
We use regex matching to check Number's Base Formats, such as "ˆ0[bB][01]+$" for BINARY.

(4) Rule + Regex matching.
This combination checks Number's Other Formats. Before leveraging regex matching, we add a parameter called BearableLength (this paper set it to 9). For this category, if an input's length is shorter than BearableLength, we regard it as readable. Otherwise, we use regex matching to check the inputs: the requirement for LONGNUMBER is that an underscore must appear after at most every three consecutive digits (e.g., 11_321_22_745); for FIXEDLENGTH, the requirement is that an underscore must exist and appear after a fixed number of consecutive digits (e.g., 1111_1111_1111_1111); for SCIENTIFIC, we checks whether the input contains an e or E. We assume the test code is grammatically correct, so as long as a number contains such symbols, it satisfies SCIENTIFIC.


§ EVOSUITE ENHANCED BY C3 (EVOSUITEC3)

The readability contexts mined by the Mining Component of C3 (Sec. <ref>) can not only used by C3's Judging Component (Sec. <ref>) to judge tests but also can help automated tools generate tests better in readability. In this paper, we propose EvoSuiteC3, i.e., EvoSuite <cit.> enhanced by C3. Fig. <ref> shows its overview. EvoSuite is a search-based test generation tool. It formulates coverage metrics (e.g., Branch Coverage and Weak Mutation <cit.>) as fitness functions and uses them to guide genetic algorithms <cit.> to evolve tests until all fitness functions reach optimum or the search budget is out. To enhance EvoSuite, we take different strategies for string and number contexts: For each string context, we provide a group of string seeds (e.g., "Beijing" and "London" for CITY) for EvoSuite. These seeds can be randomly selected by EvoSuite to be directly inserted into tests. Secondly, we provide extra fitness functions to guide EvoSuite's genetic algorithm so that it can generate tests with string values satisfying contexts. Unlike string contexts, number contexts focus on the numbers' formats instead of values. Hence, we modify the formats of numbers according to contexts after generating tests without any operations in the test generation process.


To demonstrate the fitness function for string contexts, we first introduce several auxiliary functions: getSeeds(context) gets prepared string seeds by a specific context. isSatisfy(value, context) checks whether a string value satisfies a specific context by invoking C3's Judging Component (Sec. <ref>). It returns 1 if the value satisfies the context; otherwise, it returns 0. Note that EvoSuiteC3 forces C3 to only use the regex matcher and CoreNLP without LLM for judging (see Sec. <ref>) so that EvoSuiteC3 does not waste EvoSuite's time budget due to network calls and avoids model costs. stringDistance(value1, value2) measures the distance (ranging from 0 to 1) between two string values. It returns 0 if they are identical. We use Jaro-Winkler Distance <cit.> to implement this function.

Based on these auxiliary functions, we define the readability fitness function f_c3(value, context) as follows:

    {[                           0                                        isSatisfy(value,;                                                                         context)=1,; minDistance(value, context)                                              otherwise,;                             ].

where minDistance(value, context) is a function defined as min{stringDistance(value, v)|v ∈ getSeeds(context)}.

Besides, a method may have multiple parameters with a context (e.g., the constructor in Code <ref>). To guide Evosuite in generating a method invocation satisfying multiple contexts, we define an extra fitness function f_c3invo(invocation) as:

    max{f_c3(v, c)| (v,c) ∈ getVCPairs(invocation) },

where getVCPairs returns an invocation's all value-context pairs. This function returns 0 only if all contexts are satisfied.

Based on the above definitions, EvoSuiteC3 extracts all f_c3 and f_c3invo functions for parameters with contexts and methods containing such parameters from the  class under test. It combines them with the existing fitness functions of EvoSuite to guide test generation.


§ EVALUATION


The evaluation mainly assesses C3's performance, i.e., false positive and negative rates, in mining readability contexts from real projects. Additionally, we aim to explore the differences between manual and automated tests under C3 measurement. Moreover, we investigate whether programmers agree with C3. The research questions are as follows:

∙RQ1:
What are the false positive and negative rates of C3?

∙RQ2:
What differentiates manual from automated tests in C3 measurement?

∙RQ3:
Whether programmers agree with C3?


 §.§ RQ1: C3's Performance


This RQ aims to answer what the false positive and negative rates of C3 are.



  §.§.§ Methodology

(1) Subjects.
Table <ref> presents the real projects used in this RQ. These projects come from two sources: (1) We select 254 classes from 100 projects within the DynaMOSA benchmark <cit.>; (2) To enrich the project diversity and obtain more classes with manual tests for subsequent RQs, we add 155 classes from seven open-source projects, namely, Airsonic <cit.>, BroadleafCommerce <cit.>, GoogleMapService <cit.>, OpenMRS <cit.>, OpenRefine <cit.>, PetClinic <cit.>, and Shopizer <cit.>. In summary, these classes contain 4485 primitive-type parameters.

(2) Procedure.
We employ GPT-4-Turbo <cit.> to run with C3's mining component (Sec. <ref>). Specifically, C3 uses GPT-4-Turbo to analyze every primitive-type parameter and further extract readability contexts. The default GPT-4-Turbo settings are used when invoking its APIs. Two parameters' concrete values of C3's mining component (see Sec. <ref>) are 4096 (MaxToken) and 20 (Remain). Although multiple executions of this procedure could mitigate the effects of the LLM's randomness, C3 only uses the result of one run due to model financial costs. Then, we conduct a manual review of C3's results. Following the previous study <cit.>, two authors independently examined each parameter, its associated  source code, and comments to check whether the readability context is correct (True Positive), wrong (False Positive), or missed (False Negative). They then compared the label results and found a few differences. In the end, they discussed and resolved the differences. Note that a context may be both False Positive and False Negative, as C3 might incorrectly assign a context to a parameter with a correct readability context. In such instances, they only label these cases as False Positive.

(3) Data analysis.
Firstly, we summarize the results as a Confusion Matrix <cit.>. Secondly, we calculate the Precision, Recall, and F1-Score <cit.>. Thirdly, we group C3's results by project categories to show the distribution of readability contexts. Finally, we compute the proportion of each context type.




  §.§.§ Results




 
(1) Confusion Matrix.
Table <ref> shows the True Positives (TPs), False Positives (FPs), True Negatives (TNs), and False Negatives (FNs).

(2) Metrics.

    Precision = TP/TP+FP=864/864+160=84.4%.


    Recall = TP/TP+FN=864/864+177=83%.


    F1 = 2 ×Precision × Recall/Precision + Recall=2 ×84.4%× 83%/84.4%+83%= 83.7%.

(3) Context distribution.
After manually checking C3's results, we confirmed that 1048 (23.4%) out of 4485 primitive-type parameters have readability contexts. Table <ref> presents the distribution of readability contexts across each group.

(4) Context proportion.
Fig. <ref> shows the proportions of 1048 parameters' readability contexts. Overall, string-type readability contexts constitute 57.44%, while number-type contexts represent 42.56%. The leading two context categories are Other Formats (37.35%) and Cyberspace (24.52%). The top six contexts are LONGNUMBER (35.88%), PERSON (12.88%), PATH (6.39%), LOCATION (6.11%), FILE (5.92%) and CARDINAL (4.96%).


  §.§.§ Discussion

Table <ref> shows that all project categories have a similar proportion of readability contexts (20% to 30%). Second, LONGNUMBER (35.88%) is the most common context. After a manual review, we find that GPT-4-Turbo tends to associate a parameter with LONGNUMBER if its name ends with “id” (e.g., “id” and “userId”) and concrete type is long. GPT-4-Turbo might do this because such parameters are typically used for large-value database keys. Meanwhile, such parameters are common in data-manipulating projects, leading to a high proportion of this context.

The statistical data shows C3's ability to correctly identify positive instances while minimizing false positive cases. There are still 177 FNs and 160 FPs. We classify the analyzed reasons as follows.



(1) LLM's randomness.
We find 154 (87%) FNs and 49 (31%) FPs whose projects contain many parameters with similar names, comments, and meanings that are accurately annotated with context, except for them. We assume this is due to LLM's randomness, which could be mitigated by calling LLM multiple times. We randomly select 30 cases as a sample. After calling LLM three times, 20's readability contexts can be correctly extracted at least one time. After calling LLM five times, the correct cases increase to 24.

(2) LLM's insufficient understanding.
We find that 28 (13%) FNs and 102 (64%) FPs are due to LLM's insufficient understanding. The first example is a class Playlist from Airsonic <cit.>, which stores music playlists. Its one parameter, name (the playlist's name), is wrongly recognized as PERSON. The second instance is a class Billing from Shopizer <cit.>. Its parameter, state that could be tagged as COUNTRY, is missed by GPT-4-Turbo. We assume that these cases are due to LLM's insufficient understanding, which could be solved by the LLM with better understanding in the future. On the other hand, our mine-context task can be used as a benchmark to compare different LLMs.

Another issue undermining C3 is that our contexts are incomplete. One example is a class Location from OpenMRS <cit.>. Its one string parameter, postalCode is not recognized since we do not define POSTCODE context (see Sec. <ref>). We do not count these instances as FN, as they fall outside C3's contexts and exceed its capabilities. To cope with this issue, we allow C3's users to integrate new contexts into C3 by providing their names, descriptions, and some examples for LLMs (see Sec. <ref>).
[title=Answer to RQ1,boxrule=1pt,boxsep=1pt,left=2pt,right=2pt,top=2pt,bottom=2pt]
In a sample of 4485 primitive-type parameters, C3 detects 1024 ones with readability contexts. After a manual review process, we confirm that its Precision, Recall, and F1-Score are , , and , respectively.






 §.§ RQ2: Readability of Manual and Automated Tests


This RQ aims to study the differences between manual and automated tests under C3 measurement.



  §.§.§ Methodology


(1) Subjects.
Table <ref> shows this RQ's subjects, i.e., 30 classes with manual tests whose false positive cases have been identified in RQ1 (see Sec. <ref>). We chose these classes because they have manual tests, and we aim to compare manual and automated tests in this RQ.

(2) Baselines.
We have five baselines: manual tests, EvoSuite <cit.>, Randoop <cit.>, ChatUniTest <cit.>, and EvoSuiteC3 (Sec. <ref>). We use EvoSuite and Randoop to represent two traditional test generation techniques, i.e., search-based software testing <cit.> and random software testing <cit.>. We use ChatUniTest to represent LLM-based test generation <cit.> since it is ready-to-use and can continuously guide LLM to generate syntactically correct tests. EvoSuiteC3 is EvoSuite enhanced by C3 (see Sec. <ref>).

(3) Test generation procedure.
For EvoSuite, Randoop, and EvoSuiteC3, we use their default settings to run on each class for producing tests. Due to their randomness, we repeat the procedure for 30 rounds, and each round's time budget is two minutes. Besides, we pass all contexts, including false positive ones mined by C3 (Table <ref>), to EvoSuiteC3. For ChatUniTest, we adopt the authors' settings <cit.>. For example, the test number for each method under test (MUT) is six, and the LLM is GPT-3.5-Turbo <cit.>. The only difference is that we use at most ten rounds of calling GPT-3.5-Turbo to generate and repair a test instead of six rounds to obtain more syntactically correct tests. Although GPT-3.5-Turbo also has randomness, we only execute the routine once due to financial costs.

(4) C3 measurement procedure.
After the test generation procedure, we employ C3's judging component (see Sec. <ref>) to evaluate all tests. We execute C3 with GPT-4-Turbo on both manual and automated tests produced by ChatUniTest. Likewise, we utilize GPT-4-Turbo for the first round of tests generated by EvoSuite, Randoop, and EvoSuiteC3. However, considering the substantial model financial costs brought by the tests of 30 rounds, we use GPT-3.5-Turbo to run with C3 for them.

(5) Data analysis.
We calculate two metrics for both manual and automated tests generated by ChatUniTest, as well as the first round of EvoSuite and Randoop tests: the number of covered parameters (i.e., the existence of a test that calls the MUT with an input) and parameters with readable inputs (i.e., the existence of a test that calls the MUT with a readable input). Additionally, we show the metric data by true positive contexts and all contexts (see Table <ref>), respectively, to show the effects of false positive cases from C3 on measuring test generation tools. Note that we only show string-type data for all contexts since there are no false positive cases in number-type contexts (see Table <ref>). Furthermore, we compute the average values of these two metrics across all rounds of tests generated by EvoSuite, Randoop, and EvoSuiteC3 to check whether the randomness affects their performance in generating readable inputs.



  §.§.§ Results


(1) String coverage and readability.
Table <ref> presents the performance of five baselines in covered parameters and parameters with readable inputs on 91 true positive string-type contexts. Overall, four automated tools surpass manual tests in parameter coverage (86% to 98% versus 48%). The readability rates (i.e., the ratio of parameters with readable input to covered parameters) indicate that EvoSuiteC3, ChatUniTest, and manual tests significantly outperform EvoSuite and Randoop in generating readable inputs (90%, 83%, and 68% versus 8% and 8%). Fig. <ref> summarizes test inputs of five baselines.

(2) Number coverage and readability.
Table <ref> also presents baselines' performance on 24 number-type contexts. Similarly, four test generation tools (ChatUniTest, EvoSuite, EvoSuiteC3, and Randoop) surpass manual tests in parameter coverage (88%, 71%, 71%, and 62% versus 50%). Their readability rates are close: 92% (manual tests) and 100% (four tools).

(3) Effects of false positive contexts.
Table <ref> shows the results of all string-type contexts (including false positive cases). The parameter coverage delta between all and true positive contexts only exists in Randoop (86% and 87%). The readability deltas range from 1% (EvoSuite and EvoSuiteC3) to 8% (ChatUniTest).

(4) Thirty rounds.
We run EvoSuite, Randoop, and EvoSuiteC3 30 times to eliminate the randomness. Table <ref> shows their average performance. In total, the parameter coverage deltas between one and 30 rounds range from 0% (EvoSuite's string) to 4% (EvoSuite's number). The readability deltas range from 0% (EvoSuiteC3's number) to 7% (Randoop's string).



  §.§.§ Discussion


(1) Parameter coverage.
The parameter coverage figures show that all test generation tools can generate higher code coverage than manual tests, confirming many studies' claims and findings <cit.>.

(2) String readability.
EvoSuiteC3 and ChatUniTest's tests surpass manual ones regarding string-input readability, while EvoSuite and Randoop almost fail to generate readable string-type inputs.

(3) Number readability.
All baselines achieve scores near 100% in number readability. This is primarily because all these contexts are LONGNUMBER, and C3 regards that a number shorter than 9 meets this context's readability (see Sec. <ref>). Meanwhile, such numbers are prevalent in all baseline tests.

(4) Effects of false positive contexts.
False positive contexts lead to underestimating the string-type readability performance of ChatUniTest and manual tests. The primary reason is that these two baselines perform well in string-type readability, thus making them sensitive to incorrect contexts. However, such contexts have tiny impact on EvoSuite and Randoop, as these tools fail to generate readable string inputs. Therefore, C3, even with false positive contexts, can be utilized to evaluate traditional tools. Besides, since we pass all contexts to EvoSuiteC3, its performance is nearly identical with or without false positive contexts.

(5) Thirty rounds.
The thirty rounds of data confirm that EvoSuite and Randoop stably lag far behind manual tests, ChatUniTest, and EvoSuiteC3 in generating readable string inputs.

[title=Answer to RQ2,boxrule=1pt,boxsep=1pt,left=2pt,right=2pt,top=2pt,bottom=2pt]
ChatUniTest and EvoSuiteC3's tests surpass manual ones regarding parameter coverage and C3 measurement, while EvoSuite and Randoop almost fail to generate readable string-type inputs.



 §.§ RQ3: Programmers' Readability Measurements



In this RQ, we aim to determine whether programmers agree with C3's measurement on tests' readability.



  §.§.§ Methodology


(1) Subjects.
From manual and automated tests used in Sec. <ref>, we randomly select ten pairs, each testing the same method and covering identical readability contexts. Each pair consists of a manual and an automated test. Of these ten pairs, C3 identifies manual tests of five pairs as more readable than automated ones (referred to as C3-Diff pairs). In comparison, the other five pairs are determined to have equivalent readability (C3-Equal pairs).

(2) Questionnaire. 
We arrange these ten pairs into a questionnaire, forming ten questions by applying the question framework depicted in Fig. <ref> to each pair of tests. We shuffle a pair's tests to ensure that a manual or an automated test is not always shown in the Test 1 or Test 2 of the framework.

(3) Participants.
Following the previous study <cit.>, we invite 30 participants to answer the questionnaire. Table <ref> shows their jobs and programming experiences.


(4) Data analysis.
After participants finish the questionnaire, firstly, we group the data by C3-Diff and C3-Equal, then decode and count their choices, which include manual is better, automated is better, and no difference. Secondly, for each group, we calculate the Cohen's Kappa coefficient <cit.> for every two participants to determine their agreement level. The Cohen's Kappa coefficient, ranging from -1 to 1, measures the agreement between two raters, where 1 means perfect agreement, 0.61 means substantial agreement, and 0 means random agreement <cit.>.



  §.§.§ Results




(1) Choice summary.
Table <ref> summarizes participants' choices. When C3 asserts the manual test is more readable, 88 out of 150 instances (58.7%) show agreement, while 47 instances (31.3%) display explicit disagreement. For the C3-Equal cases, explicit preferences are expressed in 52+88=140 instances (93.3%).

(2) Cohen's Kappa coefficients.
Fig. <ref> displays the levels of agreement among participants, grouped by C3-Diff and C3-Equal. The agreement levels for C3-Diff range from -0.563 to 1, whereas for C3-Equal, they range from -0.923 to 1. The median values for C3-Diff and C3-Equal are 0.063 and 0, respectively, while the mean values are 0.118 and 0.067. For C3-Diff, there are 37 instances reaching substantial agreement (values ≥ 0.61), compared to 23 such instances for C3-Equal.



  §.§.§ Discussion

We observe that for C3-Diff pairs, where C3 deems manual tests more readable than automated ones, 103 instances (68.7%) of participants' choices express at least no objection. Moreover, several metrics of Cohen's Kappa coefficients from C3-Diff pairs surpass those from C3-Equal pairs, including the minimum, median, and mean values and substantial agreement count. These findings suggest that when C3 distinguishes readable differences between two tests, programmers are more inclined to reach a consensus and concur with C3. However, 31.3% of choices still disagree with C3 in C3-Diff pairs. Concurrently, the choice summary and Cohen's Kappa coefficients of C3-Equal pairs indicate that programmers' preferences are not uniform. This is because apart from test inputs matching contexts, different programmers emphasize various other factors in readability, which are shown in the programmers' feedback.

Feedback from programmers.
After completing the questionnaire, we specifically asked participants about the importance of matching test inputs to contexts for readability and which other factors they emphasize. We categorize their feedback as follows:

(1) Programmers approve readability context matching.
Most programmers concur that matching test inputs to contexts enhances readability. Twelve participants expressed their dissatisfaction with random and meaningless inputs (generated by EvoSuite and Randoop). One participant stated that the situation deteriorates when a test calls a method with multiple random inputs, as the mapping from values to parameters becomes unidentifiable. The EvoSuite test of Code <ref> provides an example.

(2) Short and explicit tests are preferred.
Beyond input contexts, two hot keywords from programmers discussing test readability are shortness and explicitness. Shortness is a requirement for all test elements, such as variable names, statements, and literals. Explicitness implies that (1) a test should be self-contained, i.e., the overuse of methods other than the MUT is discouraged. This prevents readers from referring to different methods for understanding the test; (2) statements should be straightforward. For instance, pass-by-value is preferable to pass-by-variable when calling the MUT. Code <ref> exemplifies this: the manual test violates both principles (too long literals and calling two methods), whereas the automated one adheres to them. Hence, 21 out of 30 programmers preferred the latter, although the manual one satisfies the URL context. However, for the MUT in Code <ref> with two parameters requiring readability contexts, 22 out of 30 preferred the manual test to the EvoSuite test, indicating dominating factors vary per scenario.
Two object creation statementscode:s_e
// object creation of manual test
OEmbed oembed = twitter1.getOEmbed(new OEmbedRequest(240192632003911681L, "http://somesite.com/"));
// object creation of automated test
OEmbedRequest oEmbedRequest0 = new OEmbedRequest(1784L, "kP");

(3) Unreadable inputs are necessary.
Two participants mentioned that unreadable inputs are necessary when testing boundary conditions since many defects are triggered by unexpected inputs. This posed a challenge for C3 to measure tests since C3 cannot identify unreadable inputs that are on purpose. Still, our experimental setup avoided this issue: For a context, as long as one input meets the requirement, we assert that context is to be satisfied (see Sec. <ref>).

[title=Answer to RQ3,boxrule=1pt,boxsep=1pt,left=2pt,right=2pt,top=2pt,bottom=2pt]
When C3 identifies readable differences between tests, programmers tend to reach a consensus and agree with it, although C3 alone does not include all readability factors.
 




§ THREATS TO VALIDITY


Construct Validity.
In Sec. <ref>, we use NLP's named entities <cit.> to construct string type' contexts and number formats to construct number types' contexts, which brings a threat that such named entities are not complete to represent all contexts of real-world projects. We provide an example in Sec. <ref>. Firstly, We plan to extend C3 to mine contexts beyond prepared contexts in the source code automatically. Secondly, in practice, we allow C3's users to integrate new contexts into C3 by providing their names, descriptions, and some examples for LLMs.

Internal Validity.
The first threat to internal validity comes from two parameters (MaxToken and Remain) of C3's Mining Component (Sec. <ref>). We use MaxToken to represent the token size limit and Remain to keep the token space for LLM's response. With these two parameters, our prompt token size will not exceed the LLM's token limit. In our experiment, we used GPT-3.5/4-turbo, and their token limit was 4096 when we ran our experiment. C3' users need to reset MaxToken when their adopted LLM has another token limit. Remain is used to remain the token space for LLM's response. C3's expected response should be a small string, i.e., a context name (e.g., “EMAIL” and “PERSON”). Hence, we set Remain to a small number (20 in our experiment). However, C3's users also need to adjust this parameter's concrete value when using an LLM with a different string-tokenization algorithm from GPT-3.5/4 since a different tokenization algorithm may lead to a significantly different token size, even for a small string.  The second threat to internal validity comes from a parameter (BearableLength) of the C3's Judging Component (Sec. <ref>). This parameter is used to control the readable threshold for Number's three contexts (FIXEDLENGTH, LONGNUMBER, and SCIENTIFIC). For these three contexts, when the inputs are not too long (e.g., 12 and 12345), we regard them as readable. Hence, we add the bearable length to filter out such numbers when we apply our judger on the tests. Since many meaningful numbers (e.g., American phone numbers and credit card numbers) usually have at least ten digits, we set the bearable length at 9. The third threat is the randomness of the test generation tools (ChatUnitTest, EvoSuiteC3, EvoSuite, and Randoop) when comparing their test input readability in Sec. <ref>. To reduce this risk, We repeated running EvoSuiteC3, EvoSuite, and Randoop 30 times on the same  class. However, we only ran ChatUnitTest once because it leveraged GPT-3.5-turbo to generate tests and could bring many financial costs if we repeated running it 30 times. The last threat is due to potential bugs introduced in implementing C3 and EvoSuiteC3. For C3, we manually checked its extracted contexts and confirmed that its Precision, Recall, and F1-Score are , , and , respectively (see Sec. <ref>). For EvoSuiteC3, we extracted the inputs from its generated tests and manually checked whether these inputs match the contexts. The result confirmed EvoSuite's readability increase brought by EvoSuiteC3. Fig. <ref> demonstrates the comparison between EvoSuiteC3 and EvoSuite.

External Validity.
Experimental subjects form a threat. Many classes from the DynaMOSA benchmark <cit.> are obsolete <cit.>. Hence, we complement it by adding 155 classes from seven open-source projects (see Sec. <ref>). The choice of LLMs in our evaluation forms another threat since C3's performance depends on LLMs. Moreover, we implemented C3 in  and evaluated it using  subjects. Hence, our experimental findings are not verified in other programming languages. Nonetheless, the fundamental concepts of C3 are not confined to .


§ CONCLUSION

This paper proposes the Context Consistency Criterion (C3), an automatic readability measurement tool for primitive-type test input. C3 can mine the readability contexts for primitive-type parameters in the code under test and use these contexts to judge tests' consistency as a readability metric. Besides, this paper proposes EvoSuiteC3 (i.e., EvoSuite enhanced by C3). Our experimental results show the distribution of readability contexts and confirm C3's accuracy. Furthermore, while LLM-based test generation and EvoSuiteC3 surpass manual tests under C3 measurement, two traditional tools (Randoop and the original EvoSuite) almost fail to generate readable string-type inputs.



§ DATA AVAILABILITY

C3's implementation and the experimental data are at <https://doi.org/10.5281/zenodo.13028768>.

IEEEtran







 Paper Title: Chat-like Asserts Prediction with the Support of Large Language Model.




















































§ ABSTRACT 
Unit testing is an essential component of software testing, with the assert statements playing an important role in determining whether the tested function operates as expected. Although research has explored automated test case generation, generating meaningful assert statements remains an ongoing challenge. While several studies have investigated assert statement generation in Java, limited work addresses this task in popular dynamically-typed programming languages like Python. In this paper, we introduce Chat-like execution-based Asserts Prediction (), a novel Large Language Model-based approach for generating meaningful assert statements for Python projects. utilizes the persona, Chain-of-Thought, and one-shot learning techniques in the prompt design, and conducts rounds of communication with LLM and Python interpreter to generate meaningful assert statements. We also present a Python assert statement dataset mined from GitHub. Our evaluation demonstrates that achieves 64.7% accuracy for single assert statement generation and 62% for overall assert statement generation, outperforming the existing approaches. We also analyze the mismatched assert statements, which may still share the same functionality and discuss the potential help could offer to the automated Python unit test generation. The findings indicate that has the potential to benefit the SE community through more practical usage scenarios.





<ccs2012>
   <concept>
       <concept_id>10011007.10011074.10011099.10011102.10011103</concept_id>
       <concept_desc>Software and its engineering Software testing and debugging</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Software and its engineering Software testing and debugging






Chat-like Asserts Prediction with the Support of Large Language Model
    
Kanika Dheman1,
Marco Giordano1,
Cyriac Thomas1,
Philipp Schilk1,
Michele Magno1


1{dhemank, mgiordano, cythomas, schilkp, magnom}@ethz.ch
1Center for Project Based Learning, Dept. of Information Technology and Electrical Engineering, ETH Zürich, Switzerland

    August 5, 2024
=========================================================================================================================================================================================================================================================================








§ INTRODUCTION



Unit testing plays an important role in software testing, as it can detect and diagnose bugs in the existing system by testing small units at the early stage <cit.>. A unit test case normally consists of test inputs and test oracles (assert statements) <cit.>. The assertions are crucial in the unit testing process, as they compare the actual and expected outputs to determine whether a tested function performs as intended.  High-quality assert statements enhance the maintainability of unit tests, help better identify bugs, and assist future developers in understanding the purpose and output of the code. In addition, if a test case fails, the assert statements serve as a valuable reference for identifying and resolving faults. 


Developing unit tests for a project can be a demanding and time-consuming task for developers <cit.>. In response to this challenge, automated unit test generation has emerged as a well-established field of research. Studies have concentrated on automating unit test generation using rule-based or search-based algorithms for both Java and Python projects <cit.>. 
However, while search-based testing approaches can facilitate the automation of test input generation and improve test coverage, generating complete and meaningful assert statements that meet the actual needs of developers remains a challenge <cit.>. To be specific, these approaches either capture and assert the return values of non-void-return methods or introduce mutants into the software and attempt to generate assert statements able to kill these mutants. 
Interviews <cit.> with developers have revealed that the automatically generated assertions are often poor and simple, resulting in their inability to detect real bugs, and the lack of meaningfulness in these assertions leads developers to prefer handwritten test cases over the generated ones, ultimately undermining the purpose of automated testing <cit.>.

Therefore, recent works have started to explore automatically generating meaningful assert statements in Java given the input of the test input and its focal method (method to be tested) <cit.>. 
However, for dynamically typed programming languages such as Python, though it has become the most popular programming language <cit.>, no previous research has investigated the generation of meaningful assert statements. There is a demand for more advanced and effective tools to assist developers in this area and enhance the overall code quality. Unlike Java, Python is dynamically typed, features flexible syntax, and employs high-level abstractions, making it more challenging to accurately capture the expected behaviour or values of the focal method. 
Directly adopting current approaches to assert statement generation in Python has limitations. First, these deep learning (DL) or information retrieval (IR) based methods rely on a training set, and performance varies with the set's size and relevance. Unfortunately, Python lacks a standardised dataset specifically for assert generation, complicating the adoption of these models. Python's diverse testing practices and dynamic typing system, is challenging for automated test identification and assert extraction, further compounded by a less mature testing tools ecosystem that limits the availability of training data for such methods.
Second, we found that 71.8% of the test cases we crawled from open-source Python projects contain more than one assert statement in a test case, which is not supported by current approaches designed only for a single assert statement. Generating multiple assertions using the current single assert statement can be challenging as identified in previous work <cit.>. These methods typically treat each assertion independently, but fail to consider the connections/dependencies of the asserts in the same test cases. These limitations highlight the need for new approaches to generate multiple meaningful assert statements for Python unit testing.


In recent years, Large Language Models (LLM), such as GPT <cit.> and LLAMA <cit.>, have emerged as pre-trained neural network models capable of achieving impressive results in various tasks 
such as code completion <cit.>, documentation and comments generation <cit.>, repair programs and fix bugs <cit.>. 
Leveraging LLMs' pre-trained nature and their proficiency in understanding code context, we use them to automate the assert generation in Python.
Notably, GPT models perform well at Python code generation compared to other languages, likely due to their pre-release training set distribution <cit.>.


In this paper, we propose (Chat-like LLM-based Asserts Prediction), a novel LLM-based approach for generating meaningful assert statements for Python projects. We adopt state-of-art prompt design techniques, such as one-shot learning and Chain-of-Thought (CoT), into the assert generation prompts. To include more context of the given code, we design three prompt templates to communicate with the LLM as well as support the multi asserts generation given one test case. We execute the initial round of predictions and relay any error messages received from the interpreter back to the LLM, prompting it to correct the mistakes and refine its predictions.
We also mine GitHub and release an open-source Python assert statement dataset. Lastly, we implement the approach with OpenAI LLMs and evaluate it with the dataset. 

We evaluate the effectiveness of our prompt design and compare performance with state-of-the-art approaches <cit.>. While the prompt design demonstrates strong performance with the two OpenAI LLMs, it achieves better results on the text-davinci-003 <cit.>. The prompt techniques we integrate into can help the LLM generate more accurate assertions. In our evaluation, attains 64.7% accuracy for single assert statement generation,  and achieves 62% accuracy in generating all assert statements that no other approaches have explored. We collect the generated assertions that can improve the test code quality or fit more to the test purpose and submit pull requests to the tested projects, and 7 out of 9 pull requests has been approved.

In summary, the contribution of this work is:

    
  * The study specifically focuses on assert generation in Python, encompassing multi-assert generation and the utilization of various testing libraries. 
    
  * , a system that integrates rounds of CoT and feedback prompts to interact with the LLM, effectively addressing the assert generation task.
    
  * We make both our tool, , and our Python assert dataset publicly available for the benefit of the research community <cit.>.



§ BACKGROUND





 §.§ Unit Testing & Assert Statements


A unit test is a small and executable piece of code that exercises the functionality of a unit under test <cit.>. Unit testing plays an important role in test-driven development (TDD) as it can better help with fault detection <cit.>, contribute to developer's productivity <cit.>, and can also serve as documentation and specification <cit.>. The existence of unit tests ensures the correctness, reliability, and maintainability of the units within the code <cit.>. 

To implement unit tests, developers use specific conventions and frameworks that define how test methods are structured and recognized within different programming environments. 
In Java projects, test methods start with @Test annotation (inherent to the JUnit framework) <cit.>. However, in Python, projects do not have a universal annotation for indicating test methods. Instead, they often rely on naming conventions and specific testing frameworks such as unittest <cit.>. In Python, test methods are generally prefixed with test <cit.> and contained modules following the pattern test_module_name.py or module_name_test.py.

Test oracles are important parts of the test methods, normally appearing as assert statements. They validate whether the given conditions are met during the execution. 
In Python, there are two common ways to write assert statements. The first method involves utilising the assert keywords provided by Python, which allows for limited but straightforward checks of conditions within the code. The second approach entails leveraging other testing libraries, such as unittest <cit.>, scikit-learn <cit.> and tf.test <cit.>, which offer over 70 specialized assert statements and additional features for a wide range of testing scenarios. For example, the assertRaisesRegex checks if a specific exception is raised during the execution and if the exception message matches the regular expression pattern. Therefore, the assert statements in Python vary depending on the specific requirements and preferences of developers.




 §.§ Automatic Assert Generation


Automatic assert generation is a critical challenge in software testing. It aims to automate the creation of assert statements within unit tests. The challenge in automatic assert generation lies in effectively predicting assert statements that accurately reflect the intended functionality of the application.

The pioneering work in this area was introduced by Watson et al. <cit.> in 2020. They developed a Deep Learning (DL) approach using a Neural Machine Translation (NMT) model, named ATLAS, to generate assert statements specifically for Java. 
They proposed the Test-Assert Pairs(TAPs) dataset extracted from Java unit test method. This dataset paired unit tests with their corresponding focal methods, providing a structured way to train and evaluate the model. Since then, all works have followed this structure, generating assert statements based on the given rest of the unit test and its focal method.
Subsequent developments <cit.> have enhanced the assert generation in Java with mixed methods, including information retrieval and the tuning of transformer and large language models. Beyond Java, developments in other languages like JavaScript have also progressed; for instance, Zamprogno et al. <cit.> introduced AutoAssert, which incorporates a human-in-the-loop mechanism. The advancements highlight the increasing interest in automatic assert generation, aimed at reducing manual efforts.





 §.§ Large Language Model & Prompt Design


Large Language Models (LLMs) like GPT, with over 175 billion parameters, excel in diverse tasks such as summarizing and code completion. LLMs are Trained on extensive datasets, including various open-source projects, and built on the Transformer architecture <cit.>. They utilise mechanisms like self-attention to comprehend prompts and capture contextual relationships between words from different distances.
To control the output of LLMs, prompts are fed into the models. Prompt design influences the LLM's ability to capture the requirements and produce specific and coherent results. The concept of prompt engineering <cit.> has been proposed to describe the iterative process of crafting and refining prompts to optimize the performance of LLMs in generating desired outputs. The process may include incorporating natural language descriptions, such as “The aim of this task is to xxx”, or give the LLMs one or a few samples (one-shot or few-shots learning) to better understand the task and guide the model in formatting the output content.


 

§ METHODOLOGY

 




In this section, we introduce , a system that engages the LLM as a role of an experienced unit test expert, and constructs prompts to initiate multiple conversation with the LLM to automatically generate the assert statements based on a given Python unit test case. Fig <ref> illustrates the system overview of . 
Given a Python project, analyses the project and creates data entries that include the focal methods, test methods, and corresponding assert statements. 
Then, starts to communicate with the LLM, which involves three phases. First, we create a greeting prompt to define the task and provide one sample test case, offering LLM additional context (<ref>). In the second phase, we incorporate the target test case into the query and wait for the initial round of assert predictions (<ref>). In the third phase, we execute the predicted assert statements using a Python interpreter and, if any error messages arise, input these messages back into the LLM to request corrections to its predictions (<ref>). By interacting with the LLM, collects the automatically generated assert statements and completes the unit test case.
 


Code Sampled from: <https://github.com/scrapy/scrapy>




 §.§ Project Analysis

 
Our goal is to generate assert statements in Python projects. As of March 2023, Python has been acknowledged as the most popular programming language by the TIOBE Programming Community Index <cit.>; however, previous works have not focused on the assert generation area in Python. Therefore, begins by analysing the projects and preparing the data entries for the prompts construction. In Fig <ref>, we provide an example of how a data entry extracted from a project is presented. A data entry consists of three main parts, the focal method, unit test, and assert statements. 

Within a given project, initially identifies all unit test files. This identification is based on the naming conventions of Python unit tests, as discussed in Section <ref>, checking for specific patterns in the file names. Upon locating the files, then examines their contents to verify the use of testing libraries and the completeness of the tests.


Next, starts to identify the focal method of the giving test method. Following the focal method extraction for Java projects <cit.>, we identify the last method call from the same library or project before the assert statements as the focal method. In some cases, the method calls are embedded inside the assert statements as parameters, for example, the process_spider_input in Fig <ref>. This approach might be considered controversial, as the parameter to be predicted is already present in the prompts. However, in real-world scenarios, developers typically have a clear understanding of the focal method. Consequently, we believe that incorporating the focal method information in the prompts closely aligns with developers' actual workflow and context, providing a more realistic and practical representation of the problem at hand. retained the original comments within both the focal methods and the unit tests, as these help the language model understand the functionality of the functions. Global variables defined in the same file were also included.

Finally, extracts the assert statements from the unit test and use ⟨ AssertPlaceholder⟩ to replace the assert statements within the unit test code. To differentiate assert statements in the same test case, assigns incremental numbers to each of the statements. 
The detailed implementation is discussed in Section <ref>. 








 

Code sampled from: <https://github.com/lk-geimfari/mimesis>



 §.§ Greeting Prompt


The greeting prompt is the first prompt we sent to the LLM. As shown in Fig <ref>, it includes a persona message, a sample code for one-shot learning, a Chain of Thoughts (CoT) prompting, and the corresponding assert statement as the output.

Persona-based Prompting. The persona-based prompting approach assigns a persona or role related to the task given to the LLM <cit.>. By adopting a persona or role, the LLM can align its responses with the expertise, knowledge, and context associated with the given persona. This approach provides a clear framework and perspective for the LLM's response generation, ensuring that the output is more relevant and accurate for a specific task. In the official OpenAI documents, they also ask users to have a system message to help set the behaviour of the assistant <cit.>. 
In this study, we instruct the LLM to assume the role of a unit testing expert to assist in completing the unfinished test method by generating assert statements. The first sentence introduces the LLM to its assigned role. Then, inspired by a popular open-source repository that features a collection of valuable prompt examples <cit.>, we include a task description paragraph to the prompt. The description details the input, steps to generate the assert statements, whether or not to use a certain testing framework, and the expected output, effectively guiding the LLM in producing the desired results.

One-shot Learning.
The LLM, such as GPT-3, can achieve state-of-the-art performance on a wide variety of tasks using only few-shot prompts, in which examples of solved tasks (shots) are included in the prompt. Our approach adopts the one-shot learning prompt to generate assert statements for Python projects, enabling the LLM to learn from a single example and adapt its response generation accordingly. This method considers Python's unique language characteristics and assumes that the writing style remains consistent across the same project. 
Therefore, while retrieving this one sample, we only look for the test method that is within the same test class of the target test method as the sample. 
If there is only one test method in a class, we use the other test method within the same project as the sample input.
When more than two test methods are present in a class, we compute the cosine similarity of the method names and choose the one with the highest score.  After locating the sample code, we parse it into a format suitable for constructing the Greeting Prompt, as illustrated in Fig <ref>, and generate the corresponding CoT process. 
We carried out a small-scale experiment showing that the performance of one-shot and few-shots prompting (6 shots) in our task does not show significant differences. In fact, one-shot prompting achieved higher accuracy (69.8% over 58.1%), likely due to less disruption in the prompts. Given the comparable performance and lower cost, we adopted one-shot prompting in .

Chain-of-thought Prompting. Recent research <cit.> in prompt design has demonstrated that for complex tasks requiring multi-step reasoning, incorporating a rationale or key phrase can significantly enhance the performance of LLMs. The approach, called Chain-of-thought (CoT) prompting, is effective in guiding LLMs toward desired outputs. 
In our work, we implement CoT prompting strategies by incorporating a guiding phrase followed by rational steps for generating assert statements. This approach aims to leverage the advantages of both strategies to enhance the LLM's ability to produce accurate results. To develop the manual instruction steps, we investigated the process of writing unit tests and assert statements <cit.>. Additionally, we posed this question (i.e., What are the steps to write assertions for an incomplete unit test when provided with its focal method?) to the LLM model and combined the results to finalize the steps. The first step is to identify the test method and the focal method. Next, developers need to determine the parameters being tested and the assert type, e.g., asserting the parameter value, type/instance, or structural information. The final step involves reasoning the assert value and generating the corresponding statement.



 §.§ Query Prompt


Following the submission of the greeting prompt, we introduce the actual query prompt containing the target test cases. The prompt follows a structure similar to the sample code in the greeting prompt but remains incomplete. We also include a brief description in the CoT section to remind the LLM model of the task, generating assertions to replace the assert placeholders, and request to start the CoT process from the first step.
The LLM then responds to and produces the first round of predicted assert statements.







 §.§ Feedback Interaction and Prompt
 

Despite the LLM's ability to generate accurate assert statements, particularly with the CoT prompting which enables it to understand the testing purpose (derived from the test method and focal method names) and locate the parameter to be tested, there are instances where the LLM can not generate the correct value for the target parameters. For example, in situations where the target parameter value relies on complex math calculations (e.g., deep learning models) or is influenced by external factors not immediately evident from the test method or focal method (e.g., parameters defined outside the files or obtained through multiple function calls, self-defined util functions), the LLM may fail to predict the correct value. 


To overcome the limitations, we introduce a feedback interaction that allows the LLMs to learn from execution errors in value generation, inspired by prior research <cit.>. As shown in Fig <ref>, this process involves executing the initially generated assert statements using the Pytest library <cit.>.
If the assert passed, the LLM has identified the correct value for the tested parameter. In cases where execution fails, utilises regular expressions to automatically extract key information about the expected and actual values from the error message. then incorporates the error message into the feedback prompt, ensuring that the LLM is aware of its previous mistake and can use this information to adjust its response generation accordingly for the specific assert placeholder. Fig <ref> shows a sample piece of the feedback prompt. Through this feedback interaction, the LLM can progressively generate precise values for target parameters. After all interactions, collects the second-round predictions and saves them for evaluation.













 


§ EVALUATION


To verify the performance of , we conduct experiments to answer the following research questions:

    
  * RQ1: How is the performance of on LLMs?
    
  * RQ2: How do different prompt designs influence prediction performance?
    
  * RQ3: How does the perform in assert generation compare with the state-of-the-art approaches?
        
  * RQ4: What are the cases generated by that failed to accurately match the original test cases?
        








 §.§ Dataset




In our research, we assembled the first dataset for Python assert generation, by examining datasets from the recent popular Python research tools like Pynguin <cit.>, BugsInPy <cit.>, and CODAMOSA <cit.>. Considering that CODAMOSA utilizes data from Pynguin and BugsInPy, we focused only on these two sources. To avoid the data leakage i.e., testing data being used for model training, we only included project files that were newly added after July 2021, which corresponds to the training data cut-off date for the GPT models <cit.>. 
After the removal, we identified 9 projects from the earlier works, with 7 of them using Python assert keyword and 2 employing testing libraries. To increase variety and robustness, we employed the crawling process of ATLAS <cit.>, the first research work in the automatic assert generation field, identifying popular Python projects (stars and forks greater than 30) that also used testing libraries by Github APIs <cit.>. Combining the 5 most recently updated projects with the initial 9, we created a balanced dataset of 14 projects. These projects are equally distributed between those using the built-in assert keyword and those utilizing testing libraries such as unittest and numpy.testing.








Given a project, identifies and extracts unit test suites, we wrote automated scripts that could parse the selected repositories, traverse the directory structure, and pinpoint the test files according to the patterns we discussed in Section <ref>. Then, we excluded test methods created before July 2021 to match with the cut-off date. Following the approach used by CODAMOSA, we employed pipreqs <cit.> to identify project dependencies and configured the testing environment accordingly. We down-sampled some methods that necessitated additional setup procedures. In total, the dataset consists of 3,696 test methods and 1,298 focal methods (2.85 test methods per focal methods on average). Table <ref> provides a detailed overview of the projects included in our dataset. 

^∗ denotes the project uses testing libraries. # Test Method is the number of test methods within the project. Src is the project source, where Pyn is for Pynghuin, BIP is for BugsInPy, and is from this study.

To match the focal method with the test case, traverses through the Abstract Syntax Tree (AST) <cit.> of the unit test code using the NodeVisitor class, and identifies the last method call from the same project before or within the assert statements.
In Section <ref>, we discussed two prevalent approaches for writing assert statements. When using the build-in assert, the assert statements consistently begin with the assert keyword, while when employing popular testing libraries, the unit test methods generally inherit the test class provided by the libraries, and their assert statements start with self.assertFunctionName. In our extraction script, we combine the use of AST and regular expressions to effectively extract assert statements from the original source code. We executed the scripts across all projects and constructed the dataset for evaluation. The dataset shall also benefit relevant studies within the community.


In our testing, the LLM is capable of generating assertions corresponding to the number of placeholders provided. But in the evaluation, we aligned the number of assert placeholders with the original test cases, maintaining a one-to-one correspondence. This process is reflective of typical developer practices when crafting test methods, where developers often write assertions sequentially, having prior knowledge of the required quantity and understanding when the test case is complete <cit.>. By limiting the number of assert placeholders, we aimed to assess whether could generate assertions efficiently.






 §.§ Evaluation Metrics

We use the following metrics to evaluate the generated assert statements by the LLM.

Accurate Match (AM %). The Accurate Match metric measures the functional equivalence of the generated asserts to the original asserts. In addition to exact matches, we extracted assert statement types from Python testing libraries and identified 34 groups of assert pairs that, while not identical, perform the same function. This concept is inspired by suboptimal test smells <cit.>. Fig <ref> provides some examples of these equivalent assert pairs.
For example, in the first pair of assertions, they are functionally same, but the predicted one is preferred as it conveys the intention more explicitly, making the code more readable and maintaining consistency with standard testing practices.




LCS (%).  We measure the longest common subsequence between the predicted and original assert, calculating the percentage as in Nashid et al. <cit.>, by dividing the subsequence length by the original assert length.

Edit Distance (ED). The edit distance is a metric to measure the similarity between two strings by calculating the minimum number of single-character edits (e.g., insertions, deletions, or substitutions) required to transform one into the other. The metric has been widely used to evaluate code-related tasks <cit.>.

Assert Method Match (AMM %). The AMM is a metric to verify whether the predicted assert statement type aligns with the original one. For example, if CLAP predicts an assert statement as assertIsInstance(a, List), but the actual assert statement is assertIsInstance(a, pd.Array), this is still classified as a match under the AMM metric as they both want to check the instance of parameter a. The calculation of AMM is represented by the following formula:


    AMM = 1/N∑_i=1^N 𝕀( AssertMethodPredict_i= AssertMethodOriginal_i)
where 𝕀
(AssertMethodPredict_i= AssertMethodOriginal_i) is an indicator. When the predicted assert method is equal to the original assert method, 𝕀=1, otherwise, 0.

We use this metric because, the assert statements in Python come in various types, e.g., assertIn, assertIsNone, assertAlmostEqual, and assertGreaterEqual. The AMM metric quantifies the precision of CLAP in predicting the appropriate assert method, thereby demonstrating its ability in predicting relevant assert predictions based on the current context. Note that the metric is specifically used to evaluate projects employing testing libraries, as the assert keyword does not distinguish between different assert methods.







 §.§ Performance of (RQ1)


Experiment setup. In this section, we evaluate the prompt design across two LLM models from OpenAI <cit.>: text-davinci-003(text-davinci) and gpt-3.5-turbo. Inspired by previous works <cit.> and a small-scale pilot study, we set the temperature parameter to 0.3, striking a balance between allowing some degree of creativity in the model's responses while ensuring that the predictions remain well-defined. We understand the code-davinci is trained on more code repositories, but it was deprecated <cit.>, we have discussed this threat in Section <ref>. All requests were executed through Python scripts utilizing the official OpenAI Python APIs. 




Result. Table <ref> shows the results of the assert generation tasks for the two LLMs utilized by . We can observe that for the Python assert keyword, text-davinci outperforms gpt-3.5-turbo in both single and multiple assert generation scenarios, achieving 64.6% and 51% accurate match respectively. The prediction on the single assert from text-davinci has nearly 10% more accurate assert than the gpt-3.5-turbo model. For testing libraries, text-davinci till achieves better performance in most metrics. Conversely, gpt-3.5-turbo exhibits a lower ED for single assert generation.
Overall, we found the text-davinci-003 model generally outperforms the gpt-3.5-turbo model in generating accurate assert statements. 
Although gpt-3.5-turbo is designed for chat-like interactions, which aligns with our prompt design, it appears to be more cautious and hesitant to avoid making errors. For instance, in some cases, it fails to predict an assert statement because it seeks additional information about method calls beyond the focal methods, particularly for complex test input and multi-assert generation. In contrast, text-davinci seems capable of learning about other method calls based on their names and parameters, enabling more accurate predictions. Furthermore, the turbo architecture of gpt-3.5-turbo may impact its understanding of longer assertions, resulting in reduced accuracy.

To explore more details, we further analysed the text-davinci results at the project level. Table <ref> displays the statistics across various projects. Dataclass-json achieves a 100% AM, because it only has four assert methods. Detectron2 has the lowest AM overall, possibly due to its complex DL model structure. 
Sanic and Scrapy also show lower AMs, possibly because their web-related nature demands specific testing environments and assert parameters. For 11 out of 14 projects, the AM decreases from the single assert generation to all assert generation. However, Scrapy, Keras, and Mkdocs show an increased AM for all test methods, suggesting higher accuracy in predicting multi-assertions. This can be attributed to these three projects having some test cases with similar structural assert statements, therefore increasing the AM overall. 










 §.§ Effectiveness of the Prompt Design (RQ2)



Experiment setup.
We assess the effectiveness of the prompt design with various input configurations. Existing research has confirmed that providing examples to the LLM can improve accuracy for different tasks <cit.>. Therefore, we evaluate the other three features within the prompt design, which include persona, CoT, and feedback prompting. In the experiment, we remove the portions of the prompt that serve these features respectively and maintain all other settings. We then collect the results from different configurations and evaluate their performance.

Result.
Table <ref> presents the results of the prompt design evaluation.  Removing persona design leads to decreased performance, revealing the importance of persona prompts in generating accurate assert statements. The situation is different for the CoT prompt design validation. For single assert statements, outperforms without CoT for both the Python assert keyword and testing libraries. However, multi-assert generation shows mixed results. For Python assert keyword tests,  performs better; for Python testing libraries, 's prompt achieves higher AM and LCS.
Upon examining the multi-assert generation cases responsible for this difference, we identified the following reasons: 1) Some multi-assertion test cases test multiple parameters within a single unit test, which, although common, is considered poor practice for code quality <cit.>. For instance, a test case might use 8 assertions to test 3 parameters. With the CoT prompt design, the LLM might mistakenly use extra assertions to test the type of parameter A rather than the value of parameter B, ultimately neglecting to test parameter B and resulting in lower AM and AMM. 2) Some test cases are too long and come close to the token limitation of the GPT models, leading the LLM to generate assert statements that might not conform to the desired format. This makes it challenging for to identify the predictions. Nevertheless, the average AM of remains higher than the design without CoT. Future work should focus on improving the prompt design for multi-assert statements to address the issues mentioned above.


In general, we observe that the most significant difference across the four metrics lies between  and , indicating that the feedback interaction design has a substantial impact on performance. 
Various scenarios may cause the LLM to struggle in predicting the correct value, such as global parameters or functions defined outside the focal method, asserts requiring mathematical calculations, or focal methods with dependencies on other functions. Feedback interaction can assist the LLM in learning from error messages and correcting incorrect predictions. To gain deeper insights into which assert types benefit the most, we provide a breakdown of AM statistics for different assert types in Table <ref>. The assert types Is and IsInstance achieve the highest accuracy, possibly due to the complexity of predicting whether objects are the same and determining the object's type is low. The feedback interaction contributes to the most significant accuracy increase for Equal and True types, which is because the feedback interaction gives the LLM a better understanding to the expected values. For example, if the assertEqual predicted value is incorrect in the first round, the LLM can comprehend the correct value based on the feedback prompt and adjust its prediction accordingly.

Overall, all three prompt components contribute positively to assert generation, with feedback interaction having the most significant impact on performance, while the CoT prompt design needs to be improved for the multi-assert generation.






 §.§ Comparison with State-of-the-Art Approaches (RQ3)


Baseline setup.
We evaluate the performance of by comparing it to other state-of-the-art assert generation approaches. ATLAS <cit.> is the first research work in this field and has demonstrated strong results by adopting a learning-based method. 
We also fine-tune CODET5 <cit.>, a pre-trained model that is capable of various code-related tasks such as code completion, employing it as one of the baseline models in our study.
CEDAR <cit.> is another approach that leverages the capabilities of LLMs in combination with information retrieval techniques. Since all prior approaches have exclusively focused on single assert generation, we also evaluate them with the single assert dataset. For ATLAS and CodeT5, we followed their paper instructions to set up the models and validated the result with five-fold cross validation <cit.>. For CEDAR, we strictly followed their codebase to use the BM25 <cit.> for the information retrieval and the rest of their code to do the data preprocessing and communicate with OpenAI API. 
To minimize any potential bias that could be introduced through the IR approach, we adopted the concept of Leave-One-Out <cit.>, which is known to reduce bias while optimizing prediction results in comparison to using a single test set. Specifically, when predicting an assert statement for a given test case, we employ the remaining test cases as the training set for IR, subsequently calculating the evaluation metrics across the entire dataset. Note that since the CODEX model was deprecated, we use the text-davinci-003 as the LLM to predict. We acknowledge there are other approaches in assertion generations <cit.>. However, they have some specific settings to Java which presents a challenge to the replication of their method in this study.



Result. Table <ref> presents a performance comparison of with ATLAS, CodeT5, and CEDAR. In all metrics, outperforms the other approaches. Both and CEDAR employ LLM to generate assert statements, leading to improvements in AM. Nevertheless, the difference in AMM is relatively small, suggesting that learning-based approach can identify the correct type of assert but fail to accurately capture the specific values or parameters within the assert statement. For example, given a statement self.assertEqual(out, reqs), the output from ATLAS is self.assert Equal(self, data), in which the two parameters within the assert statement do not match any elements from the original one. Sometimes the prediction from ATLAS does not follow the Python syntax, as the learning-based approach relies on patterns observed during training, which may not always adhere to the language's syntax rules. 
CodeT5 has a similar AMM when compared with CEDAR and , but falls short in other metrics, particularly ED. We observed instances where CodeT5 failed to terminate its output, resulting in the generation of extraneous and irrelevant text.

CEDAR achieves 54.7% in AM, which demonstrates its capability to generate meaningful assert statements. However, when compared to , there is still room for improvement. 
The use of IR includes more shots into the prompt but compared with , some shots may be inconsistent with the context, affecting the LLM's prediction.  For instance, for less common assert types like assertIsInstance, can predict the correct assertions, while CEDAR might be influenced by other samples and predict the value using assertEqual or assertTrue, which are more widely-used assert types.
 Besides, the incorporation of CoT and feedback in the prompt design enables to better predict assert statement values accurately, especially for the complex value structure. For example, given the assert statement self.assertEqual(jsi.call_function('x'), [20, 20, 30, 40, 50]), can identify the correct value with CoT's assistance even before entering the feedback interaction. 

 Fig <ref> shows the distribution of correct assertion lengths for the three approaches. It can be observed that, for shorter assertions, ranging up to 16, is capable of generating better predictions in the majority of cases. This can be attributed to the prompt design that assists in comprehending the test input and enables adjustments to the predicted value if it is initially incorrect. For longer assertions, the performance fluctuates, but still can predict the correct result for long assertions.
 Nonetheless, demonstrates improvements in performance and generates more accurate assert statements with less context. 


We conducted further examination of the AM assertions generated by , specifically focusing on cases that are AM but not exact matches. Among the randomly selected 50 test cases, 10% of them are due to the assert parameters are inverted, whereas the rest 90% them are due to readability differences. One prevalent pattern in such cases involves substituting assertTrue and assertFalse with more descriptive assertions like assertEqual, assertLen, assertIsInstance, and assertNone (illustrated in Fig <ref>). Though functionally identical, the predicted assertions are preferred as they articulate the intention more explicitly, enhancing code readability and aligning with established testing practices, thereby improving the overall quality of the test code. Another pattern pertains to assertRaises assertions, where readability can either improve or decline depending on the complexity of the code or developer preference. For instance, in the final example of Fig <ref>, the predicted assertion is a concise one-line explanation, in contrast to the more space-consuming original code. However, as the lines in the raise block expand, the one-line format commonly produced by may become more challenging to interpret. The findings highlight has the ability to increase the readability and maintain consistency with standard testing practices, demonstrating its potential to not only accurately replicate functionality but also enhance the overall quality of the test code.



 



  




 §.§ Analysis of the Non-AM Assertions (RQ4)


Motivation & Experiment setup. 

In this RQ, we conduct a qualitative analysis of the assert statements generated by that are not AM to the original ones. Through a detailed examination, we can identify the strength or weaknesses of the current approach, and gain valuable insights into the complexities and challenges of automated assert generation. The assessment is conducted by the first two authors who are software engineer PhD students with substantial experience in Python development. We implement an open coding approach <cit.> to evaluate the generated assertions. 
A total of 150 test cases are randomly selected from across all non-AM test methods, satisfying a statistically significant sample size that ensures a 95% confidence level and a 0.08 confidence interval <cit.>. 
Initially, the first 30 cases are jointly examined by the two authors. Subsequently, we independently label the remaining cases. In instances where the generated asserts are better than the original asserts, we submitted pull requests to the corresponding repositories.





Result.

Through the analysis, we identified cases that should be considered as correct predictions, but are challenging to detect using automated scripts. For the assertions that were predicted incorrectly, we also summarized three typical scenarios and investigated the underlying reasons behind these inaccuracies. 





22% of the non-AM test cases in our randomly selected dataset should be considered correct predictions. 
For example, in Fig <ref> sample 1, the predicted assertions use the predefined parameter whereas the original one has a hard-coded string. In sample 2, the generated assert checks the response status, whereas the original developer checks the response text. The purpose of both is to check if the response is successful, but they assert different attributes of the HTTP response. The situation varies across different cases, but these examples illustrate that the non-AM test cases may represent alternative but valid ways of performing the assertion as well. 




We identified three categories of incorrect predictions in the assertions: value/attribute error (31%), different assert rules (22%), and unrelated (25%). Firstly, the value/attribute error refers to the generated assertions inaccurately reflecting the values being asserted or asserting the wrong attribute of a parameter (sample 3 for example). Though the feedback interaction prompt is designed to help with getting the correct value, for some complex setups, it may still fail to capture the precise value. 
Secondly, there are instances where predicted assertions enforce conditions that are either more or less stringent than the original. The fourth sample reveals a stricter predicted assertion, checking the complete equality of the entire dictionaries, as opposed to the original code's specific focus on the site_name key.
Thirdly, the unrelated cases are where the generated assertions have no clear connection to the real tested parameters, which shows the may have misapprehended the test case's objective and assessed the wrong parameters. The reasons could stem from the length of the test cases, obscured method calls, or other complexities.
In addition, although it's rare for popular open-source projects, we identified a bug in the original test code. In the last sample, the predicted assertion evaluates if date lies within the range of 1 to 31, offering an accurate condition than the original assertion's potentially flawed use of the 'or' operator.


In analyzing non-AM test cases, we found that 22% should be considered correct predictions, representing valid alternatives to the original assertions. Among the incorrect predictions, 53% were closely related to the tested parameters, falling into categories of value/attribute error or different assertion rules. These cases might require minor modifications to align with the original intention. The findings highlight that a significant portion of non-AM cases either align with correct predictions or maintain relevance, underscoring its practical potential.



Following the coding exercise, we conducted a repository-wise summary of the generated assertions that could improve code readability, eliminate test smells, or establish more accurate rules in contrast to the original statements (e.g., sample 5) from both the AM but not exact match cases and the non-AM test cases but should be considered correct predictions. We made submissions in the form of pull requests (PRs) to a total of 9 repositories, resulting in 181 proposed line changes. By the time of this paper's submission, 7 PRs had been approved or merged while the rest were still under review. We also received thankful comments from project maintainers expressed for our efforts in refining the test code and rectifying potential bugs. These validations from the real software development community highlight the practical utility and efficacy of in enhancing the quality of unit test code.




§ DISCUSSION




 §.§ Generalisation across LLMs

As LLMs gain popularity, various new models are continually being developed by different companies and groups. To evaluate the effectiveness of CLAP's performance across a range of LLM models, we extended our evaluation beyond the initially tested models. Using the same experimental setup as RQ1, we included additional LLMs, specifically OpenAI's GPT-4 <cit.> and Google’s Gemini 1.5 Pro <cit.>. Table <ref> presents the results from these models compared to text-davinci-003, which achieved the highest performance in RQ1.



The gpt-4.0 achieves higher AM across all fields, likely due to being the latest model from OpenAI. The extended token limits enables it to handle longer test cases and more multiple assertions, which might be challenging for text-davinci. A new finding from the non-AM predictions of gpt-4.0 is it tends to include additional conditions compared to the original ones. For example, where the original assertion is assert A, gpt-4.0 might predict assert A and B, with B being a valid condition, but not originally specified by the developer.

The performance of gemini-pro is not as good as the two OpenAI models but still outperforms state-of-the-art approaches for single assert prediction (in RQ3). We observed that gemini-pro sometimes failed to respond in the requested format, leading to parsing errors and stopped before the feedback interaction phase due to these errors. To enhance the performance of gemini-pro, further refinements in prompt design or a more flexible parser for LLM responses may be necessary.

In summary, the results demonstrate that the design of works well with different models, highlighting its generalisation across different LLMs. Depending on the LLMs, refinements may be necessary for specific models. These refinements might involve minor adjustments in prompt design or response parsing strategies, but do not require major changes to its core structure.




 §.§ Practical Implications of 

In this section, we discuss two practical usage of : (1) as a wrapper for LLMs, integrated within an IDE or CI process, and (2) to enhance the assert generation capabilities of existing Python unit test generation tools.

Usage Scenarios for Developers can be used as an assistant for developers to help them complete the test cases in their projects. For example, John is a developer for an opensource project. He has just written a new function and is in the process of writing a unit test for the corresponding function. He writes the test inputs but is unsure about the specific assertions needed to thoroughly test the new function. can work as a plugin to his IDE. It automatically analyse the code context and suggest the correct assert statements for John.

can also be used in continuous integration (CI) environments. When new test code is committed to a repository, can automatically analyse the test code. In cases where the test code may contain bugs or smells (e.g., tests that always pass regardless of the code behavior[<https://github.com/lk-geimfari/mimesis/pull/1385 >]), can compare the generated assert statements with those written by developers. If discrepancies are found, can notify the developers, providing both versions of assert statements and prompting them to decide which one is more appropriate.
Additionally, if the new test code fails, can analyse the code context and suggest alternative assert statements using its feedback interaction mechanism. So if the issue lies within the original assert statements, developers can quickly identify the issue and replace the asserts.


Integration with Automatic Unit Test Generation Tool Pynguin <cit.> is a research framework that can automatically generate unit tests for Python and its Github repository has received over 1.1k stars. Due to its design, it requires the target projects to have type annotations. Two of the projects in our dataset overlap with those used in their research, namely Mimesis and Sanic. We employed Pynguin version 0.33.0 with the configuration “pynguin –algorithm DYNAMOSA –assertion_generation MUTATION_ANALYSIS” to generate unit tests for these codebases. We then removed the test cases which were tagged with pytest.mark.xfail decorator. These are the cases where Pynguin catch an unexpected error while running the test case. After manual inspect, we found this is due to failed to feed the correct typed parameter of a function. Then, for test cases that failed to produce assertions or the assertions failed, we executed to generate the necessary assertions and subsequently compared the resulting code coverage.



In total, Pynguin generated 98 test cases without the xfail docstring for Sanic and Mimesis. Among them, 25 missed an assert statement. For example, the Pynguin generated cases could have assigned a parameter from the focal method without asserting the parameter.  After execution of , we successfully generated assertions for the untested parameters in all the missing cases and fixed 2 failed test cases where Pynguin generated the wrong assertions. And result in an increased 2% test coverage for Sanic and Mimesis. These results indicate that can complement Pynguin by rectifying its limitations and enhancing the quality of the automatically generated test cases. Further experiments are necessary to explore how Pynguin can better utilize 's capabilities, perhaps by integrating 's functionality into its asserts generation process.



Moreover, prior studies such as CODAMOSA <cit.> have also explored the potential of LLM models to generate meaningful inputs for the test cases generated from Pynguin, with the intention of mitigating the incidence of test cases decorated with pytest.mark.xfail and thus increased the code coverage. By integrating these efforts, there exists prospective future research to enhance the quality of automatically generated unit tests. This could involve the development of a more comprehensive framework that capitalizes on the strengths of LLMs to not only create more meaningful assert statements but also generate the entire test case. It could potentially lead to a improvement in auto-generated unit tests, making them more reliable and beneficial for Python projects.



 §.§ Multiple Asserts and Assert Roulette


Test smells, similar to code smell, are known for bad design or implementation decisions in the unit test code <cit.>. Over the years, different test smells in languages such as Java, Javascript, Python were raised by researchers. One of the earliest and most prevalent of these is Assert Roulette <cit.>, characterized by test cases that contain multiple assert statements without explanatory messages or documentation <cit.>. It can be harmful as developers need more efforts to figure out why the test case failed. It has been one of the most common test smells among languages <cit.>. Our randomly sampled data set reveals that 71.8% of the test cases include more than one assert statement. 

Howerver, recent research by Panichella et al. <cit.> suggests that the concept of Assertion Roulette may need updating due to advancements in testing frameworks. For example, the unittest framework in Python provides detailed feedback about which asserts fail and the reasons for the failure (e.g., actual vs. expected value). The message is clear enough for the developers to identify the issues. Furthermore,  the choice between using multiple assertions in a single test and spreading them across multiple tests often comes down to a trade-off between assertion roulette and code duplication. Developers often prefer multiple assertions within a single test when these assertions are strongly related, as this approach reduces redundancy. For example, in the Scrapy project, there is a test_send_html function[<https://github.com/scrapy/scrapy/blob/10a843ac1d7105d1503ab367e3113da20a2d36fc/tests/test_mail.pyL46>], two assertions check that both the body and the content type of the sent HTML meet expectations. It is more efficient and clearer to test these aspects together in a single test case rather than duplicating the setup in separate tests.

Although Assert Roulette is traditionally considered a test smell, it no longer negatively impacts unit tests as it once did, with the evolving testing frameworks and the way developers write unit tests. Therefore, CLAP’s ability to generate multiple assertions within a single test case remains a significant and valuable contribution.



























§ THREATS TO VALIDITY




Internal validity. A potential threat to internal validity in our study is the extraction of focal methods. We adopted a heuristic approach from previous research <cit.> to match focal methods with test cases. However, since Python test cases can be structured differently compared to Java, biases may be introduced in the identified focal methods. To address this issue, we modified the script to exclude cases where the focal method was unclear. Additionally, in real-world scenarios, we expect users to be developers who can provide the correct corresponding focal method. Despite this limitation, our approach demonstrates its effectiveness in generating meaningful assert statements, reflecting its robustness.




External validity.

Our study focuses on generating assert statements for Python projects that present potential limitations in external validity, specifically concerning the generalizability of our approach. While the LLMs utilized in our approach have demonstrated versatility across various languages (e.g., CEDAR <cit.>), we did not conduct an empirical evaluation on other widely used datasets, such as the Java dataset from ATLAS. This was mainly due to the need for executing real projects, which did not align with the ATLAS database's constraints. Although our method shows the capability to address dynamically-typed languages like Python that possess a greater diversity of assertion types and complex type check compared with Java, the generalization to statically-typed languages remains untested. However, because of the use of the LLM, can potentially be generalized across various programming languages with minor adjustments to the interpreter environment for feedback interaction and the prompt manager design. This indicates a promising scope for 's widespread applicability in the assert generation task for a diverse range of programming languages.

While our dataset represents the first dedicated collection for Python assert generation, its 14 projects constructed dataset presents a limitation in generalisability. Despite our efforts to ensure diversity by including projects from various domains (Python libraries, ML/DL repositories, command line tools, and applications) and balancing those using built-in assert keywords with testing libraries, the relatively small number of projects may not capture all Python testing practices. This constraint is due to the large effort required to configure and prepare each project for analysis, and the necessity to avoid data leakage for the LLMs. Future works should aim to expand this dataset to further enhance generalisability of the findings.

Another potential threat is the adaptability of our prompt design to a variety of LLMs. Given the current popularity of the LLM field, new models are being introduced frequently and old ones may be deprecated (e.g., CODEX). In the experiment and discussion section, we evaluate our approach with four LLMs, and the results demonstrate that all models can generate meaningful assert statements using the current prompt design while a minor modification will be needed for different LLMs. As LLM development continues to evolve, there may be differences in input requirements; however, the core concept of providing natural language prompts to LLMs is expected to remain consistent. Therefore, only limited efforts would be required to adapt our approach to different LLM models.

Construct validity.
Throughout the study, our aim has been to generate meaningful assert statements; however, the term “meaningful” lacks a clear definition. In line with previous studies <cit.>, we evaluate the results by comparing them with the original asserts from developers. However, it is important to acknowledge that these original asserts may have their own limitations, such as poor readability, ineffectiveness in bug detection, or the presence of test smells. There are other metrics such as the checked coverage <cit.> that can evaluate the quality of the generated test code dynamically rather than the static string-based metrics. However, as Schafer et al. <cit.> pointed out, for dynamically typed language (e.g., JS and Python), it is difficult to precisely implement and calculate the checked coverage.
This could potentially affect the construct validity of our study. We recognise this approach to be an assistant to help developers write assert statements and conduct RQ4 to analyse the differences between the generated asserts and the original ones. Future works could focus on involving a more comprehensive evaluation process and improving the generated assert code quality.




§ RELATED WORK




 §.§ Assert Generation


In 2020, Watson et al. <cit.> first introduced a Deep Learning (DL) method for generating assert statements in Java, referred to as ATLAS. Utilizing a Neural Machine Translation (NMT) based approach, their work aimed to generate meaningful assert statements, which had proven to be a challenge for automatic unit test generation. Moreover, they extracted test cases from Java projects, paired them with their corresponding focal methods, and termed this structure as Test-Assert Pairs (TAPs). They released their dataset for the benefit of future research in the field.

Subsequently, Yu et al.<cit.> extended ATLAS by integrating an Information Retrieval (IR) based approach, while Tufano et al.<cit.> fine-tuned a state-of-the-art transformer model <cit.>, both achieving better performance compared to ATLAS. Similarly, Mastropaolo et al. <cit.> trained T5 model with large Java Dataset and fine-tuned with assert code, also has achieved better performance in assert generation. Most recently, Nashid et al. <cit.> combined IR with LLM to generate asserts in Java, achieving state-of-the-art performance. Their tool, CEDAR, which implements the BM25 algorithm to find the most similar test cases from a pre-built dataset and fed into CODEX (an LLM model from OpenAI) as few-shot learning prompts to generate the assert statements. 
In contrast to previous works, which focused on generating single asserts for Java projects, our work addresses the currently most popular language, Python, which has not been previously explored in this context and tends to have more multi-assert test cases. Furthermore, all three previous works relied on pre-built datasets to support their performance, potentially introducing bias and limitations in generating assert statements. Other than Java, 
Zamprogno et al. <cit.> introduced AutoAssert, a system designed to generate assertions for JavaScript. To address the limitations of dynamic approaches in assertion generation, they implemented a human-in-the-loop mechanism by allowing developers to choose the parameters to be tested and select suitable assertions from a predefined set. Like our method, AutoAssert assumes the tested code is correct, but its support is restricted to a limited number of assertion types due to inherent limitations in its design.



 §.§ Large Language Models for Software Engineering

Recently, various LLMs have been produced by different organizations, such as RoBERTa <cit.>, GPT models <cit.>, PaLM <cit.>, LLAMA <cit.>. In the software engineering domain, researchers and practitioners have conducted numerous experiments to explore the potential contributions of LLMs to the community. 
Copilot <cit.> assists developers in code writing by reading the code context and comments to generate code suggestions. Codium.AI <cit.> utilized LLM to generate unit tests for developers and developed plugins for the IDEs.
CODAMOSA <cit.> leveraged OpenAI's CODEX to provide example test cases to help with unit test generation. Xia et al. <cit.> and Nashid et al. <cit.> discussed how the LLM can help with program repair to fix potential bugs. Wang et al. <cit.> demonstrated that LLMs exhibit a deeper understanding of source code in tasks like algorithm classification, code clone detection, and code search. And Khan et al. <cit.> used LLM to generate meaningful documents for different programming languages. Our project aims to capitalize on the LLM's understanding of source code and incorporate the latest prompt engineering methods to assist developers in generating meaningful assert statements for Python projects.



§ CONCLUSION AND FUTURE WORK


 In this study, we present , an approach that integrates rounds of prompt communication with persona design, CoT prompting, and feedback interactions to optimize the performance of LLMs in generating meaningful assert statements for Python. Our evaluation demonstrates that outperforms the state-of-the-art approach in producing accurate single assert statements by over 10%. Adding consideration of multi-asserts generation, which is more common in Python projects, is also capable to generate accurate match assertions at 62%. 
We further conducted a qualitative study that suggested the generated assertions have the potential to increase the readability of the test case and fix the unseen bugs within the original assertions. Our submitted pull requests based on the generated assertions that positively influenced the test cases were accepted by real open-source projects. In addition, we discuss the generalisation across LLMs, practical applications of , and the importance of multi-asserts generation. In the future, we plan to further refine the method to improve and assess the quality of generated assert statements and minimize test smells. Given the compliant nature of LLMs like the GPT models, we will investigate more robust strategies to ensure these systems not only accept corrections but also apply them in a way that demonstrates a deeper understanding of code exceptions and errors in the prompt engineering aspect. From the practical perspective, we intend to develop this tool into a plugin for Python IDEs to simplify access for developers. Ultimately, we would like to extend the work to automatic unit test generation to better benefit the software engineer community.




ACM-Reference-Format






 Paper Title: Quantum Program Testing Through Commuting Pauli Strings on IBM's Quantum Computers.


Fast and scalable finite-element based approach for density functional theory calculations using projector-augmented wave method
    Phani Motamarri
    August 5, 2024
================================================================================================================================




§ ABSTRACT 
The most promising applications of quantum computing are centered around solving search and optimization tasks, particularly in fields such as physics simulations, quantum chemistry, and finance. However, the current quantum software testing methods face practical limitations when applied in industrial contexts: (i) they do not apply to quantum programs most relevant to the industry, (ii) they require a full program specification, which is usually not available for these programs, and (iii) they are incompatible with error mitigation methods currently adopted by main industry actors like IBM. To address these challenges, we present , a novel quantum software testing approach. introduces a new definition of test cases based on Pauli strings to improve compatibility with different quantum programs. also introduces a new test oracle that can be directly integrated with industrial APIs such as IBM's Estimator API and can utilize error mitigation methods for testing on real noisy quantum computers. We also leverage the commuting property of Pauli strings to relax the requirement of having complete program specifications, making practical for testing complex quantum programs in industrial settings. We empirically evaluate on 194,982 real quantum programs, demonstrating effective performance in test assessment compared to the state-of-the-art with a perfect F1-score, precision, and recall. Furthermore, we validate the industrial applicability of by assessing its performance on IBM's three real quantum computers, incorporating both industrial and open-source error mitigation methods.








§ INTRODUCTION


Quantum software testing aims to efficiently detect bugs in quantum software to ensure its correctness and reliability <cit.>. Existing quantum software testing methods adopt classical software testing methods such as search-based testing <cit.>, mutation testing <cit.>, combinatorial testing <cit.>, coverage criteria <cit.>, property-based testing <cit.>, mutation testing <cit.>, metamorphic testing <cit.>, and fuzzing <cit.>. However, practical limitations exist in applying these existing quantum software testing methods to real quantum computers. 

First, the existing testing methods are incompatible with some types of quantum programs, specifically those performing search and optimization tasks. These quantum programs do not explicitly take inputs, but instead leverage the superposition of all states to find solutions to problems. In contrast, the types of quantum programs supported by existing testing methods require explicit inputs, and their testing involves finding failing inputs. 

Second, most existing quantum testing methods rely on explicit oracles to determine whether a test case passes or fails. These methods often assume that the expected output for each test case is known in advance, which is unrealistic, especially for large quantum programs. As quantum algorithms evolve to handle more complex computations, specifying expected outputs for all potential test cases becomes increasingly impractical.

Third, existing testing methods assume ideal, noise-free quantum computers. However, this is not true for current quantum computers, which are inherently noisy, with errors affecting their computations <cit.>. To mitigate noise errors, the industry has adopted error mitigation methods that can be integrated with optimization programs to solve specific tasks on real quantum computers <cit.>. However, current testing methods are not designed to accommodate these error mitigation methods effectively. They often require additional instrumentation to align with error mitigation methods, making them difficult to use with high-level APIs such as IBM's Estimator API for developing quantum programs.


This paper addresses these practical challenges by introducing a novel quantum software testing approach called , which utilizes a new test case definition based on Pauli strings that does not require explicit inputs for quantum programs, thereby ensuring compatibility with all programs, including optimization and search programs. We also propose a new test oracle that can be integrated with error mitigation methods commonly adopted in the industry. Furthermore, we employ the concept of commuting Pauli strings and Pauli families to minimize the number of expected program outputs that must be specified to assess the passing and falling of a test case, making our approach more practical in industrial settings. We evaluated our approach on 194,982 real quantum programs and compared it with the current state-of-the-art test case definition and test oracle <cit.>. The results demonstrate that our approach performs better in test assessment, achieving an F1-score, precision, and recall of 1. Furthermore, we show the industrial relevance of our approach by performing test assessment on IBM's three quantum computers, integrating one industrial error mitigation method and one open-source method. Additionally, we discuss the implications and practicality of our approach and provide insights for practitioners. 

Our key contributions are:

[1)]

  * A novel testing approach that employs a new test case definition that is compatible with all quantum programs including search and optimization programs.

  * A novel test oracle compatible with error mitigation methods most widely used in industrial settings.

  * An empirical evaluation of test assessment across 194,982 quantum programs, demonstrating the effectiveness of our approach compared to an existing state-of-the-art.

  * An empirical evaluation of test assessment on IBM's three real quantum computers, incorporating both industrial and open-source error mitigation methods to illustrate the practical application of our approach.




§ BACKGROUND



 §.§ QC Basics and Example
 
Qubits: In classic computing, the basic unit of information is the bit, which can only be in the state of 0 or 1. Quantum computing (QC), however, uses quantum bits or qubits, which can exist in a superposition of both |0⟩ and |1⟩ states, with associated amplitudes (α). α is a complex number with both magnitude and phase in its polar form. A qubit is represented in Dirac notation <cit.> as: |ψ⟩ = α_0 |0⟩ + α_1 |1⟩, where α_0 and α_1 are the amplitudes corresponding to states |0⟩ and |1⟩, respectively. The probabilities of a qubit being in state |0⟩ or |1⟩ are determined by the square of the magnitudes of α_0 and α_1, with the sum of these squared magnitudes equaling 1: |α_0|^2 + |α_1|^2 = 1.

Quantum gates: These unitary operators change a qubit's state based on a unitary matrix <cit.>. For instance, the NOT gate flips the state of a qubit. Currently, gate-based quantum computers are programmed as quantum circuits, where the logic is implemented as a sequence of quantum gates applied to qubits. For better readability, we will refer to a quantum program as a “circuit” throughout this paper. Figure <ref> shows two entanglement circuits, each involving two qubits.



Entanglement is a phenomenon where two or more qubits are linked in such a way that the state of one qubit directly affects the state of the other. The circuit starts with two qubits (q_1 and q_2), initialized to the state |0⟩ by default. First, a Hadamard gate ( images/h.pdf ) is applied to q_1, placing it in a superposition of the |0⟩ and |1⟩ states. Next, a controlled-not gate ( images/cx.pdf ) is applied. A controlled-not gate is a two-qubit quantum gate, where the first qubit (q_1) is referred to as the control qubit and the second qubit (q_2) as the target qubit. The controlled-not gate flips the state of the target qubit if the control qubit is in state |1⟩. The controlled-not gate is applied after the Hadamard gate to create entanglement between the two qubits.

Measurement and Observable: Measurement is an operation used to extract the output of a quantum circuit. There are different types of measurement operations, with the type depending on the observable being used. Observables are physical quantities that can be measured, such as the position, momentum, and spin of a particle <cit.>. In QC, observables are represented by Hermitian matrices <cit.> with predefined eigenvalues, which are used in a measurement operation to extract the outcome of a qubit. Current quantum computers utilize four observables (Z, Y, X, I), known as Pauli observables, to measure the state of a single qubit <cit.>. 
For multi-qubit circuits, the measurement operation is represented by the tensor product of single-qubit Pauli observables, referred to as Pauli strings. For instance, in Figure <ref>, the Z measurement ( images/zm.pdf ) in circuit 1 and the Y measurement ( images/ym.pdf ) in circuit 2 illustrate single-qubit Pauli observables. The overall measurement operation in circuit 1 is defined by Pauli string ZZ which is the tensor product of Pauli observable Z on qubit q_1 and q_2. The output of a quantum circuit depends on the Pauli string used for measurement. For example, in Figure <ref>, the output for Pauli string ZZ in circuit 1 is (00) and (11), each with a probability of 0.5. For Pauli string YY in circuit 2, the output is (01) and (10), each with a probability of 0.5. For an n-qubit quantum circuit, there are 4^n possible combinations of Pauli strings, where n is the number of qubits and 4 represents the choices of Pauli observables (Z, Y, X, I) for each qubit.
Pauli strings have eigenvalues associated with them, which provides information about the possible outcomes of a Pauli string <cit.>. As mentioned before, Pauli strings are tensor products of Pauli observables, and Pauli observables are Hermitian metrics with predefined eigenvalues. So the eigenvalues of Pauli strings are represented by taking the tensor product of eigenvalues of Pauli observables that makes up that Pauli string. For example Pauli observable Z has eigenvalues (1,-1). So the eigenvalues of Pauli string ZZ can be calculated by the tensor product of (1,-1) with (1,-1), resulting in (1, -1, -1, 1) as the eigenvalues of Pauli string ZZ. The eigenvalues play an important role in the calculation of expectation values (described next) of a Pauli string.


Expectation Value: The output of a quantum circuit can vary upon repeated measurements due to its inherent probabilistic nature <cit.>. The expectation value is a concept used to describe the average outcome one would anticipate from such repeated measurements. It can be calculated using the formula E = ∑_i=0^n w_iP_i, where n is the total number of possible output states of a circuit, P_i is the probability of the ith output state, and w_i is the ith eigenvalue of the Pauli string. For example, consider the two-qubit circuit in Figure <ref> with Pauli string ZZ. The eigenvalues for ZZ are (1, -1, -1, 1). Using the formula and the measurement results from Figure <ref>, the expectation value for Pauli string ZZ is calculated as E = (1 * 0.5) + (-1 * 0) + (-1 * 0) + (1 * 0.5) = 1.



 §.§ Commuting Pauli Family

Pauli family refers to a group of Pauli strings that share a specific property, such as the commuting property <cit.>, which we consider in this work. The commuting property states that the product of two Pauli strings is not dependent on the order of application. For example, two Pauli strings ZZ and ZI commute if ZZ · ZI = ZI · ZZ.

As explained in section <ref>, for an n-qubit quantum circuit, there are 4^n Pauli strings, where n is the number of qubits and 4 arises from having four possibilities for each qubit, corresponding to the four types of Pauli observables: X, Y, Z, and I. All 4^n Pauli strings can be divided into K commuting Pauli families, where K varies for different quantum circuits. Several algorithms exist for generating Pauli families for a quantum circuit, with the state-of-the-art method presented by Reggio et al. <cit.>. 
One key advantage of constructing families of commuting Pauli strings is their ability to enable simultaneous measurements. Commuting Pauli strings have the property that the measurement result of a single Pauli string within a family can be used to calculate the expectation values of all Pauli strings within that family <cit.>. This property of commuting Pauli strings is extensively utilized in current quantum computers for tasks such as error correction and accelerating quantum computations <cit.>.



§ INDUSTRIAL CONTEXT AND CHALLENGES

In this section, we present the industrial context by highlighting three main challenges associated with applying existing quantum software testing methods from the perspective of IBM's real quantum computers. These challenges are not exclusive to IBM, but also apply to other gate-based quantum computers. However, the observations presented here are from our experience in testing quantum circuits using IBM's quantum computers. The three main challenges are incompatible test cases, requirement of a complete program specification , and incompatibility with error mitigation methods.



 §.§ Incompatible Test Cases

The first major issue with current quantum software testing methods is that they are not compatible with some types of quantum circuits. Quantum computing's primary application areas, such as physics simulations and quantum chemistry, revolve around optimization and search problems <cit.>. The industry is particularly focused on developing quantum circuits that can function on today's noisy quantum computers to address various optimization and search problems. Examples of such circuits are the Quantum Approximate Optimization Algorithm (QAOA) and Grover's algorithm <cit.>. However, the existing quantum software testing methods do not support these optimization and search circuits. The major challenge existing testing methods face for search and optimization circuits is the definition of appropriate test cases. Currently, test cases for quantum circuits are defined in two main ways. The first is binary test cases, employed by several testing approaches <cit.>. In this definition, the test cases are represented as binary inputs in which qubits are initialized to either the |0⟩ or |1⟩ state. The second is quantum test cases, used by other testing methods <cit.>. Quantum test cases are represented as quantum inputs in which qubits are initialized to a random superposition state α|0⟩+β|1⟩. However, both these test case types are unsuitable for search and optimization circuits like Grover's and QAOA. Indeed, in these circuits, the qubits are initialized to specific superposition states depending on the problem to be solved. For example, in Grover's circuit, the qubits are initialized in an equal superposition of all states, which overwrites any specific initial state set by test cases. This makes traditional test cases unsuitable for testing such circuits. 

It is therefore important to have a suitable test case definition for search and optimization circuits, especially when real quantum computers are predominantly used for quantum simulations and optimization tasks <cit.>. We address this challenge by proposing a new test case definition in our approach consisting of Pauli strings that act on the measurement operation rather than
explicitly setting qubit states as traditional test cases do.



 §.§ Requirement of a Complete Program Specification

The second major issue is the impractical requirement of specifying the expected outputs for all test cases prior to executing the tests. Given that optimization circuits are intended to solve large-scale problems, it is impractical to define the output for all possible test cases. Current quantum software testing methods require explicitly stating whether a test case passes. This is done by comparing the observed test case outcome with the expected test case outcome defined in a program specification. However, creating a comprehensive program specification is challenging, particularly when quantum algorithms are designed to solve computationally intensive problems. For instance, if we only consider binary test cases, a quantum circuit with n qubits can have 2^n potential test cases, necessitating an overwhelming number of test cases to be specified. Although it may be feasible to specify the expected output of a few test cases, specifying the output of all test cases is computationally impractical. We reduce the cost of program specification by utilizing the commuting property of Pauli strings described in Section <ref>. Pauli strings can be grouped into commuting Pauli families, allowing one Pauli string from each family to determine the results for the others within that family <cit.>. By leveraging this property, the number of outcomes that need to be specified in the program specification can be significantly reduced (more details in Section <ref>).



 §.§ Incompatibility with Error Mitigation

The third issue is the assumption made by most current testing methods that quantum computers are ideal and noise-free, which is not the reality. Indeed, real quantum computers exhibit inherent noise, which significantly affects both computational processes and the reliability of circuit outputs <cit.>. However, most current quantum software testing methods assume ideal conditions, making them ineffective in noisy real-world environments. Only three methods <cit.> enable testing on real quantum computers, but these methods come with the cost of training machine learning models, which requires a large amount of data. An alternative approach for testing on real computers could involve using the error mitigation methods proposed in the literature <cit.>. However, these error mitigation methods primarily focus on reducing noise effects on the expectation values of quantum circuits rather than individual outputs, whereas quantum testing methods operate with individual outputs and lack mechanisms or oracles directly addressing expectation values. This creates an incompatibility between current testing methods and error mitigation methods. Furthermore, companies such as IBM have introduced specialized APIs, such as the Estimator API in Qiskit <cit.>, to ease calculating expectation values with built-in error mitigation methods. However, current testing methods lack oracles directly addressing expectation values and are incompatible with such APIs. We address this challenge by proposing a new test oracle (Section <ref>) based on expectation values, making it compatible with IBM's Estimator API and other error mitigation methods.



§ APPROACH

We present an approach named : Quantum prOgram testing through Pauli Strings, addressing the three main challenges outlined in Section <ref>. In , we utilize Pauli strings for testing quantum circuits.
requires two inputs from a tester: the circuit under test () and a compact program specification (). In the , a tester is required to specify the expected output of any single Pauli string belonging to a commuting Pauli family. The cost of is significantly lower than the full program specification required by other testing methods described in Section <ref>. For example, for a 5-qubit circuit under test, the total number of Pauli strings is 4^5 (1024). Given that all 1024 Pauli strings can be partitioned into 33 commuting Pauli families, the would require a maximum of 33 Pauli strings, which is significantly lower than 1024. An example of is shown in Figure <ref>, which specifies the outputs of two Pauli strings ZX and YI for the entanglement circuit in Figure <ref>.



In the next subsections, we first provide the definition of our test cases and the test oracle along with their rationale in Section <ref>. The overall working of the approach is described in Section <ref>.



 §.§ Test Case and Oracle
 
employs a new definition of test case consisting of Pauli strings that act on the measurement operation rather than explicitly setting qubit states as traditional test cases do. We believe that different Pauli strings can identify the same faults in a circuit that are captured by explicitly setting quantum states. The rationale is that traditional test cases set the initial state of a circuit to observe a specific output. However, we noticed that varying the measurement operation with different Pauli strings can also produce the same output. In Figure <ref>, we show an intuitive example by considering the entanglement circuit as the circuit under test.



Figure <ref>a shows the execution result of a binary test case 01, which sets qubit q_1 to |0⟩ and q_2 to |1⟩ for the test circuit. However, we can also observe the same execution result of the binary test case by using Pauli string YY as shown in Figure <ref>a and not setting the qubit states explicitly. Figure <ref>b shows a quantum test case execution result where the quantum state Ψ is set in a superposition of |00⟩ and |10⟩ with 75% and 25% probability for each state, respectively. We can achieve the same output result as of the quantum test case, if we execute the quantum circuit twice, once with the Pauli string ZZ and second with the Pauli string YY and merge the two outputs by a weighted sum of both distributions, with 75% weight for ZZ and 25% weight for YY. This process of merging the results of different Pauli strings is commonly used in quantum optimization circuits <cit.>. This indicates that merging outputs of different Pauli strings with different weights could be analogous to quantum test cases without explicitly setting the state of qubits. However, the combination of Pauli strings that are analogous to a specific quantum input for a specific circuit could be different for different circuits. The advantage of using Pauli strings as tests over traditional test cases is that they are associated with the measurement operation, which is independent of the initialization logic of a circuit, making them suitable for optimization and search circuits.

Test case definition: We define our test cases based on the formulation = ∑_i=0^S c_i_i, where S is a subset of Pauli strings chosen randomly from a particular Pauli family, _i is the ith Pauli string in S, and c_i is the weight assigned to the ith Pauli string. For example, consider the subset S consisting of IZ, ZI, and ZZ. Then, a test case can be written as = c_1IZ + c_2ZI + c_3ZZ. Here, c_1, c_2, and c_3 are the weights assigned to the Pauli strings to define the contribution of each Pauli string in the overall output. We use the weighted sum format to define our test case because we hypothesize that it is analogous to the traditional quantum test cases. For example, we show in Figure <ref>b, that the output obtained by executing a quantum test case Ψ can also be obtained by executing the Pauli strings ZZ and YY and performing a weighted sum of their outputs. Therefore, we can represent the same quantum test case Ψ in our Pauli string test case definition as = 0.75ZZ + 0.25YY.

Test Oracle: With our test case definition, we can randomly generate many test cases. However, we also need an oracle to determine whether the circuit under test () has failed a test case. For this, we define the oracle Expected Expectation (), which is the total expectation value of a test case after execution. This value is calculated by ∑_i=0^ c_iE_i, where is the total number of Pauli strings in test case , E_i is the expectation value of the ith Pauli string in test case , and c_i is the weight associated with the ith Pauli string in test case . Using the expectation value formula from Section <ref>, we can transform the equation as


    = ∑_i=0^ c_iE_i = ∑_i=0^ c_i∑_j=0^_i w_jP_j


where _i is the ith Pauli string in test case , P_j is the probability of the jth output state in the output of Pauli string _i, and w_j is the jth eigenvalue of the ith Pauli string. Our test case consists of commuting Pauli strings, and, therefore, the output of a single Pauli string can be used to calculate the expectation value of other Pauli strings. By utilizing this concept, we can simplify Equation <ref> as


    = ∑_(i,j)∈ I c_iw_i,jM_j


where I = {(i,j) | 0 ≤ i ≤ and 0 ≤ j ≤_i}, is the total number of Pauli strings in test case , _i is the ith Pauli string in the test case , c_i is the weight of the ith Pauli string in test case , w_i,j is the jth eigenvalue of the ith Pauli string in test case , and M_j is the value of the jth output state of the Pauli string result given in the .

For example, consider the test case = 0.3ZI + 0.4IZ; the output of Pauli string ZZ is specified in as (00, 01, 10, 11) with a probability of (0.5, 0.2, 0, 0.3) respectively. The eigenvalues of ZZ are (1,-1,-1, 1). Given that ZZ, ZI, and IZ belong to the same commuting family, we can calculate the total expectation value of test case by utilizing the output of ZZ. Substituting the values in Equation <ref>:


    = (c_0w_0,0M_0 + c_0w_0,1M_1 + c_0w_0,2M_2 + c_0w_0,3M_3)


    + (c_1w_1,0M_0 + c_1w_1,1M_1 + c_1w_1,2M_2 + c_1w_1,3M_3)


    = (0.3*1*0.5 + 0.3*-1*0.2 + 0.3*-1*0 + 0.3*1*0.3)


    + (0.4*1*0.5 + 0.4*-1*0.2 + 0.4*-1*0 + 0.4*1*0.3)=1.56







The expected expectation value of the test case would be 1.56.



 §.§ Overview

Figure <ref> shows the overview of approach consisting of two main parts, Pauli Transformation, and Test Assessment, which are further described in the following subsections.





  §.§.§ Pauli Transformation

first generates families of commuting Pauli strings that can be used for testing a quantum circuit by taking and as inputs. To create Pauli families, employs the partitioning algorithm developed by Reggio et al. <cit.>, details of which can be found in their respective work. Here, we summarize the partitioning algorithm's operation in two main steps. First, the algorithm starts by identifying a set of Pauli strings that fulfills the commuting condition. Next, for a given and the selected set of Pauli strings, it finds a matrix that diagonalizes all the Pauli strings in the selected set. If such a matrix is successfully found, the commuting set is considered one family, and the process is repeated until no more sets can be found. 

The output of the partitioning algorithm is a list of K Pauli families and the eigenvalues of each Pauli string within a family. Table <ref> shows the output of the partitioning algorithm for the entanglement circuit from Figure <ref> taken as .



The column Family shows the identified Pauli Families and the column EigenValues shows the eigenvalues associated with each Pauli string within a Pauli family. Given the measurement result of any Pauli string in a family, these eigenvalues can be utilized to calculate the test oracle for a test case consisting of any subset of Pauli string within the family as shown in Section <ref>.


Once has identified the Pauli families and the corresponding eigenvalues, it checks the to filter out non-relevant families for testing. Depending on the number of Pauli strings specified in the , not all families may be suitable for inclusion in the testing process. Ideally, the should specify at least one Pauli string for each Pauli family to ensure all families can be effectively utilized during testing. However, depending on the number of Pauli strings that can be specified by the tester, the number of applicable Pauli families are reduced accordingly. As shown in Figure <ref>, the result of the Pauli Transformation stage in is an applicable set of families. Each item in this set is represented as a tuple containing three values:  - a specific Pauli family,  - the corresponding eigenvalues of the family, and  - the measurement outcome specified in for any Pauli string from the family .



  §.§.§ Test Generation and Assessment

The second part of aims to test the quantum circuit by utilizing the applicable family set from the Pauli Transformation stage. As shown in Figure <ref>, for each family in the applicable family set, generates random test cases along with their expected outcome for a specific budget. The user defines the budget, and decides how many test cases to generate for each family. In our experiments, we set the budget to 1000. As shown in Figure <ref>, the test assessment follows a simple process. For each family, a number of random test cases are generated depending on the budget, and the oracle for each test case is automatically calculated using the . The test cases and the are then passed to IBM's Estimator API to execute the with the test case and obtain the results. In case of error mitigation enabled, the result is first parsed to the error mitigation methods integrated in IBM's Estimator API or any open source error mitigation method. The observed results are then compared with the oracle using a threshold value set by the tester. If the difference between the observed result and the oracle exceeds the threshold, the test case is considered failed, and the fault is reported to the tester.



  §.§.§ Integration with Error Mitigation

's test case definition and oracle are directly compatible with error mitigation methods as we operate with the expectation value and not with individual outputs. Figure <ref> illustrates an example of test case executed with IBM's Estimator API, using the zero noise extrapolation (ZNE) technique for error mitigation.



In Line 1, we define a test case as = 0.75*ZZ + 0.25*ZI. In Lines 2-3, we select an IBM quantum computer that is least busy at the moment. Line 4 invokes the IBM Estimator API with the resilience level set to 2, enabling the inbuilt ZNE error mitigation technique. Finally, in Line 5, we submit our test case along with the to the IBM quantum computer. Note that our test case definition is directly compatible, so we can pass the defined test case variable directly to the Estimator API. In Line 6, we obtain the error-mitigated observed expectation value of the test case, which we then compare with our own calculated expected expectation value for assertion.



§ EVALUATION AND ANALYSIS



 §.§ Research Question

We assess the effectiveness of in detecting faults in quantum circuits by answering the following research questions (RQs):




  RQ1 How comparable is to the most widely used test case definition in terms of capturing faults in quantum circuits?

  RQ2 How effective is when applied to testing quantum circuits on real noisy quantum computers?




 §.§ Experiment Design




  §.§.§ Benchmarks

We employ the MQT benchmark <cit.>, a repository of diverse quantum circuits with real-world applications for evaluation of . MQT includes 18 different quantum algorithms implemented with varying qubit sizes. We selected all 18 algorithms, but limited the number of qubits to between 2 and 10 to ensure reasonable execution times for the quantum simulator and real computers. This selection resulted in 157 quantum circuits for testing. Additionally, we created faulty versions of these 157 circuits using Muskit <cit.>, applying all possible mutation operators defined in Muskit. This resulted in 194,982 quantum circuits, comprising 30,084 equivalent circuits and 164,898 faulty circuits, which were used to evaluate . We benchmarked against the most widely used test case definition, the binary test cases, and the common test oracle based on the chi-square statistical test. These are prevalent in current testing methods <cit.>. For better readability, we abbreviate the existing test case definition and oracle as for the rest of the paper.



  §.§.§ Experiment Settings

The implementation of and the experiment results are publicly available in our repository <cit.>. We use the Qiskit framework for quantum circuit execution <cit.>. To automatically obtain the full program specifications for the benchmarks for , we use Qiskit's AER simulator to get the correct circuit outputs. We also use Qiskit's AER simulator to get compact program specifications for , specifying only one Pauli string for each commuting family for a specific quantum circuit. For execution on real quantum computers, we use IBM's three publicly accessible quantum computers (IBM-Brisbane, IBM-Osaka, and IBM-Kyoto) and employ IBM's Estimator API for circuit execution. For the number of test cases, we set budget to 1000 test cases per Pauli family, whereas for we used all possible test cases, which results in 2^n test cases, where n is the number of qubits in the circuit under test. 

Regarding error mitigation on real computers, the most common methods are probabilistic error cancellation (PEC), zero-noise extrapolation (ZNE), and Clifford data regression (CDR) <cit.>. IBM's Estimator API provides built-in error mitigation based on ZNE and PEC. Given the exponential cost of PEC <cit.> and IBM's monthly limit of 10 minutes for using real quantum computers, we selected the ZNE method as the error mitigation method in the Estimator API. Additionally, we used an open-source version of the CDR method <cit.> for error mitigation to demonstrate 's applicability with both industrial and open-source error mitigation methods.

requires a tolerance threshold to assert whether a test case fails. For a fair comparison between and , we set this threshold to the same value used in the statistical test assertion in , which is 1% for evaluation on the quantum simulator. For real quantum computers, considering that error mitigation methods have their own error margins, we used the Standard Error <cit.> as the threshold. To calculate the standard error for a quantum circuit, we executed the non-faulty version 10 times with error mitigation and calculated the difference between the ideal value and the error-mitigated value for each run. The standard error was then calculated using the formula SE = σ/√(n), where σ is the standard deviation of the differences between the ideal and error-mitigated values over 10 runs, and n is the total number of runs, which is 10. The standard error value varies depending on the margin of error of an error mitigation method. Thus, it is different for each quantum circuit executed with a specific error mitigation method.



  §.§.§ Metrics

For RQ1, we compare the test assessment results of with those of using a classification problem framework. In this context, a method's accuracy in distinguishing between faulty circuits and equivalent or non-faulty circuits indicates its effectiveness. Therefore, we compare with based on commonly used quality metrics: F1-score, precision, and recall <cit.>. These metrics are calculated using their standard formulas <cit.>. We compute F1-score, precision, and recall separately for equivalent and faulty versions of quantum circuits, comparing the results with the ground truth for both and . The ground truth is calculated based on the Hellinger distance <cit.> between the original circuit and the mutant circuit generated by Muskit. Hellinger distance is a commonly used metric for checking the equivalence of two quantum circuits based on their outputs <cit.>. All circuits that result in a zero value for Hellinger distance are classified as equivalent circuits, and others are classified as faulty circuits. For quality metric calculation, we define True Positive (TP), False Negative (FN), False Positive (FP), and True Negative (TN), in comparison to ground truth as follows: 
TP - Both the ground truth and 's ('s) test assessment say ”faulty”;
FP - 's ('s) test assessment says ”faulty”, but the ground truth says ”not faulty”; 
FN - 's ('s) test assessment says ”not faulty”, but the ground truth says ”faulty”; and 
TN - Both the ground truth and 's ('s) test assessment say “not faulty”.

For RQ2, since is incompatible with error mitigation methods, we can only evaluate on real quantum computers. To demonstrate 's applicability on real computers, we use the same quality metrics as in RQ1. We then analyze how the results differ from our findings in RQ1 when is integrated with error mitigation methods and evaluated on real quantum computers.



 §.§ Results



  §.§.§ RQ1–Comparison with 

Table <ref> compares with in terms of F1-score, precision, and recall. The columns labeled and present the test assessment results for equivalent and faulty circuits for both and when compared against the ground truth. Table <ref> shows that outperforms in accurately identifying faulty and equivalent circuits without any false positives or false negatives. Although generally performed well, it exhibited higher false positives and false negatives for some equivalent and faulty circuits compared to . Specifically, misclassified five out of 30,084 equivalent circuits, likely due to the chi-square statistical test's inherent margin of error for false positives. Regarding faulty circuits, failed to detect faults in 484 out of 164,898 cases. These undetected faults predominantly involved a specific type of fault known as a phase flip fault <cit.>. Detecting phase flip faults is particularly challenging, because it requires altering the Pauli observables in the measurement operation. The chi-square test employed by is less effective in identifying this type of fault, leading to a higher number of undetected cases. On the other hand, proves to be highly effective in capturing phase flip faults. This effectiveness comes from the fact that utilizes test cases that consist of Pauli strings, where each Pauli string is composed of different Pauli observables. Overall, Table <ref> shows that provides better accuracy in terms of identifying faulty circuits and equivalent circuits with no false positives and false negatives.



[colback=blue!5!white,colframe=white,breakable]
RQ1: Compared to state-of-the-art, is more effective in test assessment with a perfect F1-score, precision, and recall, indicating that is a more reliable and alternative approach for testing quantum circuits. 





  §.§.§ RQ2–Integration with error mitigation

RQ2 assesses whether can be integrated with error mitigation methods and successfully capture faults in quantum circuits on real quantum computers. Given that IBM's quantum computers have a monthly usage limit of 10 minutes, executing all 194,982 circuits on real computers was not possible. To fit within these usage limits, we reduced the number of circuits from RQ1. We employed the Fisher-Jenks natural break algorithm <cit.> to identify representative faulty circuits for the 157 original circuits, based on the diversity in the oracle results. The Fisher-Jenks algorithm is a widely used method for automatically selecting a representative subset from large datasets based on diversity criteria <cit.>. The diversity criterion was the expected expectation value of the failed test cases from the RQ1 experiments. This algorithm resulted in 10 faulty circuits per original circuit, yielding 1,570 circuits to be evaluated on real quantum computers. For error mitigation methods, as stated in Section <ref>, we used both ZNE and CDR methods. IBM provides four different configurations of the ZNE method: Linear (L), Cubic (C), Quartic (Q), and Richardson (R), and we treated each configuration as a separate method. This resulted in five different error mitigation methods, which we evaluated on IBM's three real quantum computers.

Table <ref> shows the results of evaluating 1,570 faulty circuits on the IBM's three real quantum computers: IBM-Brisbane, IBM-Osaka, and IBM-Kyoto.



The columns display the F1 score, precision, and recall for each error mitigation method and computer pair, calculated by comparing the results with the ground truth. From Table <ref>, we observe that performed well across all computer and error mitigation method pairs. The variation in F1 scores for each pair is attributed to the differing levels of noise in each computer and the varying effectiveness of error mitigation methods for different circuits. Notably, the best error mitigation method integrated with is the ZNE linear configuration (L), which had only one false negative on IBM-Bisbane and IBM-Kyoto and two false negatives on IBM-Osaka. The second best method is the ZNE Richardson (R) method. Conversely, the worst method is CDR, with over 20 false negatives across all computers. 

Each error mitigation method has its own time cost for filtering noise from the circuit output. Thus, choosing an appropriate error mitigation method is important. Table <ref> indicates that IBM's integrated ZNE method outperforms the CDR method, particularly with its linear configuration. Overall, can successfully perform test assessments on real computers when integrated with error mitigation methods. Testers using IBM's APIs can conduct testing activities on real computers without additional instrumentation for integrating error mitigation with . However, the performance of is dependent on the accuracy of the error mitigation methods.
[colback=blue!5!white,colframe=white,breakable]
RQ2: Compared to state-of-the-art, is directly compatible with error mitigation methods, allowing to successfully perform test assessments on real computers using APIs with integrated error mitigation methods such as IBM's Estimator API. 




 §.§ Threats to validity

Construct Validity <cit.> refers to how well a measurement evaluates the intended theoretical concept. One threat to construct validity involves the metrics used to assess the effectiveness of . A testing method must accurately differentiate between faulty and non-faulty circuits. The effectiveness of one testing method compared to another depends on the number of false positives and false negatives produced during the assessments. This work uses F1-score, precision, and recall as performance metrics. Low precision and recall indicate higher false positives and negatives, while the F1-score measures the overall performance of a test assessment method. Another concern is the selection of error mitigation methods and the real computers used. Different quantum computers exhibit varying noise levels, and different error mitigation methods have different error margins for different circuits executed on different computers. In this paper, we focused on all of IBM's quantum computers that were publicly available during the experiment. We concentrated on error mitigation methods commonly used in the industry. IBM provides two error mitigation methods: ZNE and PEC. Given our limited access to IBM's quantum computers, the only feasible error mitigation method was ZNE. Additionally, we used an open-source error mitigation method, CDR, to demonstrate the applicability of with both industrial and open-source error mitigation methods.

Internal Validity <cit.> concerns how experiments can establish a causal relationship between independent and dependent variables. One threat to internal validity involves the tolerance parameter of . The tolerance parameter determines the acceptable error margin when comparing the observed expectation value of the test cases with the expected expectation. This parameter depends on the circuit and requires domain knowledge to be set correctly. In our experiments, to ensure a fair comparison with the and , we chose the tolerance value to be the same as the threshold used in 's statistical test for passing and failing a hypothesis, which is 1%. Although using a different tolerance value could affect the outcome, we found in the RQ1 results that a tolerance value of 1% was optimal, as it resulted in zero false positives and negatives for . For evaluations on real computers, since error mitigation methods have their own error margins that vary for different circuits, a single tolerance value cannot be established. To address this, we used 1% plus the standard error value of the original circuit for 10 consecutive runs executed with error mitigation as the threshold value. The standard error is calculated automatically for each circuit and error mitigation pair before performing the full test assessment.

Conclusion validity <cit.> focuses on the statistical significance of the results derived from an experiment.
One threat to conclusion validity is using random test cases in the test assessment by . Using random test cases does not guarantee that will perform consistently if executed multiple times. However, the main objective of this study is to assess whether can identify all the faults. Therefore, we focused on evaluating a large number of circuits, which limited our ability to run the experiment multiple times. Additionally, we set the number of test cases for each Pauli family to 1000 to increase the probability of finding a test case that captures the fault in a faulty circuit. The RQ1 results indicate that , even with random test cases, could correctly identify all the faulty circuits. Multiple runs to reduce random bias are only necessary when there is a chance that a better solution exists. However, in our case, a single experiment run was sufficient to find the best solution, reducing the need for multiple experiment evaluations.

External Validity <cit.> refers to how our method can be applied to other datasets and domains. A challenge in this area is the selection of quantum circuits for the test assessment, as a diverse set of circuits would result in more generalizable outcomes. To mitigate this issue, we used the widely accepted benchmark MQT <cit.> to choose circuits representing the most commonly utilized quantum circuits. Another potential threat is the number of qubits used, as increasing the number of qubits directly adds to the complexity of quantum circuits. Current quantum simulators can only handle up to 10 qubits in a reasonable time <cit.>. In this study, our goal was to test many circuits for generalizability, which restricted the number of qubits we could use, as more qubits significantly increase the execution time of circuits on simulators. Thus, to evaluate a large number of circuits within a practical time frame, we set the maximum number of qubits to 10.




§ DISCUSSION



 §.§ Practical Implications

This study presented a new quantum circuit testing approach () to solve three practical challenges to ensure the reliable development of QC solutions in an industrial context, particularly for IBM's real quantum computers. Indeed, the incompatibility of current quantum software testing methods with circuits designed for optimization and search tasks limits their practical utility. Since the primary industrial application of QC is in optimization and search problems, this gap presents a substantial barrier. Additionally, the impractical requirement to specify expected outputs for all test cases before execution, challenges the scalability and practicality of current testing methods in real-world scenarios. Furthermore, the assumption that quantum computers are ideal and noise-free fails to reflect their actual state. In reality, quantum computers are susceptible to noise, which affects their performance and accuracy.

To address these practical challenges, especially for testers using IBM's APIs for testing, we presented a new definition of test cases and a test oracle in . The 's generated test cases are directly compatible with IBM APIs, such as the Estimator API, which is heavily used for optimization and search circuits. We utilize the commuting property of Pauli strings to reduce the need for large number of predefined expected outputs and provide a new test oracle compatible with error mitigation methods, particularly those implemented in IBM APIs. Our empirical evaluation shows that can be a potential alternative for existing quantum circuit testing methods, improving the reliability and scalability of quantum computing methods developed using IBM's quantum computers.



 §.§ Integration with Other Quantum Software Frameworks

Our approach demonstrates that the test case definition and oracle of are fully compatible with IBM's Qiskit framework. While our current implementation primarily supports IBM's API for automated test assessment, our test case definition and oracle are also compatible with other widely used gate-based quantum computers and software frameworks. These include IBM's Qiskit, PennyLane's QML, and Google's Cirq <cit.>. Although the execution of quantum circuits varies among these frameworks, the fundamental definition of the test case and oracle remains consistent. For instance, consider a test case = 0.75*ZZ + 0.25*ZI, comprising two Pauli strings: ZZ and ZI. Figure <ref> illustrates how this test case can be represented in the three most popular frameworks: Qiskit, QML, and Cirq.



All three frameworks support the concept of Pauli observables, enabling straightforward translation of the mathematical representation of test case into an executable form. In Qiskit, the test case can be instantiated using the SparsePauliOp object; in QML, using the PauliWord and PauliSentence objects; and in Cirq, using the PauliString and PauliSum objects. Each framework provides an API that accepts the test case and the circuit under evaluation, returning expectation values accordingly. Regarding 's test oracle, no transformations are necessary for different frameworks, because the oracle is computed directly from the mathematical representation of the test case. Therefore, quantum software developers using various quantum software stacks can seamlessly integrate into their development environments.



 §.§ Scalability

In terms of test assessment for quantum circuits, is a highly scalable method compared to existing testing methods. is capable of generating and evaluating K × N test cases from just K Pauli strings specified in , where K represents the number of commuting Pauli families for a quantum circuit, and N denotes the number of random test cases. Existing testing methods, which require a full program specification to determine the passing and failing of test cases, are limited to evaluating only the number of test cases explicitly specified in the program specification. In contrast, leverages the specified Pauli strings in to generate as many test cases as the test budget allows, making it a practical approach for scenarios where the program specification is limited.



 §.§ Way Forward

In this study, we introduced an alternative approach for quantum circuit testing by developing a new definition for test cases and a new test oracle. We demonstrated that quantum circuit faults can be effectively identified through randomly generated test cases. However, generating random test cases is time-consuming and resource-intensive, making it a highly inefficient method for testing. 

To enhance the efficiency and effectiveness of our method, we plan to improve the test case generation aspect of in the future. One promising solution is to employ search-based approaches such as genetic algorithms, simulated annealing, or particle swarm optimization <cit.>. These algorithms can systematically explore the space of potential test cases, prioritizing those that are more likely to uncover faults efficiently. By using these search methods, we hope to significantly reduce the time and resources required for test case generation.

Another potential research direction is to discover relationships between test cases and various types of faults. Specifically, focusing on the relation between the passing or failing of tests within the same Pauli family. For instance, if one test in a family fails, is it more likely that other tests in the same family will also fail? Moreover, even if a test does not fail, could its execution provide a hint that another test in the same family might fail? Understanding the relation between specific test cases and the faults they reveal could lead to new methods for debugging and repairing quantum circuits. This knowledge would enhance the accuracy of fault detection and provide valuable insights for developing more robust quantum circuits.



§ RELATED WORK

Several studies have explored quantum software testing, introducing various methods for testing quantum circuits, evaluating test suite quality, and providing test oracles. Various tools and approaches have been proposed in the literature for testing quantum circuits. Quito <cit.> is a tool proposed to test quantum circuits by fulfilling input-output coverage criteria. Tools like Muskit <cit.> and QMutpy <cit.> assist in mutation testing. QuSBT <cit.> uses a search algorithm to generate test cases and MuTG <cit.> us a multiobjective search algorithm to find a minimum number of test cases that can kill a maximum number of mutants. QuCAT <cit.> employs combinatorial testing methods, while QuanFuzz <cit.> utilizes fuzzing method for quantum circuit testing. Several methods have been developed to gauge the effectiveness of test suites. Coverage criteria such as input-output coverage <cit.> and equivalence class partitioning <cit.> are employed to evaluate test suite effectiveness. Currently, two common types of oracles are used for quantum circuit testing. The first type employs statistical tests to compare quantum circuit outputs <cit.>. The second type, property-based testing <cit.>, derives several properties for the circuit under test, where the violation of these properties indicates the presence of faults. Several bug identification frameworks <cit.> have also been proposed, including Bugs4Q <cit.>, which is a benchmark for real-world Qiskit circuit bugs.

Current real-world quantum computers are noisy, which greatly reduces the reliability of computations and introduces errors in circuit outputs <cit.>. The causes of noise in quantum computers are environmental factors such as magnetic fields and radiation, qubit cross interactions, and calibration error in quantum gates <cit.>. Due to noise, testing quantum circuits on actual quantum computers is a challenging task because it becomes difficult to differentiate between faults and errors from noise. Several methods have been proposed for mitigating quantum noise <cit.>, focusing on reducing noise effects on the expectation values of quantum circuits. To the best of our knowledge, only three testing methods enable quantum circuit testing on real computers. The first method is QOIN <cit.> which trains a machine-learning (ML) model utilizing three statistical features to reduce the effect of noise. The second is Qraft <cit.>, which introduced a new set of features for training ML models that can be applied to multiple circuits under test. The last one is Q-LEAR <cit.>, which introduced a new feature called depth-cut-error to address Qraft's limitation of overestimating noise in the circuit output.

The main drawback of current testing methods is their incompatibility with search and optimization circuits, as well as the necessity for full program specification. Apart from three methods <cit.>, existing testing approaches are not suitable for use with real noisy quantum computers and depend on statistical test-based oracles that are incompatible with existing error mitigation methods. Even the three methods that do support testing on real quantum computers still face the issues of incompatibility with search and optimization circuits and the requirement for full program specification.



§ CONCLUSION

To address the practical challenges of incompatible test case definition, costly requirement of full program specification, and incompatibility with error mitigation methods, we developed , which is a novel quantum circuit testing approach. utilizes Pauli strings as a new definition of test cases, eliminating the need for having explicit inputs in quantum circuits and ensuring compatibility with industrially relevant optimization and search circuits. We have also introduced a new test oracle that can be easily integrated with commonly adopted error mitigation methods in the industry. Furthermore, by utilizing the commuting property of Pauli strings, we also reduce the cost of a full program specification making quantum software testing more practical in industrial settings. We evaluated on 194,982 quantum circuits, benchmarking it against the current state-of-the-art test case definition and test oracle. The results demonstrate the effectiveness of in terms of test assessment, achieving perfect F1-score, precision, and recall. Additionally, we validated the industrial applicability of by conducting test assessments on IBM's three real quantum computers, leveraging both industrial-grade and open-source error mitigation methods. In the future, we aim to enhance the applicability of in industrial settings by integrating with search-based approaches to further reduce the time and resources required for test assessment.



 §.§ Acknowledgments

The work is supported by the Qu-Test project (Project #299827) funded by the Research Council of Norway. S. Ali is also supported by Oslo Metropolitan University's Quantum Hub. Paolo Arcaini is supported by Engineerable AI Techniques for Practical Applications of High-Quality Machine Learning-based Systems Project (Grant Number JPMJMI20B8), JST-Mirai.

ACM-Reference-Format




47


















#1     #1#1#1     #1     #1     #1     #1#1          #1#1           






[Abreu et al(2023)]
        Rui_metamorphic
authorpersonRui Abreu, personJoão Paulo Fernandes, personLuis Llana, and personGuilherme Tavares. year2023.
Metamorphic Testing of Oracle Quantum Programs. In booktitleProceedings of the 3rd International Workshop on Quantum Software Engineering (Pittsburgh, Pennsylvania) (seriesQ-SE '22). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages16–23.
9781450393355

<https://doi.org/10.1145/3528230.3529189>



[Ali et al(2021)]
        coverage
authorpersonShaukat Ali, personPaolo Arcaini, personXinyi Wang, and personTao Yue. year2021.
Assessing the Effectiveness of Input and Output Coverage Criteria for Testing Quantum Programs. In booktitle2021 IEEE 14th International Conference on Software Testing, Validation and Verification (ICST). pages13–23.

<https://doi.org/10.1109/ICST49551.2021.00014>



[Bayerstadler et al(2021)]
        application
authorpersonAndreas Bayerstadler, personGuillaume Becquin, personJulia Binder, personThierry Botter, personHans Ehm, personThomas Ehmer, personMarvin Erdmann, personNorbert Gaus, personPhilipp Harbach, personMaximilian Hess, et al year2021.
Industry quantum computing applications.
journalEPJ Quantum Technology volume8, number1 (year2021), pages25.



[Blekos et al(2024)]
        VQE
authorpersonKostas Blekos, personDean Brand, personAndrea Ceschini, personChiao-Hui Chou, personRui-Hao Li, personKomal Pandya, and personAlessandro Summer. year2024.
A review on quantum approximate optimization algorithm and its variants.
journalPhysics Reports  volume1068 (year2024), pages1–66.



[Cai et al(2023)]
        qem
authorpersonZhenyu Cai, personRyan Babbush, personSimon C. Benjamin, personSuguru Endo, personWilliam J. Huggins, personYing Li, personJarrod R. McClean, and personThomas E. O'Brien. year2023.
Quantum error mitigation.
journalRev. Mod. Phys.  volume95 (dateDec year2023), pages045005.
Issue 4.

<https://doi.org/10.1103/RevModPhys.95.045005>



[Curran-Everett(2008)]
        stderr
authorpersonDouglas Curran-Everett. year2008.
Explorations in statistics: standard deviations and standard errors.
journalAdvances in physiology education volume32, number3 (year2008), pages203–208.



[Dasgupta and Humble(2022)]
        Hellinger2
authorpersonSamudra Dasgupta and personTravis S. Humble. year2022.
Assessing the stability of noisy quantum computation. In booktitleQuantum Communications and Quantum Imaging XX, editorpersonKeith S. Deacon and personRonald E. Meyers (Eds.), Vol. volume12238. International Society for Optics and Photonics, publisherSPIE, pages1223809.

<https://doi.org/10.1117/12.2631809>



[Dirac(1939)]
        dirac
authorpersonPaul Adrien Maurice Dirac. year1939.
A new notation for quantum mechanics. In booktitleMathematical Proceedings of the Cambridge Philosophical Society, Vol. volume35. Cambridge University Press, pages416–418.



[DiVincenzo(1998)]
        basic
authorpersonDavid P DiVincenzo. year1998.
Quantum gates and circuits.
journalProceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences volume454, number1969 (year1998), pages261–276.



[Fisher(1958)]
        fisher
authorpersonWalter D Fisher. year1958.
On grouping for maximum homogeneity.
journalJournal of the American statistical Association volume53, number284 (year1958), pages789–798.



[Fortunato et al(2022)]
        Rui_mutation
authorpersonDaniel Fortunato, personJosé Campos, and personRui Abreu. year2022.
QMutPy: A Mutation Testing Tool for Quantum Algorithms and Applications in Qiskit. In booktitleProceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (Virtual, South Korea) (seriesISSTA 2022). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages797–800.
9781450393799

<https://doi.org/10.1145/3533767.3543296>



[Georgopoulos et al(2021)]
        noisesource
authorpersonKonstantinos Georgopoulos, personClive Emary, and personPaolo Zuliani. year2021.
Modeling and simulating the noisy behavior of near-term quantum computers.
journalPhys. Rev. A  volume104 (dateDec year2021), pages062432.
Issue 6.

<https://doi.org/10.1103/PhysRevA.104.062432>



[Harman et al(2012)]
        HarmanACMsurvery2012
authorpersonMark Harman, personS. Afshin Mansouri, and personYuanyuan Zhang. year2012.
Search-based software engineering: Trends, techniques and applications.
journalACM Comput. Surv. volume45, number1, Article articleno11 (datedec year2012), numpages61 pages.
0360-0300

<https://doi.org/10.1145/2379776.2379787>



[Hidary and Hidary(2021)]
        qc_libraries
authorpersonJack D Hidary and personJack D Hidary. year2021.
Development libraries for quantum computer programming.
journalQuantum Computing: An Applied Approach (year2021), pages67–86.



[Honarvar et al(2020)]
        property
authorpersonShahin Honarvar, personMohammad Reza Mousavi, and personRajagopal Nagarajan. year2020.
Property-Based Testing of Quantum Programs in Q#. In booktitleProceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops (Seoul, Republic of Korea) (seriesICSEW'20). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages430–435.
9781450379632

<https://doi.org/10.1145/3387940.3391459>



[Huang and Martonosi(2019)]
        Huang2019
authorpersonYipeng Huang and personMargaret Martonosi. year2019.
Statistical Assertions for Validating Patterns and Finding Bugs in Quantum Programs. In booktitleProceedings of the 46th International Symposium on Computer Architecture (Phoenix, Arizona) (seriesISCA '19). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages541–553.
9781450366694

<https://doi.org/10.1145/3307650.3322213>



[Jaeger(2007)]
        speed
authorpersonGregg Jaeger. year2007.
Classical and quantum computing.
journalQuantum Information: An Overview (year2007), pages203–217.



[Javadi-Abhari et al(2024)]
        qiskit
authorpersonAli Javadi-Abhari, personMatthew Treinish, personKevin Krsulich, personChristopher J. Wood, personJake Lishman, personJulien Gacon, personSimon Martiel, personPaul D. Nation, personLev S. Bishop, personAndrew W. Cross, personBlake R. Johnson, and personJay M. Gambetta. year2024.
titleQuantum computing with Qiskit.
[arxiv]2405.08810 [quant-ph]

<https://arxiv.org/abs/2405.08810>




[Kurita et al(2023)]
        Paulistrings2
authorpersonTomochika Kurita, personMikio Morita, personHirotaka Oshima, and personShintaro Sato. year2023.
Pauli String Partitioning Algorithm with the Ising Model for Simultaneous Measurements.
journalThe Journal of Physical Chemistry A volume127, number4 (year2023), pages1068–1080.



[LaRose et al(2022)]
        mitiq
authorpersonRyan LaRose, personAndrea Mari, personSarah Kaiser, personPeter J Karalekas, personAndre A Alves, personPiotr Czarnik, personMohamed El Mandouh, personMax H Gordon, personYousef Hindy, personAaron Robertson, et al year2022.
Mitiq: A software package for error mitigation on noisy quantum computers.
journalQuantum  volume6 (year2022), pages774.



[Long and Zhao(2024)]
        LongTOSEM2024
authorpersonPeixun Long and personJianjun Zhao. year2024.
Testing Multi-Subroutine Quantum Programs: From Unit Testing to Integration Testing.
journalACM Trans. Softw. Eng. Methodol. volume33, number6, Article articleno147 (datejun year2024), numpages61 pages.
1049-331X

<https://doi.org/10.1145/3656339>



[Luo et al(2022)]
        bug1
authorpersonJunjie Luo, personPengzhan Zhao, personZhongtao Miao, personShuhan Lan, and personJianjun Zhao. year2022.
A Comprehensive Study of Bug Fixes in Quantum Programs. In booktitle2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). pages1239–1246.

<https://doi.org/10.1109/SANER53432.2022.00147>



[Ma and Kim(2024)]
        peccost
authorpersonYue Ma and personM. S. Kim. year2024.
titleLimitations of probabilistic error cancellation for open dynamics beyond sampling overhead.
[arxiv]2308.01446 [quant-ph]

<https://arxiv.org/abs/2308.01446>




[Mendiluze et al(2021)]
        mutation
authorpersonEñaut Mendiluze, personShaukat Ali, personPaolo Arcaini, and personTao Yue. year2021.
Muskit: A Mutation Analysis Tool for Quantum Software Testing. In booktitle2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). pages1266–1270.

<https://doi.org/10.1109/ASE51524.2021.9678563>



[Miranskyy et al(2020)]
        MiranskyyICSE2020
authorpersonAndriy Miranskyy, personLei Zhang, and personJavad Doliskani. year2020.
Is Your Quantum Program Bug-Free?. In booktitleProceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results (Seoul, South Korea) (seriesICSE-NIER '20). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages29–32.
9781450371261

<https://doi.org/10.1145/3377816.3381731>



[Muqeet et al(2024a)]
        qlearFSE2024
authorpersonAsmar Muqeet, personShaukat Ali, personTao Yue, and personPaolo Arcaini. year2024a.
A Machine Learning-Based Error Mitigation Approach for Reliable Software Development on IBM’s Quantum Computers. In booktitleCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering (Porto de Galinhas, Brazil) (seriesFSE 2024). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages80–91.
9798400706585

<https://doi.org/10.1145/3663529.3663830>



[Muqeet et al(2024b)]
        sourcecode
authorpersonAsmar Muqeet, personAli Shaukat, and personPaolo Arcaini. year2024b.
titleQOPS source code.
howpublished<https://github.com/AsmarMuqeet/QOPS>.



[Muqeet et al(2024c)]
        qoinArxiv
authorpersonAsmar Muqeet, personTao Yue, personShaukat Ali, and personPaolo Arcaini. year2024c.
Mitigating Noise in Quantum Software Testing Using Machine Learning.
journalCoRR  volumeabs/2306.16992 (year2024).

<https://doi.org/10.48550/ARXIV.2306.16992>

[arXiv]2306.16992


[Murillo et al(2024)]
        qse_challenges
authorpersonJuan M. Murillo, personJose Garcia-Alonso, personEnrique Moguel, personJohanna Barzen, personFrank Leymann, personShaukat Ali, personTao Yue, personPaolo Arcaini, personRicardo Pérez, personIgnacio García Rodríguez de Guzmán, personMario Piattini, personAntonio Ruiz-Cortés, personAntonio Brogi, personJianjun Zhao, personAndriy Miranskyy, and personManuel Wimmer. year2024.
titleChallenges of Quantum Software Engineering for the Next Decade: The Road Ahead.
[arxiv]2404.06825 [cs.SE]


[Murphy(2007)]
        commonf3
authorpersonKevin P Murphy. year2007.
Performance evaluation of binary classifiers.
journalTechnical report: Technical Report (year2007).



[North(2009)]
        fisher2
authorpersonMatthew A. North. year2009.
A method for implementing a statistically significant number of data classes in the Jenks algorithm. In booktitleProceedings of the 6th International Conference on Fuzzy Systems and Knowledge Discovery - Volume 1 (Tianjin, China) (seriesFSKD'09). publisherIEEE Press, pages35–38.
9781424445455


[Oldfield et al(2024)]
        phaseflip
authorpersonNoah H. Oldfield, personChristoph Laaber, personTao Yue, and personShaukat Ali. year2024.
titleFaster and Better Quantum Software Testing through Specification Reduction and Projective Measurements.
[arxiv]2405.15450 [cs.SE]

<https://arxiv.org/abs/2405.15450>




[Patel and Tiwari(2021)]
        qraft
authorpersonTirthak Patel and personDevesh Tiwari. year2021.
Qraft: reverse your Quantum circuit and know the correct program output. In booktitleProceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (Virtual, USA) (seriesASPLOS '21). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages443–455.
9781450383172

<https://doi.org/10.1145/3445814.3446743>



[Pontolillo and Mousavi(2023)]
        property2
authorpersonGabriel Pontolillo and personMohammad Reza Mousavi. year2023.
A multi-lingual benchmark for property-based testing of quantum programs. In booktitleProceedings of the 3rd International Workshop on Quantum Software Engineering (Pittsburgh, Pennsylvania) (seriesQ-SE '22). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages1–7.
9781450393355

<https://doi.org/10.1145/3528230.3528395>



[Quetschlich et al(2023)]
        mqt
authorpersonNils Quetschlich, personLukas Burgholzer, and personRobert Wille. year2023.
MQT Bench: Benchmarking Software and Design Automation Tools for Quantum Computing.
journalQuantum (year2023).
MQT Bench is available at <https://www.cda.cit.tum.de/mqtbench/>.


[Reggio et al(2023)]
        fastpart
authorpersonBen Reggio, personNouman Butt, personAndrew Lytle, and personPatrick Draper. year2023.
Fast partitioning of pauli strings into commuting families for optimal expectation value measurements of dense operators.
journalarXiv preprint arXiv:2305.11847 (year2023).



[Resch and Karpuzcu(2021)]
        noise
authorpersonSalonik Resch and personUlya R Karpuzcu. year2021.
Benchmarking quantum computers and the impact of quantum noise.
journalACM Computing Surveys (CSUR) volume54, number7 (year2021), pages1–35.



[Wang et al(2021c)]
        fuzz
authorpersonJiyuan Wang, personFucheng Ma, and personYu Jiang. year2021c.
Poster: Fuzz Testing of Quantum Program.
journalProceedings - 2021 IEEE 14th International Conference on Software Testing, Verification and Validation, ICST 2021 volume2, number1 (year2021), pages466–469.
9781728168364

<https://doi.org/10.1109/ICST49551.2021.00061>



[Wang et al(2021a)]
        Combinatorial
authorpersonXinyi Wang, personPaolo Arcaini, personTao Yue, and personShaukat Ali. year2021a.
Application of Combinatorial Testing to Quantum Programs. In booktitle2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS). pages179–188.

<https://doi.org/10.1109/QRS54544.2021.00029>



[Wang et al(2021b)]
        quitoASE21tool
authorpersonXinyi Wang, personPaolo Arcaini, personTao Yue, and personShaukat Ali. year2021b.
Quito: a Coverage-Guided Test Generator for Quantum Programs. In booktitle2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). pages1237–1241.

<https://doi.org/10.1109/ASE51524.2021.9678798>



[Wang et al(2022a)]
        search
authorpersonXinyi Wang, personPaolo Arcaini, personTao Yue, and personShaukat Ali. year2022a.
QuSBT: Search-Based Testing of Quantum Programs. In booktitleProceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings (Pittsburgh, Pennsylvania) (seriesICSE '22). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages173–177.
9781450392235

<https://doi.org/10.1145/3510454.3516839>



[Wang et al(2022b)]
        mutation2
authorpersonXinyi Wang, personTongxuan Yu, personPaolo Arcaini, personTao Yue, and personShaukat Ali. year2022b.
Mutation-Based Test Generation for Quantum Programs with Multi-Objective Search. In booktitleProceedings of the Genetic and Evolutionary Computation Conference (Boston, Massachusetts) (seriesGECCO '22). publisherAssociation for Computing Machinery, addressNew York, NY, USA, pages1345–1353.
9781450392372

<https://doi.org/10.1145/3512290.3528869>



[Wohlin et al(2012)]
        Wohlin2012
authorpersonClaes Wohlin, personPer Runeson, personMartin Hst, personMagnus C. Ohlsson, personBjrn Regnell, and personAnders Wessln. year2012.
booktitleExperimentation in Software Engineering.
publisherSpringer Publishing Company, Incorporated.
3642290434, 9783642290435


[Yanofsky and Mannucci(2008)]
        Basics
authorpersonNoson S Yanofsky and personMirco A Mannucci. year2008.
booktitleQuantum computing for computer scientists.
publisherCambridge University Press.



[Ye et al(2023)]
        quraTestASE23
authorpersonJiaming Ye, personShangzhou Xia, personFuyuan Zhang, personPaolo Arcaini, personLei Ma, personJianjun Zhao, and personFuyuki Ishikawa. year2023.
QuraTest: Integrating Quantum Specific Features in Quantum Program Testing. In booktitle2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). pages1149–1161.

<https://doi.org/10.1109/ASE56229.2023.00196>



[Zhao et al(2021a)]
        bug2
authorpersonPengzhan Zhao, personJianjun Zhao, and personLei Ma. year2021a.
Identifying Bug Patterns in Quantum Programs. In booktitle2021 IEEE/ACM 2nd International Workshop on Quantum Software Engineering (Q-SE). pages16–21.

<https://doi.org/10.1109/Q-SE52541.2021.00011>



[Zhao et al(2021b)]
        bug3
authorpersonPengzhan Zhao, personJianjun Zhao, personZhongtao Miao, and personShuhan Lan. year2021b.
Bugs4Q: A Benchmark of Real Bugs for Quantum Programs. In booktitle2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). pages1373–1376.

<https://doi.org/10.1109/ASE51524.2021.9678908>









 Paper Title: Visual Analysis of GitHub Issues to Gain Insights.







Visual Analysis of GitHub Issues to Gain Insights

    Rifat Ara Proma
Scientific Computing and Imaging Institute 

University of Utah

Salt Lake City, UT, USA 

proma@sci.utah.edu
Paul Rosen
Scientific Computing and Imaging Institute 

University of Utah

Salt Lake City, UT, USA 

prosen@sci.utah.edu

    August 5, 2024
============================================================================================================================================================================================================================================================



§ ABSTRACT 
Version control systems are integral to software development, with GitHub emerging as a popular online platform due to its comprehensive project management tools, including issue tracking and pull requests. However, GitHub lacks a direct link between issues and commits, making it difficult for developers to understand how specific issues are resolved. Although GitHub's Insights page provides some visualization for repository data, the representation of issues and commits related data in a textual format hampers quick evaluation of issue management. This paper presents a prototype web application that generates visualizations to offer insights into issue timelines and reveals different factors related to issues. It focuses on the lifecycle of issues and depicts vital information to enhance users' understanding of development patterns in their projects. We demonstrate the effectiveness of our approach through case studies involving three open-source GitHub repositories. Furthermore, we conducted a user evaluation to validate the efficacy of our prototype in conveying crucial repository information more efficiently and rapidly.
Video URL : 



GitHub mining, issue tracking, source code, visual analytics





§ INTRODUCTION





Collaboration between developers is one of the most critical tasks in software development. Distributed version control systems, such as Git, allow programmers to store source code, collaborate, and track their work. GitHub is one of the most widely used online Git-based platforms, with over 100 million developers and more than 420 million repositories as of February 2024 <cit.>. GitHub offers several useful functionalities, e.g., issue tracking, pull requests, etc., that make the collaborative development process more accessible. Issue tracking, in particular, helps keep track of work done, what work needs to be done, and who is working on what. It is used for several purposes, including reporting bugs, feature requests, and enhancements.

There are many existing work in the field of GitHub data mining with many different applications, including predicting topics for repositories <cit.>, understanding user behaviors <cit.>, predicting defects in softwares <cit.>. Many of the existing works have also proposed using data visualization to enhance different aspects of GitHub <cit.>. In this paper, we have identified a problem with the current GitHub interface and attempted to solve it using data visualization.

Despite GitHub's numerous functionalities, including an insights page that utilizes visualizations to provide insights about the repository, its presentation of information regarding issues, commits, and other activities is primarily textual. This format can limit the ability to discern patterns and extract deeper insights. Data visualization effectively conveys the meaning of additional data through charts. Visualizing GitHub issues and commits makes it possible to extract meaningful information from them and use it to analyze patterns and problems in software projects. For instance, visual representations can highlight trends in issue resolution times, reveal which components of the software are most prone to bugs, and track the contributions of individual developers more clearly.

This paper presents a prototype web application that shows GitHub issue data through several visualizations. Each one offers different types of insights into the repository issues and related commits, accomplishing tasks that include:


  * Facilitating programmers in understanding the duration of an issue, both in terms of how long it has been open and the time it took to resolve;

  * Gaining insights regarding different types of issues based on labels assigned;

  * Narrowing which commits are responsible for resolving an issue;

  * Checking which files were updated to resolve issues; and

  * Determining which files are getting more updates to resolve issues.


Our approach aimed to enable developers to identify bottlenecks in the development process quickly, recognize high-risk areas of the codebase and understand the dynamics of team contributions. In turn, supporting more strategic planning and more effective issue management.

We collected data about three public repositories and analyzed them using our application to evaluate our approach. Additionally, we conducted a small-scale expert user study involving four participants that validated the efficiency of the proposed solution. The study involved developers using our visualizations to answer some questions about the repositories, and the feedback we received indicated that our tool significantly enhanced their ability to interpret and act upon the data from GitHub. This validation emphasizes the practical benefits of integrating data visualization into version control systems, making it a promising direction for further research and development.













§ RELATED WORK

Considering the expanding popularity of GitHub and its open API, several existing research studies have delved into analyzing different aspects of it to enhance different purposes. This section discusses prior work regarding analyzing GitHub projects and using data visualizations to enhance some aspects.



 §.§ Analysis of GitHub Projects

A lot of existing research collects data about GitHub repositories and uses it to enhance user experience. Zhou et al. <cit.> presented a GHTRec service that predicts topics for GitHub repositories and then gives users personalized recommendations for trending repositories. Analyzing GitHub data can also help to understand user behavior and communication patterns among collaborators <cit.>. It can also enable detecting defective data <cit.> and factors that impact project success <cit.>. Such approaches can be useful for software development and testing teams to shorten the development life cycle. 
 
Analyzing GitHub data can potentially improve programming languages like Java. Lemay <cit.> inspected publicly available Java source code that could infer usability issues with the Java language. Zhang et al. <cit.> also studied 9,476 competing pull requests from 60 Java repositories on GitHub to analyze how multiple pull requests update the same code. 

Researchers have also explored machine learning-based approaches to enhance GitHub. Montandon et al. <cit.> proposed a method to automatically recognize the technical role of open source developers, which can help to form cross-functional teams. Ticket Tagger <cit.> also used machine learning techniques to evaluate issue titles and descriptions and automatically assign labels to each issue. However, none of these existing works explored utilizing data visualization to enhance different aspects of GitHub.




 §.§ Visualization of GitHub Projects

Some existing research has also explored visualization of data retrieved from GitHub to achieve different objectives. Dubey et al. <cit.> visualized repository parameters collected from GitHub to identify the most popular domains and languages on GitHub. Celinska et al. <cit.> proposed a visualization with a weighted network of programming languages used on GitHub to help developers choose programming languages for their applications. Visualizations have also been utilized in tools like GiLA <cit.> to categorize issues based on labels and analyze developer locations and the technology they use <cit.>. 


A web-based tool, ChangeViz, was proposed by Gasparini et al. <cit.> to help developers better understand GitHub pull requests. The tool combines the existing interface of GitHub with the visualization approach of STACKSPLORER <cit.>. Another visualization tool proposed by Højelse et al. <cit.> offered multiple views for analyzing the evolution of hierarchically organized Git repositories. MyGitIssues <cit.> is a web application that lists issues of specific GitHub repositories using data visualization approaches. It makes managing different repositories easier with the support of technologies such as WebSockets and WebHooks. Fiechter et al. <cit.> also worked on presenting issues through visualization by introducing an approach called issue tales, which is an interactive visual analytics tool. It allows for analyzing all events and actors related to any GitHub issue, such as related commits, pull requests, related issues, etc. However, it does not provide any way to get an overall overview of different issue types and their timelines. The issue-related behavior of GitHub was also investigated via visualization techniques by Liao et al. <cit.>. Their analysis emphasized the importance of issue labeling and issue-related behavior analysis for better issue-handling practices. They mentioned that the time distribution of issue commits maintains a development model that roughly corresponds to the project's life cycle. 


Numerous studies have been conducted on GitHub data mining and visualization, particularly in understanding GitHub issues. However, existing approaches lack an interface that provides an overview of multiple issues at a glance and offers interactive capabilities for identifying related commits and file changes. This restricts the identification of trends within a repository. We intended to fill this gap by developing interactive visualization interfaces that empower developers to gain deeper insights into issue trends and explore their repositories more effectively. This paper proposes methods for visualizing GitHub issues to enhance developers' understanding of issues and commit management within their repositories.
















§ OUR APPROACH


To address the gap in the available tools for evaluating GitHub issues, we developed a web-based analysis tool, using D3.js <cit.>, to generate visualizations. The code for the prototype web application can be accessed at [<https://github.com/RifatAraProma/GitHubDataViz>]. The web application can also be accessed at [<https://github-issue-analyzer-d967eeb35538.herokuapp.com/>].







 §.§ Design Considerations


This work aimed to create a useful resource that enables developers to evaluate their repositories better and comprehend development trends. Initially, we identified that the current GitHub issue does not allow the exploration of issues in a visual format and proposed a series of visualizations that can enhance the process. To ensure the proposed visualization tool enables developers to analyze issues and related commits in a repository, we interviewed two software engineers to understand their requirements. The interview was open-ended, and we asked them how they utilize the current GitHub interface to manage their repositories. Specifically, we asked them how they identify issues that have been open for a long time, identify what resolved a particular issue, which files were changed during the process, and what will help them perform these tasks efficiently. After that, we showed them some of our designed visualization solutions and asked for their feedback. These interviews played a crucial role in shaping the design of our visualization solution. The software engineers actively participated throughout the design process, providing valuable insights and feedback. By incorporating their expertise and feedback, we ensured that the resulting visualization tool aligned with their needs and preferences. This collaborative approach allowed us to create a tool that facilitates the extraction of meaningful insights regarding issues and their associated commits within a repository.

We aimed to design a collection of interactive interfaces that enable the visualization of information about issues, commits, and updated files, addressing the following tasks:


    
  * Displaying the timeline of issues in a repository, showcasing their status and duration in each state.
    
  * Providing a clearer understanding of the types of issues present in a repository.
    
  * Visualizing various aspects of an issue concisely and empowering developers to identify the commits associated with a specific issue and the files that were modified during the resolution process.
    
  * Offering visualizations that facilitate comprehension of file update frequencies within a repository, specifically focusing on bug fixes. 


Throughout the remainder of the paper, we have specified when a software feature or evaluation supports one of these tasks by denoting the task identifier inside parentheses. By addressing these objectives, our visualization tool seeks to enhance the effectiveness and efficiency of repository analysis.









 §.§ Data Collection

We developed a custom Python script for this project to extract relevant data from a GitHub repository. The script retrieved information about recent activities, including issue details such as title, assignees, labels, opening and closing times, creator and closer users, commit messages, commit timestamps, updated files, and commit authors. Specifically, the script focused on designated repositories.

We employed the GitHub API to fetch information about the 100 most recent issues. Subsequently, we filtered out any issues identified as pull requests, as they were irrelevant to our analysis. In addition, we utilized the GitHub API to access the Commits page on the GitHub user interface, allowing us to retrieve information about all commits present in the Newer tab of the page. The Python script leveraged the API to retrieve and store the desired data in a CSV file. This data serves as the foundation for generating visualizations at a later stage.





 §.§ Interfaces

The tool we constructed consists of 3 main views. Initially, it presents users with the option to select a repository. Upon selection, a Timeline View is displayed (see <ref>), showcasing recent issues along with their corresponding status (open or closed). Users can use a drop-down button to switch the view and display labels associated with the issues to enhance flexibility. By clicking on a bar within the Timeline View, users can investigate the associated issue within the Issue Graph (see <ref>). Finally, a button located in the top left corner allows users to switch to the Summary of Updated Files view (see <ref>). In the subsequent sections, we provide a comprehensive discussion of each of these interfaces, elucidating their functionalities and features.





  §.§.§ Timeline View (see <ref> and <ref>)

To visualize the duration of each GitHub issue, a Gantt chart was created due to their effectiveness in showing task schedule <cit.> (T0), as depicted in <ref> and <ref>. Each issue is represented by a bar, where the horizontal axis corresponded to the date, and the width of the bar represents the number of days the issue remained open. Open issues are displayed in purple color, while closed issues are indicated by green, consistent with GitHub's default color scheme. Open issues extend to the rightmost section of the chart, indicating their ongoing status, and are represented by an arrow to signify continuity. Hovering the mouse over an issue triggers a tooltip that displays the issue title, creation date, closing date, and labels associated with it, providing additional contextual information.



The Timeline View also incorporates a feature allowing the color-coded display of issue labels (T1), as shown in <ref>. One can switch to this view using a drop-down menu on the top left. The colors and names of these labels are specific to the repository being analyzed. The color encoding in this view varies across repositories as shown in <ref>, <ref>, and <ref>, as the exact colors associated with the labels in each repository are utilized. In case an issue does not have any label, “steelblue” color is assigned to it, and the legends mark the color as “no label". It is possible that a certain repository uses the same color for multiple labels, as shown in <ref>. Both  and  use the same purple color. To distinguish two issues having different labels but the same color associated with them, we added labels in the tooltip. In cases where an issue has multiple labels, alternating striped patterns were used to represent the colors of each label. 





  §.§.§ Issue Graph (see <ref>)

Upon clicking the bars of the Timeline View, a force-directed graph is generated, displaying properties associated with the issue (T2), as depicted in <ref>. The reason behind choosing force-directed graphs for this purpose was their aesthetic <cit.> and ability to portray the connection between issues and related factors in a natural manner. However, GitHub does not automatically establish connections between commits or pull requests and issues unless they were manually linked or specific keywords were utilized in the pull request to close a particular issue <cit.>. Developers often do not utilize these keywords, or manually link pull requests to issues. Consequently, to identify the commits that contributed to the resolution of a specific issue, the search scope is narrowed down to include only the commits made between the opening and closing dates of the issue. Additionally, information such as the issue creator, assignee, and closer is provided in the graph. For each associated commit, nodes are added to represent the files modified in that commit, along with a node representing the commit creator. The color of the issue node remains consistent with the issue colors employed in the Timeline View, reflecting the issue's status. Other colors employed in the graph were selected using ColorBrewer <cit.>, with the intention of being able to distinguish different types of nodes. <ref> displays the Issue Graph representing an open issue, as denoted by the purple-colored node. We included the “closed by" node in the graph for illustrative purposes, although it does not apply to an open issue. Similarly, the absence of an assignee for this particular issue prompted us to add the corresponding node, demonstrating how the graph would appear if assignees were present. It is important to note that aside from these two nodes, all other chart elements accurately reflect real data.



Upon hovering on each node, a tooltip reveals its details, such as the name of the users, files, commit messages, and issue title. Notably, certain edges linking the issue to the commits are assigned a red color, in line with GitHub's color scheme for bugs. These specific commits are deemed bug fixes. It is customary to use the keyword “fix" to denote a commit that addresses a bug. Therefore, commits are considered related to bug fixes by checking if commit messages include the keyword. The decision to employ a force-directed graph was motivated by the desire to facilitate the formation of natural clusters during issue evaluation. This graphical representation also allowed users to interact with the graph by dragging nodes, enabling them to explore different parts of the graph at their discretion.










  §.§.§ Summary of Updated Files (see <ref>)

From the issue Timeline View, one can switch to a view showing Summary of Updated Files that contains a donut chart that shows how many lines of the files of the repository were updated in the process of resolving issues (T3), as shown in <ref>(left). We chose donut charts as they enable viewers to identify proportions quickly <cit.>. The chart shows the top 5 files with the most updated lines in the donut chart by default. One portion of the donut chart shows the total number of updated lines of all the other files except those displayed explicitly. This helps to understand the extent of the updates of a particular file. There are filtering options as shown in <ref>(right), using which one can add or remove specific files and analyze how that changes the donut chart. There is also an option to see files that are updated during bug fixes only, enabling users to identify files that are more updated to fix bugs. Upon hovering on each wedge, a tooltip appears with the name of the file and the center of the donut shows how many lines are updated in that file. The colors used for this chart were chosen using ColorBrewer <cit.>.



A histogram that shows the number of files updated, based upon the number of lines modified, as shown in <ref>(middle). The horizontal axis of the histogram represents the number of files, and the vertical axis represents the update frequency range on a logarithmic scale. The histogram also lets users interact with the donut chart. Depending on which bar of the histogram is clicked, the donut chart is updated to show files from that update frequency range, as shown in <ref>. This functionality is added to give users the flexibility to explore files of specific update frequencies. The color chosen for the histogram is consistent with the color used on GitHub's Insights page for showing commit frequencies.














































§ CASE STUDIES


To validate the effectiveness of our designed visualizations, we conducted case studies on three public repositories sourced from GitHub: freeCodeCamp by  <cit.>, Hyprland by Hypr Development <cit.>, and javascript by  <cit.>. The freeCodeCamp repository belongs to an online community that supports individuals in learning to code, while the Hyprland repository is associated with a dynamic tiling Wayland compositor. On the other hand, the javascript repository is maintained by  and serves as a JavaScript styling guide. These repositories exhibited variations in terms of programming language, number of contributors and users, and repository management practices. Thus, the case studies served to underscore the efficacy of the proposed visualization solutions across diverse scenarios.





 §.§ freeCodeCamp

Once the repository data was loaded in the proposed interface, the first page of the application (see <ref>) showed the issues of the repository in the Timeline View (T0). It could be noticed that the longest open issue, in purple,  was open for more than 17 days. The longest closed issue, in green,  took 5 days to be resolved (T0).

After switching the Timeline View to show labels of the issues (see <ref>), it could be noticed that the majority of the issues have purple color in it indicating most of the issues were either bug reports or feature requests (T1). Upon further inspection of the tooltips, it was determined that there were 7 bug-type issues and 8 feature requests. The  issue that took the longest to get resolved was a bug-type issue. The label also indicated that it required help to solve, which could be a reason why it took longer to fix.



Upon clicking the bars of the Timeline View, the Issue Graph appears. When the bar for  issue was clicked, the graph view shown in <ref> appeared. It can be seen that only two commits were made during the time period of the clicked issue, and both of them contain “fix" keyword in the message (T2). Further inspection of commit messages revealed that  commit potentially fixed the issue. We verified that this commit fixed the issue by manually checking it on the GitHub page. The Issue Graph also shows that the commit modified  file and was created by .











<ref> shows the Issue Graph for the issue that was open for the longest time. The graph pointed out that 25 commits were made since the day the issue was opened. After checking the commit message of all of these commits, it seemed like none of these commits were related to the issue. In the Timeline View showing labels as shown in <ref>, it showed that the issue was a feature request which is still being discussed. This gives a clearer idea of why the issue was open for a long time. Another interesting aspect of Issue Graph was that it seemed like there were multiple active developers in the repository involved in multiple commits. It was also easier to identify which commits made modifications to multiple files from this graph.



Upon selecting the button to transition to the Summary of Updated Files view, it becomes evident that the file with the highest count of updated lines is the   file (T3), as depicted in <ref>(left). This observation aligns with expectations, considering that the freeCodeCamp project remains active and frequently receives updates to its packages, resulting in newer versions being incorporated. The histogram presented in <ref>(middle) reveals that most files experienced minor changes, with 2-3 lines of code being updated. This pattern suggests that most files within the repository underwent relatively small modifications.





 §.§ Hyprland


For the Hyprland repository, the temporal aspects of issues are visually represented in <ref>. This view reveals a noticeable presence of recently opened issues within the repository. Notably, the issue titled  remained open for the longest duration, spanning more than 28 days (T0). Upon switching the Timeline View to display labels, as demonstrated in <ref>, it became apparent that all the issues in the repository were assigned to a single label, distinguishing it from the freeCodeCamp repository. It is noteworthy that this repository predominantly utilizes default GitHub labels. Moreover, a significant proportion of the issues pertained to bug reports (T1).



The bug report titled  stood out as the most time-consuming issue to be resolved. Upon selecting the corresponding bar in the Timeline View, it's Issue Graph was displayed, as illustrated in <ref>. The graph provides an overview of the issue lifecycle, revealing a total of 25 commits that were made during the lifecycle. Notably, the user  was attributed to 21 of these commits. Upon examining the commit messages, one specific commit with the message  emerged as a potential fix for the issue (T2). To validate its significance, verification was performed using GitHub's user interface, confirming its role in addressing the issue. The graph further highlighted that the commit exclusively modified the  file and was authored by the user . Particularly, the graph emphasized the substantial contribution made by the user , suggesting their significant involvement as one of the main contributors to the repository.







The Summary of Updated Files view for the Hyprland repository is showcased in <ref>. Among the files, the  file experienced the highest activity level, with 23 lines of code being updated, making it the most updated file within the repository (T3). Additionally, when considering all the remaining files collectively, a total of 30 lines underwent modifications. By applying the filter to unveil files updated during bug-fixing commits, the resulting donut chart illustrated in <ref> revealed that the file named  underwent updates on 19 lines during bug-fix related commits.





 §.§ Javascript


The Timeline View of the javascript repository is presented in <ref>. From this view, it is evident that several recent issues were resolved within a short timeframe (T0). Upon switching to the label view, it was observed that a significant number of these issues were labeled as  (T1). Further investigation through tooltips revealed that these issues were flagged as spam. Out of the total 50 issues, 28 were marked as spam, indicating a recent influx of spam-related challenges in the repository. The label view in <ref> also highlights that none of these issues were bug reports, and a considerable portion of them was left unassigned.

The Summary of Updated Files view, depicted in <ref>, highlights the file  as the most frequently modified file, with 232 lines updated. The green section in the figure represents the lines updated in all other files, excluding the top five most updated files, which accounted for 46 lines. The notable disparity between  and the remaining files suggests that the repository primarily focused on documentation updates (T3). The accompanying histogram in <ref> indicates that most files in the repository underwent minor changes, with 1-2 lines updated. However, two files experienced a more substantial number of modifications, with a range of 100-232 lines updated.







































§ USER EVALUATION

To evaluate the effectiveness of the proposed visualizations achieving the tasks described in <ref> compared to the existing GitHub interface, we conducted a small-scale user evaluation involving 4 participants, including two professional software developers and two graduate students majoring in computer science and engineering. Among the software developers, one had prior involvement in the development process, while the other had no prior exposure to the tool before the evaluation. All participants had prior experience using GitHub. In terms of measuring effectiveness, we mainly focused on the time participants took to answer certain questions using the proposed visualizations and the existing GitHub UI.





 §.§ Evaluation Design

We conducted online interviews with the participants, giving them a brief overview of the study's purpose. They were also given a demonstration on how to use the proposed web tool on a repository different from the one on which they were assigned tasks. Additionally, we showcased essential features of the actual GitHub user interface (UI), such as sorting issues based on different parameters, checking labels, accessing information about specific issues and commit pages of a repository, and retrieving details regarding file updates. Once the participant felt comfortable using our proposed web tool and the GitHub UI, we proceeded with the evaluation.



For the evaluation, participants were tasked with answering a set of questions for a given repository using the proposed interface and answering the same questions for a different repository using the GitHub UI. This approach was designed to minimize bias and facilitate the comparison of learning patterns. To assess performance, we measured participants' time to answer each question using a timer for both the proposed interface and the GitHub UI. The timer was started once the question was asked and stopped once participants found an answer.



For the evaluation process, we chose two repositories, namely freeCodeCamp and Hyprland. Two participants interacted with the freeCodeCamp repository using our proposed interface, while they used the GitHub UI for the Hyprland repository. Conversely, the other two participants swapped repositories, using the proposed interface for Hyprland and the GitHub UI for freeCodeCamp. Participants were asked a series of questions designed to measure the effectiveness of the proposed interface to accomplish the tasks defined in <ref>. The questions are:

    
        
  * Which closed issue took the longest time to get resolved? Who opened it? Who closed it?
        
  * Which issue has been open for the longest time?
        
  * Which labeled issue is the majority in this repository?
        
  * Which bug-related issue took the longest to get resolved? Who opened it? Who closed it? Which commit resolved it?
        
  * Which file has been updated the most?
        
  * Which file has been updated the most for bug fixes?
    





 §.§ Results


 The time taken by the participants to answer the questions is shown in <ref>. The missing values for GitHub in <ref>, <ref>, <ref>, and <ref> denote that the participant reported that answering the question using the interface will take a significant amount of time, and thus they could not provide an answer. However, the questions that were answered were correct. For Q1 and Q4, the participants reported that the GitHub UI does not display the duration it took to close an issue. Therefore, they would need to calculate each issue manually. Additionally, GitHub utilizes terms like “last week," “two weeks ago," etc., to indicate when a specific issue was opened or closed, making it challenging to determine the exact duration. For Q5 and Q6, participants had to manually inspect each file in the repository to determine the number of updated lines, requiring meticulous file tracking. This cumbersome process makes it difficult to provide answers within a reasonable time frame. However, for Q2 and Q3, the participants, on average, required less time when using the proposed interface than the GitHub UI.




            
            
            





§ CONCLUSION & FUTURE WORK


This paper introduces a set of visualizations designed to facilitate the analysis of repositories by programmers, specifically focusing on issues and their corresponding commits. The objective is to provide programmers with a straightforward means of deriving meaningful insights to enhance project quality.

Through our case study and user evaluation, we have observed that these visualizations effectively present information, enabling users to gain valuable insights quickly. One notable feature of our approach is the ability to identify files that have been updated to address bug fixes. This functionality can be expanded to encompass updates on all labels within a repository. Furthermore, while our current system detects updates at the file level, incorporating more advanced logic to identify updates at the function or code level would allow for a deeper understanding of the relationships between these elements and issues. 

In our future research, we intend to analyze comments associated with issues to extract additional valuable insights. These insights will be presented through visualizations, providing a more efficient and expedient means of exploration. It is worth noting that GitHub issues have the potential to be reopened, and our current approach does not account for this scenario in identifying or detecting reopened issues. Furthermore, we acknowledge that the current interface lacks the ability for users to view issues within a specific time interval. Incorporating these functionalities and validating them with a more extensive larger scale user evaluation is an important aspect of our future plans.

Involving professional software developers in the design and evaluation process demonstrates the usability of our proposed approach in the software development field. Incorporating their input allowed us to refine and tailor the design to meet their specific needs. We strongly believe that this tool holds great potential to assist the programming community in gaining a comprehensive understanding of their projects and identifying areas for improvement. 

High-resolution versions of all the figures in this paper can be accessed at: .





IEEEtran









 Paper Title: Interactive Diagrams for Software Documentation.








Interactive Diagrams for Software Documentation
    
    
        Adam Štěpánek10009-0008-9388-2546, 
        David Kuťák1,0000-0002-4346-6850 
        Barbora Kozlíková10000-0003-0045-0872, 
        Jan Byška120000-0001-9483-7562
    
        1Masaryk University, Faculty of Informatics, Brno, Czech Republic
    
        2University of Bergen, Department of Informatics, Bergen, Norway

    August 5, 2024
====================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Getting acquainted with a large codebase can be a daunting task for software developers, both new and seasoned.
The description of a codebase and its development should be the purpose of its documentation.
However, software documentation, if it exists at all, is usually textual and accompanied only by simple static diagrams.
It is also time-consuming to maintain manually.
Even an API reference, which can be generated automatically from the codebase itself, has many drawbacks.
It is limited to what it can extract from the codebase, is cumbersome to navigate, and fails to capture the interwoven nature of code.
We explore an alternative approach centered around a node-link diagram representing the structure of a codebase.
The diagram is interactive and filterable, providing details on demand.
It is designed for automation, relying on static analysis of the codebase, and thus produces results quickly and offers a viable alternative to missing or outdated documentation.
To evaluate this approach, we implemented a prototype named Helveg that is able to analyze and visualize C# code.
Testing with five professional programmers provided feedback on the approach's benefits and challenges, which we discuss in detail.





software visualization, software documentation, API reference, code navigation, interactive diagram








§ INTRODUCTION












Nearly every piece of software eventually reaches a point when it should be documented.
This is because technical documentation is the most sought-after learning resource <cit.>.
Well-written documentation is the hallmark of good open-source projects and can be critical for their adoption <cit.>.
Unfortunately, real-world documentation is often incomplete or confusing <cit.> due to time constraints.


Nowadays, software documentation is typically written in a descriptive textual form and is hosted on the web to be easily accessible.
The text is frequently enhanced with code snippets and occasionally with simple static charts and diagrams.
Developers often use automatic generators to extract special comments and metadata directly from the source code, reducing the hassle of writing documentation.
Such generators typically output a page or an entry for every significant code element (e.g., a type or a function).
The documentation generated this way is often collectively described as an API reference.
The automatic creation of an API reference saves precious development time and makes it easier to keep documentation up-to-date with the code.
However, an API reference has many drawbacks.
It is limited to code metadata and the contents of documentation comments.
These comments are written manually by the developers and, thus, are often omitted.
The navigation of an API reference usually relies on hyperlinks, full-text search, and a table of contents mimicking the module hierarchy.
Thus, this limited navigability makes it cumbersome to look for specific arrangements of code elements like types bound together through composition.
Therefore, authors often use multiple writing methods in practice, combining their advantages.
They write manually about key concepts spanning multiple code elements and delegate the details of each code element to the automatic API reference generators.


While documentation can benefit greatly from using visualization <cit.>, its use is currently limited.
It is usually confined to diagrams that describe the project's architecture, explain its type hierarchies, or compare the project to its competitors.
While these visualizations are undoubtedly helpful, they are seldom interactive, are often created manually, and usually act only as a support for their surrounding text.
Therefore, they make writing documentation even more demanding because authors need knowledge of a graphic editor or a tool like GraphViz <cit.>.
Even diagrams, such as those generated by Doxygen <cit.>, are only static images and thus do not utilize the interactive potential of digital platforms.


To see the challenges with software documentation, consider a scenario where a new developer is assigned to a project. 
Navigating the existing codebase can be challenging without well-maintained documentation or the help of a senior colleague—both of which are frequently unavailable.
Often, the newcomers are left only with an API reference that offers a detailed view of each code piece but not a high-level overview of the entire project.
As the newcomers are unfamiliar with the codebase, they cannot effectively utilize full-text search.
Consequently, they are left to browse the entire documentation, hoping to stumble upon the information they need.
This process is inefficient and often frustrating.
Alternatively, they can generate a static diagram showing the whole project.
However, these diagrams typically result in unreadable hairballs or, conversely, omit critical details.


In this paper, we explore an alternative to a classical API reference.
One where the documentation can still be generated automatically but is navigated through an interactive node-link diagram.
This diagram enables a high-level, exploratory analysis of a codebase without trudging through its implementation details.
However, it is also flexible enough to provide these details on demand through filtering and interaction.


To test this concept, we built a prototype tool called Helveg.
Helveg can analyze a codebase written in the C# language and automatically output a web application with an interactive diagram representing the structure of the codebase.




In summary, the main contributions of this paper are:


    
  * An alternative concept to API references for exploring codebases. The approach is based on an interactive diagram providing a high-level overview as well as details on demand.
    
  * An open-source prototype implementation of the concept and a data miner for C# codebases. Their source code is available at <https://gitlab.com/helveg/helveg>.
    
  * An evaluation of the effectiveness of the presented concept with professional C# developers.












§ BACKGROUND AND RELATED WORK







Documentation is connected to the topic of program comprehension, which many software visualization (SV) methods aim to improve.
These methods often focus on visualizing the structure of a program's source code, the program's behavior at runtime, or its evolution.
Since this paper proposes a documentation approach that concerns itself only with the static structure of a codebase, we further delve into techniques that do the same.
We also compare many existing open-source and commercial programs that enhance software documentation or visualize code structure.
Regarding SV as a whole, we refer to literature reviews by Chotisarn et al. <cit.> and Bedu et al. <cit.>.



 §.§ Visualization of Code Structure








A simple way of visualizing a large amount of source code is to scale it down.
This is how, in the code-map metaphor <cit.>, code lines become colored, elongated rectangles.
It can be used to visualize length, age, programming language, and other code file metrics.
SeeSoft by Eick et al. <cit.> is a well-known example of this method.
This technique continues to be researched in recent years, as proven by CodePanorama by Etter and Mehta <cit.>, and it is even included in the popular Visual Studio Code editor as its Minimap feature <cit.>.

A wholly different class of code structure SV methods is based on real-world metaphors.
One such metaphor is the software city, in which codebases transform into 3D cities, where buildings represent units of code, such as classes, and their dimensions and colors correspond to code metrics.
The idea has given rise to diverse implementations such as Software World <cit.>, CodeCity <cit.>, CodeMetropolis <cit.>, and BabiaXR <cit.>.
The city metaphor is further extended to islands populated by cities in IslandViz by Schreiber and Misiak <cit.>.
While the metaphors are intuitive, they frequently omit relationships between code elements <cit.> and suffer from occlusion issues common to all 3D visualizations <cit.>.

Apart from code maps and software cities, there is a plethora of 2D charts and diagrams.
These are designed for diverse purposes, spanning from visualization of software architecture and dependencies between modules to reverse engineering and optimization.
To name a few, Lanza and Ducasse <cit.> presented polymetric views, a system with multiple views capable of displaying diverse metrics.
Gutwenger et al. <cit.> displayed UML class diagrams in a new way involving the minimization of crossings, orthogonal edges, uniform edge direction within type hierarchies, and other desirable criteria.
In E-Quality by Erdemir et al. <cit.>, the codebase is displayed as a node-link diagram, where nodes are glyphs representing quality metrics.
Müller and Zeckzer <cit.> also take a glyph-based approach, in which the structure and relations in a codebase become a circular treemap.
There are also methods utilizing rectangular and circular treemaps, like Git-Truck <cit.> by Højelse et al.

A common problem all of these approaches must tackle is scalability <cit.>.
Even relatively small software projects involve a large amount of data.
Thus, visualizations must be equipped with some kind of filtering or aggregating mechanism to be readable.
The problem with scalability also applies to performance, as SV tool makers have to consider the complexity of their analysis and rendering algorithms.

To summarize, SV research in the area of code structure and software architecture offers a diverse group of techniques.
Our attempt shares some of the challenges with these techniques.
The scalability issue, in particular, is one we encounter as well.
However, we also face additional challenges stemming from our goal to offer a better alternative to API references.



 §.§ Existing Tools

























Many existing tools can be used to visualize the structure of source code or software architecture.
In this section, we mention several examples and group them based on features relevant for use in software documentation.
For a more extensive comparison, see Supplementary Material <cit.>.

Conformance to standards.
Architecture-visualizing tools typically produce diagrams that adhere to an existing standard, most commonly to UML class diagrams <cit.>.
The advantage of this approach is that users are often already familiar with how the data is visualized.
Many tools can generate UML class diagrams from source code, such as the UML Diagrams plugin <cit.> for JetBrains IDEs and Visual Paradigm <cit.>.
Apart from UML, there is also the less strict C4 model <cit.> along with the Structurizr tool <cit.>.
However, both UML and C4 were designed with static images in mind that can be, for instance, drawn on a whiteboard or printed.
These tools usually do not take advantage of the potential for interactivity present in digital media, like the ability to expand and collapse nodes or provide detailed metadata on demand.
Therefore, numerous architectural visualizers do not follow any standard, allowing them to include more language-specific details.
These include tools like the Visual Studio Code Map <cit.> (unrelated to the code-map metaphor) and all tools employing the city metaphor or a similar 3D approach.

Code analysis / Automation.
A significant distinguishing factor between the tools is whether they perform any code analysis.
In the context of this paper, we define code analysis as any data mining or processing step involving language parsing and semantic analysis.
For example, tools like Structurizr <cit.> and D2 <cit.> are text-to-diagram utilities, turning a manually written description into a corresponding diagram.
On the other hand, NDepend <cit.> and JArchitect <cit.> measure code quality and thus, apart from visualizing a codebase, also heavily analyze it.
Similarly, API reference generators, such as Doxygen <cit.> and Docfx <cit.>, can analyze a software project and generate static diagrams depicting inheritance hierarchies and attach them to relevant pages.
From the user's perspective, code analysis is the difference between describing a codebase manually and simply passing a path to a tool as an argument.
Tools performing code analysis can run automatically, and automation is a key requirement for providing an alternative to API references.

Interactivity.
While a static diagram of a small codebase can be readable, it often becomes cluttered as the codebase grows.
This issue can be largely avoided by providing details-on-demand features—letting the user interact with the visualization and choose what they want to focus on <cit.>.
For instance, the Emerge tool <cit.> features context menus, filtering, highlighting, and a force-directed graph layout algorithm.
However, Emerge focuses more on code quality than documentation and, for example, disregards documentation comments.

Web support.
Any alternative to an API reference must be ready for the web environment.
This is because software documentation tends to be hosted online, where anyone can access it.
Although many tools are interactive and can generate diagrams from data obtained by code analysis, they are distributed as binary programs with a graphical interface.
Therefore, they lose their interactivity when their diagrams are exported as static images.
These tools also often cannot run during automated builds, making them difficult to use as alternatives to an API reference.
Commercial web platforms, such as CodeSee <cit.> and Swimm <cit.>, offer a web experience, but they focus on developer collaboration and making documentation maintainable rather than on code analysis.
There are also open-source tools like Emerge <cit.> and DependenSee <cit.> that output their diagrams as self-contained web applications, which can be placed next to the documentation as is.
However, while they demonstrate their strength in embeddability, the scope of their analysis is limited to module dependencies.
The final mention goes to the Obsidian note-taking tool—a program for managing Markdown notes connected by hyperlinks <cit.>.
It also features a node-link diagram displaying the connections between files.
Despite not performing code analysis, the Obsidian development team utilized it for their API reference.




In summary, none of the mentioned tools provides a combination of features that would make it a suitable replacement for an API reference.
They either cannot be automated, are not interactive, or cannot be embedded into existing documentation hosted on the web.







§ REQUIREMENT ANALYSIS


This paper explores the assumption that interactive visualization of a codebase's structure can improve documentation and facilitate exploratory analysis.
Our solution was designed around this assumption while considering several requirements drawn from two sources.
The first source was a survey among professional programmers, programming tutors, and computer science students conducted within the scope of a master thesis <cit.>.
The other source was our comparison of existing tools and their strengths and weaknesses, which we summarized in the previous section.
Based on the collected information, we settled on the following goals.


    
  *  Output a visualization that feels familiar.
    
  *  Generate the visualization automatically using data from source code analysis.
    
  *  Allow user interactions to enable exploration and filtering of the codebase on different levels of granularity.
    
  *  Output visualization in a format that can be embedded into a website.


Apart from the high-level requirements listed above, we also had to decide on the technical aspects of the implementation, as we needed to test the concept in practice.
In particular, we needed to consider what programming languages to support, as this decision is important for data mining algorithms.
In this regard, we identified C# as an ideal target language for our prototype implementation.
As our main goal was to test the visual representations, we decided to support only one programming language.
The advantages of C# are its large user base <cit.>, numerous projects to visualize, and an API for directly inspecting the compiler's semantic model.
However, we are convinced that the concept presented in this paper also applies to other object-oriented programming languages.
The following section summarizes the specifics of the C# programming language, whose knowledge is crucial for understanding our solution and its components.






§ C# LANGUAGE AND ITS SPECIFICS














Our visualization design and its prototype implementation assume familiarity with object-oriented programming (OOP) and particularly the C# programming language.
This section explains several C# terms, constraints, and guarantees relevant to this paper.

The .NET platform.
The C# language is a part of the .NET platform, which envelops several other languages, runtimes, and related tooling.
There are three relevant pieces of tooling: Roslyn, NuGet, and MSBuild.
The .NET Compiler Platform (“Roslyn”) <cit.> is the C# compiler, providing many code analysis APIs, which allow others to build tools capable of understanding C#'s syntax and semantics.
NuGet <cit.> is .NET's package manager facilitating the resolution and acquisition of external dependencies.
Finally, Microsoft Build Engine (MSBuild) <cit.> is the build system tying Roslyn and NuGet into a pipeline that transforms a description of a C# project, its source code, and dependencies into a binary artifact called an assembly.

The project system.
MSBuild works with projects and solutions.
C# projects define what NuGet packages are downloaded, what C# files are fed to Roslyn, and what other steps are performed as part of the build process.
Visual Studio Solutions are then simply sets of projects.
MSBuild and NuGet prevent projects and packages from having circular dependencies.

The type system.
Object classes are the core concept of OOP.
In C#, classes are just one kind of types the language has to offer.
There are five type kinds in total: classes, structures, enumerations, interfaces, and delegates.
All types must be declared inside a hierarchical namespace.
The top-level namespace is the global namespace.
Types may contain type members, which fall into one of four categories: fields, methods, properties, and events.
Both types and their members have an accessibility—a degree to which other pieces of code can interact with them.
For example, public methods can be called from anywhere, while private ones are accessible only within their declaring type.
For a more thorough description of C# types, see .NET's Common Type System <cit.>.

The static modifier.
Class types and type members can have the  modifier.
This modifier has special importance since  members are accessible without an instance of their declaring type.
Static classes are then types that may contain only static members.
For a C# programmer, a static class is essentially a container for global variables and functions.
This is important since, in .NET, variables and functions cannot be declared outside of types.

The entity abstraction.
Roslyn, NuGet, and MSBuild each have a different abstraction of the codebase they act upon.
For our purposes, we unify them into our entity abstraction.
Each entity has a name and an entity kind, which can be, for example, solution, project, package, type, or method.
Entity kinds form a hierarchy since it can be said that a solution contains a project, which contains a type, which declares a method, and so on.
Some entity kinds are further specialized.
For example, types have a type kind (e.g., class, struct, enum, etc.), and methods have a method kind (e.g., constructor, getter, operator, etc.).







§ VISUALIZATION DESIGN



Following Munzner's data visualization framework <cit.>, we iterated and enhanced our visual alternative to an API reference.
The main influences on the final visualization design and interactions were the <ref>–<ref> requirements, team discussions, and hands-on experience with our prototypes.

We considered several visual metaphors, including node-link diagrams, UML diagrams, and 3D cities.
Our goal was to create a visual representation that could provide both an overview of the entire project and details of individual code elements (see <ref>).
We disregarded standardized UML diagrams because they would require different visual representations for the overview and the details, thereby increasing complexity and the learning curve, which was unacceptable given our aim to simplify documentation navigation.
Similarly, we dismissed the 3D city metaphor in favor of simplicity and effectiveness, as 3D visual channels are less effective compared to 2D channels <cit.>.

Therefore, we opted for a node-link diagram, representing the codebase as a graph, i.e., a set of nodes and edges.
Nodes correspond to C# entities, and edges represent relationships between them. Figure <ref> shows an example of this diagram.
While node-link diagrams were already used for visualization of codebases before <cit.>, to the best of our knowledge, they were never explored as primary means of navigation within an API reference.
We can also benefit from the fact that node-link diagrams are a well-known and generally understood concept, which enhances the adoption of the solution.





 §.§ Glyphs



The nodes of the node-link diagram are glyphs—compound visual elements symbolizing several data attributes at once (see Figure <ref>).
We have decided to use glyphs as they provide an ideal compromise between space-efficiency and understandability <cit.>.
All nodes are circular, as is typical for node-link diagrams.
Circles can also be arranged in a way that minimizes overlap and maximizes the use of space in the diagram, which is particularly useful for complex networks.
The node radius is calculated based on several factors.
First, the represented entity's kind determines the fixed starting radius.
For example, the starting radius of a project node is larger than that of a type node.
For type entities, the starting radius is then increased by their member count.
The radius is then scaled, employing either a linear, logarithmic, or square-root scaling method.
Users have the flexibility to configure the scaling method, which can be helpful when the diagram contains excessively large nodes.



Each glyph has an icon in the middle representing its entity kind (Figure <ref>-A).
In the case of types, the icon represents their type kind as it is more specific (see Section <ref>).
Figure <ref>, for instance, represents a  type, whereas Figure <ref>-B is a .
These icons are taken from the Visual Studio (VS) image library <cit.>, courtesy of Microsoft, which allows their use in third-party tools.
They were chosen intentionally since VS is the most common IDE among C# programmers <cit.>, which aligns with <ref>.
The icon also sets the node's tint to strengthen its kind's distinctiveness.


Another highly relevant information is accessibility of types and their members. 
Its value can range from  to  with many nuanced shades in between <cit.>.
If the represented entity is anything other than , the glyph contains a smaller icon in the lower right corner of the kind icon.
For example, Figure <ref>-B is a  type since it has no icon in its corner.
However, Figure <ref>-D is  because its corner icon depicts a lock.

Type glyphs also have up to three outlines, as visualized in Figure <ref>.
The innermost outline (Figure <ref>-B) represents the  modifier.
If the outline is dashed, the entity is ; a solid outline corresponds to a non- entity.
The two outer outlines are only present on type nodes and symbolize the number of members the type declares.
The middle outline (Figure <ref>-C) is always solid, and its width represents the number of non- members.
Similarly, the outermost outline (Figure <ref>-D) is always dashed, and its width symbolizes the number of the type's  members.
The order of the outlines is always the same.

The visual element of node outlines has been chosen deliberately, as we believe it adequately represents the information and its relevance to the type.
To explain, the presence of the  modifier is a detail that is the closest to the type and, thus, deserves to be depicted as the innermost outline.
Similarly, non-static members are more closely tied to the type than static members and, therefore, are closer to the node's center.
We further emphasize this closeness by decreasing the saturation of each outline from the node's center outward.

A subset of these rules also applies to type members—nodes with the property, field, event, or method entity kind.
On these nodes, the accessibility icons and innermost outline representing the  modifier are also displayed.
However, their size is constant and smaller than that of any type node.



Take, for example, the nodes from Figure <ref>:

    
  * is the solution node and the root of the codebase, which is immediately clear from its icon from VS.
    
  * represents a  class with only  members.
        This is made clear by the dashed innermost outline, the presence of a dashed outer outline, and the complete absence of a solid outline.
    
  * is a static property, given by the property icon and the dashed outline.
    
  * is a non- class with only a handful of instance members. It is also  since it has the lock icon.


The glyph-nodes may also exhibit one of two animated effects.
These effects symbolize Roslyn diagnostics—compiler warnings and errors—which point out problematic pieces of code. They allow a team of developers to review issues within the codebase alongside the documentation.
Specifically, if a node has an error diagnostic attached to it, it appears to be on fire (Figure ).
In the case of compiler warnings, the node emits a mere column of smoke instead (Figure <ref>-E).
These effects were chosen as fire and smoke are typically treated with caution, similar to compiler diagnostics.
We also chose to animate them, as that makes them more noticeable, which is fitting due to the diagnostics' importance.



 §.§ Entity Relationships


Edges of the node-link diagram represent relationships between C# entities.
Each edge is defined by its source and destination node as well as a relation—a set of edges—it is part of.
There are multiple relations between nodes since we have decided to visualize more than just “dependencies” among the entities to represent the codebase accurately.

The  relation mirrors the entity kind hierarchy (see Section <ref>).
As such, it most clearly represents the codebase structure.
For example, in this relation, a solution has outgoing edges into projects, as solutions declare projects.
Similarly, a namespace is linked to types it contains, which are connected to methods declared by them, and so on.
However, the  relation alone offers no advantage over a table of contents commonly found in API references.
The real benefit of the node-link diagram lies in its ability to represent multiple relations at once.
Other relations symbolize entity associations such as type inheritance (), types of fields/properties/parameters (), and project dependencies ().
For an exhaustive list of the visualized relations, see Supplementary Material <cit.>.

To distinguish between relations when depicted simultaneously, each has a different color and line weight applied to their edges.
Each color is customizable, and each relation can be disabled if it is undesirable.
This way, the amount of clutter caused by an excessive amount of edges and their crossings can be minimized.
Following this mindset, we display only the  edges between the nodes by default. 



 §.§ Scalability



A typical problem that node-link diagrams face, especially when used to explore large datasets, is visual clutter and overlapping, which can obscure important relationships and make the diagram difficult to interpret.
A common solution found in literature <cit.> is to aggregate or filter the unimportant nodes. 
In this paper, we have primarily focused on filtering (see Section <ref>) as it was the most requested feature in the initial user survey <cit.>.
Therefore, we wanted to explore to what extent filtering would suffice as the primary means of improving the diagram's readability.

We utilize aggregation to some extent by allowing users to expand and collapse nodes at will (see Section <ref>).
However, one could argue that this is a form of localized filtering based on code hierarchy, as we do not visually communicate the cumulative properties of the aggregated nodes.
This was a deliberate design choice, as we wanted to avoid bias in the evaluation of the filtering features.




 §.§ Filtering and Highlighting



As explained above, we strive to provide the most flexible search and highlight feature we can.
Therefore, we have decided on three search modes: full-text search through the entity names, regex doing the same but using a regular expression, and the JavaScript (JS) mode allowing the user to write their own search logic with a custom JS filter.
In any mode, the user has a choice to either highlight the matching nodes, graying out the rest, or isolate the matches and remove the remainder from the current state of the diagram.

Among the search modes, JS is the most flexible one.
It enables the user to filter nodes based on any piece of entity metadata and not just their name.
For instance, one might want to know which nodes contain compiler errors, possess a number of members exceeding a certain threshold, or contain documentation comments with a certain keyword.
All of these queries can be made using the JS search mode.



 §.§ Interactivity and Layout



Apart from filtering, our approach relies on interactivity to be readable even as the diagram grows (see <ref>).
By default, the diagram shows only the solution and project nodes, with the rest of the codebase hidden.
The user can use filtering or manual interaction to expand nodes and show their children, thus shifting focus to a specific area of the codebase.
Likewise, nodes can be collapsed to unclutter the diagram and gain a more high-level view.
Parts of the diagram can also be removed from the current view.
These removed parts are then excluded from filtering and further interaction until the diagram is refreshed.
Finally, users can select any node they are interested in and inspect the represented C# entity in a side panel (see Figure <ref>-D).

The layout of the node-link diagram needs to respond to these interactions and remain readable.
However, users can still reposition nodes manually if they are dissatisfied with the automatic layout.







§ PROTOTYPE IMPLEMENTATION




To test our alternative API reference, we built a prototype named Helveg.
Helveg consists of three main components: a data miner, the interactive diagram itself, and a JavaScript Object Notation (JSON) schema used to communicate between the first two components.
The C# codebase data miner is implemented in C# as a console application so that it can be readily run in a server environment (see <ref>).
The diagram is then implemented in TypeScript (TS) and wrapped in a web-based user interface (UI) (see Figure <ref>).

The console application implements a pipeline that (1) analyzes a C# codebase, (2) serializes its abstraction into JSON, and (3) outputs a web application containing the interactive diagram.
The web application can optionally be assembled into a single HTML file.
This form of deployment has the advantage of not requiring a server to run on the user's machine <cit.>.

The rest of this section describes the most important aspects of Helveg's implementation.
For additional details, see Supplementary Material <cit.>.



 §.§ Data Mining



As mentioned, the console application analyzes the visualized C# codebase—performs data mining (see <ref>).
To be more specific, the data miner implements our unified C# entity abstraction (see Section <ref>).
Therefore, it integrates with the APIs of Roslyn, MSBuild, and NuGet.
None of the official APIs were designed with visualization in mind.
Thus, the data miner must perform substantial additional processing and work around the limitations of the APIs.

One of the main issues we encountered was the unique identification of C# entities.
This is required so that the entities can be serialized to JSON and passed to the diagram.
This issue was surprisingly difficult to solve due to C#'s method overloading and generics, which prevented us from using fully-qualified names of the entities since they may contain circular references.
Our implementation avoids these circular references by relabeling the entities with a unique, hierarchical, numeric token.
These tokens are essentially number sequences, encoding the path from a  root to the identified entity.
The numeric tokens also allow the data miner to analyze the codebase in multiple threads and make the diagram's layout more consistent across multiple tool runs.

Each entity also has diagnostics, comments, and metadata appropriate for its entity kind.
Diagnostics represent compiler warnings, errors, and hints.
This allows Helveg to visually represent codebases that are not compilable and highlight the entities responsible for the errors.
Similarly, the tool extracts documentation comments, ensuring that any documentation within the codebase is available in the final web application.



 §.§ The Diagram


Helveg's visualization, the node-link diagram described in Section <ref>, is implemented in TypeScript and transpiled into JavaScript, which can run in any modern web browser (see <ref>).
It uses the Graphology <cit.> library, which provides the necessary graph theory abstractions, and the Sigma.js <cit.> library, which renders the node-link diagram itself.
While these two libraries form the basis of the functionality, Helveg modifies Sigma's layout behavior and adds its own WebGL shaders that render the glyph-nodes.

Helveg relies on two graph layout algorithms—TidyTree by Reingold and Tilford <cit.>, and ForceAtlas2 by Jacomy et al. <cit.>.
TidyTree positions the nodes in a circular dendrogram.
It executes almost instantaneously, but it requires the underlying relation to be a tree and thus is used only on .
ForceAtlas2 is an iterative, force-directed graph layout algorithm, which runs after TidyTree has been applied.
Compared to TidyTree, it is not limited to trees and is able to respond to user interaction, but its convergence may be slow, depending on the number of nodes.
Helveg relies on open-source implementations of these algorithms: TidyTree from the D3 library <cit.> and ForceAtlas2 from Graphology <cit.>.

The diagram is encapsulated by a UI implemented in Svelte <cit.>.
It consists of several distinct parts besides the diagram itself.
The Dock (Figure <ref>-C) is a vertical bar housing several tabs.
It switches between panels, which occupy the same space (Figure <ref>-D) on the screen.
Panels may contain the properties and documentation comments of the currently selected node, search results, or configuration options.

By default, the user selects nodes by clicking on them with the left mouse button.
However, this behavior can be changed by selecting another tool in the Toolbox (Figure <ref>-E).
These tools correspond to the interactions described in Section <ref>.

The design of the UI draws inspiration from commercial software that the users are likely familiar with already.
We believe exploiting this familiarity (<ref>) makes the UI more intuitive and easy to use.
The Toolbox is inspired by toolbars on the left side of the screen, which are present in many graphic editors.
Similarly, the Dock and its Panels are heavily inspired by those in Visual Studio Code.

Most of the diagram's functionality can be customized to suit the user's needs.
All of the configuration options are contained in the panels.
For example, in the layout panel, the user can use the provided checkboxes to select the entity kinds and relationships they are interested in.
Corresponding nodes and edges will be laid out once the Refresh button is clicked.
The same panel houses controls of the TidyTree and ForceAtlas2 layout algorithms.
All of the panels are described and pictured in Supplementary Material <cit.>.







§ HELVEG USER STUDY


The functionality of the Helveg prototype was evaluated through an exploratory user study.
The study consisted of semi-formal testing sessions with five software developers (evaluators E1–E5), whom we recruited by word of mouth.
They all have at least five years of professional C# experience at a software company and regularly encounter software documentation.
The testing sessions were about an hour long, during which the participants gave their feedback verbally and via a questionnaire.
A summary of each interview and the testing samples are attached to Supplementary Material <cit.>.

We wanted to direct the participants' attention to all of Helveg's features, so we provided them with a sample diagram of a project and several tasks.
The project was KAFE <cit.>, an open-source web archive of multimedia implemented in C#.
None of the participants had encountered KAFE before their interview.
On the other hand, the interviewer was one of KAFE's developers and thus could easily assess the accuracy of the testers' observations.
The participants were also deliberately not introduced to Helveg to simulate the behavior of users who had only just discovered our visualization.
They were, however, told what Helveg's purpose is and given a hint if they asked for it or got stuck.
The tasks given to them were the following questions paired with assignments:


    
  *  What are the most important projects of the KAFE solution?
        Identify what the solution's projects are and what their purpose is.
    
  *  What external dependencies do the projects have?
        Determine on which packages the projects depend.
    
  *  Are there any issues with the codebase?
        Identify any errors, code smells, or other issues with the codebase.
    
  *  What are the major changes between the compared versions of KAFE?
        Try to deduce the purpose of the changes depicted in the provided KAFE diff.


As mentioned, the tasks were chosen mainly to ensure the participants explored the diagram's primary features.
In particular, there is no “correct” solution to the tasks because we were interested in the way the developers would use the tool and their subjective experiences.
However, we believe the tasks resemble some of the questions a new developer on a project would ask themselves.
By asking the participants to complete tasks <ref> and <ref>, we guided them towards exploratory analysis, interaction with nodes, and the search feature.
<ref> was meant to foster a more narrow analysis by suggesting they look at compiler diagnostics and attempt to find any code smells (i.e., subjectively bad practices).
We included <ref>, the final task, to evaluate an experimental overlay displaying differences between two versions of KAFE.
This overlay is only tangentially relevant to API references and is, thus, outside of the scope of this paper.
It is, however, described in Supplementary Material <cit.>.



 §.§ Participant Observations


All five developers who participated in the user study made accurate observations about the codebase throughout their 33 to 45 minutes of testing.
In task <ref>, they concluded that the most important projects are  and .
Interestingly, they arrived at this conclusion from different angles.
While some argued that it should be the  project since it is the [E4]entry point to the application, others thought about the task visually and said [E5]It would be . It takes up most of the pie that has been created., referring to the circular diagram as a “pie”.
Another participant held a different opinion after examining the  project:

[E3]
    The  [project] seems like the most important project so far. I was expecting maybe only some [data transfer objects] and data structures, but it appears to contain all the services and the core of the application itself.


The testers (E1 and E3) also accurately described the  project as an integration layer with an external service, and they (E1 and E5) correctly deduced that the  project is a utility for migrating data from a legacy database.

In task <ref>, the testers had to enable the package entity kind and the  relation.
They found both standard .NET libraries and external dependencies like FFMpeg and Marten, which led them to accurately assume that the application processes video material and uses the event sourcing design pattern.
While looking for code smells (<ref>), they located the  node due to its size and argued that a class of this size is a prime candidate for refactoring.



 §.§ Results



After every concluded testing session, each participant rated their experience with Helveg in several areas on a five-point Likert scale (see results in Figure <ref>).
The participants were mostly satisfied with the visualization design.
They found the glyphs familiar since they already knew the icons from Visual Studio, and they enjoyed the general appearance of the tool as well.
Overall, the tool's visualization was well-received and considered readable and useful, as one of the participants remarked:

[E5]
I enjoyed seeing the graph and the fact that I could imagine what the solution looks like without looking at any piece of code.


One participant thought that the visualization of compiler diagnostics was especially useful.
They remarked that Helveg made it easier to see which projects contained the most errors and warnings compared to the error list found in code editors.
[E4]Right here, I was able to see [that] the smell is coming from one concrete place from one project., they said.

When asked about comparison with existing tools, the participants responded positively.
One developer considered Helveg to be well-performant, saying that [E2]Code Map in Visual Studio usually takes so long.
Another tester said that [E4]For the amount of information that is shown right here, the JetBrains UML diagram would be unreadable for me..
A developer (E5) also said they were familiar with the concept of documentation enhanced with a node-link diagram since they already use the Obsidian <cit.> note-taking tool.

Besides the positive feedback, the testers expressed dissatisfaction with the intuitiveness of the UI.
They suggested improvements, including tutorials in both written and video form, tooltips placed next to the controls, and hiding some of the options behind an expert mode.
Some found the UI's similarity to Visual Studio Code inviting, whereas others commented they felt overwhelmed by [E3]the three levels of toolbars.
The diagram's interactivity score was negatively impacted by the overall user experience as well.
One participant indicated that they felt reluctant about interacting with the nodes since [E3]there's no point because they're going to be gone upon the next refresh.
Multiple participants also disliked that the most flexible way to filter the graph was through JavaScript, a language they detested.

While Figure <ref> shows that Helveg is far from perfect, especially regarding user experience, all testers saw its potential use.
Some saw it as a helpful onboarding tool: [E5]If someone new started working with me, I would use it to explain the high-level overview and to show parts of the architecture and how they fit together..
Others said that [E4]It could replace my visualization tooling.
Even the most critical of testers approved of the tool's basic concept: [E3]I like the idea itself. I like it for the API reference especially..





§ GLYPH EVALUATION








Apart from the user study of the prototype, we also evaluated our glyph design (see Section <ref>) to test its readability.
There were six programmers whose C# programming experience ranged between 1 to 13 years, with a median of 5.5 years.
One of the participants also tested our prototype, as described in the previous section.
The evaluation was a questionnaire with questions about each glyph element: the icons, the outlines, and the effects.
During the evaluation, a researcher was present to clarify any ambiguities and to take note of the participants' behavior.
The questionnaire and its results are attached to Supplementary Material <cit.>.

We evaluated our assumption that VS icons would be familiar to C# programmers through an image depicting 18 example glyphs with most of the relevant icons.
The questionnaire asked the participants to assign each glyph to a description such as “public class” or “method parameter”.
On average, the participants were able to correctly assign 68.52 % of glyphs, with a median of 72.22 %.
This questionnaire section took the most time, from 5 to 16 minutes.

Then, the form explained the meaning of our glyphs' outlines.
Participants were asked to compare several type and type member glyphs, focusing on the presence and width of the outlines.
These glyphs were shown in a single image to allow the participants to compare them.
The questions asked about glyphs that, for example, “represented a type with the most static members” or the total number of static C# entities.
The average success rate was 61.11 %, with a median of 58.33 %.

The next two sections were dedicated to glyph effects representing compiler diagnostics.
The respondents were first asked to give their opinions on them when the glyphs were isolated and later in a large group, simulating the view in our prototype.
The answers were given on a five-point Likert scale ranging from Strongly agree (+2) to Strongly disagree (-2).
All participants agreed that the animated effects were a faithful and intuitive representation of compiler diagnostics, with both aspects receiving a score of 1.5.
The question of whether a static effect would be preferable received a score of -0.67.
Only one programmer responded that they would prefer a non-animated effect instead, clarifying that merely different colors of the glyphs would be enough for them.
The results also show that all participants noticed the presence of the fiery effect in a large group (+2) and, likewise, the absence of the smoke (-1.5).
None would consider the depicted codebase to be healthy (-2).

At the end, the questionnaire offered an optional open feedback text field.
Two participants commented on the representation of warnings, saying that In [a] larger project, it's hard to distinguish compile errors and warnings.
One respondent went into detail on their experience, citing that they were not used to look at the icons representing types in VS.
However, they also said the visual representation is nice, although they would need more time and examples to get used to it.






§ DISCUSSION


The interviews with C# programmers and the glyph evaluation show that our diagram can serve as a feasible alternative to an API reference.
However, they also evidence that our approach, and especially our prototype, has several issues.
In this section, we discuss the lessons learned from the evaluation and the work that can be done to address them in the future.



 §.§ Lessons Learned



0.98


    Our glyphs and diagram-based navigation can be used to gain knowledge about a codebase that a typical API reference can hardly provide.






The testers' observations show what can be learned from the diagram.
For example, their discovery of the need to refactor the oversized  class could be made with a typical API reference only with great difficulty.
In the API reference, the number of type members is not discernible unless one reads every type's page.
Another example is the visualization of compiler diagnostics since, to the best of our knowledge, no API reference generator includes those in its output.
In order to see those, the reader would need to download the source code, which may be unavailable, and compile it themselves, which could require a significant amount of time.


0.98


    A familiar interface and a set of icons do not necessarily make a UI intuitive.
    





Helveg's evaluation shows a deficiency in its user experience.
This result is not unexpected, considering that the user study was the prototype's first contact with real users.
It shows that although we designed the UI to feel familiar, using recognizable UI concepts and icons, this familiarity alone does not lead to an intuitive UI.
Without any introduction, the tool can seem overwhelming due to its many options.
These options are useful and make Helveg flexible, but they also make the tool more difficult to master.
This difficulty could be managed by equipping Helveg with onboarding features.
The UI already contains a guide panel offering a non-intrusive, optional tutorial.
However, none of the testers opened the panel prior to a hint from the interviewer.
Thus, we conclude that either the UI should be redesigned to be more intuitive or a more prominent tutorial should be added.

The glyph evaluation indicates that while C# programmers are familiar with the VS icons, they struggle to assign meaning to them.
After the evaluation, a discussion with one of the respondents also revealed that the icons are not used consistently within VS.
Some VS icons can even be confusing since their visual similarity may suggest a connection between them, despite there being none, as is the case with the icon for fields and method parameters.
Consequently, we believe it to be a good practice to exploit the familiarity of the VS icons but to use them only as a basis for custom visual elements.


0.98


    A filtering feature must balance its flexibility with its ergonomics.






We also learned that simple features are not always easy to use.
The JavaScript filter mode, which we implemented in Helveg, is simple in its implementation and flexible in its use cases.
However, C# developers were reluctant and felt uncomfortable writing code in JS.
This shortcoming could be remedied in several ways.
The addition of another search mode that would contain predefined filters is one of them.
With this option, the diagram could be equipped with the most common queries by default.
These queries would additionally serve as samples, making it easier for users to write their own queries.
The JS filter could also be replaced with a filter builder interface, allowing users to construct filters more ergonomically at the expense of some loss of flexibility and implementation complexity of this feature.


0.98


    The prototype neither scales well nor performs well when applied to large codebases.






Although, according to the testers, Helveg performs better than its most well-known alternatives, it does not scale well on projects much larger than KAFE.
KAFE comprises eight projects, resulting in a diagram with at most 3760 nodes when completely unfiltered and expanded.
However, C# solutions can grow to hundreds of projects and tens of thousands of nodes.
We knew about this scalability issue when we designed the visualization (see Section <ref>), so we included flexible filtering features, allowing the user to choose what to focus on.
Without the filtering features, however, the prototype currently cannot handle large solutions.

In the future, we may consider alternatives to the node-link diagram, such as a circular treemap or a hybrid of the two techniques designed for large codebase exploration.
Helveg's performance bottleneck can likely be solved by rendering the visualization using WebGPU <cit.>, a future web standard, currently a working draft, that may be able to visualize a much greater number of nodes.


0.98


    Animated glyph effects accurately represent compiler diagnostics.
    However, the multiple outlines may be too complex.






The glyph effects received an overwhelmingly positive response in our survey (see Section <ref>).
Although the survey's sample size was small, this result indicates that the effects are suitable visual representations of compiler diagnostics.
However, we believe that glyph outlines may be too complex due to the low success rate of survey questions aimed at comparing them.
While the participants suggested they would get used to them in time, we may replace them with simpler visual elements in the future.



 §.§ Future Work


There are many ways to extend our approach besides those implied by the lessons learned.
The most obvious extension would be to support another programming language.
Another potential extension can be found in the participants' enthusiastic reaction to the experimental diffing overlay (see Supplementary Material <cit.>).

The data miner can be extended as well.
For example, it could analyze method bodies, leading to a diagram depicting the method call tree, or provide more details about package references, such as whether they are direct or transitive.







§ CONCLUSION


We introduced a visual alternative to a typical API reference, where navigation is handled by a central node-link diagram.
It provides a high-level view with interactive features, allowing in-depth exploratory analysis of a codebase.
The diagram's glyph-nodes build on iconography that most users already know.
We created Helveg, a prototype implementation of this approach.
Through the evaluation with professional software developers, we conclude that Helveg is functional and can provide insight into a codebase, which would be hard to achieve using a typical API reference.
Among possible future directions are improvements to the diagram's scalability, the visual design of its glyphs, and the prototype's user interface and range of supported programming languages.







§ ACKNOWLEDGEMENTS


We thank all participants of all our surveys and user studies.
We also thank Marco Schäfer for kindly letting us modify his questionnaire based on a SurveyAMC <cit.> template.





IEEEtran





 Paper Title: Assessing Programming Task Difficulty for Efficient Evaluation of Large Language Models.




Describe, Transform, Machine Learning: Feature Engineering for Grain Boundaries and Other Variable-Sized Atom Clusters
    [
    
======================================================================================================================




§ ABSTRACT 
  Large Language Models (LLMs) show promising potential in Software Engineering, especially for code-related tasks like code completion and code generation. LLMs' evaluation is generally centred around general metrics computed over benchmarks. While painting a macroscopic view of the benchmarks and of the LLMs' capacity, it is unclear how each programming task in these benchmarks assesses the capabilities of the LLMs. In particular, the difficulty level of the tasks in the benchmarks is not reflected in the score used to report the performance of the model. Yet, a model achieving a 90% score on a benchmark of predominantly easy tasks is likely less capable than a model achieving a 90% score on a benchmark containing predominantly difficult tasks. This paper devises a framework, HardEval, for assessing task difficulty for LLMs and crafting new tasks based on identified hard tasks. The main idea of our framework is to use a diverse array of prompts for a single task across multiple LLMs in order to obtain a difficulty score for each task of a benchmark. Using two code generation benchmarks, HumanEval+ and ClassEval, we show that HardEval can reliably identify the hard tasks within those benchmarks, highlighting that only 21% for HumanEval+ and 27% for ClassEval of the tasks are hard for LLMs. Through our analysis of task difficulty, we also characterize 6 practical hard task topics which we used to generate new hard tasks. Orthogonal to current benchmarking evaluation efforts, HardEval can assist researchers and practitioners in fostering better assessments of LLMs. The difficulty score can be used to identify hard tasks within existing benchmarks. This, in turn, can be leveraged to generate more hard tasks centred around specific topics either for evaluation or improvement of LLMs. Moreover, while we focus on code generation tasks in our experiments, HardEval generalistic approach can be applied to other domains such as code completion or Q/A. 





§ INTRODUCTION

Large Language Models (LLMs) have been widely used for various code-related tasks in Software Engineering (SE) <cit.>. Although the performance of LLMs in generating code for different programming languages, such as Python, Java, and C <cit.> is promising, LLM-generated code like human-written one is not flawless. Researchers already acknowledged that LLM-generated code are prone to bugs <cit.>, and may suffer from vulnerability <cit.>, and quality issues <cit.>. 


Generally, to assess LLMs on code generation tasks and compare them to each other, well-known benchmarks such as HumanEval <cit.>, MBPP <cit.>, CoderEval <cit.>, and ClassEval <cit.> are used. These benchmarks are generally obtained by handcrafting tasks or mining existing public repositories over GitHub and filtering them. This way, the code generation capability of LLMs on programming tasks is assessed indirectly at the benchmark level based on general characteristics such as dependency level (e.g., function level being less difficult than class level) and through general metrics such as Pass@k over the benchmark <cit.>. Moreover, the assessment of those tasks is conducted via a single prompt, which is a single formulation of the task. Yet, recent studies <cit.> showed that the way prompts are formulated has a high impact on the output quality of LLMs. Thus, current benchmarks' usage and assessment paint a general yet incomplete picture as they fail to provide a more fine-grained analysis of individual tasks within a benchmark. In particular, such benchmarks' usage does not provide a way of quantifying the difficulty of a programming task for LLMs. 

As a motivating example, consider the following task to implement the function cycpattern_check from HumanEval benchmark: 

[language=python, caption=A task example from HumanEval+., label=list:example]
def cycpattern_check(a , b):
    """ Write a function named 'cycpattern_check' that checks if the second word or any of its rotations is a substring of the first word. """


To assess an LLM, one would generally run this prompt either using greedy decoding or temperature sampling <cit.>, and then test the outcome against a set of predefined tests to evaluate correctness. Multiple LLMs could even be used to offset the potential impact of the particular training/architecture on the evaluated correctness. Doing so with a greedy decoding using the 5 LLMs we use in our experiments, results in all models except one failing on this task. In this context, should the task be considered hard? To tackle this issue, we propose HardEval, a framework for the evaluation, identification, and generation of difficult programming tasks for LLMs. We show with HardEval that this task is not difficult. Indeed, using our framework, all models except one indicate a low difficulty score for the task with a low difficulty averaged over all LLMs. As it turns out, rephrasing this basic prompt slightly or providing small additional contextual information helped the LLMs to easily address the task. As such, while the above prompt could be considered hard, the task per se is not.










To guide our study, we formulate the following Research Questions (RQs):


    
  RQ1 
  RQ2 
  RQ3 


HardEval is orthogonal to the existing benchmark's usage and works on top of available benchmarks. HardEval starts by generating a variety of prompts for each task in two well-known code generation benchmarks, HumanEval+ <cit.> and ClassEval <cit.>, representing multiple possible formulations per task. For each programming task, we craft prompts with different levels of context information and distinctive phrasing to represent diverse possible wording of the task using GPT-4, resulting in 18 different prompts per task. The framework then queries an array of 5 LLMs (i.e., CodeLLama 7B <cit.>, MagiCoder 6.7B <cit.>, DeepSeekCode 7B <cit.>, CodeGemma 7B <cit.> and GPT-3.5 <cit.>) using the formulated prompts to obtain different outputs for every single task. Those outputs are then used to compute a task difficulty score by averaging overall LLMs a combination of the functional correctness (passing all available tests for the task) and the syntactic correctness (using code similarity metrics) obtained on each code sample obtained. This difficulty score separates hard from easy tasks. Such hard tasks uncover particular shortcomings of LLMs that can then be used for improving LLMs. Then, we cluster hard tasks to recognize similar ones using a topic modeling technique called BERTopic <cit.>. To further generate new tasks, we leverage hard tasks from those topics in two ways: either by transferring constraints from hard tasks onto an easy task, or by modifying an already hard task by adding an additional constraint. Our results show that while benchmark-wide metrics such as accuracy using pass/fail code can give a general trend of a benchmark difficulty, they fail at assessing the difficulty of individual tasks contrary to the difficulty score proposed in HardEval. Moreover, using the identified topics we managed to generate 15 new tasks for both benchmarks for which the difficulty score is high as an example of HardEval's capacity. To the best of our knowledge, this paper is the first attempt to investigate the difficulty of code generation tasks for LLMs, suggest metrics, and provide a way to generate new targeted hard tasks based on identified hard programming tasks.



In summary, this paper makes the following contributions:

    
  * We present HardEval, a framework for evaluating the difficulty of code generation tasks for LLMs, identifying hard tasks, and generating novel hard coding tasks for LLMs,
    
  * We show the limit of assessing difficulty based on general benchmark metrics such as Pass@1 accuracy.
    
  * We make the data and code used in our study publicly available for researchers to duplicate our results <cit.>. In particular, we also provide the hard tasks identified for further research.


The rest of this paper is organized as follows. In Section <ref>, we describe the methodology we followed to propose HardEval. Our experimental setup for running HardEval is detailed in Section <ref>. We report and then analyze the obtained results in Section <ref>. A discussion of our results and insights for the community is presented in Section <ref>. We conclude the paper in Section <ref> after discussing threats to the validity of our findings in Section <ref>.



§ HARDEVAL


In this section, we discuss the methodology followed to propose HardEval. After an overview of the framework, we describe how we generate different prompts for programming tasks. We then delve into the details of leveraging LLMs to generate code for our prompts, measuring the difficulty of tasks for LLMs, identifying hard coding tasks, and generating new hard tasks.



 §.§ General Overview of HardEval

HardEval uses existing benchmarks to operate. Such benchmarks are generally composed of an ensemble of tasks which are represented by a single prompt per task. In our study, we make the distinction: the task is the essence of the functionality to be implemented. For instance, Quicksort or Fibonacci sequence are examples of such tasks. A prompt is a way to formulate the task so it can be understood by and fed to an LLM. For instance, Write a function to calculate the n-th element of the Fibonacci sequence. or Create a function to return the n-th element of the Fibonacci sequence. The Fibonacci sequence is defined as ... are two instances of prompts for the task Fibonacci sequence. As such, each task can be expressed by many different (even infinite) prompts, including different wordings, and amount of contextual information about the task. HardEval aims to evaluate the difficulty of those tasks for code LLMs in order to generate targeted benchmarks to probe potential hard tasks of LLMs.



A general overview of HardEval is given in Figure <ref>. HardEval is operated in three steps: -.9pt1 First, transformations on the prompts are identified. Those transformations are any dimension that can lead to a variety of prompts for an LLM while keeping the essence of the task. In a way, such transformations can be seen as metamorphic transformations <cit.> as they should not alter the semantics of the task. In our study, we focus on two transformations: Rephrasing and Context Information. Those transformations are then used in conjunction with each task to have a Prompt LLM generate prompts following the transformations' specifications. -.9pt2 Secondly, the obtained prompts are fed into several Code LLMs to generate code. The aim of those LLMs is to generate code for each given prompt and give us an assessment of how LLMs fare when confronted with the different prompts. Multiple LLMs are used to encompass the diversity of architecture and training settings that could react differently to a particular prompt. From the generated code, several metrics are extracted that will help us to obtain a Difficulty Score for each task in the benchmark. This score will further help us to classify tasks into easy and hard tasks. -.9pt3 Finally, hard tasks are analyzed to generate further targeted new tasks based on those topics. In particular, such tasks can be clusterized into tasks of similar semantics (e.g., sequence tasks, and sorting tasks), i.e., topics, and fed into a Task LLM. This allows us to generate new tasks similar in semantics to the identified hard tasks (e.g., new sequence generation based tasks), effectively generating a sub-benchmark of targeted tasks which can bring a finer analysis of LLM shortcomings for code generation. In the following, we will describe each step in more detail.



 §.§ Generating Different Prompts based on Transformations


The first step involves choosing transformations to generate different prompts for the task. We want those transformations to introduce variabilities in our prompts to represent the many ways an LLM could be prompted for the same task. However, those transformations should not introduce changes that alter the semantics of the tasks. In our case, we use two types of such transformations, but any number could be used and combined. 

First, we use the Context Information transformation. Indeed, a straightforward way to act on the variability of a prompt is to simply disclose or withdraw certain information in the prompt, while not changing the task. EvoEval <cit.>, for instance, proposes also to act on the difficulty. However, they do so by adding/removing constraints in the prompt thus modifying the semantics of the task which is not desirable in our case. In our study, we instead propose to divide the information provided in the prompt into three levels. Those three levels of prompts are generated incrementally using the previous level as a starting point. We assume that prompts with more information should be more likely to lead to correct code for LLMs. The levels are defined such as: 


    
  * First level: contains a minimal amount of information that is the inputs/outputs of the task as well as a high-level description of what is intended. We could use the original prompt in benchmarks for each task as the first level, yet Siddiq et al. <cit.> showed that there can be inconsistencies across tasks' prompts in benchmarks. Thus, we prefer to generate a new prompt to have similar formatting across all tasks and levels to reduce biases. Moreover, this level is the closest one to the original prompt in benchmarks and contains the same level of information regarding the tasks.
    
  * Second level: further adds in the prompt a description of the targeted function in natural language using available oracle code. However, it does not include any direct reference to the function (variable names, helper functions used, ...) unless it was explicitly mentioned in the original prompt of the benchmark. 
    
  * Third level: complete the prompt with direct references to the function such as variable names, and helper functions used similarly to a pseudo-algorithm but in natural language.


We present an example of different prompt levels obtained for a task in Listing <ref>.

The second transformation used is Rephrasing which is paraphrasing a given prompt. Indeed, how the prompts or instructions are formulated has been shown to have an impact on LLM responses <cit.>. Thus, similarly to these studies, we try to see if different ways of rephrasing a given prompt for an LLM can affect how the LLM addresses the task. To rephrase our prompts, we will use a similar approach as done by Gonen et al. <cit.>, that is prompting an LLM to rephrase a given text. Other techniques could be used to assist in rephrasing such as Chains-Of-Thoughts <cit.> but assessing the effectiveness of different rephrasing techniques is out-of-scope of this study.

[language=python, caption=Example of different prompt levels for the task unique from HumanEval+, with the oracle code at the bottom., label=list:levels]
def unique(l: list):
    Level1: """ Write a function named 'unique' that returns a sorted list of unique elements from a given list. """
    Level2: """ Write a function named 'unique' which takes a list as input and aims to return a sorted list containing only the unique elements from the input list. The function should first convert the list to a set to remove any duplicates and then convert it back to a list which should be sorted before returning. """
    Level3: """ Write a function named 'unique' which takes an input list 'l' and returns a list containing only the unique elements from 'l', sorted in ascending order. The function should first convert "l" into a set, using 'set(l)', to remove any duplicates, then convert this set back to a list, and finally return this list after sorting it in ascending order using 'sorted()'. """
    return sorted(set(l))


Using those two transformations, we come up with general templates that can be used to generate prompts (see replication package <cit.>). Such general templates are then tailored for each task in the benchmark under consideration. Those templates are fed to a Prompt LLM (see Figure <ref>) to generate different prompts for a given task. The way prompts are generated is as follows: we start from the original docstring to have a common starting point for all prompts. Then, we apply the template to obtain different prompt levels with Context Information for a given task. Then, we apply the obtained prompts to the Rephrasing template. In our case, we first remove the examples from the original prompt, if any, before using our templates with the Prompt LLM. While removing examples can decrease the performance of LLM <cit.>, this allows for better control over the level of information we inject in our prompts. Indeed, examples can vary from simple basic examples to showcasing corner cases of the task. As such, this might artificially bias the output of LLM and the potential difficulty of the task because of the quality of the examples. 

At each step of our generation, that is after using each template on the prompts, we do a sanity check by manually checking them. This is accomplished by the first author, and cross-checked by the second author independently. We make sure that the Context Information levels are respected and the Rephrasing does not alter the semantics of the tasks. This is done in order not to have tasks that are artificially considered hard further down the line because of misleading prompts for the code LLM and not because of the task itself.



 §.§ Assessing difficulty of tasks


The second step is to evaluate the difficulty of each task. To do so, we leverage several Code LLM that will serve as our evaluators for code generation. For each prompt per task, each LLM is asked to generate code with varying seeds using the sampling mode of LLMs. The rationale is that most LLMs, especially non-open-source ones, are used with this non-greedy approach in order to generate code. Moreover, using the sampling method for the generation allows us to have a wider variety of generated code in order to evaluate the difficulty of each task. Indeed, we assume an easier task should not be as impacted by the stochasticity of the generation compared to a harder task. The obtained code is then processed to be executed. Sample codes were then executed against available test inputs in the benchmark.

Once we have obtained all code for all prompts of all tasks and executed them against tests, we collect different metrics. In our case, we are interested in two metrics: the Correctness and the Similarity to Correct Code of a code sample. Correctness simply quantifies if a given code sample is correct or not using available test cases. As such, it assesses the functional correctness of a code sample. Formally, the Correctness Cor for a code sample s is defined as:

    Cor_s = {[                      1 ∀ (i, o) ∈ T, s(i) = o;                      0                        ].

where T is the set of test cases symbolized by an input i and an output o. That is, the code sample passes all test cases, i.e., gets a value of 1. Intuitively, we expect a harder task to lead to a lower number of prompts leading to correct codes across rephrasing and different levels of context information.

Nonetheless, code samples on a task can be wrong to different degrees depending on the bugs affecting the code sample <cit.>. Indeed, a code sample could simply be wrong because of a missing corner case (e.g., forgetting an initial condition) or it could follow a completely false logic and not implement the desired tasks at all. One way to assess this aspect would be to quantify the proportion of test cases for which a code sample s is correct <cit.>. However, this can lead to a biased measure depending on the test set. For instance, if a test set has more test cases assessing one corner case rather than another one, some code samples might end up with a higher proportion of passed tests artificially. Instead, we propose to use classical similarity metrics to measure the distance between a code sample and the the most similar correct code sample. To do so, we will rely on the CodeBLEU <cit.> metric to measure the similarity, which measures the syntactical correctness of the code sample. We chose CodeBLEU as it is a widely used metric in code generation tasks <cit.>. Thus, the Similarity to Correct Code Sim for a code sample s is defined as:


    Sim_s = max_sc(s, sc), C_sc = 1

where sc are correct code samples according to Correctness. Note that, in practice, we have at least one correct code as a reference which is the oracle code of the task in its benchmark. If s is correct, the similarity is 1 (closest is itself). This allows us to assess, when a code sample is incorrect, how similar it is to a correct code. The Correctness and Similarity to Correct Code are then used to evaluate a code sample separately, or in a Composite manner by aggregating the scores.

Finally, we use those metrics to compute a score per code sample which in turn helps to assess the difficulty score of the task itself for a given LLM. We used different transformations in this study: Rephrasing and Context Information. So, for instance, we expect an LLM to be more likely to generate a correct code if given a prompt with more context information (of a higher level) than given a prompt of a lower level. To represent this, the scores of the code samples obtained for each prompt can be weighted depending on the transformation. In our case, as we apply the Context Information before the Rephrasing, the difficulty score R_i of the i^th Rephrasing is defined as:


    R_i = ∑_j s_j × score_s_j

where score_s_j is the score obtained for the j^th code sample using the i^th rephrasing, as we generate multiple code samples for a given prompt through sampling generation. s_j are weights to give more or less importance to certain code samples. In our case, all samples are of equal importance and so all s_j equals 1 over the number of samples. Then, we calculate the score L_i for a given level of prompts i:

    L_i = ∑_j r_j × R_j

where r_j are weights used to give more or less importance to a certain rephrasing. In our case, all rephrasings are of equal importance and so all r_j equals 1 over the number of rephrasings. Finally, the difficulty score of the i^th task in the benchmark for a given LLM is calculated as:

    Difficulty_i = 1 - (α× L_1 + β× L_2 + γ× L_3)

where α, β and γ are weights over the scores obtained for each level. An explanation of the choice of parameters is given in Section <ref>.

Finally, once all the Difficulty scores have been computed for all tasks and LLMs, the score of a given task is simply the average over all LLMs. One can then use the score to determine which tasks are hard and which are not. To do so, we will use a threshold of 0.5. The choice of the threshold is motivated by our definition of the score for each code sample: a score of 0.5 would mean, in the most extreme case and for the Composite score, that all generated code samples for a task are incorrect but are very similar to a correct one. In practice, a score higher than 0.5 for a task would be more reflective of a high number of incorrect codes with a lower similarity, pointing towards a hard task.



 §.§ Analyzing and Generating new Hard Tasks
 
Once the hard tasks have been identified, the third step is to generate new tasks based on the identified hard tasks. To do so, we propose to leverage a topic modeling approach with BERTopic <cit.>, as was used in similar studies when dealing with natural language data <cit.>. For a given benchmark, the topic model clusters all tasks (easy and hard) using the prompts of Level 1 before rephrasing. We use this specific prompt type to have similar formatting across tasks while being as close as possible to the original prompt as mentioned in Section <ref>. As the process will inherently label some tasks as "noise", noisy tasks are not considered in the analysis. Then, for each topic, the proportion of hard tasks among all tasks in the topic is calculated and the topics with the highest proportion of hard tasks are analyzed. Those topics constitute tasks that have a similar functionality/goal, for instance, tasks dealing with sequence generation. Finally, we aim to generate more hard tasks leveraging the tasks from those selected topics with a Task LLM to generate additional new tasks. To do so, we leverage the Self-Instruct approach <cit.> which allows generating new tasks based on a subset of sampled tasks. Nonetheless, just providing easy and hard tasks to the Task LLM and asking it to generate more hard tasks did not lead to meaningful results according to our observation. Indeed, the small number of tasks per topic makes it quite impossible to determine what constitutes a hard task. Instead, we use a combination approach, where we ask the Task LLM to augment a hard task by adding a constraint, or by adding a constraint from a hard task to an easy one. This ultimately reduces the diversity of tasks generated but increases the likelihood of the generated tasks being hard. The newly generated tasks constitute a smaller sub-benchmark of targeted tasks related to the topic that can be used to further assess LLMs corner cases for code generation tasks.



§ EXPERIMENTS






 §.§ Hyperparameters used

To have different prompts, we choose to generate 3 levels and 6 rephrasings per level. We chose those parameters as the trade-off between generating diverse enough prompts while limiting the amount to generate. For a benchmark of n tasks, this effectively means we have to generate 18 × n prompts. We further ask each Code LLM to generate 5 code samples per individual prompt resulting in 90 × n code samples for each Code LLM to generate. As an example, for a benchmark containing 200 tasks, this process yields 18,000 generated code for our evaluation per Code LLM. We make available all the prompts and code generated in our replication package <cit.>.



 §.§ Benchmark used


In our experiment, we chose two benchmarks: HumanEval+ <cit.> and ClassEval <cit.>. Those benchmarks were chosen as they cover different types of tasks and dependency levels. HumanEval+ is widely used as a standard benchmark in code generation <cit.>. It is an extended variant of HumanEval <cit.>, that contains more tests to assess the correctness of code samples. The benchmark consists of 164 handwritten Python programming problems with a function signature, docstring, body and several unit tests. HumanEval+ tasks are at the function level dependency as they do not require external libraries, besides Python standard library, or class/file contexts to be operated. For the original prompt, we use the docstring as given in HumanEval+. 

ClassEval is a benchmark of 100 handcrafted classes coded in Python. Each class contains a description and on average 4 methods each with a docstring. We consider a task to be one of the 400 methods of ClassEval. Contrary to HumanEval+, ClassEval tasks are more similar to practical code used in projects and rely on a class context to operate. ClassEval proposes three prompting strategies: holistic, incremental and compositional. In our case, we use the compositional strategy, that is, providing the class context to the LLM and asking it to complete one method of the class, only modifying the docstring of the method to complete through our crafted prompts. We do not make use of the other generation approaches as it reduces the control we have over the prompts and could introduce biases: the holistic approach, as it implies asking the LLM to generate the whole class from scratch, can impact the performance artificially. Similarly, the incremental approach requires the reuse of codes generated by the LLM for other methods when generating new ones, which could also alter the difficulty. As we need to generate a high number of code samples and prompts, we reduce the number of tasks to evaluate by doing a stratified sampling at the 95% confidence interval. For 400 methods, this gives us 197 methods to sample which we round up to 200. Those 200 methods are sampled equally across classes resulting in 2 methods per class available. Methods chosen are available in our replication package <cit.>.



 §.§ Models used


In our experiments, we use GPT-4 <cit.> as both the Prompt LLM (to generate the different prompts) and the Task LLM (to generate new hard tasks). We do so as GPT-4 is the state-of-the-art LLM and so should provide answers with the best quality which would require the minimum human supervision. For the Code LLM, we use 5 LLMs with different architecture: CodeLLama 7B <cit.>, MagiCoder 6.7B <cit.>, DeepSeekCode 7B <cit.>, CodeGemma 7B <cit.> and GPT-3.5 <cit.>. We have not used GPT-4 as Code LLM to avoid biases as it is used to generate the prompts fed to the Code LLM. For all models, we used their instructions-tuned versions so the models can handle the different prompts properly. This gives us a diverse array of architecture and training procedures. For all code generation in HardEval, as we are using the sampling generation, we fix the temperature hyperparameter to 0.8 as it is one of the temperature settings commonly used in other similar studies <cit.>. In all cases except GPT models (for which we use OpenAI API), we use the HuggingFace <cit.> implementation of the models to allow for replication. For verification, we compute Pass@1 <cit.> with the samples obtained across levels for a given task. Results are reported in Table <ref>. Ranking of the LLMs' performance follows the assessment obtained on the original benchmarks.





§ RESULTS




 §.§ Validation of the Context information transformation


We aim to assess whether the level of information in the prompts impacts the generation of the code by the LLM in such a way that is quantifiable by our metrics. While for rephrasing, the impact on the output has been analyzed in the literature <cit.>, the context information transformation as we formulate it in our work has not been so. Some studies <cit.> showed that prompt-added contents can range from no benefit to negative effect on the code generated. For instance, Ouyang et al. <cit.> showed that a longer description in the prompt could decrease performance. As such, we verify that our added context transformation has the intended functionality, i.e., the more context we provide, the better the generation.

As reported in Table <ref>, the Pass@1 rate increases with the Context Information. We now set out to show there is a correlation between the scores computed by our metric and the Context Information. To do so, we use the scores of the generated code for each rephrasing (see Equation <ref>) and compare it to the prompt level it comes from. To see the impact of both the Correctness and Similarity metric, we compute each metric individually as well as the composite of the two. As we introduce gradation in the injected information in the prompts through the Context information, one expects to observe a correlation between the level of information and the score obtained (i.e., the more information in the prompt, the higher the score). To calculate the correlation, we use the Spearman-ρ <cit.>, as we are interested in a monotonic increase between the two variables. We report the results for each Code LLM independently and for the averaged over all LLMs. Results are reported in Table <ref>.




In Table <ref>, for all cases, we obtain a significant positive correlation, indicating that increasing the amount of information in the prompt generally improves the LLM-generated code in terms of the score for all of our metrics. The Composite score (i.e., Correctness + Similarity) yields the best correlation overall on HumanEval+ while Similarity is slightly better on ClassEval. As we have a positive correlation, it means a higher score for a level 3 prompt than a level 1 prompt and so they account for our Context Information. As the Composite metric seems to be better on HumanEval+ and is tied up on ClassEval, we consider only this metric in the rest of this paper to calculate the difficulty score.

We note that the correlations are around 0.2 to 0.4, signifying a positive significant but weak to moderate relationship. We suppose the reason is the variability across tasks. Indeed, due to the particular difficulty of each individual task or rephrasing used, there might be some variability within the tasks that could impact our outcome. For instance, easier tasks are more likely to have a higher score for Level 1 prompts while more difficult tasks can end up with a low score for Level 3 prompts. This can impact the outcome of the Spearman-ρ when taking all tasks into account at once. To account for this, we calculate the Spearman-ρ for each task individually before averaging the obtained correlations. As averaging correlations directly might lead to biases, we follow the recommended approach <cit.> of converting the Spearman-ρ correlations using the Fisher Z-transformation, averaging the results and converting it back to a Spearman-ρ. Doing so, we obtain Spearman-ρ > 0.5 in all cases, even reaching 0.70 and 0.64 respectively for HumanEval+ and ClassEval when using the scores averaged overall LLMs. This results in moderate to strong relationships. As such, our Context Information indeed helps LLMs to generate correct code when we account for task variability, and so it is a valid means to calculate the difficulty score.




























 §.§ Validation of the parameters for the Difficulty Score


Without ground truth, we describe the rationale behind the choice of parameters α, β and γ of the Difficulty score. In the previous section (Section <ref>), we show that the lower the amount of information in the prompt, the less likely the model could produce a correct code. As such, we should consider that α≥β≥γ, that is, emphasizing more on lower information context. Otherwise, this could artificially bias the difficulty score towards assessing tasks as easy. Consider that this would be equivalent to having an exam where most questions would have tips to help the participants: the exam would easily be passed, but it would not necessarily be representative of the tasks in the exam. The issue then becomes which values to assign to parameters. To decide, we analyze the shape of the Difficulty score distributions obtained by varying the parameters. In Figure <ref>, we give the distributions of the difficulty score obtained on the ClassEval benchmark used during experiments, by varying α. The values of β and γ are adjusted so the sum equals 1.



As illustrated in Figure <ref>, emphasizing too much on α will give too much importance to low information context. In this case, tasks will tend to be declared harder (see Figure <ref>, orange), because LLMs will struggle more in a low information context to produce correct code fragments. However, on the contrary, the lower the α the more emphasis is put on higher information context (see Figure <ref>, blue), for which we provide more information to the LLMs than typical prompts in benchmark normally would. This instead makes more tasks be declared as easy. A good balance would be found between those two extremes: too much and too little reliance on the low information context. To do so, we assume that benchmarks are balanced. That is, we consider that LLMs should not consider that tasks are overwhelmingly easy or hard. Given that no LLMs reach very high or very low accuracy on the study benchmark (see Table <ref>), this looks like a sound assumption. As such, we can consider the shape of the distribution as an indication for the choice of the parameters α, β, γ. 

In our case, we will consider two indicators of the distribution: the standard deviation σ <cit.>, which measures the spread, and the skew g_1 <cit.>, which measures the asymmetry.











To achieve the mentioned balance, we aim to minimize the skew and the standard deviation through our parameters α, β, γ. This implies minimizing the L2 norm of the skew/standard deviation vectors after normalization. To simplify the optimization, we only vary α from 0.33 to 1.0 and adjust β and γ such as β represent 60% of 1 - α and γ represent 40% of 1 - α (i.e. we put more weights on level 2 than level 3 information context as level 3 has more information). We report the values of standard deviation and skew obtained for the different combinations in Figure <ref>.

The minimum is obtained for α = 0.6 when considering HumanEval+ and α = 0.65 for ClassEval. To have a common set of parameters to allow for comparison between benchmarks, we will set the weights at α = 0.625, β = 0.225, γ = 0.15. In practice, for these values, the skew and standard deviation values are close to the minimum for each benchmark.



 §.§ RQ1: 




The goal of this RQ is to show HardEval can reliably identify hard tasks in a benchmark. To do so, using greedy decoding (i.e. Temperature = 0.0), as done in the-state-of-the-art leaderboard <cit.> for each of 5 LLMs of our study, we evaluate each task using a single prompt to see if each LLM outputs a correct code using available tests. For the single prompt used, we chose the level 1 prompt obtained before rephrasing for each task, which is the closest to the original prompt of each benchmark. We did so for a similar reason as mentioned in Section <ref>, i.e. removing potential formatting issues and providing a fair comparison across tasks. For greedy decoding, we consider a task to be hard for an LLM if it does not manage to generate a correct output. We consider a task to be hard for all LLMs if a majority of LLMs fails to generate a correct code for that task. We use the opposite settings for easy tasks. At the same time, we collect tasks that are labelled as hard/easy by HardEval using our difficulty score (see Equation <ref>), which is when the score is above/below 0.5. Using those sets of tasks, we can compute the overlap between the sets, e.g., the tasks that are hard according to HardEval but not for the greedy decoding definition, for a given LLM or all LLMs. This allows us to see if the sets of tasks are similar across approaches or not. Then, we compute the accuracy of the greedy decoding over the tasks by splitting them into two sets of tasks, using the set of hard/easy tasks of HardEval. Similarly, we compute the median of the difficulty score of HardEval obtained on the tasks by splitting them into two sets, based on the set of hard/easy tasks from the greedy assessment, for each LLM. Intuitively, we should see a drop in performance of the accuracy of the greedy decoding on the tasks that are noted hard by HardEval and an increase in performance for the tasks that are labelled as easy by HardEval compared to the greedy accuracy of each LLM.

Computing the overlap between the sets of hard tasks for both approaches results in an overlap ratio ranging from 0.24 to 0.63 for HumanEval+ and from 0.56 to 0.62 for ClassEval depending on the LLM. Similarly, the overlap for the sets of easy tasks ranges from 0.54 to 0.78 for HumanEval+ and from 0.69 to 0.80 for ClassEval depending on the LLM. If we consider all LLMs, the overlap on the hard tasks is 0.46 for HumanEval+ and 0.62 for ClassEval, while for the easy tasks, it is 0.71 and 0.78 respectively. Thus the assessment of hard tasks between the two approaches is dissimilar while there are more similarities in the assessment of the easy tasks. The assessment seems also to be more similar for ClassEval compared to HumanEval+. Thus, while there exists an overlap between our proposed approach and the greedy traditional assessment, the overlap is not perfect. 

We provide the results regarding computing the metric of each approach on the hard/easy tasks given by the other one. Results are reported in Table <ref>. For each row of the table, we get, for an LLM, the results (median difficulty score or accuracy) on both HumanEval+ and ClassEval benchmarks using the subsets of hard/easy tasks per approach. For instance, for CodeLLama on HumanEval+, the median difficulty score on the easy tasks of CodeLLama for the greedy approach Greedy Easy, that is when the output for a task is correct, is 0.27. We also give in that case the first and third quartiles. For instance, still on HumanEval+, the first quartile for MagiCoder in Greedy Hard is 0.29 (i.e. at least 25% of the tasks have a difficulty score lower than 0.29). Similarly, for GPT3.5, the accuracy of the greedy approach on hard tasks according to HardEval is 8.33%. Moreover, we also report the accuracy of the greedy decoding over the whole benchmark for each LLM and averaged across LLMs.

Using Table <ref>, we make the following observations: 
 


Greedy decoding based majority assessment agrees with HardEval 
 


When using HardEval sets of easy/hard tasks, we see an increase/decrease in performances for both benchmark and all LLMs compared to the performance calculated over the whole benchmark (see Table <ref> Greedy Acc). This shows that the tasks identified as easy/hard by HardEval are indeed easy/hard, as the subset of hard tasks will result in lower accuracy and the subset of easy tasks in higher accuracy compared to the whole benchmark baseline. 
 


HardEval assessment on easy tasks relatively agrees with Greedy decoding 
 


We see that, for all LLMs, easy tasks for the greedy decoding (i.e., whether individual LLMs manage the tasks and, Overall LLMs, a majority of LLMs manages the task) obtain a median of the difficulty score < 0.5 showing that most of those tasks tend indeed to be easy according to HardEval. We note however that for HumanEval+, the third quartile is quite high for most LLMs (but below 0.5). As such, for HumanEval+, there is a higher variation in the easy tasks difficulty which is not noticeable when using the greedy decoding approach but that we can identify with HardEval. 
 


HardEval assessment on hard tasks disagrees with Greedy decoding 
 


For the set of hard tasks for the greedy decoding, we get a median difficulty score fairly close to 0.5, the only exception being for DeepSeek and MagiCoder on HumanEval+ (0.35, 0.38). However, the first quartile is below 0.5 (around 0.4) for most LLMs and so at least 25% of the hard tasks that are hard when using greedy decoding, i.e. pass/fail on a single prompt, are not hard according to HardEval. This however was to be expected, as we saw in Listing <ref>, using a single prompt, a task can easily be hard for one or more LLMs because of the formulation and/or underspecifications in the prompt. However, rephrasing the prompts or adding information can lead to easy tasks, something that a single prompt can not tell us.


[colback=blue!5,colframe=blue!40!black]
Findings 1: While using the correctness of the code over multiple LLMs on a single prompt (greedy decoding based) can be used to identify easy tasks, it is not reliable for identifying hard tasks. On the contrary, the difficulty score of HardEval can identify hard/easy tasks in a benchmark. 




 §.§ RQ2: 


Our goal in this RQ is to study the distribution of difficulty scores across tasks as well as the type of tasks that are difficult in the benchmarks under study. To do so, first, we use HardEval difficulty scores for each task calculated in RQ1, both for the Code LLM individually as well as all LLMs. By doing so, we can output the cumulative distribution of the difficulty score across tasks for analysis. Then, we use the BERTopic model as described in Section <ref>. The topic analysis returns 17 topics for HumanEval+ and 21 topics for ClassEval. The two first authors manually check the tasks in those topics to provide a general name to the topic that is as representative as possible of the tasks within the topic. We then calculate the proportion of hard tasks in each topic. 




The results for the cumulative distributions are given in Figure <ref> for both benchmarks. On both the benchmarks under study, 63% of the tasks for HumanEval+ and 65% of the tasks for ClassEval, Overall LLMs, obtain a difficulty score of less than 0.4 and only 21% of the tasks for HumanEval+ and 27% of the tasks for ClassEval have a difficulty score higher than 0.5 thus making them hard tasks with our definition. Thus, from a high-level point of view, both benchmarks seem to have a similar difficulty profile, which echoes the accuracy presented in Table <ref>. However, there are a number of differences which we can analyze thanks to HardEval: 1) ClassEval tends to have a high proportion of trivial tasks (< 0.1) with 37% of ClassEval's tasks while on HumanEval+ it is 16% and there are much more variations across LLMs, 2) CodeLLama and CodeGemma have the most difference in terms of performance between ClassEval and HumanEval+. For instance, 50% and 33% of the tasks of HumanEval+ are hard for CodeLLama and CodeGemma respectively compared to 33% and 28% on ClassEval. We suppose the difference comes from the fact that HumanEval+ has more algorithmic tasks while ClassEval tasks are more related to general development tasks, which could benefit those two LLMs, and 3) ClassEval has more difficult tasks overall. This echoes the subjective assessment that ClassEval is harder than HumanEval+, as it relies on a class context. Finally, while the accuracy over both benchmarks seems similar (see Table <ref> and Table <ref> Greedy Acc for the same Code LLMs), the distribution of difficulty of tasks between two benchmarks can be quite different. To verify that, we compute a k-samples (k = 2) Anderson-Darling test <cit.> to assess if the distributions between ClassEval and HumanEval+ are the same for a given LLM or overall LLMs. This leads to a significant p-value < 0.05, i.e. the distributions are indeed different, in all cases except when comparing for GPT.




We then proceed to extract topics using BerTopics as described in Section <ref> to group tasks into topics and analyze them. The output is given in Figure <ref>. One can make several observations from the obtained results. First, not all LLMs necessarily encounter the same difficulty depending on the tasks. For instance, GPT has more trouble on Topic 12 in HumanEval+ but less so on the Topic 11 of ClassEval compared to Deepseek, as highlighted by the higher proportion of hard tasks in this topic as well as the average score over the tasks. This indicates that each LLM has its strengths and weaknesses that originate from its architecture and the data it was trained on. While the prompts we use could affect this outcome, the fact that other LLMs have, on the same topic, opposite behavior instead suggests that the root cause is mainly due to training and/or models' architecture. Secondly, some topics that are easy/hard across LLMs highlight some general behaviors. For example, Topic 15 of HumanEval+ or Topic 13 of ClassEval have almost no hard tasks in all models and overall. On the other hand, Topic 16 of HumanEval+ and Topic 3 of ClassEval have a higher proportion of hard tasks and this is for all LLMs. ClassEval seems to have more topics with a higher proportion of hard tasks compared to HumanEval+ but, at the same time, a higher proportion of topics with no hard tasks. These predominantly hard or easy topics highlight limitations in the ability of the LLMs. We note nonetheless, based on the extracted topics, there does not seem to be a divide between topics with topics with all hard tasks and topics with all easy tasks. This further motivates us to use a combination of hard and easy tasks from a similar topic to generate new tasks: 1) as topics are rarely only hard or easy, combining both easy and hard tasks or augmenting hard tasks, for which we already have a result, increase the likelihood of re-obtaining a hard task, 2) by leveraging topics, we can easily make use of the thematic similarity between tasks to combine/augment them. This, for instance, prevents potential arbitrary transformation when combining tasks that are not directly related which can have an unpredicted impact on the LLMs.

[colback=blue!5,colframe=blue!40!black]
Findings 2: At a high-level, HardEval difficulty score profiles of both benchmarks are similar, which echoes the similar Accuracy obtained in the Greedy decoding. However, ClassEval exhibits more hard tasks than Humaneval+ (27% against 21%) and the difficulty profile between benchmarks on the LLMs are different, which highlights HardEval local assessment usefulness. Inside the benchmarks, while there does not seem to be a clear divide between hard/easy topics, some topics for both benchmarks are predominantly composed of hard tasks or easy tasks, which can show potential ability limitations of the LLMs.




 §.§ RQ3: 


[language=python, caption=Oracle code of a new task generated. We highlight in yellow the change compared to the original hard task used for generation. Changes are made to be minimal so the task is as similar as possible to an already established hard task., escapechar=!, label=list:new_task]
def solve_with_vowel(s):
    !def vowel_change(ch):!
        !return ch if ch not in "aeiouAEIOU" else chr(ord(ch) + 2)!
    ans, has_letter = "", False
    for ch in s:
        if ch.isalpha():
            has_letter = True
            ans += !vowel_change!(ch.swapcase())
        else:
            ans += ch
    return ans if has_letter else s[::-1]



Finally, this RQ aims to show that, using identified hard tasks, we can further generate new hard tasks that are targeted at specific problems. We do not aim to be systematic, but merely to show that, using identified hard tasks and topics, it is possible to generate new tasks. To do so, we use the result from the RQ2 (i.e., the topics) and select the first three topics in terms of the number of tasks with at least a third of the tasks being hard. We do so to have a sufficient number of LLM-easy and hard tasks for the Task LLM to be able to provide meaningful new tasks. For each topic, using the Task-LLM, we generated tasks, checking each of them manually for validity, until we got 5 different new tasks using the process described at the end of Section <ref>. Then, we manually write an Oracle code as well as unit tests for this task. When possible, we try to remain as close as possible to the oracle code of the base task used in the generation process. Finally, we assess the newly generated tasks to the same Step 1 and Step 2 of HardEval to measure the difficulty score. We thus obtain, for each of the picked topics, new tasks that are assessed. We present an example of Oracle code from a newly generated task using the Task LLM in Listing <ref>.




The results for the difficulty score obtained for the newly generated tasks are given in Table <ref>. Most tasks generated following our approach turn out to also be hard. We note that a few of them end up being below the 0.5. We see two possibilities for this: 1) either the base task was easy and we added a constraint from a hard task, which is not what makes the task hard in the first place, or not completely, 2) adding a small constraint to a hard task will modify the prompt and so, we might end up with some sets of prompts that are more favourable for the generation of codes for LLMs. However, we note that the difficulty score is fairly close to 0.5, illustrating that the task is far from trivial. The only task with a fairly low score is on Topic 9 for ClassEval. This task was obtained by adding a hard constraint from an easy task. Yet, as the constraint was not applicable perfectly, we added to relax it in the implementation, which might explain the low score. In particular, the new task-generation process can vary across topics and benchmarks. For instance, while generating a sequence with added constraints on HumanEval+ is really simple for the Task LLM, coming up with new methods for already established classes on ClassEval can be quite hard, especially when the methods to use as a base are very different from each other. In practice, this results in the large majority of the tasks being a slight modification of a precise hard task. However, while not perfect, simply reusing identified hard tasks can be a simple way to generate new tasks that are focused on some specific topics. 

[colback=blue!5,colframe=blue!40!black]
Findings 3: Using identified topics and hard tasks rated by HardEval, we can generate new tasks. Those new tasks are similarly hard, showing it is possible to generate new hard tasks using our approach.




§ DISCUSSION


With HardEval, we show that assessing the difficulty of a task from the point of view of LLMs is feasible as well as identifying hard tasks. In particular, hard tasks identified with HardEval are truly hard as they are more resilient to the potential variability of the prompt contrary to assessing using a single prompt. While relying on human judgment to define a task difficulty could be used, it is unlikely to work for LLMs as they process code and task instructions differently than humans <cit.>. Ouyang et al. <cit.> showed that the difficulty assessed by humans on code competition only weakly correlates with the test pass rate variance across code samples generated by ChatGPT. Thus, it is likely that human assessment or related metrics might not be accurate for estimating LLMs' task difficulty contrary to the difficulty score of HardEval. Our proposed approach, while applied in experiments to code generation, can be extended to other code-related tasks. Indeed, the transformations we used are easily transferable to other paradigms and new transformations could be defined.

Analyzing the tasks that were labeled as hard, we observe that LLMs tend to disregard certain parts of the instructions, which can lead to certain test cases failing thus making the code incorrect. That is why similarity helps (i.e., code is incorrect but very similar to a correct one) (Finding 1). This phenomenon happens generally as the number of constraints in the prompt (and in the function) increases or if the prompt becomes longer. This explains why coding-competition or algorithmic-like problems in both benchmarks, which require a lot of description to express the many operations and constraints, generally lead to a higher difficulty, as the LLM is overwhelmed with information. This echoes Ouyang et al. <cit.>, as they showed that description length impacts the test pass rate in ChatGPT. This however can not on its own explain why difficulty could be higher as Level 3 prompts tend to have a better score and are longer than lower-level prompts. We also note that corner cases generally have to be explicitly mentioned otherwise LLM might miss it. For instance, stating when checking a date that February can have 29 days, or when checking a condition in a string, the string can be empty, can improve generation. This type of information is generally omitted or implicit in prompts in benchmark (and so in our level 1 prompts) but is more likely to be explicitly mentioned in level 2 or 3 prompts. This also can explain why using the correctness of codes over one simple prompt is not enough to quantify the difficulty of a task (Finding 2). 




§ THREATS TO VALIDITY

Internal validity: One threat to the validity of our results comes from the way we measure the difficulty of tasks. To mitigate this threat, we combined two metrics to see if a programming task is difficult for LLMs to code using functional and syntactic correctness and showed it correlates with the Context Information level. Those metrics were used in studies investigating LLM capabilities for code generation. We used a linear combination of all levels having the minimum weight for Level 3 (with maximum contextual information) and maximum for Level 1 (with the least information) to put more emphasis on the Level 1 prompt score. While this might influence the decision on the hard tasks, it is more likely to have some easy tasks be considered as hard, as we showed that the metrics correlate positively with the level. Another threat might come from the generation of the prompts. The transformations used are motivated based on existing works. Moreover, while we have made use of state-of-the-art GPT-4 to generate them, we manually checked the prompts to make sure that the generative process did not alter semantics. Finally, another threat to validity is the fact we did not consider examples in the prompts. Not adding examples in prompts, as we mentioned, can decrease the performance of the models and so add artificially hard tasks. This was nonetheless necessary to make the comparison fair across tasks as not all examples contain the same information. However, beyond having some easy tasks be considered as hard, it does not affect the applicability of HardEval.

External validity: Our sets of tasks from the selected benchmarks could be a threat. The collection of tasks might not be representative of real-world programming tasks to assess LLMs' capabilities. Nonetheless, both HumanEval+ and ClassEval are used to evaluate Code LLM performance <cit.>. Moreover, the objective of the study was to set the scene for the assessment of task difficulty rather than dwell on the practicalities of the task. So, we believe using tasks from these two benchmarks provides a diverse set of tasks to assess the difficulty of code generation using LLMs.

Reliability validity: We detailed our methodology, and to enable other researchers to reproduce or expand upon our work, we make our code and data publicly available <cit.>. 




§ CONCLUSION

In this paper, we realized the first attempt to investigate the difficulty of code generation tasks for LLMs. To do so, we proposed a novel framework, HardEval, to assess the difficulty of code generation tasks for LLMs, identifying hard programming tasks, and creating new hard tasks. We evaluated HardEval on two well-known benchmarks for LLM-based code generation: HumanEval+ and ClassEval. A set of diverse prompts is crafted for each task including different levels of contextual information about the task and various rephrasings, yielding 18 different prompts per task. 5 state-of-the-art LLMs were then employed to solve coding tasks using the set of prompts. We proposed a difficulty score based on functional and syntactic correctness. Tasks with high average difficulty scores across all LLMs were identified as hard coding tasks. Based on thematic similarities, we clustered tasks from both benchmarks into respectively 17 and 21 topics to inspect their characteristics. For selected topics, we generated new tasks. New tasks were then assessed to see if they were really hard based on the proposed score, revealing their difficulty for LLMs to be coded. This new set of tasks can be used as a targeted benchmark to assess the effectiveness of LLMs for code generation tasks or to improve on them. We hope this research paves the way for further research on improving LLMs for code-related tasks.

In future works, we plan on expanding the evaluation of HardEval to other domains such as code completion or summarization. Moreover, we aim to improve the new task generation method so the process is more reliable as well as to be able to have better control of the difficulty level of the task generated. In particular, this would allow us to generate a benchmark with a desired difficulty profile.


This work was supported by: Fonds de Recherche du Québec (FRQ), the Canadian Institute for Advanced Research (CIFAR) as well as the DEEL project CRDPJ 537462-18 funded by the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Consortium for Research and Innovation in Aerospace in Québec (CRIAQ), together with its industrial partners Thales Canada inc, Bell Textron Canada Limited, CAE inc and Bombardier inc.


ACM-Reference-Format




 Paper Title: WebApp1K: A Practical Code-Generation Benchmark for Web App Development.

	
Candidate Distant Trans-Neptunian Objects Detected by the New Horizons Subaru TNO Survey
    [
    
========================================================================================



§ ABSTRACT 
We introduce WebApp1K, a practical code-generation benchmark to measure LLM ability to develop web apps. This benchmark aims to calibrate LLM output and aid the models to progressively improve code correctness and functionality. The benchmark is lightweight and easy to run.

We present the initial version of WebApp1K, and share our findings of running the benchmark against the latest frontier LLMs. First, open source LLMs deliver impressive performance, closely trailing behind GPT-4o and Claude 3.5. Second, model size has strong correlation with code correctness. Third, no prompting techniques have been found to lift performance either universally to all models, or significantly to a single model.

	


§ INTRODUCTION

Large language models (LLM) have been widely adopted in the software industry to enhance project velocity and code quality. Many coding benchmarks have been developed focusing on different aspects such as algorithms<cit.>, data science<cit.>, class-level code<cit.>, execution<cit.>, resolving software bugs<cit.>, etc., more efforts are needed to calibrate LLMs' ability to assist software development for consumer-grade applications. While the current norm of LLM application is in the form of copilot or code assistant to existing software developers<cit.>, we argue that LLMs already, and will continue to, empower more and more non-developers (by traditional standards) to write code. Below are some scenarios.



  * A handful of first-time startup builders brainstorm a new social network helping a group of underprivileged individuals, and bootstrap themselves to launch a prototype to the public. LLMs play pivotal role during each phase of the project, e.g. planning, tech stack choice, task assignment, code generation, and deployment.


  * A small group of employees volunteer to a greenfield project funded by a mid-sized software company. While enjoying the freedom of scope and high-level blessing, the team finds itself understaffed and under-resourced. The founding members spares no time to reach an all-hands-on-deck decision, and turns everyone into a developer with the aid of LLMs, very similar to the above startup scenario.


  * A college student in their senior year, determined to be a top competitor on the job market, decided to boost their GitHub profile by some impressive demos. They first consulted LLMs on popular frameworks and best practices, then delved into a project and continue to utilize LLMs to help maximize velocity. In particular, LLMs are used to generate, from scratch, functional code expected to be bug-free and ready-to-build.


Arguably, the ensemble of users exemplified above is a much larger group than the professional developer community. Undoubtedly LLMs will deliver tremendous and life-altering values to this group of users. However, as will be revealed by this study, the improvement space for LLMs is huge. To this end, we present a preliminary work to benchmark LLM performance on practical application development.

We aim the consumers of this benchmark to be practitioners working closely with LLMs, e.g. those who ship models (pre-training and post-training), or those who build LLM applications and decide how to choose/use their models. Lots of considerations are taken into the benchmark usability.


	
  * Marketability Since the benchmark aims to evaluate usefulness of LLMs to practical software development, the model artifact, i.e. the generated code, must meet the needs of customers or employers. As such, LeetCode-type algorithmic problems, while extremely valuable in other contexts, are not applicable here. Moreover, the evaluation would be a lot easier if the knowledge and capabilities to create marketable code is inherently possessed by LLMs, i.e. present in their training dataset.

	
  * Metric In order to present a clear direction to model hill climbing, the metric should be simple with unambiguous answer to each problem of the benchmark. Fortunately, low-hanging fruits are available because the code correctness as the most basic criteria is still an unsolved problem.

	
  * Overhead Training LLMs is a high-budget and high-risk process introducing lots of stress. To exacerbate the problem, training environments and tools are primitive, also vulnerable to deprecation given the continuously escalating model scale and the training infra it demands. Therefore, one should assume that bare-bone Linux is all we can count on. Anything more sophisticated, e.g. sandbox or virtual environment, will decrease the benchmark adopt-ability, also taking more time to run, potentially causing training delays.


We are blessed by achievements of open source contributors to address all these challenges. By building our benchmark around the most successful open-source frameworks, the generated code is widely considered marketable by the industry. Also these frameworks have abundant quality code and best practices shared on the Internet, already handily included in the training dataset of modern LLMs. Finally, the rich ecosystem of these frameworks equips us with stable and well-distributed testing tools to address the metric and overhead challenges.

We introduce WebApp1K, a benchmark to evaluate web app code generation performance of LLMs. We start with React, a JavaScript framework widely recognized as the top choice for web app development. Below are our main findings:


	
  * Open source models deliver strong results. The top performing model is DeepSeek Coder V2 which closely trails behind GPT-4o and Claude 3.5 Sonnet, the top two LLMs.

	
  * There is significant performance gap between the big model and the small model of the same family, proving that the scaling laws still apply in our evaluation context of code generation.
	
	
  * No prompting techniques have been found to universally improve performance for all models or significantly lift performance of a single model. On the other hand, the strong performance of top performers (GPT and Claude) under a very simple prompt suggests that LLM should be able to code correctly as long as the expectations (i.e. test cases) are clear.


We share three artifacts on GitHub and Huggingface: the dataset containing all 1000 problems of WebApp1K<cit.>, the script<cit.> to run WebApp1K, and the leaderboard<cit.>.

The rest of this report is organized as follows. Section <ref> details how WebApp1K was built. Section <ref> presents our evaluation methodology, various experiments, and performance results of the latest LLMs. Finally, Sec <ref>, conclude contributions and discusses future works. 


§ BENCHMARK

We choose React<cit.> as the first evaluation subject of our benchmark. Introduced in 2013, it has remained a top choice for web app builders for the quality and stability of the framework itself, productivity gains to developers, and the rich ecosystem. In addition to these marketabiliy benefits, it also constructs a higher level abstraction above HTML syntax and DOM manipulation, allowing developers to encapsulate their business logic to succinct and reusable code. These technical advantages also benefit the code generation evaluation on LLMs.

Consider a blogging website, in which a user adds comment to an existing blog post. This user journey is simulated by the unit test in Table <ref>. Here,  is a lightweight setup to mock a successful API response without running any additional software components. The following  lines simulate user actions (text input, mouse click etc.). Finally,  lines examine the expected outcome, i.e. the mocked API should be invoked exactly once and the system response of success should appear on the updated webpage. Similarly, the pairing failure case is shown in Table <ref>, where a mocked API failure is expected to lead to error message on the updated webpage.




Table <ref> shows the verbatim code generated by GPT-4o as an attempt to pass these two tests, which it does. Besides the toyish API and purposely simplified application semantics, the code is functional and conceptually reusable to a web application which needs social commenting functionality.


The WebApp1K benchmark consists of 1000 problems, each depicting an unique user journey illustrated above. The construction of this benchmark is a combination of human curating and data synthesis by LLMs.

First, human proposes 20 web apps and their feature sets. They are listed in Table <ref> of the Appendix section. Then, within the scope of each app, we ask GPT-4o to describe 50 different user journeys, then review and iterate with the LLM until quality and relevancy are satisfactory. Finally, we feed each user journey to GPT-4o, prompting it to codify the journey into a success case and a failure case. Each of them strictly follows the  sequence illustrated in Table <ref>. All test cases can be found in our GitHub repo<cit.>.


§ EVALUATION RESULTS



 §.§ Experiment Setup

The most straightforward way for us to access LLMs are public token-based APIs. For top close-sourced models, our only option is via the owners' APIs. The top open-sourced models are hosted by a few platforms, among which we choose Fireworks<cit.>.

We evaluate three closed-source, and three open-source model families listed in Table <ref>.


Below is the prompt we use. The majority of it is the test code exemplified in Tables <ref> and <ref>.

    Generate {file_name} to pass the tests below:    
       {success_test_code}{failure_test_code}. RETURN CODE ONLY.


We use pass@k, a metric defined in <cit.> and commonly accepted by subsequent coding task evaluation works. Due to budget and rate limit constraints, we evaluate each LLM-problem pair 10 times, i.e. n=10. Since k must be smaller or equal to n, we measure pass@1, pass@5, and pass@10. 


 §.§ Parameter Tuning

Although each API bears its minor difference, all APIs are heavily influenced by the design of OpenAI API. Table <ref> lists the tunable parameters exposed by each API. Since we do not know the default parameter value set by each API provider, we explicitly set the same parameter values to all LLMs under evaluation, whenever applicable. To limit the search space, we only tune temperature and top_p, the two most popular parameters available on all platforms. For other parameters, we assign fixed value to them across all LLMs.


We conducted a grid search to locate a sweet spot at which all LLMs deliver near-best results. We chose 100 random problems from the benchmark, 5 out of each application. We then choose the large model out of the five leading model families, and measured their pass@1 (n=1) on the discrete 2D space of temperature and top_n, where temperature = 0, 0.1, 0.2, ..., 1, and top_p = 0, 0.1, 0.2, ..., 1.


Table <ref> presents the lowest and highest Pass@1 value by each LLM in this grid search. Based on the results, we finalize our parameters as follows.

    temperature = 0.2 
    
    	top_p = 0.8 
    
    	top_k = 40 
    
    	presence_penalty = 0 
    
    	frequency_penalty = 0


As will be shown later, results of our full-scale evaluations also align with this small-scale experiment, except for the Deepseek coder model<cit.> whose performance exceeds expectation. Also worth noting is that open-source models exhibit larger performance variation than closed-source models.


 §.§ Prompt Experiments

We also study whether more sophisticated can lift the model performance.

The first experiment is system prompt, which assigns an explicit role to the LLM and raises its awareness. Available in all APIs we run, it complements the user prompt (Equation (<ref>)) which gives detailed instructions to LLM. Equation (<ref>) shows our system prompt.

    You are a code generator.


The second experiment is verbose comment, which aims to help LLMs better understand the semantics of  tests it tries to pass. For each of the 1000 problems, we feed its test code to GPT-4o and ask for English summary of the expectation in multiple sentences. The summary is then inserted into the test code. Table <ref> shows the verbose comment variant of the test code in Table <ref>.


 
The third experiment is error debugging. If the generated code fails the test, we add the failed code and the error log to the prompt, hoping the LLM will generate the correct code by learning from its own mistakes. Below is the prompt.

    {failed_implementation}
       The above code is the implementation of {file_name}. It failed the tests below
       {success_test_code}{failure_test_code}
       Below is the test log
       {error_log}
       Try to generate {file_name} again to pass the tests. RETURN CODE ONLY.


For all three prompt variants, we measure pass@1 (n=1) against all 1000 problems of the benchmark. Also in each experiment, we apply one prompt variant only, and compare it against the control test using the original prompt (Equation (<ref>)). Table <ref> summarizes the relative performance gains/loss of each variant.


To our surprise, we are unable to find a prompt variant delivering universally positive (or negative) impacts to all LLMs. Also we observe the huge swing in the error debugging column. The situation is unique here because this technique is not needed if the model output is correct on the first try. Strong models like GPT-4o can produce high Pass@1 (n=1) closed to 0.9, which significantly shrinks the sample size.

As such, we can not recommend LLM users to adopt or avoid any prompting technique we have experimented. More investigations are definitely needed here, especially why a seemingly harmless prompt variant should cause regression. 

Also for the sake of fair evaluation, we decide to use the original prompt (Equation (<ref>)) in our full-scale evaluation.


 §.§ Results

Table <ref> shows results of full-scale evaluation in the form of leaderboard. Models are ranked by Pass@10. where gpt-4o is the winner. Results of Pass@5 completely agrees with this ranking. Results of Pass@1 also largely agrees, except claude-3.5-sonnet which surpasses gpt-4o as the winner.


We believe the ranking of this leaderboard meets expectations of the LLM community. It is worth calling out the truly impressive performance of open source LLMs, especially Deepseek.

Also when comparing the large and small models of the same family, it is evident that scaling laws still holds. In fact, of the six open source models whose parameter counts are available, their performance ranking aligns with their size ranking, except deepseek-coder-v2-lite-instruct which itself is a coder model.



§ CONCLUSION AND FUTURE WORKS

This report presents our preliminary contribution to LLM code generation. We proposed WebApp1K, a practical and comprehensive benchmark focusing on web app development. To recap findings via this benchmark:

	
  * Open-source LLMs deliver impressive results
	
  * Model size plays pivotal role to code generation performance
	
  * No prompting techniques have been found to either universally help all LLMs or significantly help a single LLM


We plan for the following future works.

First, more frontier LLMs are released as of the writing of this report, namely Llama 3.1<cit.> and Mistral Large 2<cit.>. We will report our evaluation results as soon as possible.

Second, we need to make the benchmark more difficult, as it is already saturated by top performers, i.e. GPT-4o and Claude 3.5. While this benchmark, as it stands now, can help other models to improve themselves, we need to make improvement space for the best models, as bigger and stronger models will be released sooner rather than later.

Finally, this work produce lots of error logs which we will dive deeper. We expect to uncover useful insights, which could help models of all rankings to continue the hill climbing.


§ APPENDIX: APPLICATIONS OF WEBAPP1K


plain



 Paper Title: Predicting Software Reliability in Softwarized Networks.






Predicting Software Reliability in Softwarized Networks


    Hasan Yagiz Özkan, Madeleine Kaufmann, Wolfgang Kellerer, Carmen Mas-Machuca 
Chair of Communication Networks 

Technical University of Munich

Munich, Germany 

yagiz.oezkan@tum.de, madeleine.kaufmann@tum.de, wolfgang.kellerer@tum.de, cmas@tum.de




















    30 July 2024
===============================================================================================================================================================================================================================================================================



§ ABSTRACT 

Providing high quality software and evaluating the software reliability in softwarized networks are crucial for vendors and customers. These networks rely on open source code, which are sensitive to contain high number of bugs. Both, the knowledge about the code of previous releases as well as the bug history of the particular project can be used to evaluate the software reliability of a new software release based on SRGM. 

In this work a framework to predict the number of the bugs of a new release, as well as other reliability parameters, is proposed. An exemplary implementation of this framework to two particular open source projects, is described in detail. The difference between the prediction accuracy of the two projects is presented. Different alternatives to increase the prediction accuracy are proposed and compared in this paper.






Bug History, Version Control System, Regression Model




§ INTRODUCTION



Delivering high quality software is of utmost importance for software vendors and platforms. For that purpose, they aim at reducing both, the number of undetected and unsolved bugs, to increase the software quality and dependability<cit.>. The software development process is a cyclic process that delivers periodically releases with new functionalities, solved bugs, etc. Each new release potentially has residual bugs (i.e., undetected and unsolved bugs from previous releases as well as from the added code). A software bug can be defined as an anomaly that spoils the normal operation of a software. Hence, bugs are big thread to the reliability of the software.



Furthermore, software development is nowadays being open source. Open source software development strategy offers many advantages in cost, service and support. As a result of these advantages the number of open-source projects and  the number of contributors increase consistently. Big open source projects are mostly based on the contributions from different programmers and/or teams, which develop different parts of the same software. Although it is an efficient and low-cost development process, it is prone to contain higher number of bugs than in closed software development processes<cit.>.


The problem for both, the developers/vendors of the new release as well as the clients considering its adoption, is how to evaluate the quality and dependability of new releases. For this purpose,  we propose in this paper a methodology to estimate the number of residual bugs of a new software release.  



Most of the open source projects,

adopt bug-tracking systems to efficiently manage any detected software bug <cit.>. These bug-tracking systems facilitate the bug detection and resolution by offering transparency on providing information about all the detected bugs, which can be processed by data-mining techniques <cit.>. These techniques can be helpful to create the bug history as well as understand the relation between the bug history and code changes of the different releases.



Software code metrics can be defined as the information about the properties of a source code like for example the complexity of the functions or size of the code. These code metrics have been used to predict quality, security vulnerability, and maintainability of a software <cit.>. This information can be easily gathered with help of online tools or programming libraries. Moreover, the history of the code metrics, which relates to the project releases can be calculated and considered in the residual bug estimation as proposed in this work. 











In this paper we present the following contributions: 

    
  * We propose a framework to predict the number of bugs in a new software release. This framework combines the bug history and the code metrics history of previous release(s) of a software project.
    
  * As some new projects do not have enough previous releases to make an accurate prediction. For this purpose we evaluate the usability of cross-project prediction and evaluate the impact of the prediction accuracy.
    
  * As the software development of some projects lasts several years (e.g., the first release of ONAP project was released in 2017 and new releases are still coming), aspects such as trends in the software development strategy or changes in the developer teams can impact significantly the bug estimation. Because of these changes, the usage of all previous releases can be unfavorable for the prediction. The impact of the number of previous releases on the prediction quality will be also investigated. 

    
 

After finding the best number of releases to consider and favourable metrics to feed the model, this framework can be used to predict the number of bugs in a new software release before its code freeze, which indicates the beginning of debugging process. 

This paper is structured as follows: Section <ref> summarizes the previous works that investigated the relation between bugs and code metrics and/or the effects of bugs on software reliability. In Section <ref> the framework to predict the number of bugs is discussed. An exemplary implementation of the proposed framework is explained in Section <ref>. The numerical results of this exemplary implementation are presented in Section <ref>. Finally, the paper concludes with some future work ideas in Section <ref>.



§ RELATED WORK




In this section works that combines the bug manifestation process and the code metrics are discussed by explaining the used methodologies and their outcomes. This section also introduces works that discuss the relation between bugs and software dependability.

In <cit.> the authors used different machine learning methods like linear regression, decision tree and multilayer perception to estimate the number of bugs in a project. The machine learning models are fed by the  code metrics like number of lines of code (LoC), number of commits. Based on data from 15 different projects from <cit.> the authors found that the linear regression model outperforms the other machine learning models and machine learning models can be successfully used to predict the number of bugs before the release of the software. As machine learning model they found that the linear regression model performs better than compared models and machine learning models can be successfully used to predict the number of bugs before the release of the software. Their model only consider the prediction after the release date. A prediction before release date might offer a possibility to the developers to provide more reliable software.


The authors of <cit.> predicted the number of bugs in a file with code metrics. They compared efficiency of four different code metrics to predict the number of bugs in a file. The results of their predictions were significantly higher than real number of bugs. They found that the depth of inheritance is better than the compared metrics for the bug prediction, they only considered one single metric for the prediction. In <cit.> authors  discussed the importance of choosing the correct metrics for the prediction of the number of bugs. They compared correlation based and wrapper based metrics for metric selection. They used different prediction models with multiple inputs for the considered models. They found that metric selection has an important effect on the prediction quality. The wrapper based metric selection found to be better than correlation based selection. However, they did not investigate correlation for different releases of the same project. This is an important issue because the similarity of the different releases of same project is expected to be higher and the metric selection based on that is expected to be more reliable. 

Instead of code metrics authors of <cit.> collected metrics depending on the judgement of the project managers of 31 different projects. These metrics includes information about design, documentation, scale of new code etc. Their method found more successful to predict the number of residual bugs of bigger projects compared to smaller projects. Although they could predict the number of bugs successfully the metrics they used is not easy to obtain and depends on personal comments of the project managers.

Instead of predicting the number of bugs some works focus on either a software is reliable or not. Reddivari and Raman <cit.> applied different machine learning models for software quality prediction. They made binary decision about reliability of a system 'faulty' or 'not faulty' and classified its maintainability into three groups 'light', 'medium' and 'heavy' depending on code metrics. The Random Forrest algorithm found as a reliable classifier for both reliability and maintainability prediction. Visagan et al. <cit.> predicted either a software is reliable or not depending in its code metrics. They found that projects with high complexity have poor reliability. The binary prediction might be not considered as superficial for dependability analysis.

SRGM models are employed to predict the bug manifestation after the release date in <cit.>. It is been shown that these models can be used to predict the number of residual bugs some time after the release date of the new release. They also showed that this information can be used to make an interpretation on the reliability growth of the software.



§ PROPOSED FRAMEWORK



Inevitably, any new release of a software project comes out with some bugs, that could not be found during its development and debugging period. These bugs, referred as residual bugs, have negative impact on the reliability of the new release. The goal of this framework is to provide a procedure to estimate the number of the bugs in the new release of a software project. The predicted value can be used to foretell the percentage of the bugs that are to be discovered and how reliable a new release is. Based on this estimation, developers/vendors can decide when the code is mature enough to be released and also users/customers can decide whether to  upgrade to the new release or keep the older version depending on their reliability requirements.

Based on existing literature, newly added, removed features and modifications of the existing features have been identified as main contributors of residual bugs <cit.>. The proposed framework relies on the code metrics of the new and previous releases as well as on the bug history of this software project. Code metrics are measures that gives insights into the internal properties of the release (e.g., Number of lines, number of functions), as well as the development process of a particular software project (e.g., number of commits, number of developers). The proposed analysis relies on the correlation between these code metrics and the detected bugs of previous releases. The overview of the proposed framework is given in Figure <ref>.



Most of the software projects follows a release cycle with dates that indicates: i) the start of the release (t_s), ii) start of the debugging period, so called code freeze date (t_f) and iii) release day (t_r). Between t_s and t_f new features are added to the software. At t_f, the debugging period starts and no more new features are allowed. Thus in this work the code from t_f is considered as the code that contains all the bugs. 


Let us give an overview of the main steps to estimate the number of residual bugs on release k given the bug history and the code metrics of previous releases: 

    
  * Bug History: Many of current open-source software projects (e.g., ONAP, ONOS) use trackers like JIRA, Bugzilla and Mantis, which allow users and developers to report any bug they experience. These bug reports include detailed information such as creation date, bug description and current status of the bug (if it is still active, it has been solved, etc.). The bug history in this framework represents the distribution of the bugs among the releases based on either the creation date or the affected release information in the bug report.

    
  * Code Metrics: The code metrics give information about the source-code features (e.g., number of lines of code) or development characteristics (e.g., number of commits) in a version control system. Most open-source software projects rely on version control systems like github and gitlab. These version control systems contain the history of the development process of the software. The code metrics of a particular software on a particular date can be found from the code on that date. Tools like SonarQube <cit.> and python libraries like pydriller <cit.> support code metric calculations. 
    
    Some of the considered metrics like lines of code or number of files, provide fundamental information about the state of the code on a day. In this case the code from t_f will be used to calculate the code metrics. On the other hand some code metrics like new lines of code aim to describe the development process, in other words only the changes made for the considered release. In order to calculate these metrics we take into account the changes made between t_s and t_f. 

    
  * Bug History and Code Metrics Correlation: The proposed framework utilises the code metrics and bug history of releases 1...k-1 to predict the number of bugs in the new release k of a software project. This is done by finding a meaningful relation between the code metrics and bug history, and converting this relation into a regression model. This regression model together with the code metrics of the new release are used to find the number of bugs in a new release. For that purpose, the code metrics that correlate better along the bug history are preferred over the ones with lower correlation. Thus a valuable set of code metrics can be chosen with the help of correlation values.
    
    New projects might not have fair amount of previous releases to find a valuable correlation between bug history and code metrics. In case a project does not enough releases to introduce a meaningful correlation, similar software projects with same language, similar release cycle or code review structure can be used to fill this gap. Feeding the model with older releases might not increase the quality of the prediction, because the code structure, experience level of developers and writing style might change with time. Therefore a correlation analysis can be done before including all their releases.   



    
    
  * Fit Regression Model: Regression models are widely used for prediction and forecast <cit.>. A regression model defines a response variable y related to one or list of covariates x_1, x_2, ... , x_n. The distribution of y values is related to x values and y value can be formulated as <cit.>; 
    
    y = f(x_1, x_2, ... , x_n) + ϵ

    where ϵ corresponds to the error. From the given formula, the prediction value can be defined as y = f(x_1, x_2, ... , x_n).  In the presented framework, regression model is based on the relation between the number of bugs in older releases 1 ... k-1 (response variable ) and the code metrics of these releases 1 ... k-1 (arguments). In this work fitting a regression model means finding a function that minimizes the error value ϵ. Therefore the code metrics that offers high correlation with bug history are decent candidates to be used as arguments in the regression model.
    
    
  * Prediction with Regression Model: The fitted model then can be used to make a prediction of the target variable given the arguments of the new release. In other words, only the code metrics of the release k are given to the regression model and regression model returns the predicted number of bugs of release k.
    






§ IMPLEMENTATION



In this section as an example one particular implementation of the proposed framework will be described. This implementation aims projects that use Jira for bug tracking and github for version control. The implementation of each step from Fig. <ref> will be discussed in detail. The projects 'Open Network Automation Platform (ONAP) <cit.> and Open Network Operating System (ONOS) <cit.> are examined as example.











 §.§ Bug History


Each software project using JIRA has a URL page. For example, the Jira page of the open-source project ONAP is ('https://jira.onap.org'), which  will be used as example. Since Jira is mostly used as project management tool, the type of tracked issues includes a wide variety of different types of issues (e.g., "Bug", "Milestone", "Improvement"). Some of them are also used for documentation of the software development steps (e.g., 'Milestone', 'Improvement' and 'Project Plan'). Since our purpose is to obtain the bug history, our implementation builds the bug history by filtering all the issues of type "Bug". 

As an example, two of the bugs with their associated information have been shown in Table <ref>.





After filtering all the bugs, they are appointed to their corresponding release depending on their "First Affected Release" field information. Unfortunately, not all the bugs are properly defined and many bugs reported for ONOS do not have affected version in JIRA. In this case, those bugs are associated to the release based on their "Create Date" as most of them are found during the debugging period and coming weeks after its release date (t_r). Even if development of the new release k+1 is started, majority of the bugs after t_r are related to newly released release k. Thus if the creation date of the bug is 14 days after t_r of release k, the first affected version as considered as release k. The dates and ditribution of the unlabeled bugs to releases are visualized in Fig. <ref>.










 §.§ Code Metrics


The proposed implementation calculates the code metrics by analyzing the  code. Therefore the code has to be downloaded from Github before code metric extraction. 















Also some metrics are calculated as sum of the other metrics like number of new and modified LoC.



  §.§.§ Downloading the repositories from Github


Two dates t_s and t_f are considered to extract the code metrics. It is assumed that the bugs appear because of the newly added features and changes made between t_s and t_f. Thus all the bugs of the release is assumed to introduced before t_f in other words the code from t_f is expected to have all the bugs of the considered release.

Some of the code metrics that considered are measures of the state of the code like number of functions, total complexity of the project and number of thousand lines of code (kLoC). The code state from t_f is enough to get those information. On the other hand having the code status from one day is not enough to find some of the considered metrics which measures the changes made in the code. New kLoC, number of new functions are some examples to these metrics, that cannot be found from one state of the code. Since the changes related to the release are made after t_s, the code state from this date is considered as the old state of the code and state from t_f is considered as the new version. Therefore the code states from last github commit before t_s and the last commit on t_f are downloaded. the commits added after t_f are considered as part of debugging process. The t_s and t_f dates should be found in the website or documentation of the considered projects and given by user.

Some of the code metrics like number of commits or number of contributors cannot be found in source-code instead they should be gathered from github. The number of commits per release is also extracted during the code download process. In this work the metric 'commit' represents the number of commits between t_s and t_f like other metrics that measures change.









  §.§.§ Extracting the code metrics


After downloading the source-code of different releases from the github, they are analyzed to extract the code metrics. In this work the code metrics will be divided into two categories. The terms complexity metrics and code size metrics will be used to differentiate these categories. The complexity metrics are calculated from each function separately and they are metrics related to their cyclomatic complexity like number of functions with complexity value higher than 15. The code size metrics are the metrics which defines the length of code like number of thousand lines of code or the changes made in the code like thousands of new lines of code and modified lines of code. Two different tools are used for the calculation of the metrics of these categories. The code size metrics are found with Count Lines of Code (CLOC) <cit.>. It gives the information about number of lines of code, number of files and also calculates the differences between two folders in this manner. Lizard <cit.>, which makes function level analysis, is used for complexity metrics. It calculates the complexity of the functions and also used to calculate the number of functions and number of new functions.



  
Code Size Metrics

The code size metrics describe the metrics which relate to the length of the code and quantity of the changes made in the code. We used CLOC tool <cit.> to calculate the code size metrics, because it calculates not only the lines of code but also the differences between the code in two different directories. The CLOC tool goes through all files and returns LoC, new LoC, modified LoC, number of files, number of new files etc.. This part takes a directory as input and returns the related metrics as output. 

The projects that are analyzed in this work are Java based projects with around 70% of all the LoC is written in Java. Therefore the code size metrics are extracted also specific to files written in Java. The languages like 'yaml', or 'xml' are excluded, because they mostly do not contain logical operations. Since they are mostly do not include complex operations, it has been assumed that they are mostly not the reason behind the bugs and even some of them like might be created automatically.




  
Complexity Metrics

Complexity related information are collected with the help of the 'Lizard' Python library <cit.>. The input of our implementation is the directory of the considered project and as output it gives the complexity metrics for this directory. In this work cyclomatic complexity is used to evaluate the code complexity, which is found useful by many research and widely used <cit.>. It represents the count of linearly independent paths in a piece of code. 

The counts of the functions with complexity values higher than specific thresholds (10, 15 and 20) are one of the extracted complexity related metrics. Additionally, the summation of the complexity numbers of all functions is considered. The main purpose of this tool is calculating the complexity of functions meanwhile it goes through all of the functions in the analyzed file. Thanks to this strategy the number of functions is also calculated.

In order to get complexity metrics for new and modified functions, the 'filecmp' Python library is used. First newly added and changed files in the project are found, then all the functions in the mew files are added to the count of new and modified functions. All functions in modified files are compared and if they are new or modified, they will also added to the count of new and modified functions.





 §.§ Bug History and Code Metrics Correlation


The code metric extraction returns 43 different code metrics. Since the number of metrics is high, some of them should be preferred over the others to predict the number of bugs in the new release. A meaningful relation between code metrics and bug history has been searched to choose the code metrics for prediction. Pearson correlation coefficient (PCC) is a measure for linear correlation to evaluate the correlation between two sets of data <cit.>. In order to evaluate the correlation between bug history and code metrics PCC is used. PCC gets a value between -1 and +1, where 0 means no correlation between two sets of value. Positive PCC value indicates positive correlation and negative PCC value means that correlation is negative. Higher absolute value of PCC indicates stronger correlation, where -1 and +1 point to perfect correlation. If the absolute PCC value is higher than 0.7, the correlation is considered as "high" correlation, whereas a value between 0.4 and 0.7 indicates "significant" correlation <cit.>. The correlation is weak if the PCC is below 0.4. The metrics with higher absolute PCC are expected to be more valuable for prediction. In this work 'pandas' library <cit.> is used to calculate the correlation between bug history and code metrics. 



 §.§ Fit Regression  Model


The experience from older releases is used to calculate the number of the bugs in the new release. The code metrics that has higher correlation with number of bugs has been chosen to fit the regression model. Since PCC gives information about linear correlation, linear regression model is preferred. The formula of the linear regression model is given in the following expression <cit.>:


    y_i = β_0 + β_1x_i1 ... + β_nx_in + ϵ_i,i = 1, ..., k-1


where β_0 is the intercept, β_1, β_2, ... , β_n are the slopes of the line and ϵ_i is the error value. In this work k-1 represents the number of older releases used to fit the regression model, y_i is the number of bugs in release i and x_in represent the nth code metric value for release i. Fitting the regression model aims at finding the optimal β values that minimize the error ϵ for all the older releases. In this work 'scikit-learn' library <cit.> is used to fit the regression model. Code metrics of the older releases and number of bugs of the same old releases are given to the regression model to fit.



 §.§ Prediction with Regression Model


After the model is fitted the code metrics of the new release is used to estimate the number of bugs in the new release. The calculated β values are express the fitted regression model. The code metrics values of the new release k (x_k1, ... , x_kn) are given to the fitted model to predict the number of bugs in release k (y_k). Since the β values are chosen to minimize the error ϵ_i for the older releases, y_k value is expected to be close to the real y_k value. 



§ RESULTS



The implementation described in Section <ref> is applied to two different open source software projects ONOS and ONAP. In this section the results for each step of the framework will be given.





 §.§ Bug History


Let us start with the ONAP project, whose first release Amsterdam was released on 16 November 2017. Since then, 10 more releases have been delivered. An ONAP release from its start t_s to its release t_r takes approximately 6 months. From its start to 17 October 2022, 11.666 bugs have been reported in ONAP Jira. These bugs have been assigned to their corresponding release based on the bug's affected release label in Jira.However, around 9.5% of these bugs could not be assigned as they do not have an information about the first affected release. Those bugs are assigned to the releases based on the issue creation date. The collected information for two example bugs are given in Table <ref>. 








As of 17 October 2022, ONOS project has 24 releases. An ONOS release from  t_s to t_r takes approximately 4 to 5 months. Similarly to ONAP, the bugs have been mapped to their corresponding release according to the collected bug information. In ONOS, 45% of all 2301 bugs (17 October 2022) do not have an affected release, and hence, those bugs are associated to the release based on their create dates. ONOS also has long time support (LTS) releases, whose bugs can be found for longer time compared to normal releases. It make the distribution process of the unlabeled bugs less convenient compared to ONAP. 

The number of bugs for the different releases of the ONAP and the ONOS projects are depicted in Figures <ref> and Figure <ref> respectively, where the blue bars correspond to the bugs with release information and the brown bars correspond to the associated bugs based on the bug creation date. It can be observed that for both projects, the number of bugs per release are higher for the first releases. However, there is a difference in the number and distribution of bugs without the release label. In ONAP, most of those bugs are associated to the first release, whereas for ONOS, those bugs are distributed along most of the releases. Also first ONAP releases contain the largest number of bugs without the affected release label but this trend is not observed in ONOS.









 §.§ Code Metrics


6 examples of the 43 code metrics are chosen as example. Those are the examples of development effort (commits), code metrics based on project size (kLoC), size of changes made in the code (new, removed kLoC), changes made in Java code (new kLoC Java) and addition of the complex functions (new complex Java functions). 

The number of bugs and 6 code metrics of the 6 ONAP releases are given in Table <ref>. Release 1 is chosen because it is the first release, release 3 is the most buggy release and 10 is the last and least buggy release. Other three release are intermediate points between those releases. Here it can be seen that the release 7 has the highest number of kLoC but it does not have the highest or lowest number of bugs. On the other hand, release 3 has the most number of commits and removed kLoC also the highest number of bugs. The release with the lowest number of bugs (release 10) is the one with the lowest number of commits. It can be said that there is a correlation between the code metrics and number of bugs. From this trend it can be said that if the number of commits and removed LoC is high, number of bugs in this release is expected to be high.




Same example code metrics for the 6 releases of the ONOS project are given in Table <ref>. Also first, last, most buggy (release 8) and least buggy (release 21) releases are chosen for ONOS. Other two releases are the releases between these releases. In ONOS also the release with the lowest number of bugs (release 2) has the lowest number of commits and removed kLoC. The release with highest number of bugs (release 8) has the highest number of bugs but it does not has the highest number of commits. This shows that code metrics can be used to predict the number of bugs in a release in ONOS project.



The kLoC value gives broad information about the size of the project and number of commits and new kLoC reflects the development effort for the related release. From Tables <ref> and <ref>, it can be said that the ONAP project is much bigger and more development efforts have been made for the given releases.  In connection with the development efforts the number of bugs of the ONAP releases are also higher than the ONOS releases.



To understand the connection between code metrics and bug history better, the change of bug history and commits that corresponds to the development efforts is shown in Figure <ref>. It can be observed, that ONAP is closer to a linear dependence than ONOS. 









 §.§ Bug History and Code Metrics Correlation

The correlation between the bug history and the code metrics have been performed for ONAP and ONOS as shown in Figures <ref> and  <ref> respectively. 

As observed in Figure <ref>, the expected number of bugs increases proportionally with number of commits. In order to evaluate the correlation between the number of bugs and each code metric, the Pearson correlation coefficient (PCC) is used.

Let us focus on the correlation of the code metrics with the bugs for ONAP, which are shown in the last row of the Figure <ref>. Six out of nine metrics shown "high" correlation, and in particular metrics like commits, sum of new, modified LoC and number of complex function have a correlation coefficient higher than 0.9. Those "high" correlation values motivate the use of code metrics to predict the number of bugs in a new ONAP release.



However, the correlation of ONOS is different as shown in Figure <ref>. In this case, the PCCs for code metrics and the number of bug are lower than the ONAP project. Only the metric commits has "high" correlation and three other metrics have "significant" correlation. Since the correlation values are lower than ONAP project, the prediction accuracy is expected to be also low.






 §.§ Regression Model



Linear regression model, whose formula is given in Formula <ref>, is used to predict the number of bugs. Fitting the regression model aims at finding the optimal β values that minimize the ϵ values.





The number of commits, number of new LoC, sum of new and modified Java LoC, number of new functions and number of complex Java functions values are used to feed the regression model as they offer higher correlation values and they give information about different aspects like development effort, complexity and Java based changes. The prediction values are evaluated with an error value which calculated with given formula:


    e = |E[#Bugs] - #Bugs|/#Bugs


where e represents the error value, E[#Bugs] is the predicted number of bugs and #Bugs is the real number of bugs.

Figure <ref> shows the distribution of the error value of the linear regression for different adjustments. In the first case referred as base linear regression model (BLR), all the previous releases are considered to feed the model (e.g., to predict the bugs of the fifth release, all releases up to release four are used to fit the linear regression).

Since all the chosen code metrics have positive correlation with bug history of ONAP, the second case referred as "linear regression with positive coefficients" (LR-PC) considered only positive slope β values. Also a linear regression model without intercept value (LR-woI) β_0 = 0 (third case) is trained because the changes made in the code is considered as the reason of the emerging bugs and without any change no bug is expected. The drop in the median value of error can be seen in the Figure <ref> 

In order to get better insight of the figure, Table <ref> summarizes the mean, median, maximum and minimum error values for each case. Both median and mean values are lower for LR-PC and LR-woI compared to BLR model. A "linear regression model without intercept and positive slope (LR-PC+woI) offers even lower error values. The median value drops from 0.499 to 0.12 which shows that the prediction can become closer to the real value with finding the correct structure of the linear regression model. These results show that the LR-PC+woI can be used to predict the number of bugs in the new release of the ONAP project.



Similarly to ONAP, all code metrics except for the number of complex Java functions, have been used to fit the regression model to ONOS releases. The reason to exclude the number of complex Java functions is due to its negative correlation with the number of bugs and using a positive slope is only meaningful if all the correlations are positive. Instead sum of new, modified and removed LoC of Java is used. 

The box plot of the error value for the predicted number of bugs with different compositions for ONOS is given in Figure <ref>. It can be observed that there are more outliers than in ONAP. The not labeled data and irregular release dates might cause the very high errors.

The mean and median values of the prediction error are given in Table <ref>. The lowest error value is reached with using positive slopes and no intercept for both projects. On the other hand using positive slopes alone or no intercept does not give lower error value. The higher values show that the lower correlation value effects the prediction accuracy negatively. 



 §.§ Cross-Project Prediction































Sometimes a project does not have enough previous releases to fit a regression model and achieve a realistic prediction of the number of bugs. As an example, the ONAP project started in November 2017, 3 years later than ONOS project. Can this model be used to predict the number of bugs in a new project with the help of older project? In order to answer this question, a regression model is fitted to all the ONAP and ONOS releases which were released before the code freeze date of the predicted release of the ONAP project. As example, the first 11 ONOS releases are used to predict the first ONAP release and the regression model is fitted to fitst ONAP release and first 13 releases ONOS releases that released before second release of ONAP to predict the second ONAP release. The results from the Table <ref> shows that the predicted values can give a rough idea about the number of bugs in the release. TheSince the linear regression model with positive slope and without intercept was better for in-project prediction the results of these model will be discussed. The error value is 0.081 for the first release, 0.481 for the second and 0.11 for the third release. The cross-project strategy can be used for the prediction of the number of bugs in the first releases of a new software.































 §.§ Prediction Accuracy with Different Number of Considered Releases


From release to release, the experience level of the contributors, the release procedures or priorities of the teams can change, which may impact the number of bugs. Over time, these aspects may affect the similarity between the releases and hence, the objective of this study is to evaluate how many of the previous releases should be considered to achieve the best estimation of the number of bugs of a new release. 

In order to perform this evaluation, the first task is to evaluate the correlation of the different metrics for the different number of releases. Based on this correlation, the change in the correlation value for different number of releases will be given.



As first task, we evaluate the PCC for a different number of releases. The median and mean PCC for bug history and two code metrics, which offer higher correlation value, commits and new and modified number of Java code is given are shown in Table <ref>. The number of previous releases is the number of releases used to get the correlation coefficient. Let us give as an example the case of considering 2 previous releases. In this case, for release x, the code metrics of releases x-1,x-2 are considered.  
The median and mean values are calculated from all the correlation coefficients values that are gathered using the same number of releases.

From this Table <ref>, it can be seen that the median values are high for lower number of releases, it is because those releases are coming from a shorter time period and similar to each other. On the other hand, the mean value is lower because a smaller shift in one release causes a lower correlation value. This is because of the low mean value using 2 previous releases might give lower prediction accuracy. After 3 previous releases, both mean correlation coefficients becomes higher than 0.7 which is the limit for high correlation. Although the mean values are comparably higher for more releases the median value does not follow the same trend. The median value for the best correlated metrics drops from 0.979 to 0.974. Thus using more than 3 and less than 7 previous ONAP releases seems reasonable from the correlation values. 



The error value for prediction of each ONAP releases with different number of previous releases used to fit the regression model is given in Figure <ref>. Linear regression model with positive slope values and without intercept is used for prediction. There are more results for latest releases because for those releases more  previous releases can be given to the model. After using more than 3 previous releases none of the prediction errors reach to 0.4, on the other hand after 7 previous releases all the error values are higher than 0.1. It can be said that neither using all the releases nor using only the last one or two releases gives the optimal prediction performance. 



The mean and median error values for regression model, which feed with different number of releases is given in Table <ref>. In order to make a fair comparison prediction error for the last 4 releases is also given because the error values is comparably higher than 4th, 5th and 6th releases as it can be seen at the Figure <ref>. The best value is 0.141 of the prediction with 9 previous releases but it only has one sample and the error for this sample is not lower than prediction with 4, 5 or 6 previous releases which can be seen in the last column of the Figure <ref>. After 9 previous releases the lowest number is reached with using 4 previous releases which followed by 5 previous releases. If we compare this results with the results in Subsection <ref>, we can see that the mean value of the 4 previous releases is lightly higher but the mean value is comparably lower. 
















































































































§ CONCLUSION



This paper proposes a framework to predict the number of bugs in a new software release. First each step of the proposed framework has been described. Then an exemplary implementation of this framework has been discussed. Two open-source projects ONOS and ONAP were analyzed with the presented implementation. The prediction has been improved with different methods and cross-project prediction is also tested to predict a new software with low number of old releases. 

The results shows that the number of bugs can be predicted at the end of the process of the software. The trustworthiness of this prediction depends on the project's properties. When ONOS and ONAP projects are compared, it can be seen that the correlation between code metrics and bug history of ONAP project is relatively higher, which leads to better prediction accuracy. One reason for that can be the number of bugs, which does not have any effected release information in issue tracker, is higher in ONOS. Other possible reasons are sliding release times of ONOS project and different type of releases like normal and long time support. The results also show that the bug history of a similar project can give a rough prediction on the number of releases of the first releases of the software. The optimal number of releases that should be used for a better prediction is also investigated. 

As future work this framework can be taken forward to predict the number of bugs in the su-bmodules of a software project. Also it can be tested with different machine learning models. The distribution of the unlabeled bugs can be improved with further investigation of commits that solves the bugs. Bug reports also includes information about the priority of the bugs, the framework can be tested with the bugs of different priority. A software reliability growth model can be feed with the predicted number of bugs and the expected reliability of the software on release date can be foreseen.



§ ACKNOWLEDGMENT


This work has received funding from the "DFG" under grant numbers MA 6529/4-1 and KE 1863/10-1.








1

visagan2019building
A. Visagan et al. Building a Data Mining Based Software Reliability Estimation Model,1em plus
 0.5em minus 0.4em1st Int. Conf. on Advances in Information Technology (ICAIT), 2019.

 
 falcao2020relating
F. Falca o et al., On relating technical, social factors, and the introduction of bugs,1em plus
 0.5em minus 0.4emIEEE 27th Int. Conf. on Software Analysis, Evolution and Reengineering (SANER), 2020.
 
 yang2017analyzing
 G.Yang et al., Analyzing emotion words to predict severity of software bugs: A case study of open source projects,1em plus
 0.5em minus 0.4emProc. of the Symposium on Applied Computing, 2017.

vizarreta2020dason
P. Vizarreta et al., DASON: Dependability assessment framework for imperfect distributed SDN implementations,1em plus
 0.5em minus 0.4emIEEE Trans. on Network and Service Management, 2020.

 mamun2017correlations
M. A. A. Mamun et al., Correlations of software code metrics: an empirical study,1em plus
 0.5em minus 0.4emProc. of the 27th Int. workshop on software measurement and 12th Int. Conf. on software process and product measurement, 2017.

rajendren2021predicting
S. Rajendren et al., Predicting number of bugs before launch: An investigation based on machine learning,1em plus
 0.5em minus 0.4emIEEE 45th Annual Computers, Software, and Applications Conf. (COMPSAC), 2021.

toth2016public
Z. Tóth et al., A public bug database of github projects and its application in bug prediction,1em plus
 0.5em minus 0.4emInt. Conf. on computational science and its applications, 2016.

 gupta2015developing
V. Gupta et al., Developing software bug prediction models using various software metrics as the bug indicators,1em plus
 0.5em minus 0.4emInternational Journal of Advanced Computer Science and Applications (IJACSA), vol. 6, 2015.

 osman2018impact
H. Osman et al., The impact of feature selection on predicting the number of bugs,1em plus
 0.5em minus 0.4emarXiv preprint arXiv:1807.04486, 2018.

 
 fenton2008effectiveness
N. Fenton et al., On the effectiveness of early life cycle defect prediction with Bayesian Nets,1em plus
 0.5em minus 0.4emEmpirical Software Engineering, 2008.

 reddivari2019software
S. Reddivari et al., Software quality prediction: an investigation based on machine learning,1em plus
 0.5em minus 0.4emIEEE 20th Int. Conf. on Information Reuse and Integration for Data Science (IRI), 2019.

 vizarreta2017empirical
P. Vizarreta et al., An empirical study of software reliability in SDN controllers,1em plus
 0.5em minus 0.4em13th Int. Conf. on Network and Service Management (CNSM), 2017.

 zimmermann2009cross
T. Zimmermann et al., Cross-project defect prediction: a large scale experiment on data vs. domain vs. process,1em plus
 0.5em minus 0.4emProc. of the 7th joint meeting of the European software engineering Conf. and the ACM SIGSOFT symposium on The foundations of software engineering, 2009.
 
nagappan2005use
N. Nagappan et al., Use of relative code churn measures to predict system defect density,1em plus
 0.5em minus 0.4emProc. of the 27th Int. Conf. on Software engineering, 2005.

 sonar
https://www.sonarqube.org

 
 pydriller
https://github.com/ishepard/pydriller

fahrmeir2013regression
L. Fahrmeir et al., Regression Models, Methods and Applications,1em plus
 0.5em minus 0.4emSpringer Berlin, Heidelberg, 2013.

 onap
https://www.onap.org/

 onos
https://opennetworking.org/onos/

cloc
https://github.com/AlDanial/cloc

 lizard
https://github.com/terryyin/lizard

  ebert2016cyclomatic
C. Ebert et al., Cyclomatic complexity,1em plus
 0.5em minus 0.4emIEEE software, vol. 33, no. 6, pp. 27–29, 2016.

eid2013linear
H. F. Eid et al., Linear correlation-based feature selection for network intrusion detection model,1em plus
 0.5em minus 0.4emInt. Conf. on Security of Information and Communication Networks, 2013.

 yu2017research
Z. Yu et al, Research on software project risk assessment model based on fuzzy theory and improved,1em plus
 0.5em minus 0.4emIEEE 2nd Advanced Information Technology, Electronic and Automation Control Conf. (IAEAC), 2017.

pandas
https://pandas.pydata.org/

scikit
https://scikit-learn.org/
 










































