# The potential of LLMs for coding with low-resource and domain-specific
programming languages ††thanks: I am grateful for the helpful comments of
Steffen Remus, Allin Cottrel, Riccardo (Jack) Luchhetti and Sven Schreiber,
and for the discussion of participants at the 8th gretl conference in Gdańsk
(June 2023). Any errors or omissions are the sole responsibility of the
authors.

Artur Tarassow
<EMAIL_ADDRESS>
(August 28, 2024)

###### Abstract

This paper presents a study on the feasibility of using large language models
(LLM) for coding with low-resource and domain-specific programming languages
that typically lack the amount of data required for effective LLM processing
techniques. This study focuses on the econometric scripting language named
hansl of the open-source software gretl and employs a proprietary LLM based on
GPT-3.5. Our findings suggest that LLMs can be a useful tool for writing,
understanding, improving, and documenting gretl code, which includes
generating descriptive docstrings for functions and providing precise
explanations for abstract and poorly documented econometric code. While the
LLM showcased promoting docstring-to-code translation capability, we also
identify some limitations, such as its inability to improve certain sections
of code and to write accurate unit tests. This study is a step towards
leveraging the power of LLMs to facilitate software development in low-
resource programming languages and ultimately to lower barriers to entry for
their adoption.

Key Words: large language models; coding; programming; low-resource language;
domain-specific; gretl; hansl.

##  1 Introduction

Large language models (LLMs) have emerged in computer science and are
revolutionizing the interplay of human (natural) language and computation. The
most well-known ones are GPT-3.5 (a base LLM) and
ChatGPT111https://chat.openai.com/ (an instruction-tuned LLM), both of which
have gained significant attention for their impressive language generation
capabilities.222You may think that ”revolutionizing” may be exaggerated.
However, researchers from Microsoft Research argue that the recent GPT-4 model
”[…] could reasonably be viewed as an early (yet still incomplete) version of
an artificial general intelligence (AGI) system” [3].

Automatic code generation is a long-standing challenge in computer science
[12]. Large corpora comprising code and the training of huge language models
have fuelled research in this direction. The potential benefits of using LLMs
for programming comprise assistance with code creation, documentation,
explanation, and description. For instance, LLMs may assist with naming
variables and functions, help with code refactoring.

Previous studies have investigated the effectiveness of LLMs as a programming
assistant for various tasks typically faced by programmers. However, most of
the research has focused on high-resource general-purpose programming
languages, such as Python, C++, or JavaScript [5, 21, 19]. There has been only
limited research on low-resource programming languages, which often lack large
amounts of data, such as source code or references, making them challenging
for LLMs to process compared to high-resource languages. Recent exceptions are
[25, 2, 4] who studied the performance of LLMs for coding with low-resource
but general-purpose programming languages.

Domain-specific programming languages also exist and offer high levels of
abstraction. Due this they can be more efficient than general-purpose
languages for specific domain-related tasks, such as financial modelling,
econometrics, or scientific computing. However, current research on LLMs as
programming assistants has focused only on a limited subset of programming
languages, leaving the vast majority understudied. The aim of this paper is to
investigate how LLMs may assist with the use of a low-resource and domain-
specific programming language. By exploring the potential of LLMs in this
context, this paper aims to contribute to filling the gap in research on such
languages.

For this study, we choose scripting language named hansl. Hansl (a recursive
acronym: “hansl’s a neat scripting language”) is the scripting language of
gretl333gretl.sourceforge.net, an open source econometrics and statistics
package written in C and licensed under the GNU GPL [6].444For a practical
introduction to gretl, see e.g. [18]. It can compete with the leading paid
econometrics packages of Stata and Eviews, as well as the top open-source
statistical software project, R.555https://www.r-project.org/ Gretl is mainly
used by economists and econometricians when doing applied work as well as
simulations.666Note, in order to avoid confusion, we use gretl as a synonym
for hansl throughout the text.

While Python and R code are highly prevalent, the amount of gretl code
available is relatively limited.777We did a search on github.com to identify
files relevant to gretl and hansl. The search query used was as follows:
("gretl" OR "hansl") AND (path:*.inp OR path:*.gfn). This query searched for
files that contained either the term ”hansl” or ”gretl” within files with
either the extension inp (gretl script file) or gfn (gretl package file). The
search results showed that there were 1500 such files available on GitHub. It
is notable that the number of files available for R and Python are multiple
factors higher in comparison: Searching for language:R revealed a total of 5.1
million files while searching for Python on GitHub revealed ten times as many
files. The search was conducted on the 2023-07-18. This means that the data
available for training LLMs to perform specific tasks using gretl code is far
less comprehensive than that available for Python or R code. As a result,
existing LLMs may not be as effective when applied to gretl code tasks used in
applied research. Despite these limitations, we demonstrate that LLMs can
still be applied successfully to certain coding tasks using gretl.
Specifically, in the field of (applied) econometrics, LLMs can help users
understand user-written functions involving linear algebra and statistics.
They can also provide suggestions for coding solutions based on natural
language prompts. Overall, LLMs have the potential to save time, minimize or
avoid coding errors, and increase productivity for both programmers,
econometricians, and other knowledge workers. This indicates that publicly
existing models are able to generalize well to low-resource programming
languages [25].

We conduct a series of tasks applied coding economists or econometricians may
face such as code documentation, code explanation and description (with
special focus on the econometrics involved), variable and function naming,
code creation as well as refactoring of code. We also present examples on how
gretl code can be written by the LLM based on a docstring only. We show that
the LLM also writes gretl code for computing the Fibonacci sequence or the
root mean squared error including only minor syntactical errors. In a last
experiment, we simulate a simple coursework exercise for an introductory
econometric class and show how the LLM writes the gretl script according to
the class instructions to some extent. For this, we use a proprietary
instruction-based LLM provided for free (after registration) by www.you.com.
The model is a generative pre-trained transformer using the ChatGPT framework
which has around 175 billion parameters and is based on GPT-3.5.

It should be said that we were unable to use a "vanilla" pre-trained model.
The publicly usable model relies on human-in-the-loop (HITL) feedback to
improve its performance.888HITL is a machine learning method that involves
incorporating human feedback at various stages of training in order to enhance
the accuracy and overall model performance. This is particularly important in
the context of natural language processing, where the intricacies of language
can be difficult for models to interpret without additional guidance. Since
the beginning of June 2023, the author has actively used the model and
provided feedback on gretl code-related prompts, allowing the model’s
performance to continually improve over time. However, this introduces a
potential challenge in assessing the model’s true capabilities, as it becomes
difficult to determine whether the model can generate working gretl code on
its own, or if it relies solely on the HITL feedback to do so. While this
issue is not the focus of our paper, we recognize the potential impact it may
have on the results.

In the next section, we provide a brief background on LLMs and summarize the
recent literature on both the use of LLMs for programming generally as well as
on low-resource programming languages specifically. Afterwards, we present the
experiments of applying the LLM to gretl code and discuss the results. The
paper ends with a conclusion.

##  2 Background

###  2.1 Large Language Models

Modern LLMs rely on deep learning algorithms and large datasets to process and
understand human language and generate text. Nowadays, they use the
transformer architecture, which were introduced by Google in 2017, as the
backbone of their models [20]. The transformer architecture is a neural
network that is capable of processing long sequences of input data in
parallel, resulting in more efficient and effective training.

The key innovation of the transformer is the self-attention mechanism, which
allows the model to identify which parts of the input are relevant for each
output. This enables the model to identify and learn more complex
relationships between the input and output, resulting in better performance on
a wide range of natural language processing tasks. For details see also [15,
22]. Current architectures can be autoregressive models that have a memory of
the last $N$ word-pieces (potentially solely parts of a word). Their pre-
training objective is to predict the next token and the next sentence.

LLMs are used for many tasks such as machine translation, sentiment analysis,
text completion, summarization and extraction etc.

###  2.2 Large language models used for programming

Automatic code generation is a longstanding challenge [12]. Large corpus
comprising code [8] and the training of huge language models such as GPT-J
[21] have fuelled research in this direction.999GPT-J is a large-scale
language model that has been trained on a massive amount of text data and is
general in nature, meaning it can perform a wide range of tasks such as text
generation, translation and summarization. While GPT-J is powerful enough to
generate code, it is not specifically tailored for this task like the well-
known ”code-davinci” models are. In the following, we provide a brief overview
of recent research on the use and effectiveness of LLMs for programming.

[5] introduce Codex, a GPT language model fine-tuned on publicly available
code from GitHub, and evaluate its Python code-writing capabilities. The study
reveals that Codex significantly outperforms GPT-3 and GPT-J on a new
evaluation set designed to measure functional correctness for synthesizing
programs from docstrings.101010A docstring is a string literal specified in
the source code that is used to document a specific segment of code such as a
function. It is used to provide information on what a particular piece of code
does, its input parameters, expected return values, and any relevant usage
example

The company GitHub published its product Copilot which relies on OpenAI’s
Codex model.111111https://github.com/features/copilot/ Copilot is a
programming assistant for various general-purpose programming languages widely
used in industry. The model is trained on billions lines of code to turn
natural language prompts into coding suggestions.

[23] find that existing open-source models perform similar to closed-source
models such as Codex in some programming languages, although targeted mainly
for natural language modelling. The authors publish a LLM named PolyCoder
which is the first public model trained exclusively on code from multiple
programming languages. Also, the authors claim that training on natural
language text and code jointly can benefit code modelling.

Recently, [19] studied the potential of ChatGPT as a programming assistant on
code generation, detecting and fixing bugs as well as code summarization. They
compare the model against two benchmark approaches using Python as the
programming language being evaluated. One benchmark is using LeetCode which is
an online platform offering a diverse range of programming problems for
software engineering interviews, where new problems are constantly posted and
updated. The second benchmark is named Refactory which is a semantic-based
assignments repair tool written in Python and released in 2019. ChatGPT
dominates the benchmarks on code writing and performs equally well to
Refactory on program repair. On code summarization their results show that
ChatGPT may have problems in explaining the intention of code though.

[24] evaluate the performance of ChatGPT for writing automatic unit tests.
ChatGPT’s generated tests were analyzed to evaluate its correctness,
sufficiency, readability, and usability. The results showed that while the
tests still suffer from correctness issues, passing tests generated by ChatGPT
resemble manually-written tests in terms of coverage, readability, and
developer preference, suggesting that generating unit tests with ChatGPT could
be promising.

[3] evaluate GPT-4 for coding challenges. They find that GPT-4 does much
better on the HumanEval, a docstring-to-code dataset consisting coding
problems, compared to state-of-the-art LLMs trained specifically on code. The
authors also compare leading LLMs on the LeetCode dataset and state that
"GPT-4 significantly outperforms the other models, and is comparable to human
performance…" [3, ch. 3.1.1]. They even go further and find that GPT-4 can be
used to create complex data visualization, write a 3D game in HTML with
JavaScript, using a very high-level specification, customize an deep-learning
optimizer module involving operations such as applying SVD and linear algebra
just by giving human language description. Furthermore, GPT-4 reverse-
engineers assembly code, reasons about code execution of C code and runs a
complex Python method by simulating the steps instead of running the program
actually.

[9] is a rare exception studying the value-added of LLMs for economic research
including coding. Apart from writing Python code for computing and plotting
the Fibonacci sequence, he shares his experience that Open AI’s model name
text-davinci-003 is not yet able to write code for simulating basic economic
problems like optimal consumption smoothing. According to Korinek, current
LLMs can also be used to translate short pieces of Python into Matlab code or
debug Python functions. For more complex code human assistance is still
required, though.

###  2.3 Low-resource programming language and coding

Low-resource programming languages (LRPLs) are characterized by a low number
of records in their datasets, which means that there is little information, or
even none, comprising source code or references. This lack of information
makes it challenging for language models to process LRPL data. Although models
can be trained to achieve high accuracy for tasks such as intent
classification and sequence labelling, these tasks require large amounts of
labelled data in the target language.

However, due to their scarcity of resources, LRPLs are less studied. This
limitation is similar to that experienced in natural language processing of
low-resource languages, as elucidated by [13]. The authors study the limited
availability of natural language understanding resources for African
languages, most of which are considered low resource languages.

Despite the importance of LRPLs in software development, little research
exists on how pre-trained LLMs perform on them. Exceptions are [25] who have
demonstrated that training a model with data from multiple programming
languages enhances its performance on individual languages. This finding is
especially true for LRPLs and suggests that multilingual LLMs may be able to
generalize well across programming languages. Recent research by [2] also
supports this view, particularly in the areas of code summarization, code
retrieval, and function naming.

On the other hand, [4] focus on the low-resource (general-purpose) programming
language Ruby. The authors fine-tune both monolingual and multilingual pre-
trained language models for this language, respectively, and evaluate their
performance in code search and code summarization. Their results indicate that
multilingual models perform less well than monolingual ones.

In their recent work, [7] have introduced the MultiCoder LLM, which aims to
improve code completion performance on LRPLs. The authors achieve this by
adding new layers to a pre-existing LLM in order to enhance its performance,
demonstrating that MultiCoder outperforms their monolingual baselines.

For our study, it is relevant to examine the generalization ability of popular
LLMs such as GPT-3.5 to the programming language used in the open-source
software gretl. While existing research focuses on low-resource but general
programming languages, we put our focus on a domain-specific LRPL.

##  3 Tasks

This section explores several programming tasks that are relevant to applied
economists and econometricians. Firstly, we discuss how to improve the
documentation of gretl code by writing docstrings based on the code itself. In
the following section, we demonstrate how the LLM can be used to translate
docstrings into executable code. Moving on, we show how the LLM can help
summarize or describe code, and suggest self-explanatory variable and function
names for better readability and maintainability of the code. We then further
explore how the LLM can aid in refactoring gretl code and writing unit tests.
Finally, we provide a brief example of how the LLM can be utilized in
translating real-world student exercises from human language into gretl
code.121212All exercises were conducted in June 2023. Since mid of July 2023,
www.you.com has switched from GPT-3.5 to GPT-4.0.

###  3.1 Create docstrings from code bodies

In the software industry, the lack of high-quality code documentation is a
common challenge. Docstrings are commonly used in code to address this issue.
Docstrings provide clear and concise explanations of the code’s functionality,
simplifying collaboration among developers. Docstrings also help ensure
consistency of documentation across different code bases and modules.
Additionally, API documentation can be generated automatically from
docstrings. Moreover, docstrings can be used to create unit tests, which can
improve code quality over time. However, creating and updating docstrings is
labor-intensive, which has prompted research into automatic code summarization
[17]. This paper investigates whether it is feasible to generate docstrings
from gretl code bodies following an exercise by [5].

We examine the creation of a docstring for the gretl function addprefixtoany
in Listing 1. The function accepts an array of strings as input and adds a
prefix followed by each array item, returning the updated array. In case the
input array targets is empty, the function throws an error. To avoid
confusion, we have enclosed the relevant code in a block of triple backticks.

Listing 1: Prompt to write a docstring for addprefixtoany()

⬇

Write a docstring for a gretl function. The gretl function is provided in
triple backtics:

‘‘‘

function strings add_prefix_to_any (const strings targets, const string
prefix)

strings ret = array(nelem(targets))

errorif(nelem(ret) == 0, "Array is empty.")

if nelem(prefix) == 0

return targets

endif

loop foreach i targets

string ret[i] = sprintf("%s%s", prefix, "$i")

endloop

return ret

end function

‘‘‘

The resulting docstring for the addprefixtoany function is provided in Listing
2. The description accurately conveys the purpose of the function, while the
input parameter descriptions and types are precise. Moreover, the docstring
thoroughly explains the function’s output. Remarkably, the LLM recognizes the
error case, even though gretl does not support exceptions (specifically, of
type "ValueError").

Listing 2: Answer on writing a docstring for addprefixtoany()

⬇

Add a prefix to each element in an array of strings.

:param targets: The array of strings to modify.

:type targets: array of string

:param prefix: The prefix to add to each element in the array.

:type prefix: string

:return: A new array of strings with the prefix added to each element.

:rtype: array of string

:raises: ValueError if the input array is empty.

The first example may appear simple, and it is not surprising that the LLM can
accurately generate a docstring. However, it is important to note that (1) the
LLM is most likely not explicitly trained on gretl’s scripting language, and
(2) the datatype strings is not present in other popular programming
languages.

The next example involves a much more complex function called kernelloop,
taken from the "ketvals" package (v0.91) by [11]. The function name and the
parameter names passed are not very meaningful, as they consist mostly of
single letters (e.g. y or Z), and there are no comments. Additionally, the
function calls another private function called kernelmat, which the LLM has
not seen before. In order to write an accurate and precise docstring, deep
domain knowledge is required, as there are mathematical operations (involving
linear algebra) and a loop block. The prompt for writing a docstring can be
found in Listing 3.

Listing 3: Prompt to write a docstring for kernelloop()

⬇

Write a docstring for a gretl function. The gretl function is provided in
triple backtics:

‘‘‘

function matrix kernel_loop (const matrix y,

const matrix X,

const matrix Z,

scalar h,

scalar type[0:4:1],

matrix optparm[null])

scalar q = cols(y)

scalar k = cols(X)

scalar T = rows(y)

matrix ret = mshape(NA,T,q*k)

W = kernelmat(T, type, h, optparm)

ini = T + 1

fin = 2*T

loop t = 1 .. T

matrix PX = W[ini:fin] .* Z

matrix num = PX’y

matrix den = PX’X

ret[t,] = vec(den\num)’

ini\--

fin\--

endloop

return ret

end function

‘‘‘

The output generated by the LLM for the kernelloop function is displayed in
Listing 4. The docstring correctly identifies the function as executing a
kernel regression. Surprisingly, despite the non-meaningful and single-letter
naming of the parameters, the LLM still manages to accurately describe them.
For example, the LLM refers to the h parameter as the kernel bandwidth
parameter, which is correct.

There is only one error (coloured in red) in the LLM’s output: for the type
parameter, the LLM incorrectly states that it is an array of integers, when in
fact it is a scalar value. Although an array of integers is not supported by
gretl, the syntax scalar type[0:4:1] used in the function signature indicates
the minimum, maximum, and default values, rather than an array of integers.

Interestingly, the LLM identifies an error in the function even though no
explicit error is raised in the code body. This is because the LLM recognizes
that a linear algebra operation such as matrix multiplication can throw an
error if the dimensions of the matrices do not match.

Listing 4: Answer on writing a docstring for kernelloop(). Syntactical error
coloured in red.

⬇

Perform a kernel regression using a loop and return the parameter estimates.

:param y: The dependent variable matrix.

:type y: matrix

:param X: The independent variable matrix.

:type X: matrix

:param Z: The kernel weight matrix.

:type Z: matrix

:param h: The kernel bandwidth parameter.

:type h: scalar

:param type: An array of integer values representing the kernel type.

:type type: array of int

:param optparm: Optional parameters used to calculate the kernel.

:type optparm: matrix or None

:return: A matrix containing the parameter estimates for kernel regression.

:rtype: matrix

:raises ValueError: If the input arrays are not correctly formatted.

###  3.2 Create executable code from docstrings

Docstrings provide a clear and concise description of the code’s functionality
and purpose. By translating these descriptions into actual code, programmers
can more easily understand the intended behavior of the program and ensure
that they are implementing the desired functionality. Additionally, this
process can help improve the accessibility of code to non-technical
stakeholders or users who may not have a programming background but want to
understand the software’s functionality. Thus, translating docstrings into
programming code is relevant, necessary, and contributes significantly to the
quality and usability of software applications.

[5] have recently found evidence that current-state LLMs are able to write
executable Python code based on information contained in docstrings only. [3]
evaluate the ability of the GPT-4 model to translate docstrings to code and
show that this LLM is comparable to human levels.

In this exercise, we want the LLM to write executable gretl code based on a
docstring. As an initial example, we make use of the docstring the LLM
provided in Subsection 3.1 for function addprefixtoany(). We start by opening
a new session to clear the LLM’s memory. Listing 5 presents the prompt for
creating a gretl function.

Listing 5: Prompt to write a gretl function based on a docstring

⬇

Write a gretl function based on the following docstring. The docstring is in
triple backtics.

‘‘‘

Add a prefix to each element in an array of strings.

:param targets: The array of strings to modify.

:type targets: array of string

:param prefix: The prefix to add to each element in the array.

:type prefix: string

:return: A new array of strings with the prefix added to each element.

:rtype: array of string

:raises: ValueError if the input array is empty.

‘‘‘

Listing 6 presents the proposed function and a detailed description. While the
LLM provides a code skeleton that serves as a good starting point, the code
itself contains several syntax errors. For instance, the function signature
lacks a return type, and the datatype arraystring is not recognized by gretl -
strings should be used instead. Furthermore, arraylength() is not a built-in
function in gretl, as the LLM notes. Additionally, the initialization of the
string array newarray is incorrect since the array’s dimension is not set. As
a result, accessing a specific entry in the object (such as newarray[i] within
the loop-block) is not possible. Lastly, the string concatenation will not
work.

Despite these issues, the proposed function serves as a useful skeleton for
developers who want to structure a function that accomplishes the task, given
the docstring written in plain language. The LLM provides a highly detailed
description of the individual steps, which can be particularly beneficial for
fixing the bugs.

Listing 6: Code created for prompt in Listing 5. Syntactical error coloured in
red.

⬇

function <RETURN TYPE IS MISSING> array_add_prefix(array_string targets,
string prefix)

if array_length(targets) == 0

error ValueError: The input array cannot be empty."

endif

array_string new_array

loop i=1..array_length(targets)

new_array[i] = prefix $ targets[i]

endloop

return new_array

end function

This function takes two arguments:

targets: an array of strings to modify

prefix: the prefix to add to each element in the array

It first checks if the input array is empty, and if so, throws a ValueError.

Then, it initializes a new array new_array to store the modified strings, and
loops through each element in targets. For each element, it concatenates the
prefix with the element using the $ operator, and adds the resulting string to
new_array.

Finally, it returns the modified array new_array.

Note that I assumed array_length is a built-in function in gretl to get the
length of an array. If this is not the case, you may need to define it
yourself or use a different method to check for an empty array.

Listing 7 shows how we asked the LLM to generate a function that calculates
the root mean squared forecast error, based on a simple docstring. The
resulting function, as shown in Listing 8, has the correct signature and is
well-structured with only one syntactical error. Specifically, the gretl
function sumsq() is not available, and we need to replace it with
$e.\hat{}\;2$ to correctly calculate the sum of squared errors. Although the
LLM output requires some minor modifications, it provides almost working and
correct code. Additionally, the LLM provides a detailed description of each
line of code which is makes it easy to understand.

Listing 7: Prompt to write a gretl function for computing the root mean
squared forecast error.

⬇

Write a gretl function using the following docstring. The docstring follows in
triple backtics.

‘‘‘

Compute the root mean squared forecast error.

:param y: Series of realizations

:type y: series

:param f: Series of forecasts

:type f: series

:rtype: scalar

‘‘‘

Listing 8: Code written for computing the root mean squared forecast error.
Syntactical error coloured in red.

⬇

‘‘‘

Here’s a gretl function that computes the root mean squared forecast error:

‘‘‘

function scalar rmse_forecast_error(series y, series f)

matrix e = y - f

scalar n = nelem(e)

scalar mse = 1/n * sumsq(e)

return sqrt(mse)

end function

‘‘‘

The function takes in two series: ‘y‘ and ‘f‘, which represent the
realizations and forecasts, respectively. It then computes the error for each
forecast by subtracting the forecast from the corresponding realization, and
stores these errors in a matrix ‘e‘.

Next, it computes the mean squared error (MSE) by dividing the sum of squared
errors (‘sumsq(e)‘) by the sample size (‘n‘). Finally, it returns the square
root of the MSE, which is the root mean squared forecast error.

Note that we used the ‘nelem()‘ function to calculate the sample size. This
function returns the number of elements in a matrix or a series. We also used
the ‘sumsq()‘ function to calculate the sum of squared errors. This function
takes a matrix or a series as an argument and returns the sum of squared
values.

In another example, we requested the LLM to write a function that calculates
the Fibonacci sequence, which is presented in Listing 9.

Listing 9: Prompt to write a gretl function for computing the Fibonacci
sequence.

⬇

Write gretl code given the following docstring in triple backtics.

‘‘‘

Fibonacci sequence is a sequence in which each number is the sum of the two
preceding ones. The sequence starts from 0 and 1.

:param steps: Length of the sequence

:type steps: integer of the length of the sequence

:return: Fibonacci number

:rtype: integer

:raises: Error if the parameter steps is not positive.

‘‘‘

Listing 10: Code written for computing the Fibonacci sequence. Syntactical
error coloured in red.

⬇

‘‘‘

function integer fibonacci(integer steps)

if (steps <= 0)

errprintf("Error: steps parameter must be a positive integer.\n")

return

endif

if (steps == 1)

return 0

endif

if (steps == 2)

return 1

endif

series fib = zeros(steps)

fib[1] = 0

fib[2] = 1

loop i = 3..steps

fib[i] = fib[i-1] + fib[i-2]

endloop

return fib[steps]

end function

‘‘‘

This function takes an integer parameter ‘steps‘ and returns the ‘steps‘th
number in the Fibonacci sequence. If the ‘steps‘ parameter is not positive,
the function raises an error and returns ‘null‘. The function first checks if
‘steps‘ is less than or equal to zero. If so, the function raises an error and
returns ‘null‘. Next, the function checks if ‘steps‘ equals 1 or 2 and returns
the first or second number of the Fibonacci sequence respectively. For ‘steps
> 2‘, the function calculates each of the Fibonacci numbers up to ‘steps‘
using a loop that starts at 3 and continues until the ‘steps‘th number is
reached. It then returns the ‘steps‘th number of the sequence.

The function name and variable names used in the code are clear and
descriptive, and the structure of the code looks satisfactory at first glance.
However, there are some minor errors in the code that need to be addressed.
First, the type of value returned should be scalar instead of integer when
using gretl. Additionally, the command errprintf() does not exist and should
be replaced with the funcerr command. Moreover, the object holding the
Fibonacci series should be initialized as a vector using the command matrix
fib = zeros(steps), instead of being initialized as a type series. Once these
minor corrections have been made, the proposed code can be executed in gretl
to return accurate results.

Our docstring-to-code exercises demonstrate that LLMs can provide significant
support for generating gretl code. Specifically, they can serve at the very
least as a starting point by providing the general structure of a program.
However, as expected, LLMs may not be fully aware of gretl functions,
datatypes, and syntax, so their generated code may not be fully functional or
may require further editing. Nonetheless, the ability of LLMs to generate
accurate and helpful code to some extent highlights their potential as a
valuable tool for developers and researchers in the field of programming.
Also, it substantially lowers the entry barrier as the level of basic
programming skills required decrease (low-level programming).

###  3.3 Code explanation and summarization

Research into using LLMs for summarizing code is currently active [1].
Programmers may encounter functions that are difficult to understand due to
poor documentation, convoluted programming, or specialized mathematical
concepts that require domain knowledge. Recent studies by [19] have evaluated
ChatGPT and found that it can produce useful explanations of code to a high
degree. The authors note that even for incorrect code, ChatGPT can reason the
intended purpose of the code, although its explanation may not accurately
describe the current code.

To study the complexity of the kernelloop() function [11], we chose to use it
as an example. We ensured that our prompt-session had no previous knowledge of
this function by operating within a new session. Our goal was to evaluate
whether the LLM could accurately explain the intention of the code. Listing 11
shows the prompt requesting an explanation of the function.

Listing 11: Prompt to explain the function kernelloop()

⬇

What does the following gretl function do? Please explain. The function is
provided in triple backtics:

‘‘‘

function matrix kernel_loop (const matrix y,

const matrix X,

const matrix Z,

scalar h,

scalar type[0:4:1],

matrix optparm[null])

scalar q = cols(y)

scalar k = cols(X)

scalar T = rows(y)

matrix ret = mshape(NA,T,q*k)

W = kernelmat(T, type, h, optparm)

ini = T + 1

fin = 2*T

loop t = 1 .. T

matrix PX = W[ini:fin] .* Z

matrix num = PX’y

matrix den = PX’X

ret[t,] = vec(den\num)’

ini\--

fin\--

endloop

return ret

end function

‘‘‘

The detailed answer provided by the LLM for the kernelloop() function is
presented in Listing 12. The LLM correctly identifies the function signature
and provides a detailed description of the initialized variables. The most
noteworthy aspect of the LLM’s output is its correct identification of the
matrix W as a weight matrix, indicating that the kernelmat() function returns
a vector of weights. The LLM also correctly identifies the element-wise
multiplication of "the subset of the W matrix", demonstrating its ability to
recognize the gretl syntax .* for element-wise multiplication. Moreover, the
LLM’s overall statement that "this function computes some sort of weighted
regression using the ’y’, ’X’, and ’Z’ matrices, where the weights are
determined by the ’W’ matrix calculated using the ’kernelmat’ function" is
accurate and complete.

In another prompt session (not shown here), the LLM provided even more precise
information, stating that "the function appears to implement a kernel-based
estimation procedure. Specifically, it computes a regression of ’y’ on ’X’
using instrumental variables generated from ’Z’ based on a kernel weighting
scheme. The resulting estimates are then used to compute regression
coefficients and store them in the ’ret’ matrix." This ability of the LLM to
recognize the function’s use of instrumental variables is particularly
significant from an econometric standpoint.

Listing 12: Answer explaining the function kernelloop()

⬇

The function is called ‘kernel_loop‘ and appears to be a function defined in
the gretl programming language. Here is a brief explanation of what the
function does:

The ‘kernel_loop‘ function takes several arguments, including ‘y‘, ‘X‘, ‘Z‘,
‘h‘, ‘type‘, and ‘optparm‘. These arguments appear to be matrices and scalars
that will be used in the computation.

The function first creates a new matrix ‘ret‘ using the ‘mshape‘ function with
parameters ‘NA‘, ‘T‘, and ‘q*k‘. This will create a matrix with ‘T‘ rows and
‘q*k‘ columns, where each element is initialized to ‘NA‘.

The function then creates a new matrix ‘W‘ using the ‘kernelmat‘ function with
parameters ‘T‘, ‘type‘, ‘h‘, and ‘optparm‘. This will create a matrix that
will be used as a weight matrix.

Next, the function initializes the variables ‘ini‘ and ‘fin‘ to values ‘T+1‘
and ‘2*T‘, respectively. This is used to keep track of the elements in the ‘W‘
matrix that will be used in the computation.

The function then enters a loop that iterates through each row of the ‘y‘
matrix. For each row, the function first constructs a new matrix ‘PX‘ by
element-wise multiplying the subset of the ‘W‘ matrix specified by the ‘ini‘
and ‘fin‘ variables with the ‘Z‘ matrix. The function then performs some
matrix operations involving ‘PX‘, ‘y‘, and ‘X‘, and stores the result in the
corresponding row of the ‘ret‘ matrix.

After each iteration, the ‘ini‘ and ‘fin‘ variables are decremented, to select
a different subset of the ‘W‘ matrix for the next iteration.

Finally, the function returns the ‘ret‘ matrix.

Overall, it appears that this function computes some sort of weighted
regression using the ‘y‘, ‘X‘, and ‘Z‘ matrices, where the weights are
determined by the ‘W‘ matrix calculated using the ‘kernelmat‘ function. The
exact nature of the computation will depend on the values of the input
variables and the implementation details of the various matrix operations used
in the function.

Next, we provide a second example that uses a function from the SVAR package
(v1.97) developed by [10]. In Listing 13, we present a prompt where we ask the
LLM to describe the function maxeval(). Unfortunately, the function’s name is
ambiguous and lacks specificity, making it difficult to infer its purpose. The
use of the term "eval" could signify different things, including "evaluation,"
"eigenvalue," or other possibilities. Moreover, the function lacks a docstring
and any explanatory comments, further compounding the ambiguity. However, we
note that the function utilizes several built-in functions from gretl, such as
eigen(), to compute the eigenvalues of a square matrix.

Listing 13: Prompt to explain the function maxeval()

⬇

What does the following gretl function do? Explain. The function is in triple
backticks.

‘‘‘

function scalar max_eval (const matrix A)

n = rows(A)

p = cols(A) / n

matrix compan = p==1 ? A : A | (I(n*(p-1)) ~ zeros(n*(p-1), n))

matrix lambda = eigen(compan)

# (lambda will be complex now)

scalar maxmod = maxc(abs(lambda))

return maxmod

end function

‘‘‘

To clarify the function’s purpose, we turn to the LLM, which provides
additional information about the function in Listing 14.

Listing 14: Answer explaining the function maxeval()

⬇

The provided gretl function ‘max_eval‘ computes the maximum modulus of the
eigenvalues of a square matrix ‘A‘. Here are the steps that the function
follows:

The function takes a matrix ‘A‘ as an input and extracts the number of rows
‘n‘ and number of blocks ‘p‘ from ‘A‘. It then constructs a companion matrix
‘compan‘ based on ‘A‘. If ‘p==1‘, ‘compan‘ is just equal to ‘A‘, otherwise,
‘compan‘ concatenates ‘A‘ with an identity matrix of size ‘n(p-1)‘ on the
lower block-diagonal.

The function then computes the eigenvalues of ‘compan‘ matrix using the
‘eigen‘ function available in gretl.

Finally, it computes the maximum modulus (absolute value) of the complex
eigenvalues obtained from the ‘eigen‘ function and returns it as the output.

In summary, the function can be used to determine the maximum absolute value
of the eigenvalues of a square matrix in gretl. This information can be useful
in a variety of applications, such as system simulation or control theory.

According to the LLM’s description, the function maxeval() computes the
maximum modulus of a square matrix’s eigenvalues. However, we note that the
function can also process rectangular matrices, despite the LLM’s initial
claim. Specifically, maxeval() constructs a companion matrix, named compan,
that embeds the input matrix A in a larger square matrix. The LLM’s
explanation of the function is informative and emphasizes its potential
usefulness in system simulation.

Overall, these advanced examples clearly demonstrate the effectiveness of
using an LLM to explain econometric functions.

###  3.4 Suggestion of variable and function names

Proper naming of functions and variables is critical for code readability and
maintainability. Descriptive and self-explanatory names help to reduce the
cognitive load of understanding code and eliminate the need for extensive
comments. However, finding good names can be challenging for programmers who
would rather focus on code implementation than name selection. In this
exercise, we explore whether an LLM can help generate suitable names for
functions and variables based on the code body.

To illustrate the process, we use the kernelloop() function as an example
[11]. We start by opening a new session to clear the LLM’s memory and rename
the function to foo() to avoid giving too much away about its purpose. The
prompt is presented in Listing 15, and the output is displayed in Listing 16.
The LLM provides suggestions for function names along with a brief explanation
of their meanings.

Listing 15: Prompt to suggest function names for function foo()

⬇

Suggest three descriptive function names for the following gretl function. The
gretl function is provided in triple backtics:

‘‘‘

function matrix foo (const matrix y,

const matrix X,

const matrix Z,

scalar h,

scalar type[0:4:1],

matrix optparm[null])

scalar q = cols(y)

scalar k = cols(X)

scalar T = rows(y)

matrix ret = mshape(NA,T,q*k)

W = kernelmat(T, type, h, optparm)

ini = T + 1

fin = 2*T

loop t = 1 .. T

matrix PX = W[ini:fin] .* Z

matrix num = PX’y

matrix den = PX’X

ret[t,] = vec(den\num)’

ini\--

fin\--

endloop

return ret

end function

‘‘‘

The first suggestion, kernelregression, is descriptive, accurate, and
reasonably concise. However, the second suggestion, locallinearregression, may
be ambiguous as to what "local" entails. Regarding the third suggestion, the
naming of "h" is not precise enough to be informative. Additionally, the
suggested name "estimationregression" is too general and fails to specify the
regression type.

Overall, this exercise shows that an LLM could provide valuable insights into
suitable function and variable names, thus enhancing code readability and
maintainability for programmers.

Listing 16: List of suggested function names for function foo()

⬇

Here are three possible descriptive function names for the provided gretl
function:

1. "kernel_regression": This name reflects the fact that the function is performing a kernel regression, which is a type of non-parametric regression that estimates the relationship between variables using a weighted average of neighboring data points.

2. "local_linear_regression": This name also captures the essence of the function, which is to estimate a regression relationship using local (as opposed to global) linear models.

3. "h_estimation_regression": This name emphasizes the fact that the function is using a bandwith parameter h to determine the range of nearby data points that are used in the regression estimation.

Another example highlights how an LLM can assist in improving variable names.
We again employ a function from the "SVAR" gretl package [10] called
CheckNormalizeRd(). We keep the original comment block at the top of the code
and instruct the LLM to limit variable names to a maximum of 10 characters
(see Listing 17).

Listing 17: Prompt to suggest variable names for function CheckNormalizeRd()

⬇

Please suggest improved variable names for the following gretl function in
triple backticks. Restrict the length of variable names to 10 characters.

‘‘‘

function scalar CheckNormalizeRd (matrix *R,

matrix *d)

/*

Checks that

(1) the constraints are consistent

(2) the constraints are non-contradictory (??? non-redundant ? - Sven)

if (1) fails, an error message is printed and R and d are replaced by

empty matrices; if (2) fails, redundant rows in R and d are dropped.

*/

p = rows(R)

r = rank(R)

ret = 0

if r < p

matrix Rd = R ~ d

if r < rank(Rd)

R = {}

d = {}

ret = 1

else

matrix RR

matrix QQ = qrdecomp(Rd’, &RR)

matrix e = abs(diag(RR)) .> $macheps

QQ = selifc(QQ, e’)

RR = selifr(selifc(RR, e’), e)

Rd = QQ * RR

R = Rd[1:rows(Rd)-1,]’

d = Rd[rows(Rd),]’

ret = 2

endif

endif

return ret

end function

‘‘‘

Additionally, we found that the LLM was able to improve the overall
readability and ease of comprehension of the function by making the variable
names more intuitive and meaningful. The resulting variable names were
succinct and clear, providing a significant improvement over the previous,
less informative names. Importantly, despite the changes to the variable
names, the function itself remained fully functional and accurate. The
abbreviations "cons", "Mat," and "Val" correspond to "constrained," "Matrix,"
and "Value," respectively. This example ultimately demonstrated the utility of
the LLM in assisting coders with generating clear and concise variable names
that facilitate the understanding of code.

Listing 18: Refactored variable names of function CheckNormalizeRd()

⬇

Based on the length restriction of 10 characters, here are some suggestions
for improved variable names in the ‘CheckNormalizeRd‘ function:

- ‘consMat‘ instead of ‘R‘

- ‘consVal‘ instead of ‘d‘

- ‘numCons‘ instead of ‘p‘

- ‘rankCons‘ instead of ‘r‘

- ‘cleanMat‘ instead of ‘Rd‘

- ‘decompMat‘ instead of ‘RR‘

- ‘selector‘ instead of ‘e‘

- ‘cleanedMat‘ instead of ‘QQ‘

With these name changes, the function becomes:

‘‘‘

function scalar CheckNormalizeRd (matrix *consMat, matrix *consVal)

/*

Checks that

(1) the constraints are consistent

(2) the constraints are non-contradictory (??? non-redundant ? - Sven)

if (1) fails, an error message is printed and consMat and consVal

are replaced by empty matrices; if (2) fails, redundant rows

in consMat and consVal are dropped.

*/

numCons = rows(consMat)

rankCons = rank(consMat)

ret = 0

if rankCons < numCons

matrix cleanMat = consMat ~ consVal

if rankCons < rank(cleanMat)

consMat = {}

consVal = {}

ret = 1

else

matrix decompMat

matrix selector = abs(diag(qrdecomp(cleanMat’, &decompMat))) .> $macheps

matrix cleanedMat = selifc(qrdecomp(cleanMat’, &decompMat), selector’)

decompMat = selifr(selifc(decompMat, selector’), selector)

cleanMat = cleanedMat * decompMat

consMat = cleanMat[1:rows(cleanMat)-1,]’

consVal = cleanMat[rows(cleanMat),]’

ret = 2

endif

endif

return ret

end function

‘‘‘

###  3.5 Improving code

Refactoring is a software development technique used to enhance the quality of
existing code without impacting its functionality. This process entails
modifying the code’s internal structure to make it more efficient,
maintainable, and readable while removing duplicate code and ensuring
compliance with coding standards. Refactoring has numerous benefits for
software development projects, including decreased development time and cost,
enhanced reliability and performance, and improved codebase maintainability.
Several studies have underscored the importance of refactoring in software
development projects, including [14].

To illustrate the concept, we utilized the original CheckNormalizeRd()
function from the package written by [10], and instructed the LLM to refactor
it according to the "Clean Code" principle (see Listing 19).131313As a note,
this approach promotes the development of simple, readable, efficient, and
maintainable code, coined by software expert Robert C. Martin, a.k.a. ”Uncle
Bob.”

Listing 19: Prompt to improve function CheckNormalizeRd()

⬇

Suggest improvements for the following gretl function in triple backticks.
Make use of the "Clean Code" approach.

‘‘‘

function scalar CheckNormalizeRd (matrix *R,

matrix *d)

/*

Checks that

(1) the constraints are consistent

(2) the constraints are non-contradictory (??? non-redundant ? - Sven)

if (1) fails, an error message is printed and R and d are replaced by

empty matrices; if (2) fails, redundant rows in R and d are dropped.

*/

p = rows(R)

r = rank(R)

ret = 0

if r < p

matrix Rd = R ~ d

if r < rank(Rd)

R = {}

d = {}

ret = 1

else

matrix RR

matrix QQ = qrdecomp(Rd’, &RR)

matrix e = abs(diag(RR)) .> $macheps

QQ = selifc(QQ, e’)

RR = selifr(selifc(RR, e’), e)

Rd = QQ * RR

R = Rd[1:rows(Rd)-1,]’

d = Rd[rows(Rd),]’

ret = 2

endif

endif

return ret

end function

‘‘‘

Listing 20 shows the proposed refactoring of the code body. Apart from some
formatting issues, the LLM attempts to simplify the nested if-else block.
While in the original code, the ret=0 is returned if the rank(r) $\geq$
rows(R) this was not present in the refactored code resulting in a serious
bug. We do not further evaluate the output here. However, this shows that one
has to be cautious when using LLMs for programming.

Listing 20: Refactored function CheckNormalizeRd()

⬇

Here are some suggested improvements for the ‘CheckNormalizeRd‘ function in
triple backticks, using the "Clean Code" approach:

‘‘‘

function scalar check_normalize_rd(matrix *R, matrix *d)

/*

Checks that the constraints are:

(1) consistent and

(2) non-contradictory.

If (1) fails, an error message is printed and R and d are

replaced by empty matrices; if (2) fails, redundant rows

in R and d are dropped.

*/

num_constraints = rows(R)

rank_R = rank(R)

if rank_R < num_constraints # check for inconsistency

print "The constraints are inconsistent."

R = {}

d = {}

return 1

elif rank_R == num_constraints # check for redundancy

return 0

else

matrix Rd = R ~ d

matrix RR

matrix QQ = qrdecomp(Rd’, &RR)

matrix e = abs(diag(RR)) .> $macheps

QQ = selifc(QQ, e’)

RR = selifr(selifc(RR, e’), e)

Rd = QQ * RR

R = Rd[1:rows(Rd)-1,]’

d = Rd[rows(Rd),]’

return 2

endif

end function

‘‘‘

1. Function names should be in lowercase, so ‘CheckNormalizeRd‘ should be changed to ‘check_normalize_rd‘.

2. Add a brief description of what the function does in the comments.

3. Use descriptive variable names instead of single character variables. For example, ‘p‘ and ‘r‘ should be changed to ‘num_constraints‘ and ‘rank_R‘, respectively.

4. Simplify the conditionals for consistency and redundancy checks.

5. Use early returns instead of nested conditionals to reduce indentation.

6. Refactor the code to avoid duplicating codes within the ‘if‘ and ‘else‘ cases.

7. Add spaces between the operators and operands for better readability.

Applying these changes can make the function easier to read, understand, and
modify.

We present another example in Listing 21. The function maxeval(), sourced from
the SVAR package (v1.97), is examined. This function utilizes linear algebra,
specifically calling gretl built-in functions and a ternary operator
abbreviation for an if-else block. Despite the absence of descriptive comments
or docstrings, the function is complex and necessitates an understanding of
linear algebra.

Listing 21: Prompt to improve function maxeval()

⬇

Suggest improvements for the following gretl function in triple backticks.
Make use of the "Clean Code" approach.

‘‘‘

function scalar max_eval (const matrix A)

n = rows(A)

p = cols(A) / n

matrix compan = p==1 ? A : A | (I(n*(p-1)) ~ zeros(n*(p-1), n))

matrix lambda = eigen(compan)

# (lambda will be complex now)

scalar maxmod = maxc(abs(lambda))

return maxmod

end function

‘‘‘

The output produced is intriguing and displayed in Listing 22. The LLM
comprehends the function’s purpose, calculating the eigenvalue modulus of a
matrix, and identifies two possible scenarios: that the input is either a
square or block matrix. It enhances the function’s readability by
supplementing a docstring, adjusting variable names, and implementing an if-
else block while avoiding ternary operators. Additionally, the code is
simplified so that each line performs a single operation, in accordance with
the "Clean Code" philosophy.

Listing 22: Answer of improved function maxeval()

⬇

Here are a few suggestions for improving the ‘max_eval‘ function using the
"Clean Code" approach:

‘‘‘

function scalar max_eval(const matrix A)

/*

Computes the maximum modulus of all eigenvalues of the matrix A.

If A has multiple columns, it is assumed that it is arranged in a

stacked block matrix format with n x n submatrices, and the function

computes the maximum eigenvalue modulus of the matrix A as if it were

block diagonal. If A has only one column, it is treated as a square

matrix of size n x n.

*/

num_rows = rows(A)

num_cols = cols(A)

is_block_matrix = num_cols > num_rows

if is_block_matrix

num_blocks = num_cols / num_rows

identity_matrix = I(num_rows * (num_blocks - 1))

zeros_matrix = zeros(num_rows * (num_blocks - 1), num_rows)

stacked_matrix = A | (identity_matrix ~ zeros_matrix)

eigenvalues = eigen(stacked_matrix)

else

eigenvalues = eigen(A)

endif

max_eigenvalue_modulus = maxc(abs(eigenvalues))

return max_eigenvalue_modulus

end function

‘‘‘

1. Add a brief description of what the function does in the comments.

2. Use descriptive variable names instead of single character variables. For example, ‘n‘, ‘p‘, and ‘compan‘ should be changed to ‘num_rows‘, ‘num_blocks‘, and ‘stacked_matrix‘, respectively.

3. Add whitespace around operators to improve readability.

4. Add conditional logic to handle the case where ‘A‘ has multiple columns. In this case, it is assumed that the columns represent stacked block matrices, where each block is a square matrix of size n x n.

5. Instead of using a ternary operator to define ‘compan‘, use an ‘if‘ and an ‘else‘ clause to make the code more readable.

6. Refactor the code to reduce indentation by using an early return.

7. Add more complete documentation to describe how the function works, what type of input it takes, and what output it produces.

8. Use consistent and meaningful variable names, such as ‘is_block_matrix‘ and ‘max_eigenvalue_modulus‘.

By applying these changes, the resulting function becomes easier to understand
and modify.

Overall, these examples demonstrate that LLMs may be used to improve code
quality through refactoring. However, the output must be scrutinized.

###  3.6 Write unit tests

Unit testing is standard in the modern programming paradigm for detecting bugs
in units of a program. However, writing high-quality unit tests can be time-
consuming. [24] study the performance of ChatGPT to generate unit tests. They
find that the LLM suffers from correctness issues including diverse
compilation errors and execution failures. Still, the authors interpret the
results as promising for automatically writing unit tests.

To evaluate the LLM’s ability to write unit tests for gretl, we employed the
refactored maxeval() function from the previous subsection. In Listing 23, we
prompted the LLM to write a unit test for this function. The unit test should
contain an artificial dataset and compare the actual output with an expected
result for both the square and block matrix cases.

Listing 23: Prompt to write a unit test for function maxeval()

⬇

Please write a unit test using the gretl syntax for the following gretl
function in triple backticks.

‘‘‘

function scalar max_eval(const matrix A)

/*

Computes the maximum modulus of all eigenvalues of the matrix A.

If A has multiple columns, it is assumed that it is arranged in a

stacked block matrix format with n x n submatrices, and the function

computes the maximum eigenvalue modulus of the matrix A as if it were

block diagonal. If A has only one column, it is treated as a square

matrix of size n x n.

*/

num_rows = rows(A)

num_cols = cols(A)

is_block_matrix = num_cols > num_rows

if is_block_matrix

num_blocks = num_cols / num_rows

identity_matrix = I(num_rows * (num_blocks - 1))

zeros_matrix = zeros(num_rows * (num_blocks - 1), num_rows)

stacked_matrix = A | (identity_matrix ~ zeros_matrix)

eigenvalues = eigen(stacked_matrix)

else

eigenvalues = eigen(A)

endif

max_eigenvalue_modulus = maxc(abs(eigenvalues))

return max_eigenvalue_modulus

end function

‘‘‘

Although the output in Listing 24 appeared reasonable at first glance, closer
inspection revealed several issues. For the square matrix case, the LLM
incorrectly created the matrix using matrix A = 1, 2; 3, 4  instead of gretl
syntax.141414In gretl, creating a matrix is done by matrix A = { 1, 2; 3, 4 }.
Additionally, its expected maximum eigenvalue of 5.0 was incorrect, as the
correct value was 5.372. Furthermore, the LLM was not familiar with gretl’s
built-in assert() function for invoking assertions. Thus, the unit test would
fail even though the function returns the correct value.

For the stacked block matrix case, the LLM provided a poor example of a unit
test. It again failed to properly create a matrix using gretl syntax, and the
proposed matrix B was square rather than block, thus failing to test the
intended scenario. The expected maximum eigenvalue of 13 was also incorrect,
as the correct value was 13.15.

Listing 24: Proposed unit test for function maxeval(). Syntactical error
coloured in red.

⬇

Here’s an example unit test for the ‘max_eval‘ function in Gretl syntax:

‘‘‘

# Test for a square matrix

matrix A = { {1, 2}, {3, 4} }

scalar expected_result = 5 # max eigenvalue modulus of A is 5

scalar test_result = max_eval(A)

printf "Test 1 - Square Matrix: "

if test_result == expected_result

print "Passed"

else

print "Failed"

endif

# Test for a stacked block matrix

matrix B = { {1, 2, 0, 0}, {3, 4, 0, 0}, {0, 0, 5, 6}, {0, 0, 7, 8} }

scalar expected_result = 13 # max eigenvalue modulus of B is 13

scalar test_result = max_eval(B)

printf "Test 2 - Stacked Block Matrix: "

if test_result == expected_result

print "Passed"

else

print "Failed"

endif

‘‘‘

In this example, we run two tests. The first test uses a 2 x 2 square matrix
and verifies that the ‘max_eval‘ function returns the expected result of 5.
The second test uses a stacked 2 x 2 block matrix and verifies that the
‘max_eval‘ function returns the expected result of 13. We use ‘printf‘ to
print informative messages to the console, and we use ‘if‘ statements to check
if the function’s output matches the expected output.

Overall, while the LLM’s unit test appeared to be well-structured, it
struggled to create an artificial dataset using gretl syntax and produced
incorrect expected values. Additionally, it failed to test the two distinct
cases that the function comprised.

###  3.7 A tiny coursework example

The following prompt describes an exercise for an introductory econometrics
course. The student is asked to load a dataset, manipulate a variable, compute
summary statistics, and run an ordinary least square (OLS) regression (refer
to Listing 25).

Listing 25: Prompt to write a gretl script

⬇

I want a gretl script doing the following tasks:

1. Load the "denmark" dataset.

2. Create the first differences of the series "LRM" and "LRY".

3. Compute summary statistics of the series created at step 2.

4. Run a OLS regression. The endogenous variable is the first difference of "LRM". Regress on a constant and the first difference of "LRY".

5. Store the fitted values as "yhat".

6. Plot the fitted values over time

The output in Listing 26 is noteworthy, with only two minor errors. The script
correctly loads the time-series dataset shipped by gretl and creates the
desired variable using the built-in diff() function. The LLM is aware that
first differencing results in the loss of the initial observation and attempts
to set the sample start to the second observation, which is correct. However,
the syntax for the smpl command is incorrect; it should be smpl 2 ;  which
restricts the sample to start with the second observation. The summary command
is used correctly for computing descriptive statistics, followed by the ols
command to run the OLS regression and the call to the internal $yhat accessor
to store fitted values. Lastly, the LLM attempts to call the gnuplot command,
but the syntax is incorrect; it should be gnuplot yhat --time-series --with-
lines --output=display.

Listing 26: Proposed gretl script. Syntactical error coloured in red.

⬇

open denmark.gdt

# create first differences of "LRM" and "LRY"

series dLRM = diff(LRM)

series dLRY = diff(LRY)

# compute summary statistics of the series created at step 2

smpl 2: \--quiet # this sets the sample from the second observation to the end

summary dLRM

summary dLRY

# run OLS regression and store fitted values as "yhat"

ols dLRM const dLRY \--quiet

series yhat = $yhat

# plot fitted values over time

gnuplot yhat vs time

Next, we engage in iterative prompt development and send the prompt to the LLM
to remove the smpl command.151515In gretl all the following commands
automatically deal with missing values such that we do not need to restrict
the sample explicitly. The returned output includes the same script but
without the smpl call (not shown).

⬇

Good job. However, get rid of the "smpl" command which is not needed.

The returned output consists of the same script just without the call of the
smpl command (not shown here).

We then ask the LLM to correct the gnuplot command’s syntax to gnuplot yhat
--with-lines --time-series --output=display. However, despite the correction,
the output for this command remains incorrect (shown below).

⬇

Also, the syntax calling the "gnuplot" command is not correct. Please fix
that. Add the options to:

1. Plot the time-series as a line.

2. Output the plot on the screen.

3. And add the time-series option.

⬇

.

.

.

# plot fitted values over time

gnuplot yhat \color{red}with lines time

##  4 Conclusion

In conclusion, this study has examined the usage and effectiveness of current
LLM models for programming with a low-resource and domain-specific programming
language. As an application, we choose gretl and its scripting language hansl.
We have employed the LLM provided by www.you.com which is based on GPT-3.5
when writing this article. The findings show that publicly available LLMs
already can be a useful tool for understanding, writing, and improving gretl
code, despite the fact that only little gretl code is publicly available. This
indicates that current LLMs are able to generalize well to LRPL.

Specifically, the LLM produced useful and descriptive docstrings for gretl
functions, translates docstrings back to gretl code and vice versa, helped to
improve the readability and maintainability of code by suggesting better
function and variable names, and provided precise and technical explanations
of abstract and poorly documented econometric code. Also, we showed how the
LLM helps refactoring gretl code mainly involving linear algebra. However, the
LLM was not always successful in improving code and also failed to write a
correct unit test. Lastly, we presented a simple exercise for an introductory
econometrics course. The written script by the LLM is useful as a starting
point for students as the syntactical errors are of minor type. We have shown,
that the LLM is expected to correct some of the syntactical errors by means of
iterative prompt development.

Future research could build on these findings by exploring ways to fine-tune
LLM models for gretl code. It also would be interesting to evaluate whether a
modern LLM helps to detect and correct errors in gretl code. Lastly, LLMs may
be used to translate code from another language into gretl which is a topic
under active research [16].

Overall, this study provides valuable insights into the potential uses and
limitations of LLMs in programming with the low-resource and domain-specific
econometric language gretl.

## References

  * [1] Toufique Ahmed and Premkumar Devanbu  “Few-shot training LLMs for project-specific code-summarization”, 2022  arXiv: https://arxiv.org/abs/2207.04237
  * [2] Toufique Ahmed and Premkumar Devanbu  “Multilingual training for software engineering”  In _Proceedings of the 44th International Conference on Software Engineering_ ACM, 2022  DOI: 10.1145/3510003.3510049
  * [3] Sébastien Bubeck et al.  “Sparks of Artificial General Intelligence: Early experiments with GPT-4”, 2023  URL: https://www.microsoft.com/en-us/research/publication/sparks-of-artificial-general-intelligence-early-experiments-with-gpt-4/
  * [4] Fuxiang Chen, Fatemeh Fard, David Lo and Timofey Bryksin  “On the Transferability of Pre-trained Language Models for Low-Resource Programming Languages”, 2022  arXiv: https://arxiv.org/abs/2204.09653
  * [5] Mark Chen et al.  “Evaluating Large Language Models Trained on Code”, 2021  arXiv:2107.03374 [cs.LG]
  * [6] Allin Cottrell  “Hansl: A DSL for Econometrics”  In _Proceedings of the 2Nd International Workshop on Real World Domain Specific Languages_ , RWDSL17  Austin, TX, USA: ACM, 2017, pp. 1:1–1:10  DOI: 10.1145/3039895.3039896
  * [7] Zi Gong et al.  “MultiCoder: Multi-Programming-Lingual Pre-Training for Low-Resource Code Completion”, 2022  arXiv:2212.09666v1 [cs.CL]
  * [8] Hamel Husain et al.  “CodeSearchNet Challenge: Evaluating the State of Semantic Code Search”, 2020  arXiv:1909.09436 [cs.LG]
  * [9] Anton Korinek  “Language Models and Cognitive Automation for Economic Research”, Working Paper Series 30957, 2023  DOI: 10.3386/w30957
  * [10] Jack Lucchetti and Sven Schreiber  “The SVAR addon for gretl”, 2022  URL: https://sourceforge.net/projects/gretl/files/addons/doc/SVAR.pdf
  * [11] Jack Lucchetti and Francesco Valentini  “Time-Varying OLS/IV estimators in gretl: the ketvals package”, 2023  URL: https://gretl.sourceforge.net/current_fnfiles/unzipped/ketvals.pdf
  * [12] Zohar Manna and Richard J. Waldinger  “Toward Automatic Program Synthesis”  In _Commun. ACM_ 14.3 New York, NY, USA: Association for Computing Machinery, 1971, pp. 151–165  DOI: 10.1145/362566.362568
  * [13] Pierrette Mahoro Mastel et al.  “NATURAL LANGUAGE UNDERSTANDING FOR AFRICAN LANGUAGES”  In _4th Workshop on African Natural Language Processing_ , 2023  URL: https://openreview.net/forum?id=gWuvdFMqHM
  * [14] T. Mens and T. Tourwe  “A survey of software refactoring”  In _IEEE Transactions on Software Engineering_ 30.2, 2004, pp. 126–139  DOI: 10.1109/TSE.2004.1265817
  * [15] Alec Radford et al.  “Language models are unsupervised multitask learners”  In _OpenAI Blog_ 1.8, 2019, pp. 9
  * [16] Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot and Guillaume Lample  “Unsupervised translation of programming languages”  In _Advances in Neural Information Processing Systems_ 33, 2020
  * [17] Weisong Sun et al.  “Automatic Code Summarization via ChatGPT: How Far Are We?”, 2023  arXiv: https://arxiv.org/abs/2305.12865
  * [18] Artur Tarassow  “Practical Empirical Research Using gretl and hansl”  In _Australian Economic Review_ 52.2, 2019, pp. 255–271  DOI: 10.1111/1467-8462.12324
  * [19] Haoye Tian et al.  “Is ChatGPT the Ultimate Programming Assistant – How far is it?”, 2023  arXiv: https://arxiv.org/pdf/2304.11938
  * [20] Ashish Vaswani et al.  “Attention is all you need”  In _Advances in Neural Information Processing Systems_ , 2017, pp. 5998–6008
  * [21] Ben Wang and Aran Komatsuzaki  “GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model”, https://github.com/kingoflolz/mesh-transformer-jax, 2021
  * [22] Stephen Wolfram  “What Is ChatGPT Doing … and Why Does It Work?”, 2023  URL: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work
  * [23] Frank F. Xu, Uri Alon, Graham Neubig and Vincent Josua Hellendoorn  “A Systematic Evaluation of Large Language Models of Code”  In _Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming_ , MAPS 2022  San Diego, CA, USA: Association for Computing Machinery, 2022, pp. 1–10  DOI: 10.1145/3520312.3534862
  * [24] Zhiqiang Yuan et al.  “No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation”, 2023  arXiv: https://arxiv.org/pdf/2305.04207.pdf
  * [25] Daniel Zügner et al.  “Language-Agnostic Representation Learning of Source Code from Structure and Context”  In _International Conference on Learning Representations_ , 2021  URL: https://openreview.net/forum?id=Xh5eMZVONGF