Paper Title: A Performance Study of LLM-Generated Code on Leetcode.






















breaklines
bgcolor=LightGray









cdfCDFCumulative Distributed Functions
llmLLMLarge Language Model










formal
  
    
    width 2pt
    width 4pt
    
  -
  
  7pt
  




  


resultcounter
result
	resultcounter
	




2024
2024
28th International Conference on Evaluation and Assessment in Software Engineering (EASE 2024), June 18–21, 2024, Salerno, Italy
10.1145/3661167.3661221
979-8-4007-1701-7/24/06

none







    Univ. Lille, CNRS, Inria
    France

 0009-0003-2525-3637
tristan.coignion@inria.fr



    Univ. Lille, CNRS, Inria
    France

0000-0003-3203-6107
clement.quinton@inria.fr



    Univ. Lille, CNRS, Inria
    France

0000-0003-1771-8791
romain.rouvoy@inria.fr


§ ABSTRACT 
This study evaluates the efficiency of code generation by llm and measures their performance against human-crafted solutions using a dataset from Leetcode.
We compare 18 llm, considering factors such as model temperature and success rate, and their impact on code performance. 
This research introduces a novel method for measuring and comparing the speed of llm-generated code, revealing that llm produce code with comparable performance, irrespective of the adopted llm.
We also find that llm are capable of generating code that is, on average, more efficient than the code written by humans. 
The paper further discusses the use of Leetcode as a benchmarking dataset, the limitations imposed by potential data contamination, and the platform's measurement reliability. 
We believe that our findings contribute to a better understanding of llm capabilities in code generation and set the stage for future optimizations in the field.





A Performance Study of llm-Generated Code on Leetcode
    Romain Rouvoy
    
1Fusion@ÖAW, Institute for Theoretical Physics - Computational Physics, Graz University of Technology, 8010 Graz, Austria

2 Max-Planck-Institut für Plasmaphysik, 17489 Greifswald, Germany

3Institute of Plasma Physics, National Science Center “Kharkov Institute of Physics and Technology”, 61108 Kharkov, Ukraine

===============================================================================================================================================================================================================================================================================================================================





§ INTRODUCTION

llm have recently increased in popularity, especially with the advent of ChatGPT <cit.>.
While llm have been spreading over various application domains, such as text or image generation, certain types of llm are being developed solely for code-related purposes.
These llm aim to assist the developers by saving time and effort through the generation of code, documentation, unit tests, etc.
Many of these llm only come in a “raw" form, that is, they do not integrate themselves into the developer's coding process.
These include models, such as CodeGen <cit.>, 
StarCoder <cit.>, 
WizardCoder <cit.>, 
CodeT5 <cit.>, and Incoder <cit.>.
On the other hand, some llm are already seamlessly integrated into the developer's IDE as code assistants, like GitHub Copilot,[<https://github.com/features/copilot>] Amazon CodeWhisperer,[<https://aws.amazon.com/fr/codewhisperer/>] and Tabnine.[<https://www.tabnine.com/>]

There has been a significant amount of work dedicated to comprehending how these llm perform in various situations and defining their limits. 
For instance, several works address the security of the code generated by such models <cit.> or the prevalence of bugs in the generations <cit.>. 
Many researchers are also investigating how developers interact with llm and how such models fit into the programming workflow <cit.> 
There is also a broad research effort to measure the actual efficiency of these llm by creating common benchmarks for comparison <cit.> or by actually measuring different qualities related to the generations, such as the success rate <cit.> or the robustness of the model regarding variations <cit.>.

To the best of our knowledge, there is no research work evaluating the performance of the code generated by llm.
Yet, having code that runs faster is an often sought-after characteristic of a program. 
Indeed, programming efficiency is paramount, especially when resources are scarce or programs are deployed on a large scale. 
In today's context where the energy consumption of software systems has become a major concern,
improving software efficiency is particularly relevant since increasing the performance of a program can also lead to energy consumption reduction <cit.>.

The process of code optimization is lengthy and intricate, requiring careful attention and a certain level of expertise, especially when searching for the best-performing algorithm, selecting the most appropriate data structure, or struggling with memory hierarchy.
Yet, this process is necessary to identify opportunities for improvement that may result in minor reductions in execution time.
llm can be used as a way to make this process easier, by generating performance-improving code edits <cit.>. 
Garg  <cit.> also presented RAPGen, a method for generating zero-shot prompts to enhance performance.
However, while users seem to put a lot of trust in code generated by llm, they still have trouble reviewing it <cit.>, which can lead to slow code being shipped in production, especially if an llm generates inefficient code.

The key contributions of this paper are as follow :
(i) We study the performance of the code generated by 18 llm on 204 problems and investigate performance differences across models using a novel method for measuring and comparing the performance of llm-generated code.
(ii) We compare the performance of the code generated by llm to the code written by humans.
(iii) Incidentally, we evaluate the usability of Leetcode,[<https://leetcode.com/>] a public repository of algorithmic problems that we use as a dataset.




From <ref> to <ref>, we describe the tasks dataset and the models we selected, outline our experiment setup, and explain the methodology we followed to analyze the obtained results, respectively.
We report in <ref> on the results of our evaluation and provide a critical discussion in <ref>. 
Finally, <ref> presents the related works, and <ref> concludes the paper.



§ METHODOLOGY





 §.§ Research questions

This paper covers the performance of code generated by various llm.
In particular, we aim to answer the following research questions:

RQ1: Can Leetcode be used as a dataset and a benchmark platform for evaluating llm?
Leetcode can serve as both a dataset of problems and a tool to evaluate and measure solutions to the problems.
Particularly, we study if the dataset is subject to recitation and if the measures Leetcode provides are reliable.

RQ2: Are there notable differences between the performance of the code generated by different llm?
llm differ greatly in terms of generating correct code, so we want to know if they also differ in terms of generating efficient code.

RQ3: Is there an effect of the success rate and the temperature of the llm on the code's performance?
Having a higher temperature decreases the capacity of the llm to generate valid code, so we aim to study if this also applies to the performance of the code. 
In the same way, we also study if an llm that is very good at generating valid code on one problem, is going to generate efficient solutions to this problem.

RQ4: How efficient are the solutions generated by the llm compared to human solutions?
Comparing the llm to a set of human-authored solutions can provide insights into their position relative to humans in terms of code performance.




 §.§ Tasks & Dataset


Task selection.
The input tasks—i.e., problems specified by a prompt—we consider to generate code from various llm has to meet the following requirements:

    
  * A given problem, should offer multiple candidate solutions, whose generated code performs differently.
    This ensures one can observe differences across the various llm;
    
  * Generated solutions should exhibit variable execution times. 
    Given more complex inputs, one should differentiate 𝒪(n) from 𝒪(2n) and 𝒪(n^2) algorithms.

As a result, task datasets, such as HumanEval or Mostly Basic Python Programming (MBPP), which are classically used when evaluating llm for code assessments <cit.>, cannot be considered for our purpose.
Indeed, while they do provide unit tests to drive the generations, the size of the inputs in these unit tests remains small and fails to scale to appreciate performance issues.
Moreover, the solutions that need to be generated are often very short, which would lead to fewer possible variations between implementations.
Also, the fact that many problems are not algorithmic by nature makes them less prone to inefficient practices and performance variation.
To face such issues, we used input prompts that were built from Leetcode problems.
Leetcode is an online judge platform that suggests programming problems to registered users.
It addresses the above limitations as it provides algorithmic problems with varying levels of difficulty and test cases with large input sizes.
Leetcode also exposes a GraphQL API[<https://pypi.org/project/python-leetcode/>] to fetch relevant metadata on the problems, such as exercise instructions, code snippets containing the signature of the function to generate, as well as the difficulty and topics of the problem.

We followed an experimental design similar to the one used by Döderlein  <cit.>, while using a different set of Leetcode questions. 
To avoid data contamination, which happens when an llm is tested on data it was trained on,[You can find more details on data contamination https://web.archive.org/web/20231020160612/https://bdtechtalks.com/2023/07/17/llm-data-contamination/here.] we only considered problems that were published after January 1^st, 2023. 
As all the llm (except GitHub Copilot) we evaluated were trained using datasets older than these problems, we avoid any data contamination.
As these problems are published by Leetcode in the context of programming competitions, they are always original.
However, GitHub Copilot being an online closed-source tool, one cannot tell whether it underwent training with the problem set we employed. 
Our set was composed of 204 problems—labeled as 56 easy problems, 104 medium problems, and 44 hard problems, as classified by Leetcode upon the problem's publication.

To answer RQ1, we also performed our experiment a second time on the set of questions used by Döderlein  <cit.>, which is composed of 300 problems (95 easy, 105 medium, and 100 hard) from the most liked problems of Leetcode. 
We will refer to this dataset as the "old" dataset, and to our dataset of problems published during 2023 as the "new" dataset.
Code generation was performed in Python
for its ease of use and because of the prevalence of Python-written datasets for evaluating llm <cit.>.



Input prompts.
The instructions given by Leetcode for each programming problem contain (i) the description of the problem, (ii) examples of inputs and outputs, and (iii) constraints on the input data. 
To build the prompts inputted to the llm, we chose to only include the description of the problem.
This choice aimed to maintain the prompt's conciseness and ensure compliance with potential llm' length restrictions. 
Indeed, llm are constrained by a context size, imposing the maximum number of tokens they can process at once, encompassing both the prompt and the answer. 
In our study, the majority of llm impose a context window of 1024 tokens.
<ref> illustrates a descriptive prompt (enclosed within triple quotes) along with a solution generated using this prompt and the corresponding benchmark instructions. 
Leetcode occasionally includes an additional comment in the prompt, indicating the available methods of the elements passed to the solution (binary trees).
While adding examples in the prompt could potentially increase the likelihood of generating correct solutions <cit.>, we chose not to include them due to the complexity of representing Leetcode's data structures (such as arrays, graphs, linked lists, trees, etc.) in textual form.
On Leetcode's website, data structures are textually represented using an array notation format with brackets (“[1, 2, 3]"), which might be misleading for an llm working with data structures that differ from arrays or lists.

Canonical solutions.
Each problem was matched with a single valid solution, written by a human and fetched from various sources.
These solutions, referred to as “canonical solutions", are considered as baselines during the benchmarking process, although they may not represent the entirety of human-written solutions. 
They were used to assess the stability of the measuring process.
Most of the canonical solutions were fetched from the WalkCC repository of Leetcode solutions.[<https://github.com/walkccc/LeetCode>]
When one question did not come up with any solution in Python from this repository, we selected one solution proposed by the Leetcode community among the most upvoted ones (starting from the one with the most upvotes), which are also publicly available on the Leetcode website.
These modifications only included changes to the function name, variable names, and type hints except for one specific case, where a recursive solution provided by WalkCC was replaced by an iterative one from the Leetcode community because of the stack limit on our local setup.
We ended up with one Python-based canonical solution for each problem of our dataset.

Test cases.
Testing generated solutions is a twofold process, as both the correctness and scalability (performance) of each solution must be checked.
First, to ensure that generated solutions are correct, we provide such solutions to the Leetcode online judge system, which in turn validates them based on its test suites.
Second, we execute the valid solutions with input data to assess their performance.
To retrieve such input data, we crawled through the problems' instructions and extracted two to three examples of inputs and expected outputs provided by Leetcode.
However, such inputs were too small to exhibit significant performance differences, arrays containing only two to five elements.
To execute the generated solutions with larger input data, we took advantage of Leetcode's judge system that returns the inputs and expected output of the first failed test of a test suite.
We noticed that Leetcode tests if the submitted solution is inefficient by using a timeout system. 
Since all selected problems are algorithmic by nature, one simple way to put a heavier load on their implementations is to increase the size of the inputs.
For every problem, we thus submitted a modified version of a canonical solution that failed only when the size of the first parameter exceeded a certain threshold. 
We then set this threshold manually multiple times for every problem to extract three different inputs for each problem, resulting in more than 150 MB of fetched input data with most inputs having over 10^5 elements.



 §.§ llm Under Study

Our empirical study covers a total of 18 llm, specifically designed for coding purposes.
We selected the 18 popular code llm from Hugging Face,[<https://huggingface.co>] as well as GitHub Copilot, which is an online closed-source code assistant. The LLMs were selected based on the number of downloads and likes they exhibited.
We chose GitHub Copilot to offer a comparison between a commercial llm and open-source llm, but did not choose any other GPT models from OpenAI because of their cost.
<ref> summarizes all the llm considered in this study.
This includes variants of the same base llm—models with varying sizes (in billions of parameters) or different training data, such as variants of CodeGen, InCoder, and CodeT5.
llm belonging to the same family are models closely related in terms of training data and method. 
We first performed our experiment in March 2023 with a subset of the models presented here, with the "old" dataset. 
We then performed it a second time in September 2023 with all the models and the "new" dataset.







§ EXPERIMENT SETUP

This section describes our experiment setup to generate and validate the solutions produced by the llm. 
First, we describe how solutions were generated from each llm.
Then, we outline the three-step process used to filter invalid solutions. 
Finally, we explain how the run time of the generated solutions was measured.



 §.§ Code Generation

We generated 10 solutions for each problem from our dataset by varying the temperature of the llm used to generate them. 
Specifically, we considered 6 different temperatures (0.1, 0.2, 0.4, 0.6, 0.8, and 1.0).
As Copilot's temperature cannot be configured, we used its default temperature for generating all the problems.

Generating code with GitHub Copilot. 
Automatically generating with GitHub Copilot for a reproducible experiment proved to be a difficult task.
Firstly, the code suggestion feature of Copilot activates when typing in a text editor with the installed Copilot plugin. 
Additionally, GitHub Copilot produces code based on a context that encompasses the current file and the files previously accessed by the user, impacting the generated solutions.
Lastly, we noticed a caching mechanism on the GitHub Copilot server side, which resulted in very similar or identical solutions if we generated multiple solutions for a given problem in the same session.
To address these issues, we used a generation method similar to the one used by Döderlein  <cit.> by instrumenting the GitHub Copilot Neovim plugin[https://github.com/github/copilot.vim] and restarting the plugin between every generation to avoid the caching effect.
This method allowed us to automatically generate solutions in a quick and isolated fashion.
On top of that, GitHub Copilot provides 2 means of generation: inline generations (the suggested code is integrated with the editor) and panel generations (Copilot generates at most 10 completions and displays them on a panel next to the editor).
We chose to exclusively use inline generations, as panel generations yielded worse results in terms of functional correctness than inline generations.

Generating code with open-source models.
Regarding the open-source models, we generated solutions by deploying the models on servers provided by the Grid5000 platform <cit.>.
We used the Deepspeed library to make the generation process faster and fit larger models on our GPUs.
Concerning the sampling, we used the same methods as Chen  <cit.> and used nucleus-sampling <cit.> with top p=0.95. 
The maximum number of tokens to be generated was set to 600. 
This is because the llm we used have a limited context size of 1024 tokens (prompt included). 
Thus, to avoid exceeding the context size, we had to limit the number of tokens to generate.
We also verified it did not significantly impact the functional validity of the llm.

In total, we generated 2,040 solutions with Copilot and 12,240 with each of the eight other models, resulting in 210,120 generated solutions overall.



 §.§ Validation

Each generated solution was tested to ensure its functional correctness following a three-step process.
At every step, if a solution was found to be invalid, it was excluded from subsequent stages of the experiment.
The process was as follows:

i) Local validation. 
We filtered out code generations that included easy-to-spot errors, such as syntax errors or runtime errors, by using the small inputs we fetched earlier (see tasks - Test cases). 
While this step was not strictly necessary, it quickly reduced the number of solutions to be validated in the next step;

ii) Leetcode validation.
Next, we submitted the solutions to the Leetcode judge system using the Leetcode GraphQL API, where they underwent a rigorous test suite managed by Leetcode. 
We kept the solutions that passed all the test cases or exceeded Leetcode's allocated time limit. 
The latter was kept to ensure that correct solutions that were too slow remained included in the benchmark;

iii) Exclusion of timeouts and other errors.
Finally, we excluded the solutions that reported errors when executed with our benchmarking setup using the large inputs fetched beforehand.
We invalidated the code generations that took more than 10 seconds to run or raised an error. 
Most of the errors raised in this step were recursion errors caused by differences between Leetcode's Python interpreter and ours. 
Indeed, Leetcode's seemed to have a higher recursion limit than ours, which we set to 10,000 instead of the default 1,000 (we could not manage to set it any higher). 
Additional errors occurred because we developed our helper classes differently from Leetcode's implementation.
Although these classes are provided by Leetcode during the submission process, they are not publicly available.
Out of the 4,930 invalidated solutions that were caught in this step, 4,863 (98.6%) were due to timeouts, 20 (0.04%) to recursion errors, and 47 (0.1%) to other errors.
Following this validation process, the initial set of 210,120 generated solutions was pruned down to 7,481 (3.6%) valid solutions remaining across the 18 llm. 



 §.§ Measuring run time

We measured the performance as the run time of the generated solutions using pytest-benchmark,[https://github.com/ionelmc/pytest-benchmark] which runs pytest unit tests multiple times to obtain run times statistics.
The measurements were performed using parameters that ensured each solution ran at least 10 times and for at least 1 second in total.
We did not perform warm-up runs of the benchmarks, as we did not notice any significant difference in the measured time during preliminary testing. 
To facilitate the measurement protocol, the generated solutions were sorted into “runs", based on the specific llm and temperature that were used during the code generation.
Within each run, which was defined by a unique combination of model and temperature, the solutions were measured in sequence during a single program execution.
Furthermore, in every run, we added the canonical solutions we previously collected, which would run alongside the generated solutions.
This approach ensured that the same canonical solutions were executed in every run, allowing us to maintain measurement stability.
Specifically, we calculated the standard deviation of the canonical solution run times across all runs, thus providing a reliable measure of the variability of the measurement protocol.
We observed that over 96% (196 out of 204) of the canonical solutions had a standard deviation lower than 1/10^th of their average run time, which we deemed to be an acceptable level of variation.

The cluster we used to run the benchmark was the chiclet cluster of the Grid5000 testbed.[<http://grid5000.fr>]
It hosts 2 AMD EPYC 7301, with 16 cores per CPU and 128GB of memory.
When using the node, all the cores of both CPUs were reserved, but only one was used at a time to maximize the stability of the measurement protocol.



 §.§ Replication package

All the artifacts of this study, including our results, code, and datasets, are available in the following public repository: <https://zenodo.org/doi/10.5281/zenodo.7898304>.




§ DATA ANALYSIS

In this section, we describe the methods we adopted to analyze our results.
These methods fall into one of the two following categories: functional correctness and code performance.



 §.§ Functional Correctness

The functional correctness of an llm defines how much the llm outputs code conforming to the program contract (as specified by the input prompt).
To evaluate the functional correctness of our llm, we computed their pass@k metrics with k=1 and k=10, using the unbiased estimator proposed by Chen  <cit.>.
The pass@k unbiased estimator which, from k samples produced, considers the test as successful if one of these samples passes all the tests, is computed as follows (with n the total number of samples, c the number of correct samples and 𝔼 the expected value):



    pass@k := Problems𝔼[1- n-ck/nk]




As Chen  <cit.> suggest, we calculated the pass@k for each temperature when evaluating an llm's functional correctness and considered the best one as the pass@k for that llm.



 §.§ Code Performance

To measure the code performance, we considered three different metrics.
First, we used the memory usage reported by Leetcode. Then, we computed the median of the run times measured by pytest-benchmark for every generated solution.
Lastly, to compare the llm solutions to human-submitted solutions, we also used the rank reported by Leetcode when validating the solution. 
This rank is a number between 0 and 100 that indicates the share of submitted solutions that are slower than the current solution (if a solution has a rank of 90, it is faster than 90% of the submitted solutions on Leetcode). 

To assess the llm' performances, we conducted pairwise comparisons as follows: For each pair of llm, we identified problems where both models generated more than 5 valid solutions. 
For each identified problem, we conducted a Student t-test on the mean run time of the generations to determine if there was a significant difference. 
Then, for each pair A-B of llm, we computed the ratio of problems where A's code was significantly faster than B's and where B's code was significantly faster than A's.



§ RESULTS


In this section, we summarize the key observations from our experiment and answer our research questions. 
In <ref>, you can also find the functional validity results of the different llm on both of our Leetcode datasets.
Our results are also available in the form of a companion notebook in our replication package. 
The companion notebook offers more insight into the results and additional graphs.





 §.§ RQ1: Can Leetcode be used as a dataset and a benchmark platform for evaluating llm?




  §.§.§ Can Leetcode problems be adopted as a dataset for llm generation?





As on can observe in <ref>, the generated codes exhibit on a significant drop in functional correctness between the two datasets. 
The difference here is pretty staggering for every tested llm, reporting on a tenfold decrease in pass@k.
We believe this issue may stem from data contamination in the old dataset. 
Data contamination occurs when an llm is assessed on data that was included in the training dataset, introducing bias into the evaluation process.
In our case, a significant number of questions in the old dataset are widely known and have been extensively shared on GitHub. 
For instance, a search for the prompt of the "3sum" Leetcode problem on GitHub yields approximately 4.000 matches in public repositories.
These questions are also old enough to likely be included in the training datasets of the llm under study, as the majority of their training datasets have a cut-off date between 2021 and 2022. 
Due to this data contamination, llm tend to recite, reproducing verbatim source code when generating solutions.
This phenomenon is more pronounced when the prompt is highly specific and lacks contextual information, as seen in Leetcode prompts that closely match GitHub repositories.
The observed shift in functional validity between the two datasets could also arise from a genuine difference in the difficulty of the questions within each dataset. 
However, quantifying this last hypothesis proves to be challenging.





  §.§.§ Are Leetcode measurements reliable?


Run time.
As reported in <ref>, the coefficient of variation of the Leetcode measures (0.089) is slightly higher than the coefficient of variation of the local measures (0.035). 
This suggests that Leetcode's measuring setup is less suited to ensure accurate benchmarks.

We also study the correlation between our local measurements and Leetcode's, and we notice two issues: (1) the times measured locally and by Leetcode only slightly correlate on average (0.28). 
For some problems, the measures are highly correlated (> 0.8) while, for others, they are almost not (<0.2).
This is more apparent when we look at the scatter plot showing the measures of some problems in <ref>. 
In this problem, there are four clusters of generations with a different locally measured time, but the clusters are indiscernible in terms of Leetcode time. 
This could be due to two main reasons. 
Firstly, as previously discussed, the variance of the measures from Leetcode is higher and as such, there is much more noise in the measures, decreasing the precision. 
Secondly, the tests we employed may be more focused on performance testing than Leetcode's. 
Notably, our test suite comprises only three tests featuring significantly large inputs, potentially accounting for certain disparities in the results.

Although still usable, relying on the time reported by Leetcode introduces some limitations due to its higher variance. 
Consequently, Leetcode's measures cannot allow us to discern the differences in run time between different code implementations as precisely as locally measured time.





Memory usage.
While we did not measure the memory ourselves, we observed the variation of the memory usage measure provided by Leetcode. We notice that the memory usage for the same solutions decreases over time. There is indeed a slight correlation of -0.24 between the day of the year we tested our solution and the memory usage.
The fact that the memory usage measure evolves renders comparisons between LLMs harder. While we could theoretically offset the memory usage when we detect changes over time, it would require testing the canonical solutions alongside the generated one on Leetcode.

Leetcode rank.
The time ranking that Leetcode returns when we test a solution represents the share of submitted and valid solutions that are slower than ours. 
While this could be a great tool to rank llm among human-submitted solutions, we find that the ranking is heavily affected by our submissions and time. 
As you can see in <ref>, the overall rank of the llm we tested decreases over time. 
To verify this, we tested GitHub Copilot twice, once as the first llm, and a second time after testing all the llm.
The first test of Copilot has an overall rank of 77, and the second test ranks down to 54, despite it being tested with the same solutions.
This effect of the rank evolving becomes obvious when you consider that the rank is determined using all the previously accepted solutions, including ours.
This means that by testing thousands of our solutions on Leetcode, we are actively changing the ranks of future tests.



RQ1: The evaluation of llm using Leetcode questions as a dataset presents some challenges.
Although Leetcode's questions could serve as a valuable dataset akin to HumanEval, limitations arise due to the constraint that only the problems published after the llm's training dataset formation are usable for evaluating the llm in question.
This creates potential difficulties in reproducibility, particularly as new llm emerge, especially if they do not exclude Leetcode problems from their training datasets <cit.>. 
Additionally, while Leetcode's provided metrics, such as run time, memory usage, and rank may offer practicality in various scenarios, their usability and reliability are questioned when compared to more traditional measurement methods. 
The presence of these challenges emphasizes the need for careful consideration and scrutiny when adopting Leetcode to evaluate llm.








 §.§ RQ2: Are there notable differences in performances between llm?


The pairwise comparison depicted in <ref> reveals subtle distinctions in the performances of various llm.
Notably, some models, such as StarCoder and the CodeLlama model with 13B parameters specialized in Python, consistently exhibit slightly superior results compared to others. 
Despite these observed variations, the mean Cohen's d effect size measures a mere 0.024, a statistically insignificant magnitude. 
This suggests that the practical impact of these differences on the mean speed of code generation is remarkably small.
For instance, when comparing CodeLLama-13-instruct and CodeGen25-7B-mono, CodeLLama outperforms the latter in a statistically significant manner in 3 problems out of 8. However, it is crucial to note that the mean performance difference between these models is a mere 0.02 standard deviation.
It thus seems that improving an llm in terms of functional validity does not significantly impact the performance of the code it generates. 
This may be due to different factors, such as the fact that most llm share the same datasets or that they are trained to produce valid code and not fast code.
Improving the performance of an llm could be done by curating a training dataset of only efficient code and fine-tuning one of the foundational models we used, or by using reinforcement learning to "teach" the model to produce better code. 
Madaan produced an llm that could improve the performance of code <cit.>. 
This llm could be leveraged in a generation pipeline to directly improve the generated code. 



RQ2: Our analysis uncovers statistical differences in the performance of generated code among different llm.
However, the effect size, as measured by Cohen's d, is so negligible that it raises questions about the practical significance of these differences. 
Despite some models consistently outperforming others, the overall impact on the mean efficiency of llm-generated code appears to be minimal.










 §.§ RQ3: Is there an effect of the functional validity of the llm and its temperature on the generated code's performance?


Functional validity. 
When calculating for every problem the correlation between the success rate of the llm that generated the solution and the run time of the solution, we find that there is only a very slight negative correlation (-0.08) between the success rate and the performance.
There is close to no correlation (-0.11) observed between the success rate of the model and the variation in the performance of the generated code.

Temperature. 
There is no correlation (0.05) observed between the temperature of the generations and the performance of the generated code. 
This means that the temperature does not affect how fast the solutions are.
However, we observe that temperature is moderately correlated (0.41) with higher variations in performances. 
This means that higher temperatures tend to increase the variation in performance across generations. 
The complete distribution of the correlation for each of the 24 problems can be seen in <ref>. 
So, while increasing the temperature leads to a lower success rate <cit.>, it can help find a faster solution with an extended exploration of generations.
The fact that the temperature increases the variation in performances comforts the idea that higher temperatures lead to more diverse outcomes.



RQ3: Our analysis of llm indicates that the quality of generated code does not have a substantial impact on its performance. 
However, we observed that modifying temperature settings within an llm significantly affects the diversity of code performances produced. 
This implies that, while code validity may not be a decisive factor in performance, adjusting temperature settings can be a valuable strategy to enhance the variety of outcomes in code generation processes.








 §.§ RQ4: How fast are LLMs compared to humans ?

As previously stated, the Leetcode time ranking evolves, so we chose to compare the second model we tested on Leetcode with humans (Copilot being the first, it did not have enough generations overall because of its lack of a temperature setting). 
The results of this comparison are depicted in <ref>.
The comparison is done using the Leetcode ranking, with the assumption that most of the previous submissions were made by humans.

We observe in <ref> that the solutions generated from llm are faster than most previous submissions with a mean rank of 73%, and that it even generated some solutions that were faster than 95% of the previous submissions.



RQ4: It seems that the llm are faster than most of the human solutions on Leetcode, on average. 
If the llm we tested were in an actual competition, his valid solutions would be on average faster than 73% of the other solutions on Leetcode.





§ DISCUSSION

In this section, we discuss the results reported in the previous section, their implications, and the limitations of our study.



 §.§ Discussion of the results

On the Leetcode measures and usability.
The data contamination issue we unveiled poses a significant challenge in the evaluation of llm, as it prevents an accurate evaluation of their real performances. 
Because Leetcode's problems are not filtered from the training datasets, as research on llm continues, even the newer problems might contaminate future training datasets, thus rendering reproduction of our study harder <cit.>. 
This conclusion also holds for any study using Leetcode as an evaluation dataset, such as <cit.>. 
However, we believe that the methodologies employed and the conclusions drawn would likely hold validity with alternative sets of questions from Leetcode or other performance-oriented datasets. 
This suggests that future studies seeking to replicate our findings would primarily need to change the dataset employed for assessing llm.
Addressing the data contamination concern could involve leveraging pre-filtered evaluation datasets, such as HumanEval, already separated from the training processes, and repurposing them into performance evaluation datasets.




On functional correctness.
The ranking of the functional correctness of the llm is consistent with the previous evaluations of the llm on HumanEval <cit.>. 
However, it seems that InCoder performs worse than presented in its introductory paper <cit.>, which may be due to our experimental protocol—using it only for left-to-right generation instead of infilling like it was built for.  

We observe that Leetcode's problems seem harder for the llm to solve than HumanEval's problems. 
Indeed, StarCoder, the model that performed the best with a pass@1 of 0.09, had a pass@1 of 0.408 on HumanEval <cit.>. 
We believe this is due to multiple factors. 
First, our Leetcode prompts and expected solutions are longer than in HumanEval (the average length of solution in HumanEval is 180 characters vs 425 characters in our dataset), thus increasing the chance of a generation to fail. 
Indeed, having to generate more code can lead to a higher chance of making a mistake, as shown by a correlation of -0.30 between the solution's length and the success rate of the problem.
Second, Leetcode problems come from programming competitions and need a lot of thinking to be solved. 
The causal generation of the llm does not allow a "thinking" process to happen, which for harder problems causes a drop in functional validity. 
This could be solved by making the llm mimic human thinking with methods, such as Chain-of-Thought prompting <cit.>.


On the performance of generated code. 
To the best of our knowledge, our methodology for evaluating llm based on the performance of generated code is novel and could serve as a benchmarking approach for future studies involving new llm and datasets.
While we aimed for a singular performance score for llm, similar to pass@k, the varying rates at which llm generate valid solutions posed a challenge, leading us to employ pairwise comparisons.
An improvement to our method could involve using an optimal solution as a reference point and comparing the speed of each generated solution to the optimal solution's speed. 
Speed would thus be expressed as a factor relative to the optimal time, addressing the issue of problems having different time scales and potentially laying the groundwork for a more comprehensive "performance score."

The low success rate of llm on Leetcode posed a considerable challenge for their comparison. 
Among the 204 problems, only 24 had (i) valid solutions from at least 10 different models (out of the 18 assessed llm) and (ii) at least 10 valid solutions in total (see to the companion notebook). 
This low success rate complicated pairwise comparisons, as numerous models could not be compared due to an insufficient number of common problems with a substantial number of generated solutions.

Our discoveries offer valuable insights for developers in their choice of llm. 
For example, when developers are considering an LLM for tasks like code generation, such as with GitHub Copilot, the performance of the generated code may be an important concern.
With our findings, developers can be assured that there is no significant variance in the performance of code generated by different llm.
This means that if they aim to have fast code, 
there's no necessity to consistently opt for the largest model available; instead, a smaller one suffices.
We also hope that our findings will incentivize further research on building new llm that produce even more efficient code.





 §.§ Limits and Threats to Validity

Regarding functional correctness, although we did our best to generate code in the most optimal conditions, some changes to the input prompt (adding examples and constraints), or the configuration of the models may have changed the performance of the llm.
However, we believe that this should not significantly impact the validity of our experiment, as all the models were configured similarly.

GitHub Copilot being a closed-source tool, it might be retrained without the community's knowledge, which could potentially lead to a modification in the tool's performance in upcoming experiments.

The majority of our validation and benchmarking process relied on Leetcode's test suite and online judge system. 
Thus, it is possible that some big test cases we extracted for the benchmarking favored some types of implementations over others. 
We mitigated this issue by systematically fetching three test cases and by having a large dataset of problems to generate from.
During our experiments, we also tried to generate plausible inputs using random generators, but this method did not yield satisfying results because randomly generating data structures with specific shapes, or properties (generating valid regular expressions) is difficult to achieve. 

One gap in our study is that we do not consider memory usage at all when studying the llm. 
This is because our benchmarking setup did not allow for memory monitoring and doing otherwise would have cost us a lot of time. 
Moreover, we believe our results obtained with only the run time to be self-sufficient.

The way we compared an llm to a human using Leetcode rankings gives a good idea of where the llm stands in terms of performance. 
However, because the ranking evolves and we have no information about the population the llm is ranked against, the results here should be taken with a grain of salt.

It is also important to note that Leetcode as an evaluation dataset suffers from some issues. 
As we only evaluate the llm on algorithmic problems, the performances of the llm are hard to generalize across all programming fields. 
However, this is difficult to improve on for similar reasons to HumanEval: we only have a limited context size and have to make the llm generate in one go a completion to some code, which must be self-contained (meaning, all the information needed to generate the solution must be in the prompt). 
Also, in terms of performance, it is difficult to find another kind of self-contained code than algorithmic problems that have, such variations in performance. 
While studying the llm on SQL generation could be a good idea, it would not fit into our studies with generalist programming languages.

Regarding Leetcode as a platform, we are heavily dependent on it for validating the generated solutions and are limited by its daily rate limits of 1,000 submissions per account. 
For future studies, it would be great to consider alternatives to Leetcode, such as the project CodeNet <cit.>, which does not have as many restrictions as Leetcode.



§ RELATED WORKS

Previous research has investigated various aspects of llm for code-related tasks, including the security of their suggestions <cit.>, the prevalence of bugs in the generated code <cit.>, how developers interact with them  <cit.> or just the quality and correctness of the code they generate <cit.>. 
There have also been efforts to measure the efficiency of llm through the creation of benchmarks for comparing them, such as HumanEval<cit.>, MBPP <cit.>,  CoderEval <cit.>, APPS <cit.>, CodeXGLUE <cit.> or ReCode <cit.>. 
Xu  <cit.> also led a comparative evaluation of multiple llm for code including Codex and Codeparrot.

Leetcode, while being just a coding competition platform, is also used as a dataset to evaluate the capabilities of llm on programming tasks. 
Döderlein  <cit.> measured the performances of Copilot and Codex on Leetcode and the effects of changing the prompts. 
Nguyen and Nadi <cit.> studied Github Copilot's code suggestions on Leetcode problems and the complexities of its generated code. 
Vasconcelos  <cit.> studied the effects of highlighting the uncertainty of AI-powered code completions using Leetcode problems and Codex.

Various other methods have also been employed to investigate the impact of temperature on the generated code, apart from the approach we proposed: Chen  <cit.> evaluated the best temperature of Codex in terms of pass@k. 
Austin  <cit.> also studied the effects of the temperature on the performance of their model.
Christopoulou  <cit.> also studied the effects of the temperature and nucleus-sampling on their llm.
Döderlein  <cit.> highlight the importance of correctly tuning the temperature of a model when using it to generate code. 
The research led by Aghakhani  <cit.> shows that poisoned models suggest insecure code more often as the temperature increases.
Our results demonstrate that increasing the temperature also increases the chance of generating slow and inefficient code.

On the subject of performance, Madaan  <cit.> fine-tuned llm to make them improve the performance of code.
Multiple other techniques have been proposed for automatically improving the performance of code using llm <cit.>. 
Our contribution is the first—as far as we know—to investigate the differences in the performance of llm-generated code. 
Future llm that would be adapted with these performance-improving techniques could also be compared using our methodology.
Regarding recitation, few works have been done on this subject, but Yan  <cit.> proposed a method to detect cases of recitations in llm using inference fingerprinting. 
Jacovi  <cit.> explained why data contamination in llm was problematic and proposed methods to mitigate it.

To summarize, our paper evaluates the performance of code generated by various llm and investigates differences in the performance of the generated code across models on Leetcode problems, which, to the best of our knowledge, has not been done previously.



§ CONCLUSION

In this study, we presented a comprehensive analysis of the performance of code generated by various llm using a novel methodology that measures and compares the runtime speed of solutions to algorithmic problems. 
Our findings suggest that the performance of the generated code is largely similar across different models, regardless of their size or training data. 
Furthermore, increasing the temperature parameter during code generation leads to a greater variance in performance, though not necessarily to better or worse solutions on average.
We also critically evaluated the suitability of Leetcode as a dataset and benchmark platform for assessing llm. 
The results indicate that, while Leetcode's problems are suitable for performance evaluation, their measures should be used cautiously due to issues with stability and reliability. 
Additionally, we observed that the use of newer Leetcode problems is essential to avoid data contamination and ensure the validity of llm evaluations.


This work opens up several avenues for future research, including the development of performance-oriented training datasets and the fine-tuning of llm for performance improvement. 
As the field of AI-assisted programming continues to evolve, studies such as ours will play a critical role in understanding and enhancing the capabilities of llm. 






This work received support from the French government through the Agence Nationale de la Recherche (ANR) under the France 2030 program, including partial funding from the CARECloud (ANR-23-PECL-0003), DISTILLER (ANR-21-CE25-0022), and KOALA (ANR-19-CE25-0003-01) projects.
Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations.[See <https://www.grid5000.fr>]


ACM-Reference-Format





 Paper Title: Initial Insights on MLOps: Perception and Adoption by Practitioners.









 
 

 
 
 
 
 
 
 

 


  University of Oulu
  Oulu
  Finland

sergio.moreschini@oulu.fi


     Tampere University, Tampere, Finland, sergio.moreschini@tuni.fi 
     
    


  Tampere University
  Tampere
  Finland


david.hastbacka@tuni.fi




  Free University of Bozen-Bolzano
  Bozen-Bolzano
  Italy

andrea.janes@unibz.it




  University of Oulu
  Oulu
  Finland


valentina.lenarduzzi@oulu.fi



  University of Oulu
  Oulu
  Finland


davide.taibi@oulu.fi




















§ ABSTRACT 
The accelerated adoption of AI-based software demands precise development guidelines to guarantee reliability, scalability, and ethical compliance. MLOps (Machine Learning and Operations) guidelines have emerged as the principal reference in this field, paving the way for the development of high-level automated tools and applications. Despite the introduction of MLOps guidelines, there is still a degree of skepticism surrounding their implementation, with a gradual adoption rate across many companies. In certain instances, a lack of awareness about MLOps has resulted in organizations adopting similar approaches unintentionally, frequently without a comprehensive understanding of the associated best practices and principles. 
The objective of this study is to gain insight into the actual adoption of MLOps (or comparable, e.g. AIOps) guidelines in different business contexts. To this end, we surveyed practitioners representing a range of business environments to understand how MLOps is adopted and perceived in their companies. The results of this survey also shed light on other pertinent aspects related to the advantages and challenges of these guidelines, the learning curve associated with them, and the future trends that can be derived from this information.
This study aims to provide deeper insight into MLOps and its impact on the next phase of innovation in machine learning. By doing so, we aim to lay the foundation for more efficient, reliable, and creative AI applications in the future.





Initial Insights on MLOps: Perception and Adoption by Practitioners
    Davide Taibi
    August 5, 2024
===================================================================





§ INTRODUCTION


The past decade has witnessed an AI revolution, with significant shifts in how industries approach data utilization. Two domains stand out as key catalysts for this transformation:  generative AI (GenAI) and machine learning operations (MLOps) <cit.>. While both domains offer unprecedented opportunities for revolutionizing industries, their roles and applications differ significantly. 

GenAI represents a category of AI that is capable of generating new content <cit.>. Some applications, such as AI assistants, are gaining interest for their ability to assist developers by automating routine coding tasks, increasing productivity, and reducing technical debt <cit.>. GenAI is positioned as a valuable tool for the software development lifecycle, particularly due to its ability to provide real-time assistance and generate high-quality code snippets. These features underscore its potential to drive transformative change across a range of technology and industry sectors. By offering rapid assistance and streamlined coding procedures, GenAI becomes an invaluable partner in software development, particularly when aligned with MLOps guidelines.

MLOps provides a robust framework for managing the ML lifecycle, addressing key challenges such as scalability, reliability, and collaboration between data scientists and operations teams. Understanding these aspects enables more efficient and effective implementation of ML projects, ultimately accelerating innovation and providing a competitive edge to organizations, paving the way for more efficient, reliable, and creative AI applications in the future. 

MLOps combines machine learning and DevOps to streamline the machine learning lifecycle, from initial development through to deployment and maintenance. The ultimate goal of MLOps is to create scalable and reliable ML systems that can be integrated seamlessly into existing workflows. By automating repetitive tasks and ensuring robust monitoring and management of models in production, MLOps promises to enhance the efficiency and effectiveness of ML projects <cit.>.

Despite the growing interest and investment in these technologies, there remains a significant gap in understanding how MLOps is perceived and implemented in the real world, particularly from an industrial perspective. There is currently a high level of skepticism about MLOps guidelines and the importance of using them when developing ML-based software. Therefore, the objective of this work is to deepen the understanding of MLOps and its role in driving the next wave of machine learning innovation.
To achieve this, we aim to address the aforementioned gap in knowledge and demonstrate the importance of such guidelines by capturing the experiences and insights of practitioners from diverse organizational contexts. This will highlight both the practical challenges and benefits of adopting MLOps. 

We designed and conducted a qualitative survey among three MLOps experts investigating the perspectives of developers and leaders in large enterprises and SMEs, uncovering the implications of MLOps adoption. It addresses key issues such as tool costs, data drift, and the daunting learning curve associated with MLOps, as well as the potential for advanced tools powered by Generative AI to address these challenges. 

We can summarize our main findings as follows:


	
  * Some developers are unknowingly following MLOps guidelines.
	
  * The high cost of tools poses a barrier to experimenting with MLOps.
	
  * The daunting learning curve for MLOps practices is primarily due to mindset challenges.
	
  * Generative AI is on the verge of becoming a valuable support tool for developers.



The rest of the work is organized as follows: Section <ref> presents the design of the empirical study. It includes multiple subsections, beginning with the introduction of the goal and research questions (RQs). We will then explain how the survey has been designed and how it is structured. Finally, we will describe how the study was performed and analyzed, and how it can be replicated.
Section <ref> reports the perspective of the interviewees on each of the different aspect questioned.
Section <ref> presents the results and discuss them based on each RQ.
Section <ref> provides other relevant works within the same topic.
Section <ref> draws the relevant conclusions.  


§ EMPIRICAL STUDY


In this Section, we describe our empirical study designed as a survey according to the Wohlin et al. guideline <cit.>.



 §.§ Goal and Research Questions


The goal is to raise awareness about the current use of MLOps in organizations. By comparing the different approaches used by different stakeholders, we aim to understand not only the current state of MLOps adoption but also the associated learning curve and stakeholders' vision of future applications and opportunities.

Therefore, we derived the following Research Questions (RQ). 


RQ_1. To what extent do developers understand and embrace the concept of MLOps?


RQ_1 is designed to assess developers' familiarity with MLOps principles, their implementation practices, and the extent to which they have integrated these practices into their workflow. In particular, it seeks to determine whether developers are actively adopting MLOps practices or if these practices are being incorporated indirectly as a result of applying machine learning (ML) within DevOps frameworks.


RQ_2. What improvements could enhance MLOps by leveraging existing benefits and solving existing issues for pipelines and tools?


In RQ_2 we aim to identify the key improvements needed to enhance MLOps practices by leveraging current benefits and addressing existing issues in pipeline development and tools. Additionally, it seeks to uncover the factors that are impeding the adoption and awareness of MLOps among practitioners.


RQ_3. What is the perception of the MLOps learning curve in the industry?



RQ_3 aims to gain insight into how practitioners perceive the difficulty of adopting MLOps and to assess how hard is to learn the correct practicess. By exploring practitioners’ experiences and perspectives, we can identify the specific challenges that contribute to the perceived difficulty and how these challenges impact the adoption process.


RQ_4. What are the anticipated future trends in MLOps?


Lastly, in RQ_4, we aim to elicit insights from participants on the tools and practices that are likely to gain prominence and the directions in which the field is expected to evolve. We are interested in understanding their views on the development of advanced tools with a comprehensive understanding of the entire MLOps pipeline.











 §.§ Study Design

To answer our research questions, we designed and conducted a qualitative survey consisting of the five aspects described below. 


    
  * MLOps adoption (RQ_1). The first part of the survey is related to the current adoption of MLOps. As a starting point, we aim to understand the user's experience in the context of MLOps and its position on the classical MLOps model in literature (such as the one from Google <cit.>). In addition, it is important to understand what are the main reasons and applications for the adoption of MLOps within the engineering team, as well as the number of people that make up such a team.


  * MLOps architecture (RQ_1). The second part focuses on tools and phases. Specifically, we aim to understand which phases constitute the pipeline of the practitioner, compared to the well-known DevOps infinite loop representation. We also aim to determine the importance given to each phase in terms of priority and automation. The tools are subject to the same types of questions, not only in terms of related phases but also regarding timing, satisfaction (scale 1-5), and potential (scale 1-5). Questions about known but unused tools and related feedback loops conclude this section. 


  * MLOps issues and benefits (RQ_2). The third section is a direct consequence of the second, as at first issues and benefits of different tools are compiled following questions from the previous section. The other questions composing this section are focused on the whole MLOps pipeline and include benefits, issues, and possible improvements.


  * Comparison of MLOps pipelines (RQ_3). The purpose of this section is to compare the current state of MLOps adoption by practitioners with our vision of an ideal MLOps pipeline. After presenting our ideal pipeline, we asked for suggestions and ideas on how to improve it, as well as opinions on the division of roles between software developers and ML developers. We also asked about any difficulties in learning how to apply the proposed pipeline.


  * Future trends (RQ_4). The last section aims to understand the practitioners' vision of possible future trends concerning applications or tools. 


In the following text, we describe the participants selection and the questions we formulated to them.



 §.§ Questionnaire

The five sections outlined incorporate a combination of twentyfour questions designed as open-ended and Likert scale items with a range of 1-5 (where 1 indicates a complete lack of likeability and 5 indicates complete likeability).



  §.§.§ Participants Selection

Three participants were selected for this study based on several criteria, including their diverse professional backgrounds. This was done to ensure a comprehensive perspective on the subject matter. 
The purpose of this selection of participants is to illustrate the various challenges and methodologies encountered in diverse organizational contexts and technical domains. MLOps and AI researchers and developers working in large enterprises provide insights into the complexities of integrating systems within vast, data-rich environments. In contrast, software developers at small to medium-sized enterprises (SMEs) offer a view of the agile and innovative techniques required for developing ML/AI algorithms within a constrained and rapidly evolving setting.

The contrast between these backgrounds is crucial to understanding the broader landscape of software engineering practices and how they can best be used in a range of different environments. By examining these different viewpoints, this study will be able to provide valuable insights and highlight potential best practices that can be used in a wide variety of organizations and for different types of technical work.

The first participant is an experienced researcher and developer employed by a large enterprise. This participant has a wealth of experience from working within a larger organizational structure. This participant’s expertise lies specifically in recommender systems. This is an area that involves sophisticated algorithms and user data interactions to personalize and enhance user experiences.

The second participant is a software developer working in an SME. Given the resource constraints and dynamic nature of smaller organizations, a more versatile and adaptive approach to software development is often required in this setting. The participant's specialization is in algorithm development for ML and AI applications, focusing on the creation and optimization of algorithms that underpin intelligent systems.

The third participant is the research and development department vice president of an SME. This role encompasses a broad range of responsibilities, including involvement in multiple projects as a project manager and hands-on development contributor due to the SME’s collaborative and dynamic nature. The participant’s extensive experience in overseeing and practically contributing to the development of various projects provides a unique perspective on balancing strategic oversight with practical implementation in a resource-constrained environment.




  §.§.§ MLOps adoption (RQ_1)
 We formulated 6 questions to assess the participant’s familiarity with the MLOps concept and to determine whether the participant consciously and intentionally implements MLOps practices or if these practices have been adopted inadvertently, without explicit recognition of the MLOps framework.
It is crucial to understand this distinction, as it helps differentiate between the deliberate, informed adoption of MLOps methodologies and incidental usage that may occur due to overlapping practices in machine learning and software engineering. By grasping this differentiation, we can gain valuable insights into the level of awareness and intentionality behind MLOps adoption, which is essential for assessing the effectiveness and integration of these practices within different organizational contexts.

The subsequent questions were selected based on the participant's initial response. If the participant demonstrated familiarity with and intentional use of MLOps practices, the following questions incorporated specific MLOps terminology. Conversely, if the participant does not have knowledge on the  terminology related to MLOps but used such practices without referring them as MLOps, the interviewer continued the interview by using the "DevOps for ML" terminology. This document will focus on the terminology used in the first case.



This approach guarantees that the language used in the survey is consistent with the participant’s level of expertise and familiarity, thus enhancing the clarity and relevance of the questions. It permits a more tailored investigation of the participant’s practices and experiences, whether explicitly framed within the MLOps paradigm or understood through the more general context of DevOps applied to ML. This differentiation is essential for accurately capturing the participant’s perspectives and the specific practices they employ in their work.
In light of the participant’s demonstrated knowledge of MLOps, the subsequent question was tailored to further explore their understanding of MLOps standards and practices. Conversely, for those less familiar with MLOps, the focus was on understanding if any machine learning (ML) approach was used. If this question yielded a negative response, the interview would conclude, resulting in an unsuccessful outcome.



  §.§.§ MLOps architecture (RQ_1)
 we formulated 8 questions to ascertain knowledge and evaluate perceptions. This method aims to first ascertain participants’ understanding of specific aspects of MLOps architecture and then assess their perceptions of the importance of these aspects.

For instance, the initial question in this section focused on identifying the various phases within the MLOps lifecycle. Following this, the  participants were asked to evaluate the significance of each identified phase. This two-step approach ensures a comprehensive exploration of both the participant’s knowledge and their evaluative perspective on key components of MLOps architecture.

By structuring the questions in this manner, the survey can effectively capture detailed insights into how participants perceive and prioritize different phases of MLOps, providing valuable data for understanding the adoption and implementation of MLOps practices across different organizational contexts.

In particular (question 3.3), we asked the participant to describe the tools used in each phase of their MLOps processes, as well as to evaluate these tools in terms of their satisfaction and perceived potential. By gathering detailed information about the specific tools employed and the context of their use within the MLOps lifecycle, this question aimed to obtain valuable insights into the tools being utilized in the MLOps process. Moreover, Question 3.3 served as a direct link to the subsequent section. For each tool mentioned, participants were also asked to detail the benefits and issues associated with it in Question 4.4. This connection between the questions ensures a comprehensive understanding of the tools’ practical applications and effectiveness, as well as the challenges encountered during their implementation.

By correlating the descriptions of tools with their evaluated benefits and issues, the survey aims to provide a comprehensive view of the current tool landscape in MLOps. This approach allows for a more in-depth analysis of how different tools contribute to the MLOps workflow and potential areas for improvement in tool development and integration.



  §.§.§ MLOps issues and benefits (RQ_2)
 We formulated 4 questions to gain insights into potential areas for improvement within the MLOps processes and tools. We intentionally linked the final question directly to the previous section. Consequently, some participants may have already addressed it. 


Specifically, (question 4.3) we requested that participants consider how their current MLOps practices could be improved. This methodology enabled the survey to obtain a comprehensive range of perspectives on existing challenges and potential areas for enhancement. By encouraging participants to identify and discuss both the advantages and the issues they encountered, the survey aimed to provide a balanced and in-depth understanding of the current MLOps landscape. The feedback gathered in this section is both actionable and relevant, and it highlights specific areas where MLOps practices can be refined to better meet the needs of practitioners. 



  §.§.§ Comparison of MLOps pipelines (RQ_3)
 we designed  4 questions, each related to a figure illustrating an MLOps infinite loop pipeline. The Comparison of MLOps Pipelines section aimed to directly compare the interviewee’s conception of MLOps with the pipeline model proposed in our previous work <cit.>. 

The questions prompted participants to assess and compare their understanding and application of MLOps with the proposed pipeline model. By doing so, the survey aimed to identify similarities and differences in MLOps practices, offering valuable insights into the diversity of approaches within the field.

The open-ended nature of these questions enabled participants to provide detailed feedback on how their MLOps practices align with or diverge from the proposed model. This approach facilitated a deeper understanding of the practical application of MLOps and highlighted areas where our proposed pipeline might be improved or adapted to better fit real-world scenarios.




  §.§.§ Future trends (RQ_4)
 In The final section,  we asked the participants to answer to 2 questions, each intended to prompt informed speculation about the anticipated evolution of MLOps tools and areas of focus. During the survey, we encouraged participants to think forward and share their visions of how MLOps will evolve. The insights gained from these discussions offer invaluable foresight into the potential trajectory of the industry.

The initial question sought to identify which tools, or categories of tools, the participants believe will become more prominent or widely adopted in the future. This question aimed to gain insights into emerging technology and innovations that participants see as having significant potential to influence the future of MLOps practices. The second question was designed to identify the specific areas or applications within MLOps that participants anticipate will receive greater focus and development. By understanding which aspects of MLOps participants believe will become priorities, this question provides valuable insight into future trends and areas of growth within the field.




 §.§ Survey Execution

We collected the information through open-ended questions, running the survey as a face-to-face interview. Two authors collected the answers separately and then checked possible inconsistencies in the report. The disagreements were discussed and clarified with the other authors. We informed the participants, according to the GDPR[https://gdpr-info.eu], about their rights and that they could abandon the study anytime. Moreover, all information provided by each participant has been treated as confidential, without disclosing any sensible data.



 §.§ Data Analysis

The qualitative data analysis has been conducted individually by each author.
Moreover, pairwise inter-rater reliability was measured across the three sets of decisions to get a fair/good agreement on the first iteration of this process. 
Based on the disagreements, we clarified possible discrepancies and different classifications. A second iteration resulted in 100% agreement.



 §.§ Replicability and Verifiability

To allow the verifiability and the replicability of our study, we publish all the raw data including the transcript of each interview in the online appendix[<dx.doi.org/10.6084/m9.figshare.26408197>]. For privacy reasons, we anonymized the names of the companies. 




§ RESULTS


In this section, we report the obtained results grouped by the three experts interviewed. The answers will be summarized for each aspect of the questionnaire





 §.§ First Interview


The first interview took place on February 20th, 2024, and lasted about 45 minutes. The interviewee, an ML engineer, focuses on full-stack research and development at his company (Large Enterprise). His main field of expertise is Recommender Systems (RSs) with more than 5 years of experience.



  §.§.§ MLOps adoption
 The first interviewee claimed extensive experience with MLOps frameworks, especially in the areas of deployment and debugging. The user-developed frameworks have been used in the area of RSs (specifically for TV channels and hotel recommendations)  to improve the user experience. These frameworks do not adhere to a specific standard but are tailored to the requirements of each project. His participation in various projects across different fields has resulted in fluctuations in team sizes, depending on the specific requirements of each application. However, the size of such teams has always been fewer than five people.



  §.§.§ MLOps architecture
 When asked about the different phases composing the MLOps architecture, the interviewee identified 4 main phases of equal importance: 
    
        
  * Prototyping
        
  * Offline Training
        
  * Online Model
        
  * Monitoring
    
    Such phases rely on different tools which have been summarized in Table <ref>. From the table it can be seen that no tool is capable of covering all the different phases and a combination of these is necessary to achieve full MLOps. The most important of these tools are those used in the monitoring stage (as they are linked to the various ongoing notebooks to create the feedback loop), MLFlow, and Databricks. However, even though Databricks has the potential to be the most satisfying, it has been replaced with a combination of Airflow and a monitoring system for cost savings. 
    The interviewee also stated that Phases 2 and 3 are most important to automate first and that the main reason for retraining can be identified in Data Drift.

[3]<https://aws.amazon.com/sagemaker/>
[4]<https://aws.amazon.com/emr/>
[5]<https://airflow.apache.org>
[6]<https://www.databricks.com>
[7]<https://mlflow.org>
[8]<https://www.elastic.co/elasticsearch>
[9]<https://prestodb.io>
[10]<https://www.datadoghq.com>  
[11]<https://aws.amazon.com/cloudwatch/>  
[12]<https://www.honeycomb.io>  



  §.§.§ MLOps issues and benefits
 When starting the discussion related to the main issues in MLOps the interviewee stated: 




        “MLOps is used to prevent issues”.
     



It is however its nature of continuous improvement and the continuous addition of features and functionalities that create the most issues especially when creating new APIs. Another valuable point is also the difficulty of performing multiple ML iterations without increasing the technical debt. Despite these issues, however, the main benefits of MLOps are evident especially when analyzing the high quality of the ML model and the provided User Experience. More importantly, these systems are far from being perfect and they necessitate high effort, especially for the Offline phase. Such a phase requires high synchronization between the different teams within a company working on different projects. The main issues and benefits of tools are also reported in Table <ref>.



  §.§.§ Comparison of MLOps pipelines
 When presented with our vision the interviewee noticed similarities in every phase except for the deployment phase. He suggested specifying the different levels kind of possible deployment, namely Local, Developer Environment, Online and Smoke Test. Moreover, when specifically asked about the binomial between SE and ML engineer he stated that it is an Utopian representation, as such binomial would mostly create "translation errors" due to the missing knowledge that one of the figures might have on the project. Regarding the adoption of such models, he believes that the most challenging part is related to the mindset of the Non-ML practitioners which need to see the benefits of applying MLOps, while on the practical side, the main challenge is the creation of the whole infrastructure that would allow a full automation.
    


  §.§.§ Future trends
 When discussing future trends, the interviewee foresees an increased effort in creating LLMs for code writing support, also known as Code Copilot. The interviewee expressed excitement for the future development of a similar application applied to unit tests. For what concerns the areas that will gain the most attention he identified Healthcare and Explainable AI.







 §.§ Second Interview

The second interview took place on March 14th, 2024, and lasted about 1 hour. The interviewee has a completely different background than the first one, he is a software developer who is dedicated to the development of algorithms for the application of ML and AI within the company he works for (Small/Medium Enterprise).



  §.§.§ MLOps adoption
 The second interviewee, when asked about MLOps, claimed to have heard of such practices.
    To ease the interview process, the interviewer provided an MLOps definition <cit.>. The interviewee was unaware of any specific MLOps standards and had never used them in a project. On the contrary, their models are too heavy to be part of the final product, so their deployed code points to a file repository. This allows them to deploy the product code and the associated model separately, thereby reducing the dependency of one on the other. Specifically, when a new model is created, the developed application only needs to be pointed to the new model address. Once the model is loaded, the startup process of the application begins. 

    In this case, the division between the model (ML side) and the code (SE side) is more evident, reducing also the level of automation. Interviewees identified this division as the primary reason for MLOps, further specifying that most model development, testing, and validation are done offline without any level of automation, and automation is only initiated in the monitoring phase after deployment. Explicitly: 
    
    

    
        “Implementing an MLOps architecture” – even if the model development is not in a pipeline – means to be able to deploy code and model separately.
    
    

    
    The developed solutions have been used in speech-to-text applications and the team is composed of less than 10 people (including administrative and management).

[13]<https://elixir-lang.org>
[14]<https://neovim.io>
[15]<https://huggingface.co>
[16]<https://aws.amazon.com/pm/ec2/>
[17]<https://aws.amazon.com/s3/>
    


  §.§.§ MLOps architecture
 When asked about the different phases composing the architecture the interviewee referred to the previously provided definition of MLOps <cit.>, specifically to its maturity level. The identified maturity level was the first as most of the ML-related tasks are performed manually but the upcoming goal is to move to level 2 by obtaining human-corrected text to perform continuous training and deployment. With the first level of maturity, the interviewee identified as the most critical phase of the testing in production. Explicitly: 
    
        

        
        “Understand if the latest deployed model is not causing regression”.
    
        

        
    Also, in this case, the architectural approach relies on the utilization of several tools, as summarised in Table <ref>. Nevertheless, the satisfaction and potential of these tools have not been evaluated by this participant.  In this level of maturity, to reach the next level, the interviewee identified the deployment phase as the most important supported by automated testing and continuous training.

    When discussing the feedback mechanisms and the consequent reasons for performing retraining, the interviewee identified two feedback events, the first defined as “catastrophic events" caused by the client identifying an issue with the application and the model quality. Specifically, the second reason for the model degradation is identified in data drift.



  §.§.§ MLOps issues and benefits
 The main issue identified by the second interviewee is the cost related to performing MLOps. Specifically, those related to tool usage (Amazon EC2 has been abandoned due to idle costs) and those related to retraining. In this interview the importance of performing correct retraining has been highlighted: i.e. retraining optimization to avoid useless costs but also to improve the model when necessary. The main benefit of the approach used has been identified in the quality of the product provided which consequently increases customer satisfaction.

In this context, the interviewer asked what could be improved, and the interviewee expressed a high level of satisfaction with the proposed solution, which currently does not need to be updated. However, the future perspective of this solution is that if there is a new and more accentuated data drift, it may be necessary to restructure the approach, including automation, to increase the level of maturity.






  §.§.§ Comparison of MLOps pipelines
 Again, we presented our vision, and the first reaction was triggered by the concept of parallelizing. For the interviewee, our vision required two people working almost synchronously, while in their approach it was important to reduce the dependency between the model and the code. Instead of having two figures working at the same speed, they needed two processes that could be updated independently. He suggested explicitly considering the importance of traceability within the proposed infinite loop to increase explainability. He was very pragmatic about adopting the model, stating that it is unlikely to get adopted because they only implement what is necessary.

When asked about the comparison between SE engineers and ML engineers, he pointed out the absence of another important figure in the picture: the hardware engineer. The interviewee regards the hardware engineer as an expert in GPU and training infrastructure, without whom the other two cannot create a realistic plan.







  §.§.§ Future trends
 When discussing future trends, the second interviewee believes that monitoring tools in the future will be able to monitor the entire MLOps pipeline, including continuous training before deployment. This would enable optimization in the coding phase based on feedback received at each stage, both before and after deployment.
In terms of areas that will receive more attention in the future, the respondent believes that hardware accelerators will become more important based on current development trends.



    
    

    



 §.§ Third Interview

The third interview was conducted on July 17th, 2024, and lasted approximately 20 minutes. The interviewee's background differs from that of the previous two interviewees. He is the vice-director of the research and development department of the company where he is employed (SME and Consultancy). 

[18]<https://www.jenkins.io>
[19]<https://www.sonarsource.com/products/sonarqube/>
[20]<https://azure.microsoft.com>
[21]<https://grafana.com>
[22]<https://nifi.apache.org>
[23]<https://spark.apache.org/docs/latest/ml-guide.html>
[24]<https://hadoop.apache.org>




  §.§.§ MLOps adoption
 In response to questions about his experience with MLOps, the third interviewee asserted that he has been utilizing MLOps practices for approximately two years in three distinct project contexts. His approach to strategy and adoption was consistently tailored to the specific projects he worked on, necessitating a correspondingly adapted pipeline to align with project-specific needs.
Consequently, the primary rationale for implementing MLOps was to facilitate the continuous analysis and monitoring of data generated within the aforementioned projects. These projects were primarily related to administrative matters concerning cities and regions within the domain of smart cities. The primary objective was the development of a recommendation system for users (i.e., citizens) that would facilitate the discovery of events within their respective cities or regions.
The project development team consisted of ten individuals. Of these, five were software developers. Two were focused on IT operations, and three were assigned to specific tasks. One was responsible for the monitoring phase, one was in charge of the planning phase, and one was engaged in data analysis.




  §.§.§ MLOps architecture
 In response to a question regarding the various phases that comprise the architectural framework, the interviewee identified the DevOps infinite loop as a fundamental element. He asserted that every distinctly developed pipeline was based on this loop and encompassed all the requisite phases.
The aforementioned phases were all identified as equal, except for the planning phase, which was accorded a lesser importance due to the straightforward nature of the project requirements. The group of phases composing the Ops was considered the most important to automate first.
Following expectations, the MLOps pipelines developed in this case also relied on the utilization of several tools, as summarized in Table <ref>. As reported, the tools exhibited an overall average performance and met the anticipated standards. Notable exceptions included NiFi, which demonstrated the highest degree of satisfaction, while Jenkins, SparkML, and particularly Grafana, demonstrated potential for improvement. Furthermore, Kafka Stream was originally included in a project but was subsequently excluded due to data distribution issues, with MapReduce being replaced by SparkML. The rationale behind this latter decision was the limited number of phases that MapReduce was able to offer.
In terms of the feedback mechanism, this was entirely contingent on the report of the data analysis. Following this, the report was then provided to both developers and clients. Subsequently, clients would then proceed to provide a second feedback response to the developers, which would then be based on both the original report and the subsequent client feedback.
In their methodology, the data assumes a pivotal role, and given its inherent variability, two potential triggers—data drift and the one-month expiration—have been identified as the basis for retraining.
    


  §.§.§ MLOps issues and benefits
 The discussion on the subject of issues related to MLOps proceeded to address the social implications and general acceptance of this concept, rather than focusing on the practical aspects involved in the utilization and advancement of pipelines. 
In particular, the interviewee highlighted the absence of a consensus surrounding the concept of MLOps as a primary factor deterring investment from stakeholders and companies regarding MLOps. This ultimately leads to a shortage of individuals adequately trained in the concept, which further distances the concept from such realities. To mitigate these issues, the interviewee proposes a focus on the interaction between human and MLOps pipelines, to facilitate the explainability of the pipeline. Nevertheless, the advantages of timeliness and an uncomplicated work schedule are significant when MLOps are utilized. 
    
When the focus is on the tools (as detailed in Table <ref>), the most significant benefit and desired quality of specific tools is the ease of use and configuration. While various types of issues have been identified for different tools, the one that presented the greatest challenges was Azure, which was perceived as both costly and having a "steep learning curve." During the interview, the concept of a "steep learning curve" was introduced as a metaphor that likely reflects a common perception that navigating a steep hill is more challenging than traversing a long, gradual incline <cit.>. To not cause any potential confusion, it will then be reported as daunting.
   


  §.§.§ Comparison of MLOps pipelines
 In this instance, when our vision was presented to the interviewee, it was observed that the proposed MLOps pipeline exhibited certain similarities with their developing pattern. In response to the question regarding potential improvements, he once more emphasized the necessity for procedures that facilitate explainability and identified the ML debug phase as the optimal context for such improvements.
    
The following question pertained to the complexity of transitioning to an infinite loop. In alignment with the preceding discussion, he highlighted resource constraints as a key challenge. He proposed a classification of these resource constraints based on their human and economic dimensions. From a human perspective, there is a notable lack of knowledge, while from an economic standpoint, the cost of tools such as Azure reduces the likelihood of implementing these approaches through experimentation. Consequently, the initial investment is likely to be high before any return is seen.

The final question of this section addressed the potential for collaboration between software engineers and those with expertise in machine learning (ML). The interviewee not only endorsed the dualism but also proposed the addition of a third role. This third role, which would be filled by an individual with a background in theoretical software development, would ensure adherence to best practices and reduce technical debt.
    


  §.§.§ Future trends
 The discussion on prospective trends centered on the significance of LLMs and Generative AI. The interviewee suggested that the application of this new technology will facilitate the coding phase. He anticipates that the tools will enable the automation of mechanical development processes. Furthermore, the introduction of LLM will enhance the debugging process, leading to more transparent code and a reduction in the time required for debugging.





§ DISCUSSION



In this Section, we summarize and discuss the obtained results answering to our RQs.



 §.§ MLOps concept understandability (RQ_1)
 
The interviews detailed in the previous section provide a comprehensive overview of perspectives on MLOps, highlighting distinct viewpoints influenced by organizational size and maturity in adopting MLOps practices. One participant from a large organization and one from an SME explicitly implemented MLOps to enhance automation within their workflows. In contrast, the remaining participants from an SME employed MLOps practices indirectly and at a lower maturity level. Despite these differences, all projects were facilitated through the strategic use of diverse tools tailored to specific goals. 
One notable insight from the interviews was regarding the existence of an "offline" phase, where models are trained and tested before deployment without automation assistance. This practice highlights the critical importance of model reliability, which cannot always be represented by a single objective metric, like accuracy. For example, automated pipelines that use tools like MLflow <cit.> for continuous deployment must consider multiple evaluation criteria that go beyond the use of basic measures.



 §.§ MLOps improvement (RQ_2)
 
The interviews revealed a consensus on the dual nature of MLOps, presenting both significant benefits and notable challenges. All participants reported that adopting MLOps practices enhanced product quality and increased client satisfaction. However, the cost of tools was identified as a potential obstacle, prompting some to switch to more cost-effective alternatives. Furthermore, data drift was identified as a pervasive issue, necessitating frequent model retraining to maintain performance and relevance.



 §.§ MLOps perception (RQ_3)
 
A common theme across all interviews was the learning curve associated with adopting MLOps, primarily influenced by the need for a mindset shift among practitioners and stakeholders. This shift often required a catalyst, such as the promise of increased benefits or meeting specific organizational requirements. In many cases, the anticipated higher return on investment served as this trigger, though it often took considerable time to realize.



 §.§ MLOps future trends (RQ_4)
 
Considering future developments, participants speculated on the potential for more advanced tools with a comprehensive understanding of the entire MLOps pipeline. These future tools aim to provide enhanced support to developers, either by improving the coding and testing phases or by offering holistic pipeline feedback mechanisms. Participants envisioned these advancements driven by the evolution of Generative AI, which could also propel the development of Explainable AI, thereby enhancing accessibility and interpretability within the MLOps field.

A summary of the individual perspective on the different RQs is reported in Table <ref>. The table included in the study highlights that at least two out of the three interviewees agreed on each key point of the research questions (RQs). This consensus among the participants serves to validate the relevance and accuracy of the RQs, demonstrating that the questions were well-formulated to capture essential insights into the benefits, challenges, and future directions of MLOps practices.




§ RELATED WORKS



Several studies have attempted to define guidelines for MLOps from various standpoints. In one of our previous work <cit.>, we outlined our view of MLOps guidelines as a logical progression of DevOps. This involved transforming the traditional infinite loop into a new model where software engineers and ML engineers work closely together.

Among the most widely recognized and universally accepted are the guidelines proposed by Google. In their Google Cloud Architecture Center blog <cit.>, researchers have outlined different levels of automation to classify maturity in MLOps. This framework provides a structured approach to scaling MLOps, ranging from manual processes to fully automated CI/CD pipelines.

Additionally, Sculley et al.<cit.> from Google presents a vision of ML systems through the lens of MLOps, emphasizing the importance of accounting for technical debt in the software development process . This work illustrates how failing to address technical debt can result in significant and prolonged maintenance challenges, ultimately leading to reduced system performance. It is essential to consider this perspective to gain insight into the long-term implications of MLOps adoption and the necessity for robust, scalable solutions.



Other works have been focusing on the importance of the taxonomy related to MLOps and other similar terms. Steidl et al. <cit.> compared terms such as DevOps and CI/CD for AI, MLOps, and Continuous Delivery for Machine Learning (CD4ML). Moreover, they investigated the factors that could trigger retraining or pipeline execution, as well as the potential challenges associated with MLOps development. However, Ianniello et al. <cit.> conducted a review of the existing scientific literature and proposed a taxonomy for clustering research papers on MLOps. Among the findings of the work, is that the application of DevOps principles to ML and the use of MLOps in industrial environments remain relatively under-researched in academic literature. 

Matsui et al. <cit.> identified 5 steps to guide researchers and practitioners toward understanding and adopting MLOps. The first and most important step in the process is to encourage a mindset that is open to learning and change. They highlight the importance of cultural aspects for the successful implementation of MLOps, noting that without a shift in mindset and organizational culture, technical advancements alone are insufficient for achieving effective MLOps adoption.
Other works have specifically focused on challenges.


Diaz-de-Arcaya et al. <cit.> provides a comprehensive survey of the challenges and opportunities in both MLOps and AIOps (AI for IT Operations). This study presents a roadmap for future developments in these fields, emphasizing the necessity for more robust integration and collaboration between MLOps and AIOps practices. They conducted a comprehensive literature review, automating the article retrieval process to identify key areas for improvement and potential advancements. Their work offers valuable insights, with one key finding being that MLOps is particularly beneficial in challenging environments, such as industry, while AIOps excels in challenging circumstances, such as 5G and 6G. 

Faubel et al. <cit.> also identify challenges as a key topic in the context of MLOps for Industry 4.0. The authors distinguish three primary categories of tasks that present challenges, namely data, models, and support activities. Notably, support activities are situated between the former two categories, highlighting their unique characteristics. Consequently, the four support activities identified in the article—namely, infrastructure, tools, versioning, and automation—are regarded as the most challenging. 

On a completely different perspective described in the review performed by Haertel et al. <cit.>, MLOps is seen as a solution to the development and production of ML models specifically for Data Science (DS) projects. The systematic literature review of 52 papers on MLOps for DS revealed that, although the academic field is still in its early stages, approximately half of the work describes practical use cases. Furthermore, while MLOps literature primarily addresses technical aspects, it often overlooks organizational considerations. Achieving success in this field depends on the effective management of both analytical and technical expertise.

Another valuable SLR is the one presented in John et al <cit.>, which in this case is supported also by a grey literature review. The reviews have led to the creation of a framework and maturity model, which have subsequently been validated in three different company cases. The cases demonstrate that the companies interviewed were still in the first two phases of the presented maturity model. This indicates that while two of the companies were automating data collection, a third company was also incorporating deployment into its automation pipeline. However, none of those were implementing any kind of automation for monitoring. 

Finally in John et al <cit.> the authors conducted an extensive multi-case analysis involving nine professionals from seven different companies to uncover the key trade-offs that organizations face when adopting MLOps. This study systematically categorized these trade-offs using the BAPO model, which stands for Business, Architecture, Process, and Organization. Each of these categories represents a critical dimension of MLOps adoption, reflecting the multifaceted nature of integrating MLOps practices into existing workflows. The results of this multi-case analysis not only highlighted these trade-offs but also led to the identification of possible mitigations. These mitigations are strategies that organizations can employ to balance the competing demands of the BAPO dimensions. 



§ CONCLUSION


This research study presents a survey on the existing use of MLOps in various organizational settings. 


The interviews yielded insights into the varying perspectives on MLOps adoption, influenced by organizational size and maturity. The interviewees offered varying perspectives as users, providing insights that were both consistent and divergent. Participants from a large enterprise and an SME utilized MLOps explicitly for automation, while another SME participant employed it indirectly at a lower maturity level. All leveraged diverse tools tailored to specific goals, emphasizing the significance of tool diversity.

While continuous training and deployment are not yet fully developed ideas, MLOps practices have been shown to improve product quality and client satisfaction. However, high tool costs and data drift have posed significant challenges, with the latter specifically necessitating frequent model retraining.

The learning curve presented a significant challenge, often requiring a shift in mindset that was prompted by the promise of increased benefits or specific organizational needs. The introduction of advanced tools with a comprehensive understanding of the MLOps pipeline, driven by Generative and Explainable AI, is anticipated to enhance developer support.


In summary, the study identifies the necessity for enhanced integration and support tools within MLOps, addresses key challenges such as the cost of tools and data drift, and emphasizes the importance of a cultural shift to facilitate smoother MLOps adoption. Addressing these issues can lead to greater awareness and adoption of MLOps within organizations.



§ ACKNOWLEDGMENT


This work is funded by IndustryX (Business Finland), 6GSoft projects (Business Finland), and Mufano project (Business Finland).



ACM-Reference-Format

























 Paper Title: Understanding Feedback Mechanisms in Machine Learning Jupyter Notebooks.











A. Shome Delft University of Technology

  a.shome@tudelft.nl
  L. Cruz Delft University of Technology

  l.cruz@tudelft.nl
  D. Spinellis Delft University of Technology

  d.spinellis@tudelft.nl
  A. V. Deursen Delft University of Technology

  arie.vandeursen@tudelft.nl






Understanding Feedback Mechanisms in Machine Learning Jupyter Notebooks
    Arumoy ShomeLuís CruzDiomidis SpinellisArie van Deursen

    Received: date / Accepted: date
=======================================================================



§ ABSTRACT 
  The machine learning development lifecycle is characterized by iterative and exploratory processes that rely on feedback mechanisms to ensure data and model integrity. Despite the critical role of feedback in machine learning engineering, no prior research has been conducted to identify and understand these mechanisms. To address this knowledge gap, we mine 297.8 thousand Jupyter notebooks and analyse 2.3 million code cells. We identify three key feedback mechanisms—assertions, print statements and last cell statements—and further categorize them into implicit and explicit forms of feedback. Our findings reveal extensive use of implicit feedback for critical design decisions and the relatively limited adoption of explicit feedback mechanisms. By conducting detailed case studies with selected feedback instances, we uncover the potential for automated validation of critical assumptions in ML workflows using assertions. Finally, this study underscores the need for improved documentation, and provides practical recommendations on how existing feedback mechanisms in the ML development workflow can be effectively used to mitigate technical debt and enhance reproducibility.






§ INTRODUCTION


The machine learning development lifecycle (MLDL) encompasses a multifaceted series of entangled stages. Significant effort is invested in gathering and assembling the required data from diverse sources. The data then undergoes cleaning and preprocessing to ensure high quality and usability, in the subsequent stages. MLDL is dominated by the model development phase which is highly experimental and iterative. This stage involves exploratory data analysis (EDA) to uncover underlying patterns, fitting the data to various models, and rigorously analysing the results. Often, this stage may lead to additional rounds of feature engineering, where new attributes are created based on insights gained from initial models to enhance model performance. Practitioners may also engage in comparing and contrasting different models to identify the most effective approach, followed by fine-tuning selected models to optimize performance. The final model is then deployed into production where it is continuously monitored, and periodically retrained to adapt to new data or changing conditions <cit.>.

The MLDL is fundamentally iterative, closely resembling Agile software engineering principles, which emphasize incorporating early feedback and working in small, manageable iterations <cit.>. Similarly, CRISP-DM—the dominant process model in data mining—outlines a cyclical process that evolves from understanding the business objectives and data, to data preparation, modelling, evaluation and finally deployment and monitoring <cit.>. Both systems advocate for a dynamic, iterative approach that enhances adaptability and effectiveness in dealing with complex, changing systems like those encountered in ML projects.

Computational notebooks—specifically Jupyter notebooks[https://jupyter.org/]—have been ubiquitously adopted by the ML community for developing ML enabled systems <cit.>. Jupyter notebooks provide an interactive computing environment that allow users to break the development of complex workflows into smaller, more manageable code chunks. Each notebook is composed of a series of code cells, which can be executed independently in a read-eval-print loop (REPL) style of interactive software development. This REPL approach allows users to write a piece of code, run it, and immediately see the results thus facilitating an incremental and iterative style of software development. This modular structure of notebooks provides users the flexibility to experiment with different approaches and algorithms. Users can test hypotheses, debug issues, and make incremental changes with immediate feedback. The ability to visualize data and results inline and document the process in a narrative form further enhances their utility <cit.>.

Notebook users can obtain feedback using implicit or explicit mechanisms. Implicit feedback mechanisms require manual validation by the user. For instance, a common prerequisite for all ML models is that the features in the dataset are all numerical. This can be verified with a print statement in a code cell and manually checking the output. This code statement however, will continue to execute even if the data type of a feature changes in subsequent batches of training data.

In contrast, explicit feedback mechanisms document the expected conditions of the code and halt execution if these conditions are not met. For instance, the implicit validation method mentioned above can be replaced with an explicit assertion: assert all(df[col].dtype in ['int64', 'float64'] for col in df.columns), "Not all columns are of numerical type". This assert statement will stop the execution if the condition is not satisfied and provide immediate feedback with a clear message explaining the issue.

As machine learning pipelines developed within Jupyter notebooks become business-critical, they are frequently transitioned into automated scripts for production environments <cit.>. In these contexts, it is crucial to shift from implicit feedback mechanisms to explicit ones. This transition reduces the reliance on manual verification, thereby enhancing the robustness and maintainability of ML pipelines.

To the best of our knowledge, no prior studies have been conducted to understand how feedback mechanisms are used by practitioners when developing ML pipelines inside Jupyter notebooks. To address this gap, we conduct a series of empirical studies to understand the role of various feedback mechanisms within Jupyter notebooks. We focus on implicit form of feedback obtained from print statements and output generated by the last statement of code cells, as well as explicit form of feedback obtained from assertions. The research questions along with the contributions of this study are presented below.


  
  RQ1. What feedback mechanisms are employed in the development of ML Jupyter notebooks?

    We mine 297.8 thousand Jupyter notebooks written in Python from GitHub and Kaggle. We analyse 2.3 million code cells obtained from the notebooks and identify three key feedback mechanisms used in ML notebooks—explicit feedback from assertions and implicit feedback from print and last cell statements.

    We create a public dataset of 89.6 thousand assertions, 1.4 million print statements and 1 million last cell statements. We perform a descriptive and lexical analysis of the dataset and report the key characteristics of feedback mechanisms used in ML Jupyter notebooks. The dataset along with the replication package is shared publicly under the Creative Commons Attribution (CC-BY) licence[https://doi.org/10.6084/m9.figshare.26372140.v1].

  
  RQ2. How is explicit feedback from assert statements used to validate ML code written in Jupyter notebooks?

    To gain a deeper understanding of Python assertions written in Jupter notebooks, we conduct individual case-studies with 82 assert statements. We analyse the assertion along with the surrounding code to understand its purpose. Additionally, we analyse the entire code cell, the previous and next cells along with the purpose of the notebook to bring in rich context.

  
  RQ3. How is implicit feedback from print statements and last cell statements used when writing ML code in Jupyter notebooks?

    Similarly, we conduct individual case-studies with 44 print statements and 27 last cell statements. During each case-study, we analyse the code statement along with the raw output to understand its purpose.


Our findings indicate that implicit feedback mechanisms are extensively utilized for making critical design and implementation decisions during the MLDL. While manual validation of these outputs is common, it is also prone to inconsistencies thus highlighting the need for automated validation mechanisms. Assertions are identified as a potential solution, capable of verifying data distributions and ensuring model performance metrics meet predefined thresholds. However, the study also notes the limited use of assertions, with only 8.5% of the notebooks analysed incorporating them, suggesting a significant gap in current testing practices within the ML community. The findings underscore the necessity for developing ML-specific testing tools that are deeply integrated into the ML development workflow to ensure consistent and reliable validation processes.



§ BACKGROUND


This section introduces relevant prior concepts for this study.



 §.§ REPLs, Interactive Computing and Project Jupyter


A Read-Eval-Print Loop (REPL) is an interactive programming environment that takes single user inputs (Read), executes them (Eval), returns the result to the user (Print) and then waits for the next command (Loop). REPLs offer an immediate feedback loop for quickly testing and debugging code snippets. The Python standard distribution includes a Python REPL with a command line interface to execute Python code in real-time. IPython—a more advanced alternative to the Python REPL—extends the capabilities of the standard Python shell with additional features such as powerful introspection, additional shell syntax, tab completion, and command history. Jupyter notebooks (and its successor Jupyter lab) leverage the IPython kernel to execute Python code and extends its capabilities with a graphical user interface (GUI). Resembling the literate programming paradigm <cit.>, Jupyter notebooks offer a web-based interface where users can create and share documents that contain live code, equations, visualizations, and narrative text.



 §.§ Anatomy of Jupyter Notebooks




Figure <ref> shows an example Jupyter notebook with markdown and code cells. The data comprised within Jupyter notebooks is stored using the JSON schema specified by nbformat[https://nbformat.readthedocs.io/en/latest/index.html], and thus can be parsed programmatically. A notebook consists of cells which can be of three types—“markdown”, “code” and “raw”. All cells of a notebook are stored as a list under the top-level [language=]cells key in the JSON representation. Within this list, each cell is represented as a JSON object. The contents of the cell are stored as a list of strings inside the [language=]source key.

Only code cells in Jupyter notebooks can generate outputs. As shown in Figure <ref>, outputs can come from three main sources in the source code—print statements, the last statement in the code cell, and statements that create a visualization. The outputs produced by code cells are stored as JSON objects under the [language=]outputs key in the corresponding JSON representation of the notebook. Text outputs (such as those produced by the last statement of a cell or print statements) are stored under the [language=]text key. Visual outputs have a sightly different JSON schema and are stored under the [language=]data key. The image itself is represented as a base64 encoded string which is stored inside the [language=]data object, under the [language=]image/png key.



 §.§ Python Abstract Syntax Tree


The Python ast module enables the parsing, manipulation, and analysis of Python source code by converting it into an abstract syntax tree (AST)[https://docs.python.org/3/library/ast.html]. The ast module can parse source code strings into AST objects, traverse and modify the tree and compile it back into executable code, providing a powerful tool for dynamic code manipulation. In Python, assert statements are represented in the AST by the Assert class. The Assert node includes the test attribute—which contains the condition being asserted—and the msg attribute—which holds an optional message displayed if the assertion fails. Print statements are treated as a call to the built-in print function. Function calls are represented in the AST by the Call class. The Call node includes the func attribute representing the print function, and args which lists the arguments passed.



§ METHODOLOGY




 §.§ Data Collection




Figure <ref> illustrates the data collection process employed in this study to gather Jupyter notebooks written in Python from GitHub and Kaggle. We used GitHub's advanced search syntax[https://docs.github.com/en/search-github/searching-on-github/searching-code], with the following search query: [language=]language:"Jupyter Notebook" to mine 49.9 thousand Jupyter notebooks from public repositories on GitHub[Collected on June 22, 2023]. We used the pre-existing dataset KGTorrent <cit.> for Kaggle since it does not support advanced code-based search like GitHub. To the best of our knowledge, KGTorrent is the largest dataset of Python Jupyter notebooks obtained from Kaggle, consisting of 247.9 thousand notebooks. Consequently, we started with 283 GB of data comprising 297.8 thousand Python Jupyter notebooks.

Since the focus of this study is on the analysis of Python code, we only include notebooks that have a valid underlying JSON structure and contain at least one code cell. Finally, to focus on Python code written specifically for ML projects, we only include notebooks that import at least one of the following popular ML libraries—Scikit Learn <cit.>, Pytorch <cit.>, Tensorflow <cit.> and Keras <cit.>. The libraries are selected based on the import analysis conducted by <cit.> on 6 million Jupyter notebooks collected from GitHub. We use the Python ast module to extract the assert, print and last statements from all code cells. Prior to conducting the analysis for RQ2 and RQ3, we remove duplicate data points resulting in a data set of 27.1 thousand assertions, 498.3 thousand print statements and 373.6 thousand last cell statements.



 §.§ Case Studies


For RQ2 and RQ3, we allocated a fixed time resource of 200 hours to conduct the case study analysis of all candidate assertions, print statements and last cell statements. To surface interesting candidates for the case studies, we apply text processing techniques as described below.

The assertions, print statements and last cell statements are first tokenized—special characters and alphanumeric words shorter than two characters are removed. Two stop words namely “assert” and “print” are removed since they appear in all assertions and print statements respectively. The term frequency (TF) for all tokens is calculated and then normalized using their inter-document frequency (IDF) such that tokens that appear less frequently are assigned a higher value. We apply stratified random sampling to identify the candidates for the case study analysis. The subgroups are created by adding TF-IDF of the tokens in each candidate to produce an aggregate value. The candidates are then divided into quartiles based on the aggregate value. A candidate is randomly drawn from each bin and analysed as an in-depth case study. The analysis is stopped when the time resource is exhausted.

During each case study, we analyse the code of the candidate to understand its purpose. Additionally, we analyse the entire code cell, the previous cell, next cell and the notebook's purpose to bring in rich context. For case studies where the first author was unable to determine the purpose of the candidate, it was additionally analysed by the second and third authors. Further discussions were held with all authors until a consensus was reached.



§ RESULTS




The results of the study are presented in this section. For the case studies conducted for RQ2 and RQ3, we group the candidates based on their similarity to each other. This is done only to structure the paper in a meaningful manner and for ease of reporting the results. Figure <ref> provides an overview of the various feedback mechanism groups identified in this study with additional details provided in Section <ref> and Section <ref>. Throughout the text, we reference the candidate (A)ssertions, (P)rint statements and (L)ast cell statements using their unique identifiers as examples. These can be viewed on our online appendix[https://arumoy.me/shome2023notebook].



 §.§ (RQ1) What feedback mechanisms are employed in the development of ML Jupyter notebooks?
 

25 thousand (8.5%) notebooks contained at least one assertion. From the 8.5% notebooks, we collected 89.6 thousand assertions. 4.8 thousand assertions (5%) used methods from external testing libraries. 85 thousand assertions (95%) however were written using the assert statement provided by the built-in Python standard library. We removed duplicate assertions for the remainder of the analysis, thus resulting in 27.1 thousand assertions.



Figure <ref> presents the 10 most common methods used in the 5% assertions written using external testing libraries. We manually tracked the source of the methods to the numpy, pandas, torch and unittest libraries. The most frequently used methods include assert_almost_equal and assert_equal, each with over 250 instances. Methods such as assert_almost_equal, assert_array_almost_equal and AssertAlmostEqual accommodate the stochastic nature of ML, where results can vary slightly due to inherent randomness in data processing and model training. We also find methods such as assert_array_equal and assert_frame_equal that are specifically tailored for comparing high-dimensional data structures such as numpy arrays and pandas dataframes.

Finding 1.1 25 thousand (8.5%) notebooks contained at least one assertion. 95% of the 89.6 thousand assertions were written using the built-in Python assert statement and 5% using external testing libraries. Methods such as assert_almost_equal and assert_array_equal from the numpy and pandas libraries were frequently used to accomodate for the stochastic nature of ML algorithms and comparing high-dimensional data structures.





We focus the remainder of the analysis on the 95% of the assertions written using the assert statement. Figure <ref> shows the most frequent AST nodes that occur in assert statements. The figure indicates that the most common test performed in the assert statements include a comparison check and a function call. Table <ref> provides a few examples of assertions for each of the tests above. In Figure <ref> we see the most common operators used in assert statements that perform a comparison check. We observe that overwhelming majority of the asserts ensure that two values are equal to each other. Figure <ref> and Figure <ref> show the most common bi-grams that appear in the LHS and RHS of the comparison statements respectively. The figures collectively show that the shape and size of data structures are frequently checked.



Of the 95% assert statements, 28.4 thousand (31.7%) have a failure message. Figure <ref> shows the most frequent bi-grams that appear in these failure messages. Similar to the analysis presented above, we removed the duplicate messages prior to analysing the bi-grams resulting in 5.9 thousand unique messages. The bi-grams indicate that the shape and data type of data structures are frequently verified.

Finding 1.2 31.7% of the assertions written using the built-in assert statement have a failure message. These assertions frequently perform a comparison check or call an external function. The comparison checks frequently verify that the shape and size of data structures is equal to a specified value.





We collected 1.4 million print statements from 180 thousand notebooks (60% of the total notebooks analysed in this study). Unlike assert statements, print statements do not have a fixed lexical semantic since the Python ast module interprets them as regular function calls. Additionally, print statements can be written in several forms—the print function may have multiple arguments, any form of string interpolation supported by Python or output from function calls. Figure <ref> shows that the most common AST nodes inside of print statements are constants followed by function calls. Figure <ref> further validates that the constants are almost entirely of type string. Based on these observations, we dissect the print statements into two components—text written in natural language (from the Constant nodes) and code (from all other nodes). Figure <ref> and Figure <ref> present the most common bi-grams that appear in the natural text and code of print statements respectively. Both figures collectively indicate that print statements are frequently used to print various performance metrics of ML models.

Finding 1.3 Print statements are most commonly used to display various performance metrics of ML models.



We collected 1 million last cell statements from 119 thousand notebooks (40% of the total notebooks analysed in this study). Figure <ref> shows that Call nodes frequently occur in the AST of last cell statements. Figure <ref> shows the most common Python modules and objects while Figure <ref> show the most common functions used inside these Call nodes. The figures highlight the prominence of data visualization libraries seaborn (sns) and matplotlib (plt) and visualization functions such as plot and countplot in the notebooks. The data analysis tool pandas (represented by df and pd) also features prominently with many data exploration and preprocessing functions such as head, value_counts and describe being used frequently. The presence of train and model Python objects along with the fit function indicates the use of ML libraries.

Finding 1.4 Libraries and functions for data visualization, data exploration and machine learning are most common in last cell statements.







We further analysed the most common functions called on pandas dataframes by isolating the last cell statements that contain data or df objects. This is shown in Figure <ref> which reveals a focus on exploratory data analysis and statistical operations. The frequent use of functions such as value_counts, head, and unique, indicate an emphasis on understanding the structure and content of the data. Visualization functions like plot and hist suggest that practitioners often create quick visualizations directly from dataframes. Descriptive statistics are also prominent, with functions like describe, mean, and sum being commonly used.

Similarly, we also analysed the most common visualization functions used in notebooks by isolating the last cell statements where the module is sns or plt. This is shown in Figure <ref> which indicates that categorical data visualization appears to be a primary focus, with countplot and barplot ranking high among the most frequently used functions. The prominence of heatmap and boxplot functions indicates a regular need for visualizing correlations and statistical distributions. For continuous data, functions like distplot, scatterplot, and scatter are commonly used. The presence of imshow suggests that image-related visualizations are also significant in many notebooks.

Finding 1.5 Common functions called on pandas dataframes focus on exploratory data analysis, statistical operations and quick visualizations. In visualizations, functions for categorical data, correlation and distribution are frequently used.



 §.§ (RQ2) How is explicit feedback from assert statements used to validate ML code written in Jupyter notebooks?
 



  §.§.§ Data shape check (N = 26)


Data shape checks can be considered the “Swiss army knife” of assertions, as they are ubiquitous and serve multiple purposes throughout the MLDL. We encountered a vast array of assert statements that verify the dimensions of various data structures, including input features (A17, A90), labels (A5, A29), predictions, images (A76, A93), sequences, and embeddings (A76, A93). These assertions ensure that the data adheres to the expected format and dimensions required by different components of the machine learning pipeline.

Finding 2.1 26 assertions verify the dimensions of various data structures such as input features, labels, predictions, images, sequences, and text embeddings. These checks ensure data consistency and compatibility throughout the ML pipeline, preventing runtime errors and ensuring accurate model training and evaluation.



  §.§.§ Data validation check (N = 14)


We found assert statements that perform various validation checks on data structures, such as NumPy arrays and Pandas dataframes. Common data validation checks include verifying the presence of specific values or ranges within arrays or columns(A41, A65) and verifying the equality or closeness of arrays to a target value within a specified absolute tolerance (A44, A46). Some assertions also focus on validating the uniqueness or cardinality of values (A52) while others validate the presence of specific values or conditions within data structures (A73).

Finding 2.2 Fourteen assertions perform validation checks on data structures, ensuring the presence of specific values, verifying value ranges, and confirming uniqueness or cardinality of values. These checks catch potential errors or inconsistencies early, ensuring data meets specific criteria or constraints for robust ML applications.



  §.§.§ Model performance check (N = 11)


[caption=Assertion A58 used to check that the accuracy of a model is close to the specified value. The use of np.isclose allows for small deviations in the accuracy thus accounting for the stochastic nature of ML models., label=lst:A58]
assert np.isclose(accuracy, 0.9666666666666667)


This study finds several assertions used to test model performance against predefined thresholds for key metrics such as accuracy, precision, recall and F1 score. Range checks confirm that the performance metrics fall within acceptable boundaries, indicating good predictive performance without overfitting (A38). Direct comparisons of model outputs with expected results, validate the model's predictive accuracy and reliability under operational conditions (A15). Other assertions ensure that the learned parameters align closely with theoretically or empirically derived values, further cementing the model’s statistical validity (A19). Finally, assertions are also used to check for the expected number of neighbours as an indirect measure of model performance in specific scenarios like clustering (A7).

The use of np.isclose in Listing <ref> is particularly noteworthy. This method accounts for the stochastic nature of many ML models, where slight variations in performance metrics can occur due to differences in initial conditions, random seed settings, or inherent randomness in algorithms. By allowing a small tolerance in the comparison, np.isclose ensures that the model's performance is consistently close to the expected benchmark despite the stochastic elements.

Finding 2.3 Eleven assertions evaluate model performance against predefined thresholds for accuracy, precision, recall, F1 score, and other metrics. These checks validate the model's predictive accuracy and reliability, ensuring it meets expected performance standards despite the stochastic nature of ML models.



  §.§.§ Existence check (N = 8)


Existence checks are primarily used during the data preprocessing stage in the MLDL to ensure data integrity before analysis and modelling. These checks primarily focus on verifying the presence of necessary columns and the absence of missing values within those columns (A23, A42, A79, A86). Such validations are especially crucial after data preprocessing steps, where transformations might inadvertently introduce NaN values (A50, A51, A63) or result in an empty dataframe (A43). These checks are also crucial for maintaining the accuracy and efficiency of data handling processes, ensuring that the datasets are ready for robust ML applications.

Finding 2.4 Eight assertions ensure the presence of specific columns in the dataset and the absence of missing values. These checks prevent operations on empty datasets and confirm the absence of  values, maintaining data integrity and reliability for model training.



  §.§.§ Resource check (N = 7)
 

[caption=Assertion A37 used to ensure that an ML model has not reached an inconsistent state due to out-of-order or re-execution of code cells., label=lst:A37]
assert svm.fit_status_ == 0, 'Forgot to train the SVM!'


Resource check assertions are used to ensure the availability and validity of essential resources, preventing runtime errors, and maintaining the integrity of the data, models, and visualizations. One set of assertions verify the existence of files on the file system, such as pre-trained models (A10) or data files (A74). Subsequently, assertions are employed to validate the successful loading of models (A14) and the completion of model training processes (A37).

A unique aspect of working with notebooks is the ability to re-run cells while experimenting, which can lead to unintended consequences. Assertion A37 presented in Listing <ref> addresses this scenario by checking if the SVM model has been properly trained before proceeding with further operations. This check prevents inconsistent model states from executing cells out of order or multiple times. Other interesting assertions unique to computational notebooks include verifying the presence of data in visualizations (A60) and explicitly verifying the versions of critical libraries (A18, A67).

Finding 2.5 Seven assertions confirm the availability of essential resources like pre-trained models, data files, successful model loading, and completion of training processes. These checks prevent runtime errors and ensure proper code execution, maintaining the reliability and robustness of the ML workflows.



  §.§.§ Type check (N = 5)


Type check assertions are used to maintain data integrity, ensure compatibility between different components of the ML pipeline, and preventing errors and unexpected behaviour. Type checks are commonly performed to ensure that the input features are PyTorch float tensors or NumPy arrays, a requirement for many deep learning models and scientific computing operations respectively (A2, A88). Type checks are also applied to verify the type of the ML models (A40), data structures and intermediate objects (A35) and individual columns or elements within a dataset (A81).

Finding 2.6 Five assertions validate the data types of input features, models, and intermediate objects. These checks ensure compatibility with ML models and operations, preventing errors from incompatible methods or attributes, thus maintaining the reliability of the ML workflow.



  §.§.§ Mathematical property check (N = 4)


When working with neural networks and other statistical models, it is imperative to ensure that mathematical properties are preserved throughout operations involving arrays and matrices. This rigour is captured though assertions that ensure convolution operations are dimensionally consistent (A3), checking for the symmetry of matrices (A64) and monitoring the standard deviation of outputs to decide the appropriateness of using batch normalization (A25). These assertions are not just precautionary but are vital for confirming that the underpinnings of ML algorithms align with expected mathematical principles.

Finding 2.7 Four assertions validate key mathematical properties of data and model outputs. These include ensuring dimensional consistency in convolution operations, monitoring standard deviation for batch normalization suitability, verifying uniformity in computations, and checking matrix symmetry. These checks confirm the robustness and reliability of model computations.



  §.§.§ Batch size check (N = 3)


In the context of neural networks, the practice of using batch processing is a key strategy to enhance computational efficiency and hardware utilization. Batching stabilizes the learning curve by updating the model weights based on the average gradient of a batch thereby optimizing learning and enhancing generalizability of the model. Ensuring the batch size matches the hardware capacity is crucial to avoid memory overflow issues that can halt training. Assertions are used to check that the batch size divides evenly into the training dataset size which ensures that each training epoch is computationally efficient (A21). Additionally, assertions are used to ensure that the size of the training data is larger than the batch size (A70) and that image dimensions are suitably divisible by the patch size which is essential for certain convolutional networks (A28).

Finding 2.8 This study identifies three assertions validating batch size configurations. These assertions ensure batch sizes are divisible by dataset sizes, image dimensions are appropriately divided by patch sizes, and the number of images exceeds the batch size. These checks are crucial for optimizing computational efficiency and ensuring stable and smooth model training.



  §.§.§ Network architecture check (N = 3)


When integrating pre-trained models or leveraging transfer learning techniques, it is crucial to ensure the compatibility of the custom neural network architecture with the pre-trained components. This is particularly important when combining convolutional layers from different sources, as the spatial dimensions and channel configurations must align correctly. Checks include assertions that verify the output and input dimensions of consecutive convolutional layers match (A11), checking that shape of the activations have the right number of channels (A62) and that the appropriate regularization parameter is set (A75). Such architectural checks are crucial for preventing potential issues during the forward pass of the neural network and ensuring that the custom and pre-trained components are compatible with each other.

Finding 2.9 Three assertions verify the consistency of neural network architecture. These include ensuring weight dimensions match across layers and that activation shapes and regularization parameters are as expected. These checks ensure that the neural network is correctly configured and integrated with pre-trained models, maintaining the integrity of data flow during training.



  §.§.§ Data leakage check (N = 1)


[caption=Assertion A33 used to ensure that the training and validation sets do not contain any overlapping values., label=lst:A33]
assert len(
  set(tr_df.PetID.unique()).intersection(valid_df.PetID.unique())
) == 0


Ensuring the absence of data leakage between training and validation datasets is a fundamental best practice in ML to prevent overfitting. To guarantee that the model can generalize effectively to new examples, it is crucial to verify that the training and validation sets are completely distinct with no shared examples. By strictly separating these datasets, the validation phase provides an unbiased evaluation of the model's ability to generalize to data similar to, but not identical to that which it was trained on. This is demonstrated by Assertion A33 in Listing <ref> which ensures that there are no overlapping PetID values between the training set and the validation set, thereby preventing any possibility of data leakage.

Finding 2.10 This study identifies one assertion to ensure there are no overlapping values between training and validation sets to prevent data leakage. Preventing data leakage is essential to avoid overfitting and ensure the model's ability to generalize to new data. This practice maintains the integrity and validity of the model evaluation metrics.



 §.§ (RQ3) How is implicit feedback from print statements and last cell statements used when writing ML code in Jupyter notebooks?
 



  §.§.§ Model performance check (N = 33)


During the model development phase, the performance of an ML model is often printed to facilitate comparisons against other models or variations. Throughout this continuous experimentation phase, authors may adjust model parameters or modify the data by engineering new features. The model is then re-trained to check if these changes lead to performance improvements. Besides the accuracy of the model (P3, P18), we also see checks for the Root Mean Square Error (RMSE) (P6) and the use of the classification report (P50) provided by scikit-learn[https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html] which reports the accuracy, precision, recall and f1 score for all labels present in the target feature. In addition, we find the use of heatmaps (L3) to visualize the confusion matrix of a multi-label classification task and custom functions (L52) to manually evaluate an image classification model on a random sample of input images.

Finding 3.1 The study identifies 33 implicit checks to monitor the performance of trained ML models. These include outputs that print key performance metrics, a normalized heatmap used to visualize the confusion matrix of a model's predictions and a custom function written for manual evaluation of model predictions on random input samples. These checks are used throughout the iterative model development phase, enabling practitioners to compare different models or variations, adjust parameters, and refine features to improve performance.



  §.§.§ Data distribution check (N = 7)
 

Understanding the distribution of data is crucial for making informed decisions about necessary transformation steps in data analysis. For instance, visualizations can efficiently determine if scaling, normalizing, or handling outliers is needed. During the exploratory data analysis phase, visualizations and pandas dataframes are commonly used to assess the distribution of specific columns (L9), helping identify features related to the target variable (L2, L14)and informing feature inclusion in model training. Additionally, descriptive statistics are used post-transformation to manually verify the effectiveness of data transformation steps and ensuring that the data conforms to the expected format for further analysis (L48, L25).

Finding 3.2 The study identifies seven implicit data distribution checks, including visualizations like categorical plots, kernel density estimate plots and count plots, and statistical summaries from pandas dataframes. These checks are used for making informed decisions about data transformations such as scaling, normalizing and handling outliers, ensuring data integrity and improving model training reliability during the exploratory data analysis phase.



  §.§.§ Resource check (N = 7)


Similar to the assertions identified in Section <ref>, we also find print and last cell statements that validate the availability of essential resources on the system where the notebooks are being executed. This includes checks for the availability of compute resources such as a GPU or a TPU (P68, P82, P86), verifying that a dataset or a pre-trained model has been successfully loaded into memory (P107, L66) and ensuring that a certain version of an external library is present on the system (P71).

Finding 3.3 The study identifies seven implicit checks used to verify resource availability on the systems where the Jupyter notebooks are being executed. These checks include confirming the availability of compute resources like GPUs and TPUs, ensuring datasets or pre-trained models are successfully loaded, and verifying the presence of specific library versions. These checks are used for ensuring that the computational environment is correctly set up, preventing execution errors, and facilitating smooth workflow execution.



  §.§.§ Spot check (N = 5)


Performing value or spot checks can be an essential practice for ensuring data integrity and model accuracy at various stages of development. These checks are crucial for verifying that operations such as data transformations, model predictions, and feature engineering are functioning as expected. Common checks include verifying the number of features in the data matches expectations after applying feature selection or dimensionality reduction techniques (L60), assessing the correctness of one-hot encoding (P114) and ensuring that data retains its expected properties after manipulation (P64).

Finding 3.4 This study finds five implicit spot checks performed at various stages of the MLDL. Spot checks are used to ensure data integrity by verifying data transformation and feature engineering steps meet expectations.



  §.§.§ Model training check (N = 4)


During training, it is common practice to monitor the progress periodically and use this feedback to adjustment training parameters or stop training early to prevent overfitting. This can be done by checking the best parameters found through tuning methods (L42), fitting the model to the training data (L31) or printing the training loss and accuracy to give real-time feedback on the learning efficacy of the model (L8).

Finding 3.5 The study identifies four implicit checks used to monitor model training progress. These checks provide real-time feedback on training loss and accuracy, validate model adherence to expected data patterns, and identify optimal parameters through tuning methods. These checks facilitate timely adjustments during the training process to prevent overfitting and ensure model performance optimization.



  §.§.§ Missing value check (N = 3)


Checking for missing values is a critical step in the preprocessing phase of an ML project because it significantly influences the quality and performance of the model. Missing data can lead to biased or incorrect conclusions if not handled properly, potentially skewing the model's performance by training on incomplete or non-representative samples <cit.>. Many ML algorithms demand complete numerical datasets to perform calculations such as matrix multiplication. Missing values interrupt these calculations, leading to errors or the inability to execute algorithms entirely. These checks can be performed in code (P74, L36) and using visualizations such as a heatmap (L12).

Finding 3.6 The study identifies three implicit checks for missing data. Missing values can lead to biased or incorrect conclusions as many ML algorithms require complete numerical datasets for accurate calculations. Identifying and addressing missing data during preprocessing is essential to avoid errors and improve model reliability.



  §.§.§ Shape check (N = 3)


Ensuring that data dimensions align with expectations is important particularly following data pre-processing or transformation steps. Primarily, the number of features in the training set must match those in the testing set (P4, P32, P117). This alignment is crucial because statistical ML models are trained on specific data dimensions and expect the same dimensional structure during inference to perform accurately. Similarly, in neural network architectures, the configuration of input layers depends directly on the shape of the training data, dictating the number of input neurons needed. Furthermore, the correspondence between the number of test examples and their respective labels is essential for accurately computing performance metrics.

Finding 3.7 The study identifies three implicit checks used to validate the shape of data. These checks are used to ensure data dimensions align with expectations, particularly after preprocessing or transformation steps.



  §.§.§ Data relationship check (N = 2)
 



Linear ML models achieve optimal performance when the target variable can be expressed as a linear combination of the input features. However, features within a dataset that exhibit a linear relationship are considered redundant as they convey the same information to the model during training. Consequently, the feature engineering stage often involves removing such features to create a more efficient training dataset <cit.>. Table <ref> illustrates two last cell statements found in this study for identifying linear relationships among features in the data. In addition to the code, the raw output of the cell is also shown since both case-studies produce a visualization.  In L6, the practitioner assesses the linearity between the features  and . L10 is a visualization of the feature  alongside the target variable , accompanied by a fitted linear regression model.

Finding 3.8 The study identifies two last cell statements used to verify linear relationships between features in the data. Visualizations such as a scatter plot with a regression line and a line plot are employed to assess the linearity between features. These checks are used for identifying and potentially removing redundant features or performing feature selection to ensure that the training dataset is optimized for linear ML models.



  §.§.§ Type check (N = 2)


Type checks are crucial not only for confirming the current state of the data but also for determining the specific transformations required to make the data suitable for subsequent modelling steps. Ensuring data types align with model requirements is a fundamental step in the exploratory data analysis phase. Since all ML algorithms perform mathematical operations, they require the input data to be entirely numerical to avoid computational errors (P43, L71). The process of checking data types helps in identifying the necessary pre-processing or transformation steps to convert the data into a form that is compatible with these models. For example, numerical features often require normalization to bring them within a specific scale and categorical features must be converted into a numerical format through methods like one-hot encoding to ensure they can be effectively integrated into the model.

Finding 3.9 The study identifies two implicit checks used to verify the data types of features in the dataset. These checks are used to determine the necessary preprocessing or transformation steps required to make the data suitable for training. Type checks are also used to ensure that all features are numerical prior to performing mathematical operations such as matrix multiplication, which is used in several ML models.



  §.§.§ Execution time check (N = 1)

[caption=Print statement P66 used to check the total execution time of training an ML model with various hyper-parameter configurations., label=lst:P66]
print('Total Run Time:')


While training multiple ML models for a single task is common, choosing the best model considers not only its performance but also its training speed. Faster training times are more cost-effective for iterative tasks and promote sustainability in the long run <cit.>. Listing <ref> presents print statement P66 that prints the total execution time of a code cell which trains an ML model with various hyperparameter configurations using a grid-search.

Finding 3.10 The study identifies one print statement used to check the total execution time of training and cross-validating an ML model. The print statement can be used to evaluate the efficiency of model training, especially when multiple models are trained iteratively.



  §.§.§ Network architecture check (N = 1)


Printing the structure of a neural network (P92) serves crucial purposes such as verifying the correct implementation of the architecture, aiding in debugging by ensuring layer compatibility, and providing a clear, human-readable format for documentation and educational insights. It helps in verifying configurations like layer dimensions and operations before initiating computationally intensive training and ensuring that the model is set up efficiently for the intended tasks.

Finding 3.11 The study identifies one print statement used to show the architecture of a neural network. Such outputs can be used during the model development phase to ensure compatibility between the layers of a neural network, for understanding and learning about the architecture of pre-trained models and for documentation purposes.



§ DISCUSSION


Drawing from the results of this study, we discuss implications and future work in this section.



 §.§ Documentation and Technical Debt


Jupyter notebooks are primarily used to explore different ideas or approaches, often neglecting software development best practices in favour of rapid experimentation <cit.>. Once the code is finalized, it is typically transferred into a script that is run in a production environment. Alternatively, the notebook may contain significant exploratory work such as developing a model or performing data exploration. To effectively communicate the information present in the notebooks with clients, collaborators or teammates, the practitioners add annotations and structure to the notebook using markdown cells <cit.>. In both scenarios, practitioners need to revisit the notebooks to obtain up-to-date and accurate requirements, their rationale and design decisions, and tests—artefacts which are usually missing in the original notebooks <cit.>.

Our findings indicate that structured information can be effectively derived from the feedback mechanisms available in Jupyter notebooks. The results demonstrate that practitioners utilise assertions, print statements, code cell outputs, and visualisations to inform data preprocessing techniques, detect erroneous assumptions, create new features to enhance model performance, and verify hardware and software dependencies. As data, personnel, and system environments inevitably change, undocumented decisions can lead to accumulation of technical debt. This not only prolongs time required for debugging and troubleshooting, but also incurs additional costs and resource expenditures <cit.>.

Our results from RQ3 reveal that implicit feedback from print and last cell statements are used in Jupyter notebooks to make design and implementation decisions throughout the machine learning development lifecycle. Unlike assertions, undocumented decisions based on implicit feedback mechanisms incur higher technical debt since they don't stop the execution despite violating the underlying assumptions of the code.

We recommend that ML practitioners document insights derived from implicit feedback mechanisms, especially for visualizations. Visual cues and trends in visualizations are open to interpretation, meaning that different practitioners may interpret the same plot differently <cit.>. By documenting these visual insights, practitioners can ensure that the knowledge encapsulated in the visualizations, along with its interpretation and the subsequent decisions made, is preserved and consistently applied throughout the ML lifecycle. Although writing documentation from implicit and explicit feedback mechanisms still requires significant manual effort and resources, the dataset contributed by this study provides a valuable resource for ongoing efforts to automatically generate documentation from code written in Jupyter notebooks <cit.>.



 §.§ Automated Data Validation


Catching data errors are critical since ML models trained using dirty data will lead to poor and incorrect predictions. This is a cause for concern since these models may be deployed in production and their output may be used by other services which will then also perform poorly. The output of the ML models are often also used to capture new training data, thus leading to a feedback loop which over time will degrade the model performance <cit.>.

<cit.> view ML training as analogous to compilation and recommend that training data be tested, similar to code in traditional software. They propose a rubric to assess the production readiness of an ML pipeline, which includes several recommendations for data testing. In subsequent work, <cit.> introduce TensorFlow Data Validation, an automated tool for data validation that works on principles borrowed from the database community. Data expectations are specified in a schema, and each new data batch is validated against this schema to ensure it meets the expected criteria or prompts schema updates if requirements change.

The results from RQ2 indicate that assertions can be used to automatically validate critical assumptions in ML workflows. Embedding assertions into the code helps ensure that any deviation from expected conditions is promptly flagged. This reduces the reliance on manual verification thus enhancing the robustness and maintainability of ML pipelines. Assertions also serve as an executable form of documentation that captures the assumptions and decisions made during model development. This is particularly important in collaborative environments where multiple practitioners might work on the same project. Assertions ensure that all team members have a clear understanding of the requirements and constraints across the entire ML pipeline, facilitating smoother transitions and handovers. Moreover, assertions may enhance the reproducibility of ML experiments by ensuring that the same validation checks are applied consistently, regardless of the execution order of the notebook cells <cit.>.

Results from RQ2 and RQ3 show that assertions, print statements and last cell statements are predominantly used in the upstream, data-intensive stages of the MLDL. For instance, this study identifies several assertions that perform data validation checks to ensure data meets specific criteria or constraints. Similarly, we find assertions, print statements, and last cell statements that conduct type, shape, and distribution checks to safeguard against data inconsistencies. These results show a missed opportunity in data validation tasks. We argue that when integrating dedicated data validation tools into an operational ML pipeline, the existing feedback mechanisms in notebooks can be effectively used to write the data validation schema.



 §.§ ML Testing Literacy


Despite the potential benefits, our results from RQ1 reveal that assertions are not widely used in Jupyter notebooks. The relatively low adoption rate compared to implicit feedback mechanisms, suggests that there is significant room for growth and improvement in the integration of assertions into ML development workflows.

We hypothesize that this limited use of assertions is due to a lack of knowledge among practitioners regarding testing approaches and the absence of adequate tooling to support the assertion of data and model properties. Guidelines and curricula on ML coding practices have historically focused on building better models. We argue that the focus must shift towards building reliable ML-enabled systems by incorporating decades of research from the software engineering community with ML. Our recommendation is evident in the recent developments within the software engineering research community that steer education and research towards addressing the specific challenges faced when developing ML-enabled software systems <cit.>. Fostering a deeper understanding of testing practices and providing tools tailored to ML development can help promote more rigorous and reliable testing methodologies within the community.

The assertions analysed in RQ1 and RQ2 further reveal two significant challenges in the current landscape of testing in ML Jupyter notebooks. First, the assertions identified in this study lack the support of dedicated testing libraries. This suggests that the culture of testing ML code is still in its infancy, and ML practitioners are not yet fully exploring or leveraging software testing libraries. Secondly, existing testing libraries are not specifically tuned to the needs of ML code, leading practitioners to rely on basic built-in approaches. This highlights the necessity for developing specialized testing frameworks that cater to the unique requirements of ML projects.



 §.§ Machine Learning Bugs


The findings of this study align with prior research on real-world defects in ML systems. <cit.> propose a taxonomy of faults in DL systems, while <cit.> curate Defects4ML, a dataset consisting of ML bugs. Both studies identify defects arising from the use of incorrect data types and shapes, deprecated APIs, and missing GPU resources. Our study identifies several explicit assertions that can catch such data errors. Additionally, we find several implicit checks performed using print and last cell statements to verify the presence of compute resources and libraries, monitor model training, and check the architecture of neural networks.

This study further reveals several avenues for research. We primarily focused on identifying feedback mechanisms used when writing ML code in Jupyter notebooks and subsequently exploring a limited subset of them. Further research is required to explore the feedback mechanisms in more detail and organize the knowledge into a taxonomy based on specific ML validation tasks. Future work can focus on mapping the feedback mechanisms in Jupyter notebooks to the ML defects identified by <cit.> and <cit.> which can aid practitioners in debugging or preventing faults using the appropriate feedback technique.



 §.§ Code Quality and Conformance to Standards


Results from RQ1 show that 68% of the assertions do not contain a failure message. This can lead to problems such as unclear error diagnostics, difficulty in debugging, and increased time to identify and resolve issues. <cit.> investigated the testing culture in open-source R packages and identified several anti-patterns that lead to the accumulation of testing technical debt. The dataset contributed by this study can be used by researchers to replicate the <cit.> study using assertions written in Python.



§ THREATS TO VALIDITY




  
Internal Validity This study examines Jupyter notebooks written in Python, specifically focusing on machine learning code. We do this by only considering notebooks that utilize popular ML libraries. Consequently, we may miss notebooks that are ML-related but do not use these specific libraries. Additionally, notebooks that import popular ML libraries but do not actively use them in the code could lead to misclassification. To mitigate this threat, we specifically look for ML libraries that facilitate model training rather than more general-purpose libraries such as pandas, numpy, and scipy, which can be used for a variety of non-ML tasks.

We enforced a time resource limit of 200 hours for conducting the case studies in RQ2 and RQ3, potentially constraining the depth and breadth of our investigation. The stratified random sampling used for selecting the case study candidates mitigates this threat by ensuring that a representative cross-section of the population is sampled within the time constraint. While this sampling method enhances the generalizability of our results, the time limitation should still be considered when interpreting and applying the study's findings.

The case studies conducted in RQ2 and RQ3 are subject to bias from the first author's individual perspective and understanding which could limit the interpretation of data and potentially overlook alternative viewpoints or explanations. To mitigate this threat, we implemented a rigorous review process involving multiple rounds of discussion with the second and third authors. These discussions continued until a consensus was reached on the interpretations and findings.



  
External Validity Our analysis is based on ML notebooks publicly available on GitHub and Kaggle. Consequently, the findings may not be generalizable to Python notebooks that do not concentrate on ML. Additionally, the results may not be applicable to notebooks authored in other statistical programming languages (such as R or Julia), or to proprietary notebooks hosted on different platforms.



§ RELATED WORK




 §.§ Assertions and Software Bugs


<cit.> conducted a partial replication of a prior study on assertion usage, expanding the analysis to 185 Java projects from GitHub. <cit.> conducted a comprehensive study on the unit testing practices within 177 systematically selected open-source R packages hosted on GitHub. The research addressed the quality of testing, identified testing goals, and pinpointed potential sources of testing technical debt.

<cit.> developed a comprehensive taxonomy of faults in deep learning systems through the manual analysis of 1059 artefacts from GitHub and Stack Overflow, complemented by structured interviews with 20 researchers and practitioners. The study categorized faults into five top-level categories, encompassing model, data, training, deployment, and infrastructure-related faults. <cit.> addressed the challenge of assessing the quality and performance of tools designed to improve the reliability of machine learning components by creating a standard fault benchmark called defect4ML. This benchmark consists of 100 reproducible bugs reported by ML developers on GitHub and Stack Overflow, focusing on the TensorFlow and Keras ML frameworks.

In contrast to prior work, our research focuses on the feedback mechanisms used within Jupyter notebooks for ML development. While previous work has provided insights into assertion usage and fault taxonomies, our study specifically examines the role of implicit and explicit feedback in ensuring data and model integrity within interactive development environment provided by Jupyter notebooks.



 §.§ Computational Notebooks and Software Engineering


<cit.> provide an overview of the evolving landscape of computational notebooks by analysing six million Python notebooks, two million enterprise Data Science (DS) pipelines, source code and metadata from over 900 releases of 12 important DS libraries. Their findings can be used by system builders for improving DS tools and also by DS practitioners to understand the current trends in technologies they should focus on. <cit.> mined 1.4 million notebooks from GitHub to conduct an empirical study on the coding practices in computational notebooks. Based on their analysis, the authors propose guidelines on improving reproducibility of computational notebooks. To enable future research on computational notebooks, <cit.> mine Kaggle and present KGTorrent, a public dataset consisting of approximately 250 thousand Jupyter notebooks written in Python. The dataset also contains a relational database dump of metadata regarding publicly available notebooks on Kaggle.

Studies with human subjects have been conducted to gain a deeper understanding of the challenges faced by ML practitioners when developing ML models inside notebooks. The results indicate that ML practitioners work in a highly iterative fashion, often experimenting with multiple strategies to analyse the data or produce meaningful visualizations <cit.>. Studies have also been conducted to understand how practitioners generate, evaluate and manage alternative hypothesis, visual designs, methods, tools, algorithms and data sources to arrive at the final implementation <cit.>. The findings from these studies can be used as guidelines for improving existing notebook technologies or designing new ones.

To manage and prune multiple versions of code that accumulate over time when developing in notebooks, <cit.> propose a code gathering tool that allows practitioners to review and only keep the relevant version of code. Other tools such as WrangleDoc and VizSmith have also been proposed to aid ML practitioners when working within computational notebooks <cit.>.

Jupyter notebooks have been widely adopted by the DS and ML communities to develop ML pipelines <cit.>. Computational notebooks provide a rich source of data in the form of natural text, code and visualizations. Computational notebooks also allow ML practitioners to present the knowledge gained from the analysis in a narrative that can be shared with others <cit.>.

Our study builds on this foundation by focusing on the specific feedback mechanisms used within Jupyter notebooks during ML development. We examine both implicit and explicit feedback methods, providing a detailed analysis of their usage and effectiveness in maintaining robust ML workflows.



 §.§ ML Testing


Existing scientific literature on ML testing broadly focuses on two aspects. First, on functional properties such as correctness and robustness of the model towards unseen data. And second, on non-functional properties such as fairness, interpretability and privacy <cit.>.

To test the correctness of ML models, several improvements over existing test adequacy metrics have been proposed. Tools such as DeepXplore and DeepGauge propose new test adequacy metrics such as neuron coverage adapted for ML enabled software systems <cit.>. Formal verification methods have also been proposed that try to provide formal guarantee of robustness against adversarial examples. Such methods however are only feasible for statistical ML models and become computationally expensive for more complex models such as deep neural networks <cit.>. Several studies have been conducted on generating and detecting adversarial inputs for ML models. Data augmentation techniques based on fuzzing, search based software testing and mutation testing have been proposed to generate adversarial examples that can be used during model training to improve its robustness <cit.>. It is however not possible to include all variations of adversarial examples into the training data. Thus, methods have been proposed to detect adversarial inputs during runtime <cit.>.

Despite these advancements, there is a notable gap in practical, day-to-day testing practices within ML development environments. Our study addresses this gap by analysing the feedback mechanisms in Jupyter notebooks, highlighting how assertions and print statements are used to validate data and model performance, and proposing ways to integrate these practices more effectively into the ML development lifecycle.



 §.§ Visualizations and Machine Learning


As ML augment software systems in safety-critical domains, emphasis has been put into explainability of ML models. ML models and the underlying data is complex and multidimensional. To combat the “curse of dimensionality”, visual analytics has been widely adopted by the ML community to understand the data and the internal workings of ML models <cit.>.

Prior studies have been conducted to understand how visual analytics techniques are currently being used in ML. <cit.> conduct a systematic review of 259 papers and propose a taxonomy of visual analytics techniques for ML. <cit.> conduct a survey of visual analytics techniques for Deep Learning Models. The findings of the study indicate that visual analytics has been widely adopted in ML for model explanation, interpretation, debugging and improvement.

Several tools have been proposed by the visual analytics research community to aid practitioners in understanding how their ML models operate. <cit.> propose What-If, a visual analytics tool to explore alternative hypotheses, generate counterfactual reasoning, investigate decision boundaries of the model and how change in data affects the model predictions. To reduce the cognitive load of ML practitioners during model building phase, <cit.> propose ModelTracker. Given a trained model and a sample dataset, ModelTracker presents all the information in traditional summary statistics along with the performance of the model.

ESCAPE, GAM Coach, Angler and Drava are a few other tools that have been proposed to handle specialized use-cases such as identifying systematic errors in ML models, generating counterfactual explanations for Generalized Additive Models, prioritizing machine translation model improvements and relating human concepts with semantic dimensions extracted by ML models during disentangled representation learning <cit.>.

Unlike prior work that focuses on standalone visual analytics tools, our study explores how visualizations within Jupyter notebooks are used by ML practitioners. We analyse the motivations behind creating these visualizations and the insights they provide, offering a nuanced understanding of their role in the iterative ML development process.



§ CONCLUSION


This study provides a comprehensive analysis of feedback mechanisms employed within Jupyter notebooks during the machine learning development lifecycle. Our findings indicate that while implicit feedback mechanisms such as print statements and last cell statements are extensively utilized, there is a significant gap in the adoption of explicit feedback mechanisms like assertions. This reliance on manual validation highlights the potential for automated data validation to enhance the robustness and maintainability of ML pipelines. This study also underscores the necessity for better documentation practices to mitigate technical debt and ensure the reproducibility of ML experiments. Feedback mechanisms are used throughout the ML development workflow to achieve more reliable and efficent model development processes. Future work should focus on developing specialized tools and frameworks to support these practices, thereby advancing the field of ML engineering.


spbasic      


   



 Paper Title: Effective Black Box Testing of Sentiment Analysis Classification Networks.
















	Effective Black Box Testing of Sentiment Analysis Classification Networks
    Parsa Karbasizadeh1,
		Fathiyeh Faghih2, and Pouria Golshanrad3 

	College of Engineering, Department of Electrical and Computer Engineering 
 University of Tehran, Tehran, Iran 

		Email: 1parsa.karbasi@ut.ac.ir,
		2f.faghih@ut.ac.ir,
		3pouria.golshanrad@ut.ac.ir
    August 5, 2024
=============================================================================================================================================================================================================================================================================








§ ABSTRACT 
Transformer-based neural networks have demonstrated remarkable performance in natural language processing tasks such as sentiment analysis. Nevertheless, the issue of ensuring the dependability of these complicated architectures through comprehensive testing is still open. This paper presents a collection of coverage criteria specifically designed to assess test suites created for transformer-based sentiment analysis networks. Our approach utilizes input space partitioning, a black-box method, by considering emotionally relevant linguistic features such as verbs, adjectives, adverbs, and nouns. In order to effectively produce test cases that encompass a wide range of emotional elements, we utilize the k-projection coverage metric. This metric minimizes the complexity of the problem by examining subsets of k features at the same time, hence reducing dimensionality. Large language models are employed to generate sentences that display specific combinations of emotional features. The findings from experiments obtained from a sentiment analysis dataset illustrate that our criteria and generated tests have led to an average increase of 16% in test coverage. In addition, there is a corresponding average decrease of 6.5% in model accuracy, showing the ability to identify vulnerabilities. Our work provides a foundation for improving the dependability of transformer-based sentiment analysis systems through comprehensive test evaluation. 


Journal of IEEE Software
Shell et al.: Testing Smart Contracts Gets Smarter

	

		
AI Dependability, Test,Transformers, Coverage Criteria,NLP
	








§ INTRODUCTION


The impact of neural networks on various fields, including image recognition, natural language processing, and self-driving cars, has been significant <cit.>. In the context of Natural Language Processing (NLP), neural networks have gained notable recognition due to their ability to enhance accuracy <cit.>.
Sentiment analysis, also known as opinion mining or emotion AI, is a natural language processing (NLP) technique used to determine the emotional tone or subjective opinion expressed in text.  This procedure utilizes NLP methodologies to analyze and interpret human language, enabling diverse applications across various domains. For instance, in marketing, sentiment analysis can be leveraged to evaluate customer opinions about products and services, providing valuable insights for brand reputation management. 
Various tools, including NLTK <cit.>, are employed to interpret customer sentiment and capture embedded emotions. 

Traditional emotion recognition methods often struggle with the inherent complexities of human language. These complexities include word ambiguity, where a word's emotional connotation can shift depending on context (e.g., “scared” can be positive in an exciting situation or negative in a threatening one). In addition, sarcasm and negation pose challenges in identifying true sentiment.  Furthermore, the process of understanding emotional meaning must include evaluating words that are not directly close to each other in a sentence, which is a step away from traditional methods.

Transformer-based neural networks effectively handle the ambiguity in emotion sentiment analysis through their attention mechanisms and large parameters <cit.>. Parallel text processing allows them to analyze text data efficiently and capture the hidden emotional factors that can exist across longer distances within a sentence. Through pre-training on extensive datasets, transformers are capable of learning complex language patterns and the subtle details of human emotional expression <cit.>.

To guarantee the efficacy of these complex models, comprehensive testing protocols are essential. A significant challenge lies in evaluating the sufficiency of test suites specifically designed for Transformer-based emotional sentiment analysis networks. Given their complex architecture and the large number of parameters, conventional methods like neuron coverage <cit.> seem to be impractical. To address this limitation, we introduce in this work a novel set of coverage criteria designed to evaluate test suites for the assessment of Transformer-based emotion sentiment analysis networks. We employ the principle of input space partitioning, segmenting textual data into emotionally-relevant categories. This strategy enables us to concentrate on particular areas of the input space and generate targeted test cases.

To achieve high coverage with the least number of test cases, we employ the k-projection coverage method <cit.>. This technique utilizes dimensionality reduction by examining smaller feature sets at the same time. Concentrating on these smaller subsets allows for an efficient evaluation of the network's performance across diverse emotional categories within the input space. This strategy ensures comprehensive coverage with fewer tests, thereby eliminating the necessity for extensive test suites.
Additionally, we introduce a set of specialized features specifically designed for emotional sentiment analysis. These features capture the emotional details within text data, thereby aiding the network in accurately identifying the intended emotions.


Upon applying our emotion-specific k-projection coverage method to a dataset designed for emotional sentiment analysis, we observed a 16% average increase in coverage of the generated sentences. This enhanced coverage led to 6.5% decrease in the accuracy of the System-Under-Test (SUT) on the new test suites, demonstrating the effectiveness of our approach in uncovering additional faults.



Related Work:This paper explores concepts related to test generation, black-box testing, and model coverage in DNN models. It summarizes the current state of coverage and performance measures for DNN models and discusses previous research on utilizing diversity in black-box DNN testing to evaluate DNNs and conventional software systems <cit.>.


Transformers are preferred over traditional DNNs due to their superior accuracy, particularly in AI domains like sentiment analysis <cit.>. Evaluating the adequacy of test suites for emotional sentiment analysis networks poses a significant challenge. In software engineering, test suite effectiveness is assessed through black-box and white-box testing <cit.>. Applying white-box criteria to transformer-based NLP systems is limited by their complexity and internal access requirements. Alternative testing methods such as black-box strategies or input diversity metrics may be necessary to ensure their reliability <cit.>.

Input space partitioning is an effective approach for black-box testing, particularly in text data, where emotionally relevant features can be used to partition the input domain into smaller subsets for precise test case design <cit.>. 

k-projection coverage, employing dimensionality reduction, shows promise in image recognition for evaluating self-driving car perception systems using CNN-based neural networks and GAN-based models <cit.>. However, its potential in text-based emotional sentiment analysis remains unexplored.

The contributions of our study are as follows: 

    
  * We introduce novel coverage criteria based on k-projection for black-box testing of sentiment analysis systems. This metric serves a dual purpose: evaluating the effectiveness of existing test suites and guiding the generation of new test cases to augment them.

  * Our experimental results demonstrate that enriching test suites using this proposed coverage criterion leads to a significant reduction in model accuracy. This finding has two key implications:

    
  * The coverage criterion is a valid metric for assessing test suites in this context, as higher coverage scores correlate with a greater ability to uncover faults and decrease model accuracy.

  * The coverage criterion can be effectively utilized to generate effective test cases that reveal failures and thus improve the overall quality of test suites.




The remainder of this paper is organized as follows. We explain the proposed methodology in Section <ref>, followed by a presentation of the experimental results in Section <ref>.  Section <ref> concludes the paper by summarizing our findings and future work.




§ METHODOLOGY


The performance of a neural network is typically evaluated by calculating its accuracy on a test suite. However, accuracy alone is not a sufficient measure of model quality. The effectiveness of testing and the reliability of the results heavily depend on the adequacy of the test suite. Coverage criteria are metrics used to assess how effectively a test suite exercises different aspects of a system under test. In traditional software testing, common criteria include statement, branch, and path coverage <cit.>. However, these traditional criteria cannot be directly applied to learning models due to their fundamentally different architecture <cit.>. To overcome this challenge, we propose a set of black-box metrics for evaluating test suites specifically designed for sentiment analysis tasks.

Our approach originates from Input Space Partitioning (ISP), a black-box software testing technique. ISP involves dividing a system's input domain into distinct subsets or partitions, where each partition represents inputs expected to result in similar system behavior. Test cases are then strategically selected from each partition to ensure comprehensive coverage of the input space.



 §.§ Input Space Partitioning



The primary objective of this study is to assess the effectiveness of utilizing the input space partitioning technique to evaluate test suites for emotional sentiment analysis in Transformer-based neural networks. This technique utilizes the features of the input data to partition the input space into distinct regions, facilitating organization and understanding of the data. <cit.>

The architecture of the neural network model and the nature of the input data require the identification and selection of appropriate features for input space partitioning. Effective evaluation of a test suite requires developing a feature space that covers a wide range of scenarios and real-world instances. 
An effective feature space can be established by adherence to the guidelines proposed in Cheng et al.'s study <cit.>, which outlines specific autonomous driving scenarios that the test cases should encompass and which will be defined subsequently. 



To establish a clear foundation for the framework discussed in this study, we introduce the following notations. These notations are essential to understanding the subsequent mathematical definitions and analyses.



  * Input Domain:
The input domain includes all textual inputs the neural network can receive. All text variations the network must process and respond to are included.
    
    D = {d_1, d_2, …, d_n}

where d_i represents an individual input text.

  * Partition:
In order to understand the existing components and assess the test suite, it is crucial to establish specific partitions and map the input domain within those defined partitions. In this study, partitions are defined as follows:

    P = {P_1, P_2, …, P_k}

    where P_i ⊆ D and P_i ∩ P_j = ∅ for i ≠ j, and ⋃_i=1^k P_i = D.
    

  * Features:
Through the analysis of the input domain and the consideration of the functionality of the system under test (SUT), we can determine the important features that will be used to establish partitions.

    F = {F_1, F_2, …, F_m}

    

  * Test Suite: Representative tests should be selected from each partition to form a test suite.
    
    T = {t_1, t_2, …, t_l}

    where t_i ∈ P_j for some P_j ∈ P.





 §.§ Emotional Features


Input space partitioning relies on specific features to establish partitions, crucial for evaluating test suite effectiveness in neural networks. The upcoming section will outline the specific features employed in this study.

The meaning and sentiment of a sentence can be identified through crucial components such as verbs, adverbs, adjectives, and other linguistic elements. Therefore, features are determined according to these important elements of a sentence.

Verbs strongly influence a sentence's sentiment. For instance, in "She laughed when I saw her," the verb "laughed" implies joy, highlighting its importance in sentiment analysis.

Besides the verb, the adverb in a sentence is another influential feature affecting its sentiment. For example, in "He bites his nails nervously," the adverb "nervously" alone can convey fear or anxiety, making its presence significant for sentence categorization.

Adjectives shape a sentence's emotional tone by describing specific feelings. For example, "She received a joyful gift" conveys happiness, whereas "She received a disappointing gift" indicates sadness.

Nouns contribute significantly to a sentence's emotion. In "The old man sat alone, his eyes filled with tears, mourning the loss of his dearest friend," nouns like "tears" and "loss" contribute to its "sad" emotional label.

In this research, the emotional attributes of nouns were integrated as an additional feature alongside verbs, adjectives, and adverbs.
The sets of features presented in this paper are denoted as:



    F = {S_V, S_ADJ, S_ADV, S_N}


where:

    
  * S_V represents sentiment scores for verbs, categorized into emotional labels: 
    S_V : V →{joy, anger, sadness, fear, surprise}

    
  * S_ADJ represents sentiment scores for adjectives,  categorized into emotional labels: 
    S_ADJ : ADJ →{joy, anger, sadness, fear, surprise}

    
  * S_ADV represents sentiment scores for adverbs,  categorized into emotional labels: 
    S_ADV : ADV →{joy, anger, sadness, fear, surprise}

    
  * S_N represents sentiment scores for nouns, categorized into emotional labels, including a neutral category: 
    S_N : N →{joy, anger, sadness, fear, surprise, neutral}



When dealing with multiple features and their numerous potential values, the number of possible combinations can grow exponentially. This can make generating comprehensive input data, encompassing all features at once, extremely complex. k-projection coverage is a technique used in input space partitioning to overcome this challenge. It aims to systematically explore the input space by focusing on a subset of `k' parameters at a time, while keeping the remaining parameters fixed.

To illustrate this method, consider an example of four features {F_1, F_2, F_3, F_4}, each taking two values {0, 1}. resulting in a total of 2^4 = 16 possible combinations.
Applying k-projection coverage with different values of k results in the following:

    
  * k = 1 (Single Feature Projection): We focus on one feature at a time, keeping the others fixed.

    
  * Number of projections: 4 (one for each feature)
    
  * Number of cases per projection: 2 (the two possible values of the feature)


  * k = 2 (Pairwise Projection): We focus on pairs of features, keeping the other two fixed.

    
  * Number of projections: 6 (combinations of 4 features taken 2 at a time)

  * Number of cases per projection: 4 (2 values for each of the two features)


  * k = 3 (Three-way Projection): We focus on groups of three features, keeping the remaining one fixed.

    
  * Number of projections: 4 (combinations of 4 features taken 3 at a time)

  * Number of cases per projection: 8 (2 values for each of the three features)


  * k = 4 (All Features Projection): We consider all features simultaneously.

    
  * Number of projections: 1

  * Number of cases per projection: 16 (all possible combinations)




 Expanding on the previous example, this approach systematically selects sets of k features and strives to achieve comprehensive coverage within a newly formed reduced dimension. This technique of dimensionality reduction enables efficient test generation, as it requires fewer test cases to achieve complete coverage of the input space.
 
 Generally, to calculate the coverage rate for k = n, where n is the number of problem features, the following formula is employed:


    COV = C(D)/α^n

In the above equation, C(D) represents the coverage rate of the test data over possible states, where α is the range of each feature and n is the problem's dimensionality. In our context, α=6 and n=4, requiring 1296 unique test data points for full coverage. This approach, however, is computationally demanding and requires substantial test data.


 Alternatively, when considering the effect of k, the following formula is used:


    COV = C(D)/nkα^k


 Considering the inverse correlation of the denominator, it is evident that the denominator, which is affected by variables such as k, α, and n, greatly decreases the number of tests needed to achieve complete coverage. 

 Incorporating all four features in sentences can be challenging and costly, reducing sentence clarity. Thus, this approach is ineffective. In order to tackle this issue, the dimensions of the feature coverage space are reduced, with the variable k being set to values of 2 and 3 in this study.The values for the variable k are compared in Section <ref> to evaluate their individual effects.


After measuring the coverage achieved by the test suite to assess the effectiveness of the k-projection metric, we attempt to increase coverage by generating test cases for uncovered scenarios. Ultimately, we explore whether improving coverage aids in the detection of hidden flaws in existing sentiment analysis networks.



 §.§ Evaluation of the Coverage Criteria

Our approach to evaluating the proposed coverage criterion involves examining whether increasing a test suite's coverage score uncovers more failures in the sentiment analysis model. We introduce a method for augmenting test suites with additional test cases specifically designed to improve their coverage score. This evaluation method has two key advantages:

    
  * Validation of Coverage Criteria: Observing a decrease in model accuracy when using the augmented test suite confirms the effectiveness of the coverage criteria for evaluating sentiment analysis test suites. A higher coverage score indicates a greater ability to identify faults and lower model accuracy, thus validating the criteria.
    
  * Practical Test Case Generation: The proposed method offers practitioners a valuable tool for generating effective test cases in sentiment analysis tasks. By focusing on increasing coverage, practitioners can systematically identify and address potential weaknesses in their models.


To uncover unexplored regions within the test suite based on our proposed coverage criteria, we propose a strategy for generating novel sentences. Manual sentence creation, while a viable method for producing desired test cases, is both time-intensive and unsuitable for generating large quantities.

Our idea in this study is to use Large Language Models (LLMs) <cit.> to generate the required sentences. LLMs require training using huge quantities of textual data, which allows them to imitate human-like text production. Examples of such LLMs include GPT-3 <cit.> and LaMDA <cit.>. In this study, we employed Claude 3 Opus <cit.>, an LLM produced by Anthropic, which is recognized for its exceptional sentence generation capabilities.

When utilizing LLMs for test generation, explicit prompts play a crucial role in guiding the model to produce specific types of test data that align with the research objectives. By evaluating the datasets using defined features in <ref>, it is apparent that this dataset does not contain inputs with mixed emotions. For instance, after analyzing the test suites, we identified that the test suite lacks sentences where a verb is labeled with sadness and an adverb is labeled with joy. We generate such a test case using the following formulaic approach:

Let S represent the set of all sentences in the test suite, and define subsets V and A as follow:

    V = {s ∈ S |verb(s) = sadness}


    A = {s ∈ S |adverb(s) = joy}

The required test case would be s' ∈ V ∩ A. To generate the missing test case s', we utilize:

    s' = LLM("Generate a sentence with a verb labeled as sadness


    and an adverb labeled as joy")


This approach ensures that the generated sentence s' fills the identified gap in the test suite, thereby increasing the coverage of the test suite according to our proposed criteria. By adding s' to S, we obtain an augmented test suite S':

    S' = S ∪{s'}


The augmented test suite S' now includes sentences that exhibit the desired feature combinations, leading to a more comprehensive evaluation of the model's capabilities. Additionally, as stated in the evaluation report, these types of situations might occur in user experiences and are viewed as real-life scenarios <cit.>. Therefore, it is crucial to include these generated tests in the test set that showcases these real-life, uncovered scenarios.


For the purpose of generating test cases, we selected the Claude language model <cit.> due to its capacity to generate varied and innovative test sets that are in line with our research goals. This model's attributes enable the generation of test cases that meet research requirements and highlight the robustness and reliability of sentiment analysis.

The generated sentences lack an emotion label, making them unsuitable for use as a test. In order to assign labels to the generated tests, we utilized six of the most accurate sentiment analysis models that were identified in the research literature <cit.>. with their accuracy mentioned in table <ref>. The input sentences were fed into these models for labeling, and the resulting labels were multiplied by their accuracy on the dataset (coefficient). The sentiment with the highest score was assigned as the label for each generated sentence, a method known as “Differential Testing” <cit.>. The system under test was then evaluated using a combination of original and newly generated tests to assess its accuracy on the updated test set.




§ EXPERIMENTAL RESULTS AND DISCUSSION








The dataset utilized for assessing the results is denoted as CARER <cit.>. This dataset contains six primary emotions expressed through text, enabling a thorough assessment of the neural network's ability to accurately classify emotions.
The dataset is split into training and testing sets. The training set comprises 16,000 tweets, each associated with six output labels. Similarly, the test set consists of 2,000 tweets, each labeled with six output categories. A total of 18,000 tweets are chosen as the test set and assessed using the proposed coverage evaluation method. For this study, the dataset is divided into smaller subsets, each consisting of 200 tweets, in order to analyze the coverage capability of each subset. Additional tests will be set based on the uncovered features.

Our model is compared to the existing test dataset since, based on our findings, no other model conducts test generation for neural networks in text categorization in the same way. This study applied the Claude language model <cit.> for test generation and 6 sentiment analysis models for differential testing <cit.>. A distinct arrangement of the DISTILBERT model <cit.> was employed for word-level analysis.The process of learning and loading was executed on Google Colab, utilizing a central processing unit (CPU) with a capacity of 12.7 gigabytes of random access memory (RAM).

We tried different values of k in our implementation to evaluate the effect of k in our approach.
The results presented in Fig. <ref> illustrate the differences in coverage among various implementations of k. When k=2, the coverage increases on average by 14% (Fig. <ref>). However, when k=3, the average increase in coverage is 18% (Fig. <ref>).
The initial coverage for k=3 is 32%, while the initial coverage for k=2 is 62%. This indicates that the test suite lacks sufficient coverage for complicated sentences that involve three combinations of features. Although full coverage was not attained, the sentences generated noticeably raised the total test coverage in comparison to the original data. 

Figure <ref> demonstrates a decline in accuracy for both test generation implementations, suggesting that the generated tests expose previously unexplored areas and have the potential to improve the model's functionality.
When k=2 (Fig. <ref>), the accuracy experiences an average loss of 5%, whereas for k=3 (Fig. <ref>), there is an average decrease of 8%. These findings indicate that the tests created for k=3 demonstrate a higher number of defects in the model's decision-making capabilities.

The average accuracy after test generation for k=3 models is 85%, whereas for k=2 models it is 88%, which is lower than the model's accuracy on the base test suite, which is 94%. This indicates a decrease in the model's precision when tested on these more challenging data points. This drop results from intentionally pushing the model to its limits in order to uncover areas for enhancement.
Based on the experimental results for  k = 3 the augmented test suite exhibits a greater decrease in accuracy compared to  k = 2. This can be attributed primarily to the production of more intricate sentence structures. Nevertheless, when k = 3, there is a greater time complexity and increased costs involved in generating these sentences. The sentences produced for k = 3 exhibit a significant level of complexity and deviate considerably from real-life situations, thereby making it challenging to evaluate the model's performance in practical settings.




§ CONCLUSION AND FUTURE WORKS

This study introduced novel coverage criteria for evaluating test suites of transformer-based sentiment analysis neural networks. By using input space partitioning focused on emotionally relevant linguistic features and the k-projection coverage metric, we efficiently developed test cases and analyzed key feature subsets.
Our methodology demonstrated a 16% average increase in test coverage for all methods and a 6.5% decrease in model accuracy on augmented tests. 
The proposed criteria balance bug detection capability and computational efficiency, surpassing existing techniques. By emphasizing emotional dimensions through input space partitioning, our approach ensures thorough evaluation across diverse real-world scenarios.

Future work could extend our coverage criteria to other transformer-based NLP tasks like text summarization or machine translation. Expanding linguistic elements beyond verbs, adjectives, adverbs, and nouns could improve the detection of subtle emotional nuances, including negations and emojis. Finally, investigating the scalability and efficiency of our methodology on larger datasets and wider applications will be essential for real-world, industry-scale applications.

IEEEtran
 


 Paper Title: An Empirical Study on Challenges of Event Management in Microservice Architectures.
















0000-0003-2771-7477

	University of Copenhagen
	Copenhagen
	Denmark

rnl@di.ku.dk


Work done while employed as a postdoc at the University of Copenhagen.
0000-0003-0936-1542

	State University of Rio de Janeiro
	Rio de Janeiro
	Brazil

ana.almeida@ime.uerj.br



0000-0002-7557-9091

  North Carolina State University
  Raleigh, North Carolina
  USA
wguezas@ncsu.edu


0000-0002-7578-8117

	University of Copenhagen
	
	Copenhagen
	Denmark

zhou@di.ku.dk














§ ABSTRACT 


Microservices emerged as a popular architectural style over the last decade. Although microservices are designed to be self-contained, they must communicate to realize business capabilities, creating dependencies among their data and functionalities. Developers then resort to asynchronous, event-based communication to fulfill such dependencies while reducing coupling. 
However, developers are often oblivious to the inherent challenges of the asynchronous and event-based paradigm, leading to frustrations and ultimately making them reconsider the adoption of microservices. To make matters worse, there is a scarcity of literature on the practices and challenges of designing, implementing, testing, monitoring, and troubleshooting event-based microservices. 

To fill this gap, this paper provides the first comprehensive characterization of event management practices and challenges in microservices based on a repository mining study of over 8000 Stack Overflow questions. Moreover, 628 relevant questions were randomly sampled for an in-depth manual investigation of challenges.
We find that developers encounter many problems, including large event payloads, modeling event schemas, auditing event flows, and ordering constraints in processing events. This suggests that developers are not sufficiently served by state-of-the-practice technologies. We provide actionable implications to developers, technology providers, and researchers to advance event management in microservices.




























	<ccs2012>
	<concept>
	<concept_id>10010520.10010521.10010537</concept_id>
	<concept_desc>Computer systems organization Distributed architectures</concept_desc>
	<concept_significance>500</concept_significance>
	</concept>
	<concept>
	<concept_id>10011007</concept_id>
	<concept_desc>Software and its engineering</concept_desc>
	<concept_significance>500</concept_significance>
	</concept>
	</ccs2012>


[500]Computer systems organization Distributed architectures
[300]Software and its engineering













An Empirical Study on Challenges of Event Management in Microservice Architectures
    Yongluan Zhou
    August 5, 2024
==================================================================================





§ INTRODUCTION






The emergence of cloud computing as a paradigm for large-scale deployment of services has prompted industry practitioners to rethink how applications are architected and deployed <cit.>. In particular, we witness a growing adoption of microservice architectures <cit.>. Microservice architectures promote designing components as independent building blocks that are deployed independently and interact with each other via network protocols <cit.>. This architectural style brings more flexibility for the development and deployment of large systems, in contrast to conventional architectures, in which software components are strongly coupled with each other and deployed as a whole unit <cit.>. 







The main benefit of microservices is that development teams can develop and operate their services independently <cit.>. Microservices should be designed as self-contained software units that operate autonomously <cit.>. To realize business capabilities, microservices need to communicate, which ends up creating dependencies among their data and functionalities <cit.>.

To manage such dependencies while maximizing decoupling, developers often rely on asynchronous event-based communication between microservices, instead of using traditional remote procedure calls <cit.>. The rationale is that, through events, producers and consumers are not directly coupled. For instance, producer microservices can trigger operations in other microservices or communicate their own state updates through events. 
Such an approach may positively influence the application's ability to evolve over time, limit the propagation of faults, and favor increased scalability of individual components <cit.>.


Event-driven architecture has been rapidly gaining industry popularity <cit.>. However, despite the benefits of an event-driven design, developers frequently report challenges in managing events in industrial settings <cit.>. 
Not surprisingly, it is common to encounter developers seeking support on how to implement event-driven microservices appropriately, such as this comment found on Stack Overflow (a popular Questions&Answers forum) <cit.>: “Implementing an eventually consistent distributed architecture has turned out to be a pain. There are tons of blog posts telling stories about how to do it, but not showing (code) how to actually do it. One of the aspects I'm suffering is having to deal with manual retries of the messages when they haven't been acknowledged.”
To make matters worse, in our study, we observed that there is a scarcity of literature on the practices and challenges that the adoption of an asynchronous paradigm brings to the designing, implementing, testing, monitoring, and troubleshooting of microservices. The challenges of managing communication in asynchronous and event-based systems have been explored in the fields of databases and distributed systems <cit.>, but this knowledge is not widely disseminated among developers <cit.>, making them sometimes try to “reinvent the wheel” when addressing common challenges. This limited understanding of existing practices and how to address challenges in architectures based on asynchronous communication leads to frustrations and ultimately makes companies
reconsider the adoption of microservices.[Recent empirical studies have reported cases of companies moving from microservices back to monoliths <cit.>.]
















There has been extensive work investigating the adoption and implications of microservice architectures <cit.>. However, these pieces of work mostly focus on REST (representational state transfer) architectural style, describe the benefits of using microservices, or characterize data management and security practices without a holistic view of the practices and challenges. Furthermore, studies target specific companies or a limited number of practitioners, reducing the generalizability of their findings. 

To the best of our knowledge, no work has systematically investigated the practice and challenges brought about by an event-driven design in microservice architectures. 
As a result, although event-driven microservices form an important portion of microservice deployments today <cit.>, it is unclear what particular challenges developers face when developing and operating them. The need to address these issues is even more relevant when we consider studies suggesting there is a tension between the use of asynchronous microservice designs and application safety <cit.>.




This work aims to characterize the practices and challenges faced by developers when adopting event-based microservice architectures.

More specifically, we first investigate the practices adopted by developers to manage events in microservice architectures. We aim to comprehend current technological trends and common implementation patterns related to the adoption of event management. We also seek to identify the functional and non-functional requirements that are met or supported by implementing event management in microservices. Secondly, we want to identify recurrent challenges faced by developers when managing events in microservice architectures. Focusing on these two perspectives, we can equip developers with knowledge of existing practices for event management in microservices and make them aware of the challenges they will face when developing asynchronous and event-based microservices. 



To achieve our goal, we perform, to the best of our knowledge, the first empirical study to comprehend the practice and identify the challenges in managing events in microservice architectures from developers' perspective. 
To this end, our study is based on repository mining <cit.>. We mine and analyze relevant questions from Stack Overflow (SO),[<https://stackoverflow.com/>] one of the most popular Questions&Answers forums for developers who seek technical advice or assistance <cit.>. 
We collect more than 8,000 SO questions associated with managing events in microservice architectures, ensuring the diversity, representativeness, and quality of our dataset. Via a mix of keyword analysis and manual examination of tags, questions, and related responses (i.e., posts), we collect the most popular patterns and non-functional and functional requirements mentioned in the context of event management in microservices.


Then, we randomly sample 624 relevant SO questions for manual analysis to identify the challenges. We carefully extract the challenges behind each question through a peer-reviewed process. We classify the uncovered challenges into different properties of distributed systems and functional and non-functional requirements. These ultimately represent key areas that microservice developers struggle with. 


In the quantitative analysis, we observe that event management has been gaining increasing attention in practice, indicating the emergence of this architectural paradigm and the timeliness of this study.
Results show the popularity of patterns and requirements related to achieving data consistency and loose coupling. This suggests microservice developers seek to strike a better balance between consistency and decoupling in their microservice architectures by employing asynchronous events, especially in computations that span microservices. 
Furthermore, we observe that developers report different categories of patterns in their implementations. For instance, observability (e.g., distributed tracing), performance (e.g., circuit breaker), and security (e.g.,  access token) patterns are reported being applied in questions involving event management. 


In the qualitative analysis, we find that microservice practitioners face a myriad of challenges in managing events. While events are supposed to maximize the decoupling of microservices <cit.>, developers often surprisingly necessitate synchronizing events for correct event processing and coordinating microservices to allow for software evolution, contradicting the alleged benefits of an event-based architecture <cit.>. Besides, although events can serve as natural progress markers of microservices <cit.>, we observe that developers find little benefit in using events to monitor and troubleshoot microservices. In particular, developers have a hard time tracking down the result of their computations, which often span a network of dependent microservices and exhibit additional difficulties in replaying past events for debugging purposes.
Safeguarding security properties is also an emerging challenge in event-based microservice architectures. Developers struggle to synchronize event management and security technologies to ensure that only events generated by secured channels are processed and that only authenticated microservices can consume them.


This paper's contributions are manifold. We evidence that microservice developers are insufficiently served by state-of-the-practice technologies, including messaging systems, frameworks, databases, and cloud providers. They end up implementing their own ad hoc solutions to fulfill their requirements, which ultimately leads to errors and bugs, rendering adoption of a microservice architecture frustrating. 
Characterizing this emerging practice well informs general software developers, especially those unfamiliar with the challenges of designing distributed systems, about the hidden dangers of asynchronous, event-based microservice designs. We derive actionable implications for each class of event management challenges to researchers and message, database, cloud, and framework providers. These relate but are not limited to, devising novel microservice programming models, event management architectures, tools, and methods to advance the state-of-the-art practice in managing events in microservice architectures, making this a timely and valuable study.
Furthermore, the results of our work can benefit different stakeholders involved in the development and management of event-driven microservice architectures:

    
  * For practitioners: the challenges reinforce the importance of reliable sources of information and the need for careful analysis of the documentation of the technologies they adopt. Practitioners should question gray literature with general and high-level instructions that often do not embrace the complex nature of problems faced in production settings. Practitioners should not overlook problem-prevention measures, which include running, testing, and deploying their applications under the expected workload prior to production. Besides, developers must be cautious about “reinventing-the-wheel” approaches, since for every possible design attempt, there are reports of previous attempts. 

    
  * For researchers: software engineering, system engineering, and database communities must work closely to each other. Leveraging the knowledge from other fields can help developers address their challenges. Software engineering researchers may find no appropriate solutions for their problems in tools from system engineering, requiring only appropriate interfaces while safeguarding the application. Also, researchers must embrace/meet practitioners' expectations that were analyzed and discussed in the paper, solving real-world problems without focusing on “in vitro” experiments. 

    
  * For tool builders: documentation must pair up with/highlight the problems faced in practice. They should also provide actions that can be taken to mitigate the shortcomings of the existing tools. Furthermore, vendors must be clear about what requirements and guarantees cannot be properly achieved.
        
    
  * For educators: we advise incorporating distributed systems concepts in software engineering courses, in particular algorithms and systems guarantees and their relationship with modern application development in the cloud. Yet, training students on how event management relates and makes its presence in modern application architectures, such as serverless, microservices, SaaS, to name some. 


In addition, the dataset used in this
study is made available [<https://zenodo.org/records/13149520>] as an additional contribution to the research community, allowing other researchers to further investigate the theme.





The remainder of this work is organized as follows: Section 2 describes the role of events in modern microservice architectures and typical issues developers encounter. In Section 3, we present the methodology employed in this work. Section 4 presents the state of practice. Section 5 presents the specific challenges microservice developers encounter in managing events and their implications for the research community and industry providers. Section 6 discusses the threats to validity. Section 7 presents the related work. Lastly, Section 8 concludes this work.









§ BACKGROUND








To discuss the role events play in microservice architectures, we use Figure <ref> to illustrate an ecommerce platform. We base on a popular microservice open-source project <cit.> that incorporates real-world event processing patterns.



 §.§ Context


Upon a customer's checkout request, the Cart microservice spawns the execution of a checkout through generating a  event. The event is then consumed by the Order microservice, which processes the cart items and generates the  event. Next, the Payment microservice is responsible for processing the payment and generating the  and  events for downstream consumption by the Stock and Cart microservices, respectively.

It is noteworthy that, in synchronous communication paradigms, such as through REST APIs and RPCs, the requester microservice gets blocked until the request terminates in the remote microservice and the response is received. It is natural to deduce that such blocking times can grow arbitrarily when business transactions traverse several microservices. In the example above, Cart would necessarily wait until Payment terminates the payment processing. Through the asynchronous paradigm, though, once the  event is generated, Cart microservice is free to allocate computational resources (e.g., threads, CPUs, and memory) to serve other requests. This usually favors higher efficiency in event processing 
 <cit.>.

As exhibited in Listing 1, the communication through events is only possible because events carry semantic data that is used by microservices in distinct ways <cit.>. For instance, although the payment data is sent by the customer in a checkout request, such data is only used by the Payment microservice. That requires both Cart and Order microservices to pass along payment data in the  and  events, respectively.



[caption=Example of application-generated event types,xleftmargin=.25,label=lst:event_types,language=rock,captionpos=b,basicstyle=0.1]]
product_update: product_id, type, description, ... 
price_update: product_id, old_price, new_price, ... 
checkout_cart: customer_id, items:[ product_id, qty,  .. ], 
    card_num, exp_date, address, discount 
update_stock: product_id, qty, timestamp, ... 








 §.§ Problem Statement


Besides the apparent attractiveness of a decoupled design, managing events appropriately renders challenges. As events are generated asynchronously, that is, producers do not wait for consumers' reception, developers must account for the anomalies that possibly arise in event delivery, as described by the microservice adopter Nubank <cit.>:





“you have late-arriving events, you can make a payment that we receive that should be credited on Friday, we need to time travel back, re-play the events and figuring out the balance that does not invalidate the invariants we have”


Besides, it is often reported that developers necessitate making sure anomalies in the event order does not lead to problems in their microservices, as explained by an Uber team <cit.>:


“If it consumes an event that is not in sequence, our processing logic identifies the version mismatch and we retry the event a number of times.”





Turning to our example scenario, if  events are processed in different orders, that can lead to incorrect product prices. Besides, supposing that the  is not delivered to and processed by both Order and Stock, that leads the system to an incorrect state. In addition, if  is delivered more than once, that can lead to overcharging customers. An abnormal burst of events can also pose challenges to the system if consumers cannot cope with the arrival rate of events. In this work, we delve into these and several other types of challenges that affect event management in microservice architectures.





§ STUDY DESIGN


Although event-driven microservices are a popular architectural style in industry settings <cit.>, existing studies mostly outlook the practical implications of adoption and the challenges brought about by an asynchronous and event-based design to microservices. Thus, the goal of this work is to identify the practices adopted and challenges faced by practitioners developing software systems using event-driven microservice architectures.




To achieve this goal, our study focus on answering two research question, as follows:


    
  * RQ1. What is the state of the practice on managing events in microservice architectures?
    Rationale: Despite the popularity of event-driven microservice architectures, in the literature it is little known on why practitioners adopt this architectural style. Also, the literature is scarce on describing what are the practices adopted by them to manage events in microservice architectures, allowing practitioners to understand technological trends and popular implementation patterns associated with event management adoption. Thus, with this RQ, the insights we want to gain is threefold: (i) Understand the popularity trend of event management in microservice architectures; (ii) Collect the most popular patterns associated with event management adoption; (iii) Identify what functional and non-functional requirements are either enabled or facilitated by managing events in microservices.
    
    
  * RQ2. What challenges do practitioners encounter when managing events in microservice architectures? Rationale: Once we understand the current practices on developing software systems using event-driven microservice architectures, we focus on identifying what particular challenges developers face when developing and operating them. In this question, we delve into the specific issues and impediments microservice developers face in practice while managing events.


To answer these two research questions, we use Stack Overflow (SO) <cit.>, a popular question-and-answer online platform where practitioners seek technical assistance on issues they face in their day-to-day activities to understand the challenges of managing events in microservice architectures.
Our methodology is similar to the ones adopted in previous studies <cit.> to collect, filter, and analyze questions. However, our focus is on event management in microservice architectures.
We detail the procedure of our study below.











 §.§ Data Collection








  §.§.§ Downloading Stack Overflow dataset


We started by downloading the entire Stack Overflow dataset S_all from the official Stack Exchange Data Dump <cit.> available when we started this study (September 15, 2023). The dataset includes 23.199.461 questions dated from July 31, 2008 to December 31, 2022.[We excluded the year 2023 from our analysis as it was still a year in progress when this work was carried out] Every question in the dataset has title, body and a “tag” metadata, which denotes the topics on which the question lies.












  §.§.§ Exploring initial tag set


We start with a general tag set to include as many relevant questions as possible. Thus, we use T_ini = {“microservice”}, resulting in 8369 questions, from which we performed an exploratory search. We observed that many questions were unrelated to event management, requiring significant effort to filter out. Also, we observed that relevant questions would mention the keywords “microservice” and “event” in either the title or body.



  §.§.§ Filtering by relevant keywords


With the above insight in mind, we filtered the questions in T_ini that contain the keywords “microservice” and “event” in their title or body, leading to a total of 1407 questions (denoted as S_rel). 

We extracted 566 tags from S_rel, which we denote as T_key. Table <ref> shows the distribution of questions per tag in S_rel (only the tags with five questions or more). 

The first two authors jointly examined the candidate tags and observed that many of them relates to techniques, patterns, and technologies applied to managing events in microservice architectures, such as event-driven architecture, pubsub, message queue, and event sourcing. Even tags unrelated to event management (e.g., authentication, logging, and database) were often accompanied by event management-related tags, indicating possible correlated issues. Both observations gave us the confidence to proceed with further analysis.










 §.§ Analyzing the State of the Practice









  §.§.§ Patterns Trend


Based on a popular collection of patterns for microservice architectures <cit.>, we extracted from the questions in S_rel the patterns for microservices mentioned by developers. It noteworthy we also included in the analysis the different users' posts (i.e., responses) for each question, totaling 3142 entries.
We took extra care analyzing the dataset to embrace all possible forms of writing the same pattern (including typos), synonyms, and acronyms. For instance, we also identified the pattern "Command Query Responsibility Segregation" (CQRS) via "cqrs" or "CQRS." Section <ref> answers RQ1(b).





  §.§.§ Non-functional Requirements Trend


We also analyzed the most common non-functional requirements (NFRs) mentioned by practitioners.
We use as reference the NFRs listed in Chung et al. <cit.>. We started searching keywords related to NFRs terms in S_rel (including posts) and incrementally introduced acronyms and synonyms.
After introducing each keyword, we evaluated randomly selected related questions to validate whether the classification of the NFR fit the questions' context. We continued this process until exhausting the set of possible NFRs.

We understand that an NFR keyword search does not necessarily mean that such a requirement is part of the practitioners' problem. However, our analysis observed that such cases are often exceptions rather than rules, thus not undermining the analysis of the most pursued NFRs regarding event management in microservices. We respond to RQ1(c) in Section <ref>.



  §.§.§ Functional Requirements Trend


We were also interested in understanding the specific functional requirements pursued by practitioners that are either enabled by or facilitated by managing events. Unlike NFRs, though, we noticed that uncovering functional requirements often necessitated a thorough, detailed analysis to uncover the application scenario and, consequently, the functional requirement related to event management.

Relevance Filter. Given the high number of questions, manually examining all of them would demand a substantial amount of time, which could make reporting this study's evidence in a timely fashion impracticable. Therefore, at this point, we decided to apply a relevance filter.
First, we use a relevance heuristic ν to filter out questions with tags that are employed to a much lesser extent compared to others. As previous studies <cit.>, we considered only the tags whose ν is higher than 0.005, leading to a set of 1298 questions and 93 associated tags.


    ν = # of questions with tag t in S_rel/# of questions in S_rel


Next, we focused on prioritizing the analysis of questions tagged with event management technologies (e.g., rabbitmq, kafka, masstransit) and techniques (e.g., cqrs, event-sourcing, event-handling) rather than questions tagged with technologies not directly related to event management (e.g., python, java, php, elasticsearch). Although the former also presents problems in the event management domain, thus part of our problem scope, we observe filtering by related event management tags led more often to questions revealing challenges within our study scope. 

These two filters combined led to 53 tags (shown in Table <ref>) with a total of 925 questions. To proceed with the manual examination, we randomly sample 628 questions, which represents more than 2/3 of the relevant questions filtered.




Manual filtering. 
To ensure we consider in our analysis only questions that are representative to our study goal, we randomly sampled more than 2/3 
of the 925 questions to examine manually. 
624 randomly sampled questions.
267 included as challenges

In particular, we were interested in discarding questions unrelated to event management that were not embraced by the previous steps. For instance, we excluded questions seeking support to migrate from monolithic to microservice architecture that does not incorporate event management as part of the problem setting. Existing studies <cit.> have already explored this problem, and it lies outside our scope.
We also discarded questions lacking technical issues but including general inquiries about well-known concepts such as event sourcing and CQRS. That would prevent us from effectively understanding real-world technical challenges practitioners face.










Procedure. We adopt an open coding procedure <cit.> to analyze the sampled questions to pinpoint the specific functional requirements sought by practitioners inductively.

The first two authors, both of whom have years of software and data engineering experience, jointly participate in the manual examination of the questions. They analyze the sampled questions multiple times to familiarize themselves with them. In this process, many elements of the question were taken into consideration for inspection, including the title, body, code snippets, URLs, and the author's responses to other users' inquiries, which can ultimately clarify unclear points of the original problem statement (i.e., content in the body). 

We observed that the user explicitly expressed the functional requirements being sought in most of the questions. For instance, in data replication cases, users would often mention that an event is generated based on some update (e.g., a user credit card score), so other microservices can also apply this change to their databases (see S3 in Section <ref>).

However, in other cases, the specific functional requirement is not made explicit. For instance, different terms, like "consistency," are used to denote replicating data as a functional requirement, like the example as follows <cit.>:



Customer and Order microservice both have customer details, though in Order microservice customer infos are striped to only required fields. I understand there is a way to maintain consistency of data across microservices using events.


Therefore, upon clarifying the functional requirement, the authors give terms to represent the requirements of the questions, using the terms used by developers whenever possible. We also found cases where a question contains more than a functional requirement. This way, we count the number of functional requirements independently of the number of questions. Section <ref> answers RQ1(d).



 §.§ Characterizing the Challenges




We use the same randomly sampled question set used above to analyze specific issues that developers report and characterize the challenges of managing events in microservice architectures.

Again, the first two authors adopt an open coding procedure and analyze all the elements of each question, including comments from users other than the author's question, to carefully extract insights about practitioners' challenges. The detailed procedure is as follows.

First, we ensured the practitioner's requirements were properly clarified. The reasoning is that we noticed the user often reports a problem faced subsequent to or in the context of expressing the non-functional and functional requirements sought. Since we already had the questions labeled with requirements through the previous methodology steps, our primary task remained to understand the challenges from the problem description and other question elements.

In many cases, the challenges could be identified by questions raised in the form of "how" or "what," as in other studies <cit.>. For example, one user asked <cit.>: "How can I process the event with the user credentials?" 
; and another user inquired <cit.>: "What's maybe not a good idea in trying to recover the current state of your domain model by replaying an arbitrary set of your events 
."

We also observed, though, in other cases, the users would not raise a question, but rather: (a) express the willingness to achieve certain functionality <cit.>: "I want to create a third microservice that is responsible to join the data of ProductService and StoreService." 

(b) describe the possible cause of a problem <cit.>: "In my system two different sources can cause creating specific type of event. [...] due to replication lag" 

(c) inform a desired correctness criteria not being met <cit.>: "The end price should be 100 for that product, but sometimes these events are processed in random order." 


In a few cases, the question's body lacks detailed information about the user's application scenario. In those cases, we searched for additional comments from the author's post made as a response to questions raised by other SO users. In the absence of further details about the user's problem scenario, the authors made the best effort to characterize the reported issue, jointly discussing whenever a disagreement was in place.

Second, once the problem statement was clarified, the authors started grouping similar problems into categories. The authors jointly iterated multiple times over the questions and categories. Whenever conflicts were observed in grouping the questions, a third arbitrator was introduced to discuss and reach a consensus. The third arbitrator has more than fifteen years of experience in cloud computing and data engineering. Lately, all questions come to an agreement and the final categories (i.e., the challenges) are confirmed by all the participants. Section <ref> answers RQ2(a).






  §.§.§ Determining Key Computing Areas.


We aim to establish which are the major disciplines that require advancements to effectively fulfill practitioners' needs in the near future. Thus, based on the ACM Computing Classification System (CSS) <cit.>, we group the specific challenges into computing areas. 

We use the ACM's guide <cit.> on how to classify works and the metadata extracted from the questions, as explained in the previous methodology steps, to group the challenges into computing areas. We use a process similar to the one used in classifying the challenges, that is, iterating over the questions' labeled metadata and engaging in consensus upon disagreements.

We explain our reasoning and describe the areas in the context of event management in microservice architectures in Section <ref>. 












§ STATE OF THE PRACTICE (RQ1)



In this section, we describe the state of practice based on the discussions found in SO. 




 §.§ Popularity Trend



Figure <ref> shows the trend, in terms of number of questions and users, related to event management in microservices in SO. We observe that the topic has been gaining increasing attention since 2014, achieving its peak in 2021. The period between 2014 and 2017 shows exponential increase of questions and stabilizes from 2018 on. 
We conjecture that the popularization of and the subsequent increased offering of message brokers and event streaming platforms in major cloud providers [Kafka started being offered by Amazon AWS in 2018 <cit.> and Service Bus on Azure in 2016 <cit.>], 
motivated developers to adopt event-driven architectures for microservices, thus leading to the observed trends.

















 §.§ Patterns Trend



Table <ref> exhibits the most recurrent patterns that appear in questions related to event management in microservices architectures [For improved presentation, we show only those patterns appearing in 20 or more questions.]. In the top ten most mentioned patterns, four are directly related to event management (messaging, event sourcing, domain event, cqrs), one related to synchronous communication (RPC), two related to deployment patterns (database per service and service-per-container), and three related to typical microservice architectural patterns (API gateway, service registry, and aggregate).

We can observe that although event management patterns dominate the list, patterns unrelated to event management also appear substantially. For instance, remote procedure invocation, a synchronous communication technology, appears as the second most cited pattern. As events are processed asynchronously, the employment of synchronous communication mechanisms contrasts with the pursued benefits of events, which are often related to more efficient usage of computational resources and decoupling <cit.>. This trend is mainly due to the need for developers to reply to users the result of asynchronous computations, as exemplified in a developer's quote below:


“How is it possible to send the results stored in the kafka topic back to the requesting WebClient_X?”


These challenges are further discussed in Section <ref>. Popular microservice patterns appearance in the top 10, namely, API gateway and service registry, can be justified by the need to allow web clients to initiate asynchronous computations and expose certain APIs to the external world, and to allow for service discovery

, respectively. It is worth noting these are not concerns event management tackles in an event-driven architecture.

We also observed that the patterns Database per Service and Service-per-Container are often used synonymously by developers. The popularity of these two database deployment patterns suggests developers tend to avoid deployments that prescribe two or more microservices sharing the same database. For instance, the messaging pattern is only mentioned together with Shared Database is 12 occasions. We further discuss this trend in Section <ref>.




Highlight: Adopting a shared database pattern would jeopardize the benefits of decoupling through events. The microservices would compete for computational resources of the machine (or container) hosting the database server, leading to decreased performance effects.





To aid our analysis, we grouped the patterns into pairs and count the number of times they are cited together in questions, as exhibited in Table <ref>. In general, the pairs' appearance is aligned with the individual numbers of Table <ref>. Event management patterns, such as messaging, event sourcing, CQRS, and domain event, appear substantially is pairs followed by Remote Procedure Invocation and Database Per Service. Other interesting insights are also confirmed, like the popularity of an authentication mechanism together with service discovery.
Furthermore, the combination of pairs sheds light on the most common set of patterns considered in questions involving event management in microservices. Messaging, event sourcing, Database per Service, CQRS, Service Registry, and Third Party Registration are substantially mentioned in conjunction in questions, suggesting a pattern adoption trend.

To further aid our analysis, we grouped correlated patterns into categories shown in Figure <ref>. Patterns associated with data management form the most prominent category, followed by communication patterns, application architecture, support-service, deployment, security, and observability. We observe three subgroups within data management, including patterns associated with data consistency enforcement, database deployment, and data querying.
Interestingly, in data consistency, we observe not only event-related patterns, such as domain event and event sourcing, but also SAGA and eventual consistency. That suggests developers also look for event management as an alternative to traditional, synchronous-based mechanisms for implementing consistency patterns, confirming the preliminary findings of <cit.>. 






Highlight: Asynchronous events can enhance the performance of data consistency patterns by allowing for non-blocking interactions across microservices. That can lead to higher performance.




In the application architecture realm, event-driven architecture (EDA) is substantially mentioned, an expected trend given the scope of the study. However, Domain-Driven Design (DDD) also appears to a lesser extent. We found that only twelve questions mention EDA and DDD in conjunction, suggesting these two patterns are not popularly adopted together when it comes to event management in microservices.




Highlight: As EDA often prescribes events trigger specific functions <cit.>, our analysis suggests that this direct mapping between an event type and an application function (aka business logic) demotivates the modeling of extra modularity layers as prescribed by DDD.




We also observe the representativeness of patterns associated with deployment, such as sidecar and service-per-container, security with access token, and observability with log aggregation. The categories and related patterns exhibited in Figure <ref> form the bulk of patterns cited by microservice developers. 
We explore further these phenomena in Section <ref>.



The most cited patterns (more than 50 posts) in posts about event-driven architecture involve communication style, application architecture, data, security, deployment, cross-cutting and observability concerns. Figure <ref> shows this summary and table <ref> details the number of posts by pattern.
We extended the pattern diagram available in pattern language for microservices[https://microservices.io/patterns/index.html, accessed October 2023] and placed it in boxes with a gray background in the figure.

We consider that this set of popular patterns needs greater attention from the microservice community since these patterns are discussed when exposing the difficulties presented by developers. Perhaps there is a lack of more explicit examples in source code so that developers may use them in practice. For example, the following fragment, extracted from the SO, exposes this deficiency in the eventual consistency pattern:
[42230797]
Implementing an eventually consistent distributed architecture has turned out to be a pain. There are tons of blog posts telling stories about how to do it, but not showing (code) how to actually do it.











Finding 1: There is a specific subset of patterns that appear frequently in event management questions. These mainly relate to database resource isolation, data consistency, and asynchronous messaging.
Questions involving event management are also accompanied by mentions of patterns not directly related to event management, suggesting users often attempt to blend heterogeneous patterns in their deployments to solve cross-cutting concerns such as observability and security.










  §.§.§ Eventual Consistency


There are users who explain that their domains support the use of eventual consistency, allowing events to notify about data updates even if they are a little late:
[57222357]
A brief delay in synchronization is acceptable (eventual consistency).


But implementing eventual consistency is not a trivial task because developers lack implementation examples for it. This is the case described by the developer in the following fragment extracted from SO:
[42230797]
Implementing an eventually consistent distributed architecture has turned out to be a pain. There are tons of blog posts telling stories about how to do it, but not showing (code) how to actually do it.


On the other hand, there are users who are using microservices architecture but need the consistency to be strong. This is the case for the following user:
[59698363]
[...]but in this scenario [betting/accounting], the use case of creating a bet heavily relies on a non-dirty read of the users projected account balance (strong consistency).


We would like to bring a discussion here about the real need for these domains to have strong consistency. One of the SO users received an interesting comment about his/her question:
[53807274]
[...] Lots of people have issues with eventual consistency, but they don't blink an eye when introducing a cache. But the cache is never 100% in sync with the data-store, so the data that you're retrieving from the cache is already not 100% consistent with the data-store and thus eventually consistent. [...]





 §.§ Non-Functional Requirements



Table <ref> exhibits the extracted non-functional requirements from SO. We discuss the results along with key correlations found among NFRs as follows.







Consistency and Decoupling. Consistency appears to be the most cited NFR in event management questions in sync with the microservice patterns collected in the last section. A popular concern we noticed in this regard is the unmet expectation of certain events to arrive. For instance, one developer mentions a consistency problem observed whenever the delivery of events lags behind <cit.>:





"When an event isn't received in the worker, the counters start to "drift away" from the true MySQL count.[...]"


We observe that most developers express awareness that there is a natural delay to be expected. In a few cases, though, concerns over "how up to date" a given microservice is  are implicitly raised, as exemplified by the same developer, as follows: 

[]

It is expected that there is a consistency delay for this type of data. How up-to-date the data is can even be figured out and included in the responses for stats data.








We further discuss the tension between the eventual delivery of events and application correctness in Section <ref>. Furthermore, we found that decoupling is the NFR most correlated with consistency. Analyzing the posts in which consistency and decoupling are mentioned together (45 posts), we observed that developers often identify trade-offs in pursuing the two NFRs in conjunction. The following extracted SO thread exemplifies this trend. A developer presents its application requirements starting with decoupling concerns <cit.>:


While each microservice generally will have its own data - certain entities are required to be consistent across multiple services. 


I do not want shared database architecture, where a single DB manages the state across all the services. That violates isolation and shared-nothing principles.


And then highlights a possible mechanism to ensure cross-microservice consistency <cit.>:


"I do understand that, a microservice can publish an event when an entity is created, updated or deleted. All other microservices which are interested in this event can accordingly update the linked entities in their respective databases." 


However, the developer realizes some inner drawbacks <cit.>:


"however it leads to a lot of careful and coordinated programming effort across the services. Can Akka or any other framework solve this use case? How?"


Another developer responds the thread acknowledging the tension between decoupling and consistency and the lack of principled solutions to the problem <cit.>:


I think there are 2 main forces at play here:decoupling - that's why you have microservices in the first place and want a shared-nothing approach to data persistenceconsistency - if I understood correctly you're already fine with eventual consistencyI don't know of any framework to do it out of the box, probably due to the many use-case specific trade-offs involved.







In the same vein as the response above, another developer stresses an incompatibility between maintaining consistency and having decoupled microservices:

[65983977]

[...] For example, the last item that was promised to the customer was damaged during packaging. It would require the order to be canceled or delayed. 


[65983977]
Handling of such a situation is not much different from the sales service handling not up to date information about the item availability. 


[65983977]
Handling all such edge cases is practically impossible when using queues and independent services.



Performance and Scalability. Consistency and decoupling NFRs lead the number of post appearances in SO. However, taken together, they are outperformed by the pair performance and scalability. We observe developers express concerns on varied performance topics. For instance, both developers below face a problem when the event processing rate exacerbates the processing capacity of consumer microservices. We delve into performance issues on managing events in Section <ref>.


(DEV#1) Service A listens to a rabbit queue and sends http request to service B (which takes a couple of seconds). Both services scale based on the number of message in the queue.
The problem is that the requests from A to B are not balanced. [...]
That obviously causes low performance and timeouts. <cit.>



(DEV#2) My problem is the queue. I can't get an easily scalable queue that guarantees ordering of the messages. It actually guarantees "slightly out of order" with at-least once delivery [...] <cit.>


The same developer continues then:


"But it turns out that using this solution, it will destroy performance when events are produced at high rates (I can use a visibility timeout or other stuff, the result should be the same)."








Fault tolerance. We also observe developers consider the adoption of event management technologies as a mechanism to increase fault tolerance in their microservice architectures. In the example below <cit.>, a developer inquiries other users about whether Kafka can provide appropriate fault-tolerance support. We discuss about failures and their possible impact on microservices in Section <ref>.



"I am working in a project that starts creating independent deployable services. The service we are creating should be resilient with an 24/7 uptime." 



"In this case Kafka will be used as an event system. What do you think about the requirements and the usage of Kafka to get a highly available and resilient application?"





Finding 2: Consistency, decoupling, and performance appear as the most mentioned non-functional requirements. However, developers suggest there are trade-offs on properly meeting them in event-driven microservice architectures.






Performance and Scalability (Others examples)
(56920178) ganho de performance-> scale!
(75550874) partitioning -> throughput/performance -> scale! (event)
(31989715) pattern -> atingir ambos requisitos + resiliency

Consistency - another example
For instance, a developer explains a data inconsistency scenario when a failure occurs and asks how to prevent or enforce consistency at the application level:
[30213456]
User Bob is created but before our Wallet can be created, our API gateway hard crashes. We now have a User with no wallet (inconsistent data). [...]
What solutions are available to prevent this kind of data inconsistency from happening? [...]
Or should I enforce consistency in my application code


We noticed that most users who express concern about system performance is related to querying data present in more than one microservice. This statement is more evident when we analyzed pairs of non-functional requirements cited together. The most cited pair is performance and decoupling (16 questions), which suggests that possibly decoupling services may impact system performance if not well designed. For instance, a user highlights the ease of performing a query in a monolithic system and the impact on performance when the same query needs to be performed in the microservices architecture.
[32217639]
I want to do a simple query : retrieve the customers from US that bought product A. Given that 3million people bought [Order Management Microservice] product A [Product Management Microservice] and we have 1 million customers [Customer Authentication Microservice] in the US; How could you make that reasonably performant? (Our current DB would execute that in few milliseconds)


When developers mentioned performance and decoupling, we identified that many questions were about data replication through events or the high number of requests between microservices to retrieve data.

Load Balancing and Scalability are most cited with the performance non-functional requirement. For example, a user of SO is working in a project with a level which load balance the request and s/he needs that the performance achieves 200 transactions per second:
[68452646]
Now we've three data centers (DC-1, DC-2 and DC-3) on top of this DC we've AKAMAI level which load balance the request and its works like round robin. [...] 
Or can we think of Redis implementation or Apache Kafka ? Will that be so performant to cater the 200 TPS request?


An interesting example of a question from an OS developer is described below:
[53270770]
This seems to me to guarantee high scalability and availability. 
My problem is the queue. I can't get an easily scalable queue that guarantees ordering of the messages. It actually guarantees "slightly out of order" with at-least once delivery: to be clear, it's AWS SQS. [...]
But it turns out that using this solution, it will destroy performances when events are produced at high rates (I can use a visibility timeout or other stuff, the result should be the same).

The developer makes it clear that he is looking for scalability and availability. Additionally, the developer demonstrates the difficulty in maintaining good performance given the high number of events produced for the queue and the need to maintain the order of events.




 §.§ Functional Requirements



In this section, we discuss the most recurrent functional requirements sought by microservice developers on which event management is applied (denoted by FR[0-N]). Table <ref> summarizes the number of functional requirements per question.

r8cm

    Relevant Functional Requirements extracted from Stack Overflow
      
        Functional Requirement     # of Questions 

        FR1. Propagation of state updates     78 

        FR2. Multi-microservice workflows     66 

        FR3. Data Integrity Maintenance     33 

        FR4. Replaying of events     22 

        FR5. Query Processing     20 

        FR6. Data replication     17  

        FR7. Cache management     12 

        FR8. Task Scheduling     5 

        
  


FR1. Propagation of state updates. We observe that the majority of questions contains a requirement related to propagating state updates in form of events. A developer summarizes this practice as follows <cit.>:


"A microservice can publish an event when an entity is created, updated or deleted. All other microservices which are interested in this event can accordingly update the linked entities in their respective databases."


Here we only take into account the questions where the developers' intention of propagating the events are either not expressed or not clear from the discussions. However, it is natural to deduce these state updates in form of events are used by other microservices to achieve other requirements, as we discuss next.





FR2. Multi-microservice workflows. We also observe a substantial number of developers reporting use cases where a business transaction require a composition of microservices <cit.> via events. 

For instance, question 42140285 exhibits that an event generated by a microservice triggers an operation in another microservice <cit.>:

"Order [microservice] receives an order request. It has to store the new Order ([record]) in its database and publish a message so that Payment service realizes it has to charge for the item"


Differently from the previous scenario though,
we observe developers in this case have the expectation about the completion of the triggered computations.

In sum, in this functional requirement, the generation of an event act as a command, a call for action, for downstream microservices, manifesting the need to perform operations, often involving the operations in their private states.

FR3. Data Integrity Maintenance.



Case 2. Invariant enforcement.



Similarly, by subscribing to state updates in the form of events, microservices can ensure constraints are held in its own state.

For example, in question 64491114, a developer explains how associated records are identified and removed after an event is received:

[...] when project service emits "Project deleted event", task service would not only clear all task with given projectId but also removes stored projectId so task creation is not possible.


In a similar way, question 45167593 highlights that

every time a user is promoted to host in EventService, an event would be posted [...]. PostService could subscribe to the event and remembering this fact when a request to delete a post is received.




We observe three types of constraints 
in the dataset: uniqueness (56730517), referential integrity (aka foreign keys), and increment. TODO explain each.
Although the idea of enforcing constraints based on the incoming events shows apparent harmless, we find many cases suggesting the opposite.

By resorting to events to update their private states, microservices inherit ordering-related challenges, as summarized by a developer:






There are X questions (x%) on which developers express uncertainty when using events as an abstraction to ensure data integrity.

Case 3. Cache freshness semantics.

53237306
57222357



We also found questions that developers highlight how an event is used to maintain the integrity of a microservice's state. An example is provided as follows <cit.>.


"I have a web-api-endpoint that receives orders that an OrderMS is responsible to handle. When order is put[,] Inventory must be updated so OrderMS publish an event to subscribers [...] and InventoryMS will update the inventory due to it is subscribing to current event/message"


Another developer explains the intention to achieve functional dependency across microservices via events propagated <cit.>:

"Customer and Order microservice both have customer details, though in Order microservice customer infos are striped to only required fields. I understand there is a way to maintain consistency of data across microservices using events."



FR4. Replaying of events. We observe that developers seek replaying (i.e., resending) past application-generated events in two cases:


(i) Synchronizing states. Microservice applications evolve over time. 
As new microservices are incorporated into the topology, it may be the case that these are required to synchronize with the current application state, as described by a developer <cit.>:


 
"When I add new service to a set of already running services, I need upstream dependencies to send all the messages from the past to the new service so it could align its state with the one of the whole system."



(ii) Troubleshooting and Auditing.
As the communication abstraction in event-driven microservices, it is natural that the re-execution of the event flow may assist in further understanding the application behavior, as explained by the following developers:


(DEV #1) I want to replay events on an invoice where the I want to see all actions done by a specific employee on the balance. <cit.>



 
(DEV #2) The intent for this pattern [(event sourcing)] is to provide an audit trail of all events that took place while the patient was in the hospital. <cit.>


FR5. Query processing. Developers tend to report the implementation of query processing operators at the application level. In this case, they implement queries that operate over the payload of subscribed events, as exemplified by the following SO question <cit.>:



"I want to create a third microservice that is responsible to join the data of ProductService and StoreService in order to retrieve all the available products. Here, CQRS pattern seems like the best solution: I will create a materialized view and I will synchronize it using domains events published by the other 2 microservices."


FR6. Data replication. We observe that the propagation of state updates is also an enabler of data replication. By subscribing to state updates in the form of events, consumers can maintain their own view of the state managed by other microservices without requiring to pull updates, thus avoiding the overhead entailed by synchronous calls (e.g., RPCs) <cit.>.

For instance, in the following question <cit.>, the developer explains that

"Service A could raise an event whenever a new student is registering. Service B will consume the event and stores student info in its [own] db."


Based on our analysis, it is worth noting that not all cases of propagation of state updates will decidedly lead to data replication, but all cases of data replication via events necessarily require propagation of state updates. 

FR7. Cache management. In a similar way to FR6, another reported scenario lies on maintaining caches based on subscribed events, as explained by a developer as follows <cit.>.



"When C starts up it needs to load all of the current data from P into its cache, and then subscribe to change notifications. (In other words, we want to synchronize data between the services.)"











FR8. Task Scheduling. We also found cases about events being used as abstractions to manage the life cycle of long-running jobs. As example scenarios, we highlight the following:





"That job info is placed in a RabbitMQ message and sent off by the RabbitMQ Producer [..]
A RabbitMQ Consumer receives message with the job info and calls the class that is responsible for Executing the long running job, the job status is updated to IN-PROGRESS" <cit.>




"The idea is to have the REST API immediately post a message to a queue, with a background worker role picking up the message from the queue and spinning up multiple backend tasks, also using queues. REST API [..] generates a GUID and attach that as an attribute on the message being added to the queue" <cit.>






Finding 3: Event management is applied to fulfill varied functional requirements, such as communicating data updates, composing microservices functionalities, and processing queries, indicating the heterogeneous applicability of events.









§ EVENT MANAGEMENT CHALLENGES (RQ2)







































Having discussed the state of practice of event management, we focus on the challenges developers face when managing events in their microservice deployments. Figure <ref> summarizes the challenges we extracted from SO. The leaf nodes represent the specific challenges, whereas their parent nodes represent the categories to which a challenge belongs. For example, the Performance (<ref>) category comprises three specific challenges: event processing overhead (CP1), large event payload (CP2), and fluctuating event rate (CP3). In total, our analysis led to five categories and sixteen specific challenges, indicating the heterogeneity of the problems microservice developers encounter when managing events. Next, we discuss and exemplify each specific challenge by their categories. Along the discussion, it is worth noting that we refer to messaging technology as any system that enables microservices to exchange events through publishing and subscription mechanisms. 
























 §.§ Safety and Liveness






Safety and liveness are properties inherent to distributed systems <cit.>. 
While safety properties prescribe that nothing bad happens, liveness properties prescribe that something good eventually happens. For instance, a traditional example of safety property is found in the 2-Phase Commit (2PC) protocol <cit.>. In 2PC, the effects of the operations of a transaction that cuts across nodes are only made available to subsequent transactions if all participating nodes agree. A common liveness guarantee example is eventual consistency <cit.>. In this consistency model, nodes eventually converge to the same outcomes (e.g., data item versions). When this is achieved, it is often unbounded, making it a weak consistency model.

In this section, we use these two properties to discuss related problems in the context of event management in microservice architectures. We reveal microservice practitioners' expectations when managing events, particularly when publishing, consuming, processing, reverting the effects of, and replaying events.




 §.§ Application Liveness


While in application safety, we covered developers' challenges when attempting to ensure nothing bad happens [http://www.bailis.org/blog/safety-and-liveness-eventual-consistency-is-not-safe/], in this section, we cover another property inherent to distributed systems: liveness. 
According to Lamport, a liveness property in multi-process programs relates to guaranteeing that something must happen. In distributed applications, however, like microservices, due to the message distribution delay, liveness turns into guaranteeing that something "good" eventually happens (CITE bailis). Next, we explore what are these good outcomes microservice developers expect and the challenges faced when these are not realized.






  §.§.§ Publishing events safely (CAS1)


Safety.

The functional requirements investigated in Section <ref> highlight that generating events is the cornerstone of event-based microservice architectures. 

Events are often raised in response to local operations, typifying a causality relation. Therefore, failures in local operations must automatically withdraw the existence of a resulting event.
However, we find that event's publishing semantics are not always crystal clear for microservice developers. We highlight as follows the most common types of issues using developers' own quotes: 





(i) Developers seek to acknowledge whether the event published has reached out to the message broker, for instance <cit.>: 

(DEV#1) "This operation [(publishing the event)] is asynchronous and ALWAYS returns true, no matter if the broker is down. How can I know that the message has reached the broker?"



(ii) Developers wonder whether publishing an event as part of a transaction is possible, for instance <cit.>:


(DEV#2) "[...] does actually spring-kafka support JTA transactions and would it be enough to wrap RDBMS and Kafka Producer into  [(ORM annotation)] methods?"


(iii) In other cases, developers wonder how to ensure atomicity semantics when publishing more than an event as part of an operation, like the user below <cit.>:


(DEV#3) "Let's say a command needs to append an event to both, the public and private user stream. How can you make sure that both events have been appended? Does the event store publish both,  and , to the event bus?"


(iv) Developers look for alternative ways to publish events out of the critical path of the application, such as asynchronously detecting database updates and publishing these as events, like the case below <cit.>:


(DEV#4) "Should I have some kind of background timed process which will scan events table and publish events to SQS? Can this be process within WebApi application (preferable), or should this be a separate a process?"


Challenges related to ensuring events are correctly published appear in 13.79% of  challenges and reflect 8.60% of the total, indicating the representativeness of the problem. 

Discussion: We observe practitioners seem unfamiliar with the guarantees provided by messaging technologies. As a result, they may fail to foresee the possible issues associated with ensuring consistency between two systems, in their case, the (producer) microservice and the message broker.



Recent Change-Data-Capture (CDC) tools like Debezium [https://debezium.io] soften this challenge by dedicating a system to capture updates in a microservice private state (e.g., relational tables), offloading developers from explicitly handling the publishing of events. Each update is then eventually delivered to consumer microservices. However, apart of the need to manage an additional system, when or whether this delivery is performed is unknown, which can lead to an additional challenge as we see in Section <ref>.

Furthermore, some database systems like Oracle [https://www.oracle.com] and PostgreSQL [https://www.postgresql.org], offer mechanisms to publish events atomically as part of transactions. However, the benefits of state encapsulation  would be jeopardized given all consumer microservices would depend on specific producer-managed databases, thus leaking the private microservice state abstraction.









  §.§.§ Roll-backing states (CAS2)



Safety.

Once an application-generated event is published (i.e., reach out to the message broker successfully), the event becomes available for consumption by other microservices, which in turn can consume and start processing the event. However, events may be consumed and processed by multiple microservices independently, which makes it challenging to track down the possible errors across multiple microservices and react accordingly. As a result, we observe that developers express uncertainties on how to appropriately deal with failures, such as the following cases: 



(DEV #1) "[...] the state modification is actually a complex operation across multiple microservices using the saga pattern, which needs to be rolled back if something fails. [...] How to cleanup the state if the service modified it, but failed in the end, e.g. due to system shutdown?" <cit.>





(DEV #2) "When my Transaction is committed, normally I will dispatch  (e.g. to the queue), but there is possibility that this queue is down as well, so previously just-committed transaction has to be "reverted". How?" <cit.>





In this sense, many answers received for related questions suggest the use of compensating actions triggered via additional events; however, some developers acknowledge that these inherit the same properties of the originating problem, as summarized by the following quote: 


(DEV#3) "If 3.a ([an operation triggered via event]) fails[,] a compensation action is performed; but what about if it fails?" <cit.>


We observe this challenge is significant in the context of  challenges (13.22%), spanning across 8.24% of the total questions.

Discussion: When the event is published, the event producer loses control over downstream processing due to the independent failure of microservices. Similarly, consumers are unaware of which microservice produced the event stream to which they are subscribed. In this case, we found some developers design specific event streams to communicate failures in the workflow, as we discuss in Section <ref>.

However, publishing events that semantically represent the occurrence of a failure can also fail. That can lead "dirty" intermediate states (i.e., the effects of canceled operations) to remain exposed indefinitely. On the other hand, even though upstream microservices acknowledge a downstream failure, once the local operations are committed as part of a transaction in the database, it is often not possible to rollback the transaction. Thus, upon failure, it is unclear for developers how to properly revert those changes. 









  §.§.§ Processing event dependencies (CAS3)

Liveness.



Events often carry a schema, or an event type, and are published in a stream (aka topic or queue) on which all events published must adhere to such schema. We observe an interesting trend in which practitioners attempt to match distinct but correlated events at event processing time. 

The trend is exemplified by a developer <cit.>:


(DEV#1) It is very straightforward to implement a [(event)] listener when an action depends on one single event ([stream]). [...] The problem arises when OrderService has to wait for more than one event [(from different streams)]


The developer contextualizes the case and expresses the problem then <cit.>:


(DEV#1) I am pooling both queues
  and
 , and both events has to be present so I can finish an order. How can I coordinate so that OrderService sees both events as only one?


However, these dependencies across distinct events may require ordering guarantees from the messaging technology, which may not be present, as noted by another developer <cit.>: 



(DEV#2) If  comes after 
 the normal flow requires to throw an exception because the user doesn't exist yet. [...] So, the ordering problems are:


No order guaranteed across events from the same event stream.No order guaranteed across events from the same ES [(event store)].No order guaranteed across events from different ES (different services).


These ordering problem can ultimately lead to concurrency bugs in the application code, as noted by the former developer <cit.>:


(DEV#1) There is a minimal chance of receiving both events at the same time[,] generating race conditions.









We observe that this challenge appears substantially, spanning across 20.11& of  challenges. Besides, it spans 12.54% of the total questions, highlighting the practical significance of this event processing pattern.











 
Discussion: Developers express some events are semantically related and thus cannot be processed independently. However, developers fall prey to weak ordering semantics across event streams. Another challenging factor is that the delivery of events can experience arbitrary delays due to network partitions, overload of computational resources, network jitter, and even the failure of producer microservices. These aspects together pose a challenge in enforcing event dependencies across distinct event streams.

This event processing characteristic may find resemblance with the so-called workflow patterns, posited for business processes and web service compositions <cit.> a few decades ago. Recent microservice frameworks, like Dapr [https://docs.dapr.io], has included in their API support for some workflow patterns, such as fan-out/fan-in <cit.>. However, workflow patterns often require either a domain-specific language or a common framework where all microservices are implemented with, as well as the explicit specification of workflows, including event producers and consumers. These contrasts with the polyglot nature of microservices and the decoupling through events seek by microservice developers. Thus, it is an open question whether and how workflow patterns can mitigate the challenges of processing event dependencies. 




  §.§.§ Processing events in order (CAS4)


Safety.

While analyzing the questions, we noticed that practitioners often expect events to be processed in order for every consumer. However, in many cases, this expectation is not met. Unlike the previous challenge, this challenge affects single event streams (i.e., all conforming to the same event type) rather than distinct events. A developer exemplifies the problem as follows.









(DEV#1) Suppose a product is ordered and it is id 80, and a series of sequential update events fired from product service to order services for that particular product [...]
The end price should be 100 for that product, but sometimes these events are processed in random order <cit.>


The developer expects that the product price observed by the Order microservice must eventually be in sync with the product's state in the producer (i.e., Product microservice). Therefore, any processing order not leading to a price of 100 as the final state is deemed incorrect. In some cases, practitioners acknowledge that an in-order processing guarantee cannot be enforced in consumers due to the uncertain semantics of producers:






(DEV#2) Since each microservice has its own event table and asynchronous worker, we cannot guarantee that events will be sent in the sequence in which the corresponding state changes occurred in their respective microservices. <cit.>


This challenges appears in 16.67% of  challenges and accounts for 10.39% of the total challenges observed, highlighting the difficulty of ensuring events are processed in the expected order.


Discussion: Although the majority of messaging systems guarantee the delivery of events from the same stream in order, such as Kafka through the concept of a topic partition <cit.>, that does not exclude producers and consumers from still processing the events accounting for the expected processing order semantic. For example, for producers consisting of concurrent threads writing to the same partition, even though Kafka serializes the messages, the correct order that the consumer must process the messages is not well defined. Similarly, suppose multiple consumer threads pull messages concurrently from Kafka. In that case, if the order of messages across different pulls matters, it is complex and error-prone for general developers to reassemble the original stream order at the consumer side.



In most questions analyzed with this particular challenge, we could not identify whether the message broker delivers the messages out of order or the microservice processes the event arbitrarily (not accounting for the delivery order). As the quote below exemplifies <cit.>, event processing semantics appear unclear to some microservice developers.



(DEV#3) In [(omitted technology,)] messages are best-effort ordering, still no idea of what they mean. [...]
Does it means that[,] giving n copies of a message[,] the first copy is delivered in order[,] while the others are delivered unordered compared to the other messages' copies? Or "more that one" could be "all"? 




  §.§.§ Synchronizing states via event replay (CAS5)

Safety.



As described in Section <ref>, the need for replaying events often arises due to new microservices being introduced in the system. In this context, a developer explains an interesting challenge <cit.>:





(DEV#1) Service "A" creates event message to inform other services about changes [...]
Newly introduced Service "D" also needs to replicate data coming from service "A". Service "D" needs all historic data[.]


The developer then continues expressing the impedance found.


([...]) other services were already running for a while, and service "A" only broadcasts new changes. What would be the correct solution to populate newly added service with historic data?




Furthermore, as microservices supposedly fail independently, it may be necessary to replay events from the point in time the subscriber microservice(s) failed, as described by another developer <cit.>:


(DEV#2) I'm wondering what should happen if one of these events can not be delivered due to an error [...] On republishing events, should all messages be republished to all topics or would it be possible to only republish a subset?




Challenges involving replaying events also appear significantly in  challenges (14.37%), accounting for 8.96% of the total challenges, suggesting mechanisms for safely replaying events are strongly seek by developers but missing in practice.

Discussion: Along the analysis, we find that the need for replaying events arises not only from software evolution, including adding new microservices and migrating to a microservice architecture but also when recovering from failures and fixing the outcome of bugs. However, developers express uncertainties about how to proceed properly in the cases above and end up implementing several ad-hoc mechanisms at the application layer to fulfill these. As a result, these mechanisms often result in rework and create additional issues in the application.





  §.§.§ Continuous queries over events (CAS6)

Liveness.




We observe that practitioners commonly carry out continuous queries consisting of various query operators (e.g., filter, join, and aggregations) <cit.> over the payload of incoming events to extract valuable information. However, the nondeterminism of asynchronous events (i.e., the arrival of events may experience large delays or even never arrive) poses challenges in ensuring correct query processing results, as described by a developer <cit.>:


(DEV#1) When I update a Product entity or Store entity there is no problem to sync the view because I already have data on it. But what about when I receive a  event? 


This event has only information about product and nothing else, so , ,  will be NULL. 



How can I save this event in my view? Should I save incomplete data somewhere else and update my view when I will receive complete data?



Furthermore, as software evolves over time, changing requirements can also undermine providing correct query results based on events, as exposed by another developer <cit.>:


(DEV#2) I have new requirement - calculating maximum all time temperature per sensor. 



I have prepared new microservice that creates  ([Kafka table abstraction]) aggregating temperature (with max) grouped per sensor. 


[67074772]
Simply deploying this microservice would be enough if input topic had infinite retention, but now maximum would be not all-time, as is our requirement. [...] How to design the solution?



The impediment found above relates to the inner characteristic of stream processing engines, that necessarily rely on windows and notions of time to avoid querying all the historical data in order to provide fast responses <cit.>. Continuous queries over events appear frequently in Safety&Liveness challenges (12.07%) and corresponds to 7.53% of the total questions, highlighting the significance of this challenge.

Discussion: For over two decades, continuous queries and stream processing systems have been a mainstream research topic in the data management community <cit.>, resulting in industry-strength solutions like Flink [https://flink.apache.org] and Kafka Streams [https://kafka.apache.org/documentation/streams]. These systems allow users to declare a query that is continuously updated based on incoming streams. However, we notice microservice developers often do not mention stream processing systems and prefer patterns like CQRS, not realizing that providing correct query results with this pattern incurs additional challenges. Besides implementing the query operators, developers must ensure that duplicate events (we further discuss in Section <ref>), crashes, and delays do not impact the query results.

Stream processing systems are designed to handle the gamut of problems that arise in maintaining continuous queries. We conjecture microservice developers are either unaware of or may find difficult to map their application logic to stream processing operators.



























  §.§.§ Weak delivery semantics (CAS7)


Liveness.



















Messaging technologies often guarantee at-least-once delivery by default, leaving the responsibility of achieving at-most-once or exactly-once delivery semantics to application developers <cit.>. However, by resorting to at-least-once delivery semantics, developers can face unpredictable challenges, as exemplified by the developer as follows <cit.>:






(DEV#1) I was able to collect the data [(published as events)] from another microservice, but I noticed that if I terminate the [(consumer)] process and run it again, I get the data back, PLUS another copy of the same data right under it. [...] I got the payload sent to me twice, and I only want to see it once.


As the developer's quote shows, a problematic aspect of at-least-once delivery is that events can be delivered more than once. That forces consumers to either make their 
microservices

idempotent or fall prey to the undesired effects of duplicate event processing. The following microservice developer expresses another interesting case <cit.>: 





(DEV#2) The only way I have to know that the payment has been processed by Payment service is by expecting an answer event ().


However, in the absence of an event that confirms that a payment has been processed, the developer comes up with its own customized solution:


If it hasn't gotten an answer in some time, [that] forces me to implement a retry mechanism in the Order server, [that is,] retry with a new  event.


The developer ended up dealing with an unexpected outcome:


[̇However,] this also forces me to take care of duplicated messages in Payment service in case they were actually processed but the answer didn't get to the Order service.


Another popular issue on resorting to events for asynchronous processing lies on the fact that computations are often initiated based on client requests. These online requests require timely response to users and challenges developers to match the eventual arrival of asynchronous events (i.e., the response of the client-triggered asynchronous computation) with the corresponding waiting client. A developer describes the problem below <cit.>:


After publishing the event[,] my booking service can't block the call and goes back to the client (front end). 



How does my client app will have to check the status of transaction? Does it poll every couple of seconds? Since this is distributed transaction and any service can go down and won't be able to acknowledge back. 



In that case how do my client (front end) would know since it will keep on waiting.


Dealing with weak delivery semantics is the most popular challenge in  category (27.59%). By appearing in 17.20% of total the questions, that evidences the difficulty of managing events under weak delivery semantics.

Discussion: The loss of events may be caused by many issues. Besides consumers crashing, during a rebalance in Kafka brokers, consumers can experience delays in event delivery <cit.>. Besides, the network can become unstable, leading to delays in network package deliveries. 


In an attempt to ensure the completeness of operations across microservices, developers often rely on custom-made, ad-hoc solutions, such as the retry mechanism shown above. However, retries can lead to the additional burden of dealing with duplicate events, which forces developers to implement additional mechanisms at the application layer to impede duplicate events from being processed, only exacerbating the problem.


























Finding 4: Developers encounter a myriad of challenges on ensuring events are processed correctly. These span the entire life-cycle of an event in a microservice architecture, including queuing, delivering, storing, processing, and synchronizing events. 







orthogonal. common underlying issues.
It is noteworthy this challenge affects C4 is some way. While in C4 we consider the distinct events to be matched arrive at some point (problem is coordinating them at runtime), use cases involving processing event dependencies can also be affected by never-arriving events, since matching distinct events would be compromised.


Contrast C4 (app safety is more about late event, data race, tx isolation and here no) and c5 (can affect the final state, specially if state is incremental) c6 (expectations of users, can affect an in-progress query)

perhaps bring C3 and reasoning about convergence here.
in some sense, C4 may also be mentioned here, because it fits the idea of missing an event too. if a dependence never arrives, something "good" does not happen.



scenarios must eventually complete in the presence of failure.

the cause can be bottlenecks in the microservices. cannot keep up with the rate of incoming events.

73613249








C8. Convergence. 


In the previous challenge, we found that the non-arrival of a particular event after some time would indicate, from the developers' point of view, the unlikeliness of something "good" to happen eventually.
However, we also observed that in many cases, there is no expectation of events that semantically represent an "acknowledgment," making it more challenging to determine whether generated events have been processed to a sufficient degree.


A user describes the issue in a social media application domain:

A new tweet appears, server A stored in its own database some data, generated new events and fired them. Server B and C didn't get these events at this point and didn't store anything in their databases nor processed anything.


The user continues then

The user that created the tweet wants to edit that tweet. To achieve that, all three services A, B, C should have processed all events and stored to db all required data, but service B and C aren't consistent yet. That means that we are not able to provide edit functionality at the moment.









In a similar manner, developers report their issues on reporting the result of asynchronous operations, often spanning multiple services, to clients. For instance, a question presents the following scenario (arrows represent the request flow):


WebClient_X–request—>LoadBalancer–forward–>A-Service_1–request—>ThirdPartyService

A_Service_1 needs to poll the ThirdPartyService for the results and writes them into kafka.

A_Service_1—>Kafka


And the person completes wondering


How is it possible to send the results stored in the kafka topic back to the requesting WebClient_X?


Scenarios like above span across X% of questions, suggesting developers encounter issues on reasoning about the completeness of operations.





  §.§.§ Implications



Given the wide range of safety and liveness challenges, we discuss the implications in two groups. The first covers event publishing and delivery semantics and the second relates to the challenges of synchronizing microservices' events. 

CAS 1, 2, 4, and 7. Developers lack a comprehensive specification to guide them with the variety of event processing semantics and in dealing with failures in asynchronous, distributed microservices, suggesting they are in need for supporting tools. For messaging technology providers and framework developers, their documentation can emphasize the publishing and delivery semantics offered more clearly, including, but not limited to, the possible failure scenario microservice developers must consider and example workarounds to mitigate some of their effects in the microservice state. In addition, these should be provided in a language suitable for developers who are not familiar with distributed and asynchronous systems.

For database providers, explicit APIs for rolling back microservices' underlying databases to a state prior to processing a given event can help alleviate the burden of dealing with online failures in event-based workflows. Researchers can systematically characterize the guarantees provided by popular messaging technologies and enhance code analysis tools to better alert developers about potential hidden shortcomings of the different guarantees.




CAS 3, 5, and 6. The need to process and match distinct event streams generated by different producer microservices spans different functional requirements, such as replicating data, processing distinct but correlated events, and maintaining queries over event streams. Together, they form 26.83% of the total challenges, highlighting the practical significance of these event processing patterns. To fulfill these, developers often hold the assumption that events will arrive in a timely and exactly once manner. However, events may either take arbitrary time to arrive or never arrive. Besides, even if events arrive, they may be repeated. The mismatch between expectations and reality leads to frustrations in implementing the requirements above. Thus, microservice developers must be vigilant with designs that favor the existence of such event processing patterns. Rethinking service boundaries <cit.> may lead to some of the events involved in intricate event processing patterns being merged or eliminated.

Providers of messaging technologies and cloud computing services, and framework developers can offer further guidance at the documentation and API levels to alert developers regarding the potential dangers of some event processing patterns. Potentially, message technologies can devise better APIs to support developers making sure event dependencies are met. For example, an API to wait for multiple dependant events instead of individual events.



Researchers can improve static and dynamic analysis tools to detect certain code properties and execution traces that may lead to the harmful scenarios discussed. 
Another direction could be developing automatic tools to enforce safety properties on event processing. For example, Lesniak et al. <cit.> 

extend message queuing system with a programmable interface where developers can specify event processing order which will be enforced automatically.




All CAS. Researchers can devise or extend existing programming models to account for use cases that necessitate correlation between distinct event streams and abstract publishing and delivery semantics from the microservice code. For instance, they can take inspiration from the dataflow model <cit.>, commonly used in stream processing engines, to accommodate the missing dynamic topology, non-blocking, isolated failure model found in real-world microservices. That can simplify the programmability of microservices and favor the adoption of tricky event stream processing patterns. Additionally, this programming model can unify the design and deployment of event-based microservices across cloud providers and execution platforms to facilitate the migration and hybrid execution of microservice applications across message technology providers, an emerging trend in cloud computing <cit.>.





 §.§ Event Schema Management


Microservice practitioners model events for external interaction. In other words, the decoupling nature of a microservice design requires producers to encode all the necessary data so that consumers do not need to query the producer for additional data when processing such an event. Furthermore, the events generated by microservices must comply with a data type to be correctly processed by consumers, and, as with any other data type, it can evolve over time. This section discusses the challenges associated with modeling and evolving event schemas in microservice architectures.

































  §.§.§ Modeling event schemas (CEM1)



We observe that how events are modeled and what they represent semantically may dictate the design and performance of microservices. For example, a developer looks for recommendations for event design explaining the following scenario <cit.>:



Say you have a micro-service architecture where multiple services produce and consume unit_statuses. There are multiple ways to design this.


The person continues explaining some options in mind:

1. Create a generic topic  and make services consume and produce messages on this topic.  [...] 2. Create a specific topic for each status, for example , 
, , , etc. 



In sequence, the practitioner explains the drawbacks of each:


1. This has the consequence that you consume your own messages and have to filter them.
2. Requires a code or configuration change in potentially all service when a new status topic is added.



In another question <cit.>, a similar inquiry is presented where a practitioner wonders about two event design approaches:



(1) A sends a message which contains event and related entity id like:  { : 1234 }. B consumes this message and if it needs further information, it fetches this from A with  (2) The message not only contains the information above, but also metadata like:  { : 1234, : someFieldValue, ... }


The practitioner continues addressing the pros and cons of each and wonders which one to choose:

(1) Pros: Less network usage; Always the same structure of messages. Cons: If information from A is needed on demand there must be some mechanism to catch, e.g. network failures. (2) Pro: Information is already there. Con: What if the attached information is not enough?


Although practitioners seem aware of some of the consequences different event modeling options bring to the application design, the excessive number of questions, 19.71% of the total, suggests developers encounter uncertainties in deciding on an optimal event design.

Discussion: Differently from relational model <cit.>, which provides a formal foundation that allows database designers to reason about the possible performance implications of a data model (e.g., the level of normalization dictates whether additional  operations are necessary to retrieve certain information from the database), event design is still an open problem. As a result, developers navigate through many possible event designs that, if not carefully thought out, may impact the design of the application and the performance of event-triggered operations.










  §.§.§ Evolving event schemas (CEM2)



We also observe some microservice practitioners share uncertainties on how to deal with the possible effects of event schema evolution properly. In particular, as events are often parsed into microservices' own data models, practitioners express concerns over the impact of event schema changes in their microservice states. For instance, upon a new attribute being incorporated into an event's schema (user's ), a developer wonders how to update a microservice state retroactively <cit.>:




(DEV#1) I can update my event and now include the , but it will only work for new users, the old ones will have null addresses. Should I scan the whole database and manually dispatch an event for each user?


On the other hand, although events are considered abstractions to decouple microservices, some practitioners report event schema evolution impacting dependent microservices <cit.>:



(DEV#2) services are tightly coupled by [event] schema and can causes errors and this contradicts with event sourcing goal. How can we address this problem?


Evolving event schemas also appear as an important challenge, corresponding to 20.59% of the questions about managing event schemas in microservices and accounts for 5.02% of the total questions.

Discussion: We observe that developers are aware of the risks associated with schema changes and their potential impact on microservices dependent on the modified event type. If the type of incoming events does not match the expected event type in a consumer microservice, the processing of this event processing may fail. However, developers have not found a systematic solution to either minimize the impact of or comprehensively manage the evolution of event schemas across the network of dependent microservices.


State-of-the-practice data serialization formats like Apache Avro [https://avro.apache.org] can potentially ease this challenge. However, such tools require a global adoption across all microservices, forcing every event producer and consumer to employ schema and data contract enforcement. Besides, adopting such tools is not always possible because the messaging technology adopted may not provide native support. In overall, this remains an open challenge in practice.


























Finding 5: Developers find no principled ways to model events accounting for performance and decoupling trade-offs. Systematic support for managing event schema changes is also an open problem.











  §.§.§ Implications
 With the lack of automatic support for event schema evolution, microservice developers can isolate failures caused by event schema mismatches and log them appropriately for later reconciliation. Messaging technology providers can supplement their documentation with more technical examples of how developers can isolate the microservices' application code from the impact of event schema changes. Further guidelines on appropriately rolling out event schema changes without impacting individual microservices in the context of their technologies can also benefit developers <cit.>


Researchers can develop analysis tools that holistically map the microservice event topology, matching producer and consumer microservices through the events exchanged. The tool can incorporate metrics to aid developers in reasoning about the trade-offs of different event designs. Furthermore, researchers can provide a formal foundation for modeling events in microservice applications. The model must allow for reasoning about the performance trade-offs of including specific attributes in events. Lastly, automatic support for event schema evolution across microservices is an open and complex problem that researchers could tackle.



 §.§ Performance



Performance is systematically cited as an important factor for adopting microservice architectures <cit.>.

Developers seek to reap the benefits of having the application decomposed into independently scalable building blocks <cit.>. In an architecture based on events, though, the performance of individual microservices is tightly coupled with its capacity to process and dispatch application-generated events at a rate that does not compromise the expected performance of the downstream microservices in the event flow. As a result,

disruptions in the event flow can lead to critical performance impacts. In this section, we analyze such cases from the lens of microservice developers.














  §.§.§ Event-processing overhead (CP1)
 



Developers report particular event processing scenarios that ultimately lead to a performance penalty. For example, in connection with CAS5, some practitioners report the need to replay events to fulfill a given requirement, but that ends up introducing additional load to the application <cit.>:
















The validation of a reservation request needs to know the previous and following reservations (within 3 hours from the booking time of the incoming request)



if I ask for a reservation for tomorrow, the system will replay reservations from 6 months ago that usually are not related with the incoming request 



The developer then highlight the overhead:

This leads to inefficiencies over time as result of the huge amount of unnecessary events that are replayed. I thought to solve it using daily snapshots but it seems the wrong way to do it.


In a similar way, in connection with CAS3, other developers acknowledge there could be a performance penalty on leveraging events for replicating data across microservices. For example <cit.>:



The Invoices microservice must listen to all 
 and  events in order to know if the given recipient id is valid.



Then I'd have thousands of Contacts within the Invoices microservice, even if I know that only a few of them will ever receive an . Is there any best practice to handle those scenarios?













Practitioners express uncertainties in dealing with the overhead of processing events and their possible performance impacts. This challenge is the most prominent in  the  category (53.57%), and spans 5.38% of the total questions. This suggests that microservices may execute with suboptimal performance in real-world deployments.

Discussion: At first sight, the overhead of the first case can be potentially mitigated by storing the events processed in the microservice's private state. However, this technique introduces hidden dangers. For example, keeping track of the last stored event is nontrivial and can ultimately lead to losing an event due to crashes. Besides, developers tend to favor the data shipping paradigm <cit.>. By offloading state management to another system (e.g., database system, message broker, or cloud object storage), applications can remain stateless, decreasing their complexity. In the second case, the overhead is triggered by the lack of systematic support for data replication through event management.



  §.§.§ Large event payload (CP2)
 



We observe developers also present concerns over the size of the event payload generated and consumed by different microservices. For instance, a common concern lies in whether publishing events with large payloads into the message layer entails a good practice:







(DEV #1) My question is, is it wise to stream such file over event bus from Service A to API gateway? (File may get as large as 100 MB) <cit.>




(DEV #2) If an event needs to pass a very large volume of data to the next Saga event, how is this done in terms of the request structure? Is it divided into multiple Sagas for example (as a result pagination type)? <cit.>



(DEV #3) I think we will need to go with second option i.e. not sending data in the event due to the potential size of the event data.  <cit.>


To circumvent the potential performance degradation incurred by processing large event payloads, some developers look for alternatives to the solo event-driven approach but end up encountering other issues:



(DEV #3) I was also thinking about adding the URI in the event so that the service just needs to call it. Only issue I can see is that how would the service know what type to deserialize the response to? <cit.>


The performance impacts of generating, processing, and storing large event payloads account for 17.86% of  challenges. This challenge represents 1.79% of the total questions and highlights that developers find difficulties on dealing with large event payloads and their consequences to microservice performance.

Discussion: Message brokers and modern log processing systems are not designed to handle messages with large payloads natively. Message brokers have historically targeted systems' integration cases, which usually do not require large event payloads <cit.>. Log processing systems, like Kafka, for example, were initially designed for the timely processing of logs extracted from processes running in distributed servers <cit.>. Similarly to message brokers, though, it was never a design goal to accommodate the processing of large log payloads. Native support for these types of objects is found in cloud storage services and specific column-oriented database systems, like  type in PostgreSQL. [https://www.postgresql.org/docs/16/largeobjects.html] Therefore, the questions analyzed in StackOverflow suggest that microservice developers may benefit from either redesigning their computations to manage large objects in appropriate storage systems or rethinking the granularity of their event payloads when they grow arbitrarily. This may involve breaking down application components into smaller, finer-grained tasks to decrease the size of events transmitted across microservices.





  §.§.§ Fluctuating event rate (CP3)




Developers describe challenges associated with dynamic workloads and their impact on the observed event processing rate.



(DEV #1) In case of pressure over the system, can service B communicate to service A to slow down, and A will react to this by not accepting more requests from clients, till B decides it can continue? Is this something that can be achievable using [(omitted library)]? <cit.>





(DEV #2) I want to restrict the users from publishing a new message on this topic to prevent my system from choking after a certain limit.



For example, if the number of unacknowledged messages in the topic is already more than or equal to 10000, then I want to give a bad input exception or something to restrict users from flooding my queue. <cit.>







(DEV #3) Traffic is increasing and we've noticed that events spend a lot of time in queue. We need to process events faster. [...] Is it possible to implement circuit breaker or a failover mechanism for an async call? <cit.>






The challenges about adapting microservices to handle increased influx of events represent 28.57% of  questions. Representing 2.87% of the total of questions, this additional performance challenge strengthens the perception that microservice developers encounter difficulties on reaching performance goals in event-driven microservice architectures.

Discussion: With traditional synchronous communication paradigms, like RPC and HTTP requests, an application can usually define a threshold of the maximum allowed concurrent connections. In event-based architectures, though, there is an indirection: the events are often stored in the event management layer first and only later forwarded to (or pulled by, depending on the system) consumers. This decoupling introduces challenges for consumer microservices to communicate whether producers must decrease the event generation rate or even refrain from generating new events. On the other hand, modern event management systems, like Kafka, provide abstractions to parallelize the consumption of events through partitions. However, developers must configure the unit of parallelism according to their requirements, which may not be trivial for newcomers. 










Finding 6: Developers encounter challenges on managing the event rate across producers and consumers. This only exacerbates in the presence of event replays and large event payload sizes.






  §.§.§ Implications

Message technology vendors can provide efficient and proactive event-processing techniques and algorithms that mitigate or decrease the impact of fast producers and slow consumers. As mentioned in the discussion of individual challenges, developers may benefit from rethinking the granularity of their microservices and the content included in their event payloads to escape from adding overhead to their event processing pipelines. Developers may also be aware that, as events flow through microservices, the events generated by each microservice may include additional data. That suggests developers may track the event payload sizes more carefully as they flow across possibly many microservices. 

Researchers can investigate whether state-of-the-art approaches for scaling distributed systems also apply to event management in microservices. Besides, researchers can develop systematic tools to mitigate the effects of large payload sizes in event-based microservice workflows. That could involve optimal deserialization of event payloads (e.g., only deserializing the subset of attributes that the microservice requires), advanced compression tools that mitigate the overhead of growing events, and transparent removal of unused event attributes.




 §.§ Observability



Observability is a critical concern in modern software development because it allows developers to track important application behavior and metrics, such as application exceptions and memory usage, respectively <cit.>. In distributed systems, such as microservice architectures, observability becomes more critical since problems can possibly span across multiple components that execute independently in a decentralized manner <cit.>.

In our analysis, we were surprised to find that microservice developers leverage events not only as an abstraction to process operations asynchronously but also to track the progress of operations across multiple microservices and troubleshoot the complex interplay of multiple microservices. In this section, we describe these patterns and their associated challenges.






  §.§.§ Event flow observability (CO1)
 


Although tools to observe synchronous requests (HTTP or RPC-based) crossing microservices enjoy apparent consolidation, developers express challenges in monitoring the asynchronous event flow across microservices, as exemplified by the two following developers' quotes:


(DEV #1) What is something we can do to have an end to end understanding from producers to consumers, etc? Mainly for troubleshooting purposes/change management. <cit.>



(DEV #2) What is the best way to track the JSON as it flows through many microservices down stream in an event driven way? <cit.>


Another example presents a developer wondering whether an observability tool can embrace events:

(DEV #3) Will [(omitted tool)] be able to trace requests done over the eventbus? The tracing page says that headers need to be propagated through in http or grpc - but the eventbus sends messages via tcp – does that mean that [(omitted tool)] will not be able to trace requests and show the visualisation tools[?] <cit.>





Furthermore, we found that issues in the message broker only exacerbate the challenge of monitoring how healthy the event flow is, as explained by a developer <cit.>:









For any reason the messages get stuck after some time [...] There is nothing in the log files nor the OS event log. I have to restart the ([omitted technology]) service in order to "reanimate" it. 



Afterwards[,] all stuck messages will be processed and everything is working fine until the next "accident". [...] Does anybody has an idea what I could check additionally to find out what the problem is?


Observing the event flow appear significantly in Observability challenges (89.47%).

This highlights the difficulties developers find to understand the progress of events as they flow across microservices.

Discussion: Application-generated events serve as natural progress markers. It is a natural choice for developers to obtain an end-to-end overview of the application execution based on these events. However, developers encounter a twofold problem. On the one hand, developers find that state-of-the-art observability tools do not natively support tracking application-generated events. 
On the other hand, when disruptions in the event flow are not caused by crashes or bugs in the microservices themselves, developers have difficulties in finding the root problem. 
Although messaging technologies are only part of the intricate and heterogeneous components in microservice deployments <cit.>, they are a core enabler of event-based interactions in microservices. Thus, timely detection of message broker failures and their relationships with microservice disruptions are key to observability in event-based architectures.






  §.§.§ Auditing via event replay (CO2)



In CAS5, we discussed how developers leverage events in an attempt to synchronize data across microservices. As a related pattern, we also found that 
developers use the events as an abstraction to reproduce some application behavior. Similar to CAS5, developers end up facing difficulties in ensuring the correctness of the process, as explained by a developer <cit.>:



(DEV#1:) If the VisitId was deleted across all services we could just replay the events one at a time, in order, and reproduce an exact copy of the original record.


The developer then mentions an impedance commonly expressed by another practitioner across the questions:


I've been using [(omitted message technology)] for the stream itself [...] 

The issue though is that this is not suitable for a replay store - only delivery of the event messages.


Another developer warns about the hidden dangers of such a method <cit.>:


(DEV#2:) What's maybe not a good idea is trying to recover the current state of your domain model by replaying an arbitrary set of your events.



[..] Remember, an event isn't usually a complete representation of the state of the model after the change, but rather a description of the things that changed. 



Auditing past microservice executions through events to observe and further understand their behavior appear in 10.53% of  challenges. 

This suggests that current technologies offer insufficient abstractions to support event replay effectively.

Discussion: As events often represent intermediate microservice states, they appear as convenient abstractions to developers seeking to troubleshoot past application behavior. However, in the same vein as CAS5, this practice incurs hidden challenges due to the lack of appropriate support from current messaging and framework technologies. For instance, as discussed in CAS7, mismanaging the processing of duplicate events and their possible effects on the application can lead to corrupting a microservice state. Another example, based on CEM1, is
that microservices may even fail to replay events due to the divergence of event schemas.




Finding 7: Developers find difficulties on observing and thus reacting upon disruptions that occur in the event flow across microservices. These disruptions may not only be caused by microservices, but also message brokers and network partitions.






  §.§.§ Implications

Messaging technology providers can evolve their systems and APIs to provide native integration with industry-strength observability tools. The integration must account for the needs of developers in their real-world cases as discussed above, including but not limited to specific metrics such as event processing and acknowledgment delays from consumers.

For developers, as state-of-the-art abstractions render limited support for safely replaying events, developers can use a "staging" environment (similar to testing environments with testing, staging, and production  <cit.>) to audit their event-triggered operations. However, generating an application and message broker state from a previous point in time for replaying can be challenging.

In the tool development landscape, opportunities for researchers are:
(i) They can co-design static analysis tools and dynamic analysis of event streams for a holistic monitoring of microservice event streams. In particular, this can aid developers by automatically matching correlated events and associated service disruptions in the event flow.
(ii) They can develop techniques for isolating re-execution of events from impacting the actual microservice states. Recent tools <cit.> allows for replaying operations in database-backed applications. However, it is unclear how the approach can be applied to events and distributed states of microservices.



 §.§ Security



Security is another critical concern in event management in microservices. In this section, we discuss developers' requirements and challenges when ensuring that event ingestion, processing, and storage meet security constraints.














  §.§.§ Authentication (CS1)









Although events are often an abstraction that is only internally recognized by the microservices, some events may be generated by end users. That requires making public APIs available, which necessarily exposes the microservice to security vulnerabilities, as exemplified as follows:



(DEV#1) Because I don't check authorization on websocket open, in theory this approach is vulnerable to a dDos attack, where an attacker simply opens as many sockets as they can. <cit.>





On the other hand, even when APIs are safeguarded with authentication methods, developers are concerned about aligning the validity of the authentication mechanism with the event flow.



(DEV#2) In case of event driven application, it's possible to ensure the token is valid? In case of failure, the user clicks on button and an event is written but the processing of this will be hours later. How can I process the event with the user credentials? <cit.>









(DEV#3) if I change my communication mechanism to using async messaging (e.g. RabbitMQ), how would I now authenticate the event to the scope of the user who initiated the event. <cit.> 



























Challenges on ensuring event-based workflows and event-triggered computations execute securely appear substantially in  challenges (76.92%). Spanning across 3.58% of the total questions, that suggests developers fall short on a holistic solution to secure their event-driven microservices.

Discussion: Aligning authorization mechanisms and event processing is non-trivial. One obstacle is that the authorization and messaging mechanisms are often provided by different systems, resulting in a system integration issue. In particular, developers working with microservices need to handle authentication token life cycles and make sure that any events related to the token are in line with its validity. This process can add complexity to the solution.



  §.§.§ Data Privacy (CS2)




Data privacy emerged as a key requirement in modern applications to prevent leakage of user data and potential violation of data privacy laws such as the General Data Protection Regulation (GDPR) [https://gdpr.eu/]. Data privacy laws prescribe that users can request the removal of their data at any moment, and firms are usually given a deadline for fulfilling the request. As discussed earlier, in microservices, data flows from the multiple event streams to many microservice consumers, which in turn apply operations on their private states. This heterogeneous data placement and movement can make managing data privacy challenging. As exemplified by the following quote <cit.>, we observe that microservice developers express uncertainties about handling data privacy in event management.









deleting user data in an event-sourced system [...] To which stream is the tombstone event written?
The event stream of the specific user? Or is there an event stream specifically for tombstone events?


to keep some data (e.g. the users' id), he advises to split the user stream into a public and private event stream. [...] How can you make sure that both events have been appended?








Managing data privacy in the context of event streams accounts for 23.08% of security challenges.

This suggests that developers find uncertainties on how to properly manage data privacy in events exchanged between microservices.

Discussion: We find that developers often manage data privacy in event streams using ad-hoc solutions, such as separating privacy and non-privacy data into different streams. However, these mechanisms do not enforce by design that data leakages do not occur, such as user data contained in events that are not supposed to be processed by certain consumer microservices. Besides, these mechanisms can lead to consistency problems when it is up to the developer to ensure privacy and non-privacy streams are in sync.




Finding 8: It is unclear for microservice developers how to properly safeguard security properties in the context of event processing in microservices.






  §.§.§ Implications
 
Messaging technology providers could provide technology-specific guidelines on properly handling authentication and data privacy in their systems, accounting for the mechanisms that prevent exposing unintended event payloads. Cloud providers and framework developers can offer custom deployment templates to facilitate meeting certain security criteria. However, these should not be oblivious to the event management layer, suggesting that an appropriate cross-system integration is necessary to fulfill security challenges.

Microservice developers can, by principle, never include sensitive data in their generated events (i.e., privacy-by-design). However, this can jeopardize the decoupling benefit brought by event-based architecture, requiring consumers to contact the producer for the missing sensitive data. Researchers can extend existing code analysis tools to alert developers about bugs and configuration mistakes that can lead to unintended access to events and exposure of private data in event streams. Besides, researchers can investigate privacy-by-design event management methods without impact decoupling. For instance, protocols that automatically encrypt sensitive data in events without developer intervention.



















Summary: Microservice developers encounter a myriad of problems when managing events, including dealing with large event payloads, evolving event schemas, auditing and validating security tokens through events, and processing events in order.




    




§ THREATS TO VALIDITY



In this section, we discuss the threats to the validity of our empirical study.



Selection bias of the source. 
Similar to previous research <cit.>, our work uses SO as the data source to study the challenges developers encounter in practice. 
Although other data sources could supplement the findings, the heterogeneity of challenges, application scenarios, and developer expertise found in SO enables a trustworthy reflection of the state of the practice. For instance, the findings are aligned with several blog posts that report similar challenges <cit.>.





Construction of tag set. We select a set of tags to filter SO questions associated with event management in microservice architectures. The relevant keywords and metric applied may omit some tags associated with event management in microservices. To mitigate this threat, we rely on manual inspection conducted by independent researchers, and we adopt the lowest threshold used in previous work <cit.> to include the highest amount of tags as possible.

Subjective of researchers. Since we adopt manual analysis to identify and categorize challenges, that may threaten our findings' validity. To minimize this threat, the first two authors analyzed and classified each filtered question in isolation. Upon conflicts, an experienced arbitrator coordinated the discussion to reach an agreement. 







Generalizability of our findings. We recognize that the application scenarios, requirements, design choices, event processing patterns, and the challenges extracted from SO may not be generalizable to all possible cloud and data platforms to which event-driven microservices can be deployed to. However, as SO is an industry-strength questions & answers platform for developers, their questions tend to shed light on the most adopted industrial practices. Thus, the 10-year question range covered in this study strengthens our perception that the findings highlight a substantial portion of the challenges in the state of the practice.











§ RELATED WORK



In this section, we summarize related work on challenges that developers face in microservice architectures.

Benefits of microservice adoption.
Zhang et al.  <cit.> conduct interviews with microservice practitioners to understand the benefits brought about by microservice adoption. They find organizational transformation, decomposition, distributed monitoring, and bug localization as prominent challenges. Wang et al. <cit.> interview and survey practitioners to collect and categorize best practices, challenges, and their related successful solutions employed by practitioners. They find managing API changes, lack of support for monitoring, and finding microservice granularity as common challenges.

Microservice issues.

Zhou et al. <cit.> performed a survey to characterize typical faults, debugging practices, and the challenges entailed by troubleshooting failures in microservice architectures. They find microservice developers can benefit from improved trace visualization tools, specially in cases related to microservice interactions. Ramírez et al. <cit.>
mine SO posts to identify common issues on
development and testing microservice applications. They find missing parameters as a prominent issue in communicating with other microservices, wrong library versions as the main impediment for service discovery, and connecting to other microservices with correct user credentials as an example of authentication & authorization challenge. Waseem et al. <cit.> employ a mixed-method empirical study to understand the types of issues microservice developers experience. Similarly to Ramírez et al. <cit.>, they find programming errors, missing artifacts, invalid configuration and communication as the main causes behind the issues. 

Data Management in Microservices.
Laigner et al. <cit.> characterize data management challenges in microservice architectures through a mixed-method empirical study. They focus on analyzing developers' pitfalls when implementing data management logic in the application layer and discussing the limitations of state-of-the-art database systems that prevent them from better serving microservice architectures.

Event Sourcing Pattern.
Overeem et al.  <cit.> study the challenges practitioners experience using event sourcing. Through interviews with 25 engineers, they find event system evolution, steep learning curve, lack of available technology, rebuilding projections, and data privacy as the main issues affecting the development of systems that employ the event sourcing pattern. From 19 systems mentioned by interviewees, only 8 apply the microservice architectural style.

Security in Microservices.
Nasab et al. <cit.> performs a mixed-method empirical study to understand the security practices in microservice systems. They collect 28 security practices open-source repositories and SO posts, which were later confirmed through a survey with practitioners.

Exploratory Studies in Microservices.
Hacaloglu and Demirors <cit.> study the usefulness of events for software size measurement. They find preliminary evidence that events can supplement software size measurement techniques. Lazzari and Farias <cit.> reports an exploratory study that compares event-driven and REST architectural styles in the context of modularity. They find preliminary evidence that event-driven architecture improves the separation of concerns. 




Overall, we observe that challenges related to event management are overlooked in the literature. For instance, although few works report developers mentioning possible issues with message technology systems <cit.> and asynchronous task invocation  <cit.>, they are often addressed as general microservice configuration or invocation problems, preventing a proper characterization of the problem. Therefore, although event management in microservices has been rapidly gaining industry popularity <cit.>, it has not received due attention from the research community. In this work, we make the first attempt to investigate the specific challenges that developers face when managing events in microservice architectures.




§ CONCLUSION


    
In this paper, we mine and analyze several relevant Stack Overflow questions to characterize the state of the practice and the challenges microservice developers face while managing events. We identify key patterns developers use while attempting to realize their functional and non-functional requirements, suggesting a tension between achieving requirements related to data consistency, loose coupling, and performance. 

To further understand these tensions, we manually examined 628 sampled questions and identified key challenges microservice developers face in varied application scenarios. These include issues related to queuing, delivering, and processing events at the application level, as well as monitoring and securing the event flow. Based on the myriad of practical findings, we provide actionable implications for messaging systems, framework maintainers, cloud providers, and researchers. 

We hope that the results drive the reflection of microservice developers and researchers to escape from the dangers of asynchronous, event-based designs and to build event management technologies that meet the expectations of microservice developers respectively.



	We thank Jean Mello and Marcos Antonio Vaz Salles for early discussions and initial drafting of this manuscript.
	
	This work was supported by the PAPRICAS.org project - Independent Research Fund of Denmark (Number 9131-00077B). Part of the computation done for this project was performed on the UCloud interactive HPC system.


ACM-Reference-Format

  


 Paper Title: Towards Automated Continuous Security Compliance.
















angermeir@fortiss.org
0000-0001-7903-8236

  fortiss and Blekinge Institute of Technology
  Munich and Karlskrona
  Germany and Sweden



fischbach@fortiss.org
0000-0002-4361-6118

  fortiss and Netlight Consulting GmbH
  Munich
  Germany

  

fabiola.moyon@siemens.com
0000-0003-0535-1371

  Siemens Technology and Technical University of Munich
  Munich
  Germany



daniel.mendez@bth.se
0000-0003-0619-6027

  Blekinge Institute of Technology and fortiss
  Karlskrona and Munich
  Sweden and Germany













§ ABSTRACT 
Context: Continuous Software Engineering is increasingly adopted in highly regulated domains, raising the need for continuous compliance. Adherence to especially security regulations – a major concern in highly regulated domains – renders Continuous Security Compliance of high relevance to industry and research. 

Problem: One key barrier to adopting continuous software engineering in the industry is the resource-intensive and error-prone nature of traditional manual security compliance activities. Automation promises to be advantageous. However, continuous security compliance is under-researched, precluding an effective adoption.

Contribution: We have initiated a long-term research project with our industry partner to address these issues. In this manuscript, we make three contributions: (1) We provide a precise definition of the term continuous security compliance aligning with the state-of-art, (2) elaborate a preliminary overview of challenges in the field of automated continuous security compliance through a tertiary literature study, and (3) present a research roadmap to address those challenges via automated continuous security compliance.







<ccs2012>
   <concept>
       <concept_id>10002978.10003022.10003023</concept_id>
       <concept_desc>Security and privacy Software security engineering</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10011007.10011074.10011099</concept_id>
       <concept_desc>Software and its engineering Software verification and validation</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10011007.10011074.10011099.10011105</concept_id>
       <concept_desc>Software and its engineering Process validation</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Security and privacy Software security engineering
[500]Software and its engineering Software verification and validation
[500]Software and its engineering Process validation









Towards Automated Continuous Security Compliance
    Daniel Mendez
    August 5, 2024
================================================





§ INTRODUCTION

Over the last two decades, cse, entailing Agile and DevOps <cit.>, transformed how software is developed, with the promise of creating software faster, providing value earlier, and, thus, resulting in higher project success rates. Some of the success factors are flexible adoption of requirements, early customer integration, enhanced team collaboration (e.g., between Dev, Ops, and Sec), and continuous experimentation and improvement <cit.>. In the light of highly regulated domains such as healthcare, critical manufacturing, or transport systems, the same benefits entail challenges for providers, seemingly rendering cse and highly regulated environments incompatible. Providers in highly regulated domains have to adhere to regulations like laws (e.g., GDPR [<https://data.europa.eu/eli/reg/2016/679/oj>]), standards (e.g., ISO 27001 <cit.>), internal policies, contracts or best practices to ensure the quality of their systems over disciplines including security, safety or privacy. Non-compliance with applicable regulations can negatively impact the quality of a system, with various risks for providers. These risks range from reputation damage over monetary loss[<https://web.archive.org/web/20230708131758/https://edpb.europa.eu/news/news/2023/12-billion-euro-fine-facebook-result-edpb-binding-decision_en>] to competitive disadvantage <cit.>. 
Providers try to reduce those risks already in requirements engineering where regulations provide a source for requirements to be extracted, analyzed as vague and volatile subjects (e.g., <cit.>), and implemented. To ensure compliance with regulations, companies must assess and demonstrate the effective fulfillment of identified requirements. In the context of cse, requirements extraction, analysis, implementation, and demonstration (e.g., by verification) have to be performed frequently during development instead of following big-bang schedules, e.g. during releases <cit.>, resulting in Continuous Compliance. However, manual compliance activities are too resource-intensive to keep up with the ever-changing nature and high development velocity of cse. Already in 2014, Fitzgerald and Stol acknowledged Continuous Compliance as a major challenge for highly regulated domains in adopting cse <cit.>. Since then, more and more providers of software-intensive systems in highly regulated domains have adopted cse, exacerbating the challenges and pressing need to overcome them. 

Problem Statement. Automation promises to be advantageous. However, the multi-faceted field of Continuous Compliance is still vastly under-researched. Even though aspects such as automated compliance checking gained attention over recent years <cit.> with promising approaches such as <cit.>, the need for more extensive and structured research on automation approaches is highlighted by Castellanos Ardila et al. <cit.>: It is difficult to guarantee industrial adoption when there is nonexistent or loosely coupled tool support [...]. Thus, it is crucial to provide adequate and complete tool support for automatically [sic] perform compliance checking <cit.>. While automation approaches have gained attention over the last years, the compliance challenges of cse in highly regulated domains and the structurization of such seem very much neglected. From a security compliance perspective, Moyón et al. <cit.> offer preliminary insights. However, further research seems necessary to provide an extensive landscape of challenges in employing cse while adhering to security regulations (for now referred to as Continuous Security Compliance). Exacerbating the issue, to this day, little research has been performed on the requirements and their systematization for automation solutions in continuous compliance, even less Continuous Security Compliance. Instead, researchers and practitioners too often follow a solution-oriented approach <cit.> where they focus on solutions while leaving requirements underspecified and poorly understood. To our knowledge, Kellogg et al. <cit.>, and Santilli et al. <cit.> are those that stand out in literature by describing five and 13 general requirements for Continuous Security Compliance (covering automation aspects) - in a structured way from an industrial perspective.

Objectives. With our research, we operate at the intersection between requirements engineering, security, and continuous software engineering. We aim to contribute to introducing automation in Continuous Security Compliance in a problem-driven manner. To this end, we formulate the following research goals (RG):

    
  rg1 understand the challenges of adhering to security compliance regulations in continuous software engineering projects
    
  rg2 understand  requirements and constraints for automation in Continuous Security Compliance as suggested in literature and practice
    
  rg3 analyze the potential and limitations of automation as treatment towards the identified challenges and its implications for manual involvement
    
  rg4 develop and evaluate – along the previously determined potential and limitations – sensible solutions containing automation in industrial settings to treat respective challenges


Contributions. In this manuscript, we define the terms Continuous Compliance and Continuous Security Compliance as required for our research context. We provide the first insights into ongoing research through a tertiary study, in which we investigate and validate industrial challenges reported by practitioners. We conclude by presenting our research roadmap towards achieving our objectives and bringing together requirements engineering, security, and continuous software engineering. One hope we associate with this contribution is to foster a critical reflection and discussion on contemporary challenges in automated Continuous Security Compliance and engage the research community to join our research endeavor.

Data Availability. The protocol of our literature study and the analyzed and processed data is shared in our online material <cit.>.


§ STATE OF RESEARCH

Based on the current state of research, we lay the foundation for future research by defining the term csc in <Ref> and contributing to rg1 through an analysis of relevant challenges reported in the literature in <Ref>.



 §.§ csc

The field of csc has gained more interest over the last years. To effectively operate in this field, researchers need a definition they can identify with and refer to. While the phrase csc is used in some resources (<cit.>), to our knowledge, it has only been defined explicitly once by Moyón et al. as combining CC [Continuous Compliance] and CS [Continuous Security] through the holistic view of standardisation that spans across people, processes, and technology. Regulatory requirements are utilised to derive security activities and integrate security into a process while making it standards-compliant <cit.>. This definition references two concepts, continuous compliance and continuous security, defined by Fitzgerald and Stol in the seminal framework for continuous software engineering Continuous * <cit.>. While the existing csc definition coined the field as part of the cse research agenda, it lacks preciseness concerning terminology in the compliance field (e.g. reference to regulatory requirements and compliance to standards), and has limitations on the applicability for practitioners (e.g., security compliance solely mentioned as process security compliance, assuming product security compliance is a consequence of process compliance). An updated definition is required to improve understandability based on a broader analysis of the current state-of-art, beyond Continuous *, and, therefore, extend the concept's applicability.

We aim for a csc definition that fits contemporary research while considering practice observations gathered during the last +5 years of our research on csc, together with our industrial partners. 
To that end, we conducted an ad-hoc literature review (as defined by <cit.>) on the terminology related to continuous compliance. We reviewed 12 resources mentioning or defining continuous compliance, including theses and grey literature (<cit.>). We extracted relevant parts from each of the manuscripts, providing insights into the concepts covered in definitions and the actual definitions (see <cit.> for details). Most resources framed continuous compliance in light of their contribution. This confirms the observations by Santilli et al. <cit.>, who independently analyzed 13 resources, of which three overlap with ours. Of the resources we reviewed, only two provided contribution-independent definitions. One is the original definition by Fitzgerald and Stol <cit.> from 2014, and the other was recently introduced by Santilli et al. <cit.>. Santilli et al. also scanned the field for a proper definition and provided their version. They define continuous compliance as the application of approaches and techniques to guarantee compliance with standards throughout a system's lifecycle. Continuous compliance activities span the various development phases [...]. Their definition covers the most relevant aspects we could distill from the other resources as well as from our own experience (1) Continuous execution of compliance activities (e.g. assessment, update of artifacts), (2) adherence to regulatory requirements, and (3) compliance over the entire development life-cycle. However, none of the newer definitions explicitly oriented on cse and its principles and goals, which is the fourth aspect we add to our definitions. In our experience, the goal of continuous compliance is as manyfold as the goals of cse ranging from the enablement of compliance in fast-paced development, over enablement of organizations to respond quickly to changing compliance requirements, to continuous improvement of the compliance state.

Given these four aspects, we define the term continuous compliance ourselves and base the definition of csc on it. The goal of the definitions is to not only provide a contribution-independent, yet concise updated definition but also to respect the background under which it was originally introduced: continuous software engineering. We propose to define Continuous Compliance as a set of practices to ensure compliance targets adherence to requirements derived from relevant regulatory sources, integrated holistically into the product development life-cycle, following continuous software engineering principles and goals.

The benefits of the updated continuous compliance definition are its alignment with current state-of-the-art practices, its shortness, agnosticism regarding the regulatory source and product development methodologies, as well as the explicity orientation on continuous software engineering principles and goals. The definition covers requirements from various regulatory sources such as standards, laws, policies, or contracts and is not restricted to a certain domain (e.g. security, safety) or compliance target (e.g. software development process). While the definition ensures that the original background (cse) is preserved, it does not further stipulate the product development methodology. Additionally, given that large parts of regulated domains haven't and might never fully transition to cse, we do not prescribe the employed compliance process or its execution frequency. This flexibility enables practitioners to relate to csc already in the early stages of the continuous improvement journey in their organization (e.g. following the stairway to heaven model of <cit.>). The main downside of the flexibility in our definition is that it does not provide direct guidance toward implementing continuous compliance (e.g. compliance process execution frequency). However, this flaw also applies to existing definitions, and concrete guidance requires further in-depth research. With security as one application domain for continuous compliance, the before discussed advantages and disadvantages also apply to the csc definition. For csc the compliance target is usually products or processes and the requirements to comply with are derived from regulatory sources that are relevant to security and the respective organization.


We define csc as a set of practices to ensure product and process adherence to requirements derived from relevant security regulatory sources, integrated holistically into the product development life-cycle, following continuous software engineering principles and goals.



 §.§ Challenges in Continuous Security Compliance


Large parts of security compliance research focus on traditional software development. This state does not reflect the ever-growing adoption of cse methodologies in the industry. Consequently, effective research is difficult without structured overviews of the challenges experienced in practice when following CSE principles while pertaining to security regulations. This state is exacerbated by the difficulties of cooperating with industry on such delicate matters and publishing the findings. One instance of such research is Moyón et al., who present 15 challenges experienced in three different companies <cit.>.

Contributing to rg1, we research the pertinence of challenges on csc in literature, validating and extending the challenges of Moyón et al. <cit.>. We performed a tertiary literature study on three major scientific databases: IEEE Xplore, ACM Digital Library, and Scopus. We followed Kitchenham <cit.> for the planning and execution of the literature study. The literature review protocol and other relevant artifacts for traceability are available in our online material <cit.>. The search initially yielded 110 hits, one of which was a duplicate. Manual filtering based on the abstract and the full text left three relevant manuscripts (<cit.>).
Following Wohlin's guidelines <cit.>, we performed forward and backward snowballing following the references in each of the manuscripts and employing the citation feature of Google Scholar, yielding one more publication <cit.> (referenced by <cit.>).

The identified manuscripts investigated 15 security challenges in large-scale agile software development <cit.>, 23 challenges while adopting DevSecOps <cit.>, four security compliance challenges in agile software development <cit.>, and 20 challenges while developing secure software using agile under the consideration of security compliance. In total, we extracted 62 challenges from those manuscripts.

Validation:
Nine of the 15 challenges reported by Moyón et al. <cit.> (C1, C3, C4, C6, C7, C8, C9, C13, C14) could be found in literature. Leaving six challenges unvalidated. Those challenges are not necessarily irrelevant, as further research might validate them. The challenges of <cit.> and the number of challenges in the literature that validated them are listed in <Ref> (denoted as C1-C15). The pertinency of challenges C6, C7, C8, and C14 in literature (see <Ref>) highlights and aligns with Moyón et al.'s perspective on the need for research about automatic continuous security compliance.

Extension:
The remaining 47 challenges from the literature were used to extend the challenges given by Moyón et al. <cit.>. To this end, we performed a thematic analysis, analyzing and aggregating similar challenges across varying granularities into 12 common new challenges (NC). The online material <cit.> details the analysis and aggregation steps, and <Ref> depicts all new challenges with the number of supporting challenges from literature (denoted as NC1-NC12). For illustration purposes, we briefly describe one such new challenge. NC4: Security compliance evidence generation and documentation too resource consuming. NC4 is based on three challenges that are thematically connected: 

    
  * Security compliance requires documented evidence which increases delivery time <cit.>
    
  * Security assessment favors detailed documentation <cit.>
    
  * Teams need extra effort [...] to involve security experts and assessors to evaluate accuracy of documentation <cit.>


As becomes apparent, security compliance evidence is the common subject here, with either the generation of evidence or the documentation of evidence being the activities that imply high resource demand (e.g., an increased delivery time or extra effort). Three challenges in literature could not be aggregated: Customer readiness for applying frequent releases to production setup <cit.>, Lacking guidance on the implementation of scaling agile frameworks <cit.>, and Overcome security expert shortage at scale <cit.>.


Usage of Challenges & Preliminary Evaluation:
Of the new challenges, the following seem to be relevant for automated csc: NC1, NC2, NC4, NC5, NC7, and NC8. These challenges are part of the basis for our research roadmap detailed in <Ref>.

As a preliminary evaluation, we interviewed a security expert with more than 10 years of security experience in the industry to evaluate the understandability of the challenges and get their perspective on the most relevant challenges for automated csc. 
To that end, we took a snapshot of the existing relevant challenges at that time (C6, C7, C8, C14, NC1, NC2, NC4).
We first asked for understandability of the new challenges and, afterward, which were the most relevant to them. All new challenges were perceived as easily understandable, and no clarification was necessary during the interview. The security expert perceived C6, NC1, and NC2 as the most pressing challenges in the subset providing indicators for the focus of future research (see <Ref>).




Nine challenges delineated by Moyón et al. <cit.> were validated in literature: C1, C3, C4, C6 - C9, C13, C14. The list of challenges was extended by 12 challenges.
Challenges relevant for future automated csc research: C6, C7, C8, C14, NC1, NC2, NC4, NC5, NC7, NC8.







§ RESEARCH ROADMAP

This research has four goals, as outlined in <Ref>. In the following, we will give an overview of the research environment and outline our research vision to address the research goals. <Ref> summarizes the endeavor, highlighting research activities, the environments in which they are carried out, and research outcomes.



 §.§ Research Environment

Our research is performed with an industry partner, a large enterprise operating in highly regulated domains, aligning and building on top of their strong research foundation of the last 6 years (selected examples: <cit.>) in the area of compliance to security standards applicable to the development and operation of software-intensive industrial automation and control systems. To ensure generalizability, we extend and validate our findings wherever possible with industrial companies partnering with Blekinge Institute of Technology as part of Software Engineering ReThought[https://rethought.se/], a large-scale academia-industry collaboration network entailing diverse industrial and academic backgrounds.



 §.§ Research Vision

We structure the research according to the dsm <cit.>, offering an iterative framework towards the problem investigation, treatment design, and treatment validation. DSM further enables the technology transfer to real-world application, closing the engineering cycle described by Wieringa. Since the dsm describes a cyclic approach, we will perform steps 2 and 3 iteratively, adapting to changing requirements, challenges, and technologies. 

Step 0. Preliminary Challenges lays the foundation for the problem investigation of the dsm. We analyzed and reported the challenges in csc (rg1) in <Ref>. These preliminary challenges are used in the next step of our research.

Step 1. Challenges, Root Causes, Requirements contributes to the problem investigation and addresses parts of the treatment design of the dsm. We analyze the challenges in csc and their root causes (rg1), the potential of these challenges for automation (rg3), and the requirements for automation in continuous security compliance (rg2). 

In the first stage, we perform a survey to gather a broad perspective on the challenges and their automation potential from different stakeholders. This activity is performed internally at our industry partner and, to scale up, in parallel in the entire network of the large-scale academia-industry collaboration Software Engineering ReThoughtnote1 as well as with selected external partners. Based on the survey findings, in the second stage, we design and perform in-depth interviews with cse stakeholders targeting the root causes of the challenges, their automation potential, and respective limitations and requirements for their automated treatments. In all of these cases, we will thoroughly monitor literature (e.g. <cit.> reports on automation limitations, <cit.> lists relevant requirements).

Step 2. State of the Art Analysis and Solution Design to progress in the treatment design of the dsm. Together with our industry partner, we decide on a subset of challenges to focus on. To scan for available solutions to address those challenges, we will perform a technology screening through a systematic literature review. Given the existing gap between research and industry, we will additionally review the potentials, limitations, and use cases of available solutions on the market (such as Drata[https://drata.com/], Vanta[https://www.vanta.com/], Scrut[https://www.scrut.io/]) in the hope of better understanding and ultimately bridging this gap. We will continuously experiment to test the applicability of chosen automation approaches to the practical context. This shall provide a basis for our treatment design in which we will adopt existing approaches or design new ones (rg4). We then continue with the treatment validation in step 3.

Step 3. Treatment Validation to close the dsm cycle. We validate existing and newly developed treatments for (1) the satisfaction of the requirements identified before, (2) the impact on csc, and (3) the value added by the treatments to cse projects. For that purpose, we not only integrate the treatments into the processes of cse projects at our industry partner but also measure their impact on cse principles and consequently on their development process. Further, we analyze with security experts (e.g. auditors) whether the treatments' outputs improve csc, e.g. provide valid evidence for compliance. This way, highly regulated organizations can integrate our proposed treatments into their projects while ensuring the treatment's acceptance by auditors. 


We perform this research in an industrial setting iteratively in three stages: (1) research of challenges in practice, (2) analysis and design of treatments for these challenges, and (3) validation of the treatment designs.



§ THREATS TO VALIDITY

We describe the threats to validity structured by the three contributions reported in this manuscript. First, the definition of continuous security compliance is based on an ad-hoc literature review. To put it in Ralph's and Baltes' way: The limitations of ad hoc review are the unsystematic nature that might be subject to sampling bias and hinders replication <cit.>.
However, given the large content-wise overlap in-between literature and also with our insights gained from the industrial environment, we are confident that the threat is a minor one.

Second, the extension and validation of challenges in continuous security compliance. As with any systematic (tertiary) literature review, the exclusion of relevant resources is a major threat. We experimented with and improved the search strategy in four iterations before we reviewed the title and abstract of each manuscript. We only dismissed manuscripts if we were certain about their irrelevance to us. Otherwise, we read the entire manuscript. After our analysis and synthesis, the first and the third authors discussed the challenges stemming from the extension to reduce experimenters' bias. In addition, we interviewed a security expert to ensure the validated and extended challenges were suitable for further research in practice.

Finally, the research roadmap describes our long-term research projects spanning multiple years which might be subject to change. As with any research roadmap, it is exposed to various (subjective) factors influencing its creation. Although our roadmap is rooted in industrially relevant challenges and corroborated with literature results, strengthening our confidence, we cannot guarantee its relevance. As an additional means to mitigate related threats, we intentionally share our roadmap with this paper to receive feedback and support from the research community. 



§ CONCLUSION

The growing adoption of cse in highly regulated domains raises the need for csc. Traditional compliance activities are slow, labor-intensive, and costly, rendering them unsuited for the fast-paced development and release of contemporary software development. Automation could contribute to transitioning security compliance from traditional, rigid, and slow to continuous, flexible, and fast-paced. However, to this day, automated csc is under-researched. This state forces companies operating in highly regulated domains to trade off between fast-paced development following cse principles and reduced security risks by adhering to security regulations. We contributed to closing this gap by 1. providing a precise yet agnostic up-to-date definition of csc reflecting its four main components: continuous execution of compliance activities, adherence to relevant security regulatory sources, compliance over the entire development life-cycle, and its continuous software engineering background. 2. we offered a preliminary extension and validation of industrial challenges at the intersection of requirements engineering, security compliance, and continuous software engineering. We also distilled six new challenges for future research of automated csc. 3. we outlined a research roadmap entailing the industrial research environment, research activities, and the outcomes required to treat the identified challenges successfully. With this manuscript, we hope to foster a critical reflection of the contemporary challenges and potential treatments of automated csc and engage the research community to join our research endeavor. 




ACM-Reference-Format









 Paper Title: Analysis of Functional Insufficiencies and Triggering Conditions to Improve the SOTIF of an MPC-based Trajectory Planner.
















Analysis of Functional Insufficiencies and Triggering Conditions to Improve the SOTIF of an MPC-based Trajectory Planner
    Mirko Conrad[samoconsult GmbH, Französische Str. 13, 10117 Berlin, Germany], Georg Schildbach[University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany corresponding author ()]
    July 2024
=============================================================================================================================================================================================





§ ABSTRACT

Automated and autonomous driving has made a significant technological leap over the past decade. In this process, the complexity of algorithms used for vehicle control has grown significantly. Model Predictive Control (MPC) is a prominent example, which has gained enormous popularity and is now widely used for vehicle motion planning and control. 
However, safety concerns constrain its practical application, especially since traditional procedures of functional safety (FS), with its universal standard ISO 26262, reach their limits. Concomitantly, the new aspect of safety-of-the-intended-function (SOTIF) has moved into the center of attention, whose standard, ISO 21448, has only been released in 2022.  Thus, experience with SOTIF is low and few case studies are available in industry and research. 
Hence this paper aims to make two main contributions: (1) an analysis of the SOTIF for a generic MPC-based trajectory planner and (2) an interpretation and concrete application of the generic procedures described in ISO 21448 for determining functional insufficiencies (FIs) and triggering conditions (TCs). Particular novelties of the paper include an approach for the out-of-context development of SOTIF-related elements (SOTIF-EooC), a compilation of important FIs and TCs for a MPC-based trajectory planner, and an optimized safety concept based on the identified FIs and TCs for the MPC-based trajectory planner.



§ KEYWORDS

Automated Driving System (ADS) ,  
Autonomous Vehicles ,  
Trajectory Planning ,  
Model Predictive Control (MPC) ,  
Functional Safety (FS) ,  
Safety of the Intended Functionality (SOTIF) ,  
Functional Insufficiencies (FIs) ,  
Triggering Conditions (TCs) ,  
SOTIF-related Element out of Context (SOTIF-EooC)




§ INTRODUCTION


Since the beginning of the 21st century, automated and autonomous vehicles have become a major subject of research across the globe, in academia and industry <cit.>. The technological progress is fueled by ever more powerful hardware for computation, communication, and sensing. Another relevant factor is the rapid advances in mathematical algorithms, especially in Artificial Intelligence (AI), Machine Learning (ML), and Numerical Optimization (NO). A plethora of new and powerful methods have been developed and successfully demonstrated in actual vehicles on the road, among them Model Predictive Control (MPC). 

MPC is a popular computer-based control method <cit.> that has been frequently tested in automated driving system (ADS) prototypes <cit.>. Thus far, however, it has not been considered a viable option for series production. The main reason is its relatively high complexity, causing relatively high efforts in the development process and increasing the computational hardware requirements. Arguably, an even bigger challenge results from safety concerns related to MPC. These concerns are similar to other state-of-the-art methods of AI, ML, and NO. 

Due to the complexity of these methods, and the complexity of the corresponding automated driving tasks, Functional Safety (FS), which is concerned with safety in the presence of malfunctions, is no longer the single most important aspect of safety. Instead, Safety of the Intended Function (SOTIF), which is concerned with the  safety of the nominal function even if all system elements work as intended, becomes equally important. The intention of this paper is hence to present a contribution towards a viable SOTIF concept for an MPC-based motion planning module in automated driving systems (ADSs).



 §.§ Model Predictive Control


Over the past decades, MPC has developed into a mainstream approach for the control of complex and safety-critical systems, including ADSs. The basic idea is to formulate and solve a finite-horizon optimal control problem (FHOCP) as a numerical optimization program in real time <cit.>. The FHOCP contains a model of the system and naturally integrates constraints on the model states and control inputs. It can be used for motion planning <cit.> and trajectory tracking control <cit.>, where feedback is introduced by a receding-horizon implementation. It can also be used as a hybrid module, covering some or all aspects of planning and tracking control <cit.>. It has even been used to cover some aspects of tactical planning, such as lane change decisions <cit.>.

The current trend is to use MPC as a motion planner, such as in the commercial product ProDriver by Embotech <cit.>. The goal is to integrate as many aspects as possible into the motion planning module, in order to slim down the module interfaces. Interfaces always require additional engineering efforts and they are a common source of functional, and thus also safety problems. The MPC module should thus be capable of making tactical decisions, by introducing integer variables, and ideally should also provide a basic level of vehicle and actuator control.

Recent research in the context of MPC for ADSs has begun to focus on interaction-aware motion planning <cit.>. The basic idea is to consider the predictions of dynamic objects in the environment not as invariably fixed, but as interdependent with the other objects and also dependent on the actions of the ego vehicle. Hence the future trajectories of dynamic objects are included in the model-based predictions in the MPC. The interactions themselves can be modeled by different approaches, such as game theory <cit.> or neural networks <cit.>. 

A common idea is to represent the intention of other drivers as modes <cit.>, corresponding to lane keeping or a lane change, for example. The remaining uncertainty regarding the exact trajectory of the dynamic object can be considered as stochastic noise. Another noteworthy approach is Branch MPC <cit.>, where the modes of each dynamic object and the ego vehicle may change at several specific instances over the prediction horizon.



 §.§ Safety of the Intended Function (SOTIF)


SOTIF is a comparatively new branch of safety that is becoming increasingly important for SAE driving automation levels 3 or higher. The first edition of the applicable SOTIF standard ISO 21448 has only been published in 2022 <cit.>. 
Due to its relatively recent introduction, scientific literature on the application of SOTIF to ADSs is still  sparse. 
Zhu et. al. <cit.> emphasize that the research on the SOTIF of automated and autonomous driving systems is still in its infancy. Their paper focuses on safety analysis methods that could be applied to SOTIF, but it does not cover specifically the topic of identifying and analyzing functional insufficiencies (FIs) or triggering conditions (TCs).


Qidong et. al. <cit.> state that the identification and systematic analysis of FIs and TCs remains a major challenge for the SOTIF. Using an Adaptive Cruise Control (ACC) system as an example, the authors propose a scenario-based analysis method of SOTIF triggering conditions. In this approach, the TCs are analyzed according to elements of SOTIF scenarios that are formed in view of FIs. 
The approach is then applied to the 'Sense' portion of the ACC system. The 'Plan' part is not addressed in this paper.



Junfeng et. al. <cit.> apply the System-Theoretic Process Analysis (SPTA) method to study the SOTIF of a Lane Keep Assistance (LKA) system. The study intends to verify the feasibility of STPA as part of the SOTIF lifecycle.
The paper proposes a set of TCs for the LKA, but not all of them seem to be TCs in the sense of ISO 21448. A subset of these TCs will be used as input for the SOTIF analysis of the MPC-TP. 
In the analysis, there is also no specific focus on the 'Plan' part.



 §.§ Content and Contributions


This paper describes a possible approach for the analysis of FIs and TCs, with the goal of improving the SOTIF of a generic planning module for an ADS. The trajectory planning module is based on the Model Predictive Control (MPC) approach. It is assumed that the MPC-based trajectory planner (MPC-TP) can be integrated into different vehicle-level functions and that it is re-usable for different vehicles within a specified vehicle class. Hence it is treated as a SOTIF-related element out of Context (SOTIF-EooC), which is a particular novelty compared to the existing literature.

The main objective of this paper is to demonstrate how the framework of ISO 21448 <cit.> can be used for improving the SOTIF of a modern, state-of-the-art planning algorithm. The system architecture and functional specifications are outlined in Section <ref> and the details of the MPC-TP are provided in Section <ref>. The relevant SOTIF background is outlined in Section <ref>. The main part of this paper is the SOTIF analysis in Section <ref>, which describes the application of various methods and techniques from ISO 21448 <cit.> towards the identification of FIs and TCs for the MPC-TP. The analysis leads to a refined specification and design of the MPC-TP, as described in Section <ref>. The conclusions of this work are presented in Section <ref>. The main contributions of this paper can be summarized as follows:


  * Application of the abstract / generic framework of the new SOTIF standard ISO 21448 to a realistic case study, with a focus on early parts of the SOTIF lifecycle;

  * Interpretation and application of generic methods in ISO 21448 for the determination of FIs / TCs, in the context of a specific use case;

  * Derivation of a structured list of FIs / TCs as a consolidated result of the SOTIF analysis (in a tabular format);

  * A refinement of the notion of an out-of-context development in the scope of SOTIF.










§ INITIAL CONCEPT FOR VEHICLE LEVEL CONTROL SYSTEM






The starting point for the design of the planning module as a SOTIF-SEooC are assumptions about the encompassing vehicle control system into which the module is to be embedded, as well as the related interfaces. Furthermore, specifications are needed about the overall functionality and the operational design domain (ODD). 



 §.§ Functional Architecture


The functional architecture of a generic ADS is  shown in Figure <ref>. The MPC-TP is shown in the middle (red). The task of the MPC-TP is to compute a reference trajectory, which is then used by the trajectory controller (blue) and lower-level modules to control the vehicle's actuators. The actuators are an integral part of the chassis, including, in particular, the drivetrain and braking systems for longitudinal motion control and the steering system for lateral motion control. 

The MPC-TP receives information from centralized estimation modules (green), which may serve also other functions and/or other modules in the architecture <cit.>. The key point is that they are developed independently of the MPC-TP. These modules provide a local map, including the topology and road geometry and information about the infrastructure, such as traffic signs (Map Fusion). They also include the relative position of the vehicle on the map, the vehicle's dynamic states, a list of relevant vehicle parameters, and diagnostic information about the vehicle's sensors and actuators (Ego Motion Fusion). Furthermore, a list of relevant traffic objects in the vicinity of the vehicle is provided, including their classification, position, speed, and other states, as well as a situation analysis and prediction(s) about (possible) future evolution of the situation (Object Fusion and Prediction). The sensors also deliver important information about the environment, such as ambient temperature or road surface (Environment Fusion). Finally, the MPC-TP may receive high-level commands by the human driver or the ADS, such as the reference speed, which lane or road to use, or where to take a turn (Tactical Planner).









	






 §.§ Encompassing Vehicle Control System


The MPC-TP is treated as a SOTIF-related element out of context (SOTIF-EooC), as detailed in Section <ref>. Hence it shall be possible to integrate the MPC-TP into different the SOTIF-related systems as well as different vehicle types. For this paper, it is assumed that the MPC-TP can be used in the context of the following systems for a given class of passenger cars:

    
  F1) Energy Efficient Lane Centering and Adaptive Cruise Control (E2LCACC): E2LCACC is a SAE level 2 function that combines the classic Lane Centering (LC) and Adaptive Cruise Control (ACC). It performs the lateral and longitudinal control of the vehicle, in order to keep the vehicle on a reference path in its lane and to follow the preceding vehicle. E2LCACC contains AI-based elements for energy efficient driving, in particular to determine the optimal speed profile. 
    
  F2) Active Lane Change Assist (ALCA): ALCA is SAE level 2 or 3 function designed to help drivers changing lanes safely by automatically steering the vehicle into the adjacent lane. ALCA operates in combination with E2LCACC and also provides longitudinal and lateral control of the vehicle during overtaking. The driver can issue lane change requests to the left or the right, e.g., by operating the indicator lever. The ALCA then checks whether a safe lane change is possible within 10 seconds. If it is possible, it performs the lane change; if it is not possible, it keeps the vehicle in the current lane and continues with E2LCACC.
    
  F3) Highway Pilot (HP): The HP is a SAE level 4 function that takes full control of the vehicle during highway driving. Only high level objectives are provided by the user, such as the destination, the desired route, the targeted arrival time, or the driving style. The driving itself, including all decisions, is performed autonomously by the HP.





 §.§ Operational Design Domain (ODD)


The vehicle-level functions F1, F2, F3 are only to be used within the conditions and limits of the ODD () specified in Table <ref>. The ODD documents the specific conditions under which the ADS is designed to function <cit.>. Its description is based on the structure provided in DOT HS 812 623 <cit.>.





 §.§ Assumptions about the Functionality and the Vehicle Class


Since the MPC-TP is considered as a SOTIF-EooC, specific assumptions are required about its integration into the vehicle and the vehicle-level functions F1, F2, F3, as well as its interfaces to other modules. An overview of the upstream and downstream modules can be gathered from Figure <ref>. The perception modules make use of appropriate sensors and communication infrastructure. The control modules are connected with the relevant vehicle actuators. It is beyond the scope of this paper to specify these modules here in detail. A list of the most relevant specifications for the ego vehicle (EgoV) and the module interfaces is provided below.


    
  Spec:ODDj) The EgoV belongs to a specific class of passenger vehicles, consisting of a finite number of vehicle types.
    
  Spec:ODDk) All possible EgoV types are known at the time of the MPC-TP development process, including their physical design and specifications, chassis and suspension systems, and chassis control systems (Anti-lock Braking System, Electronic Stability Control, etc.).
    
  Spec:ODDl) Further parameters of EgoV (such as the vehicle mass, location of the center of gravity, moments of inertia, and tire parameters) are known to be within specified value ranges.
    
  Spec:ODDm) The loads inside the EgoV are within the limits specified by the respective vehicle type.
    
  Spec:ODDn) There are no external loads attached to the EgoV, such as a trailers, roof racks, or any type of load carriers. 




§ INITIAL CONCEPT OF THE MPC-TP


The specifications for the MPC-TP are based on the overall architecture in Figure <ref>. The MPC-TP is (initially) given by the single block displayed in Figure <ref>. The specifications concern the input signals, the output signals, and the contents of the block itself, and they are presented along this structure in the remainder of this section.





 §.§ Input Interface / Perception


For the interface to the map fusion module, the 6 layers road model specified in ISO 34503 <cit.> is considered. See Figure <ref> for an illustration.






  ) Map Fusion: The MPC-TP receives a layered map every 50 ms, including

    
  Spec:MapFusiona) geometry and topology of the road, lane and road boundaries (layer 1),
    
  Spec:MapFusionb) traffic signs, speed limits, traffic guidance, construction barriers (layer 2),
    
  Spec:MapFusionc) temporal modifications, e.g., due to construction sites (layer 3).


For the sake of the SOTIF-EooC, it is assumed that all of this map data is always fully accurate. The MPC-TP must plan its trajectories such that they conform to the applicable traffic rules (e.g., speed limit, no passing zone).



  ) Object Fusion and Prediction: The MPC-TP receives a complete list of dynamic objects in its relevant vicinity every 50 ms, including

    
  Spec:ObjectFusiona) other traffic participants (layer 4),
    
  Spec:ObjectFusionb) animals and other temporary obstacles, like dropped cargo, rocks, etc. (layer 4).


All dynamic objects include additional information about their classification (vehicle, truck, motorcycle, pedestrian, animal type, obstacle, etc.), position, velocity, and heading direction, if applicable. Furthermore, a reliable prediction for all of these objects is provided over the next 6 seconds. The uncertainty in these predictions is accounted for by the use of intervals (e.g., for the velocity) or bounding boxes (e.g., for the position).

No assumptions are made on specific sensors, their field-of-view, or their operational status. All assumptions are based on the interfaces to the sensor fusion modules. If the information flow on these interfaces cannot be guaranteed, the vehicle-level function will hand over to the driver (SAE levels 1,2,3) or transfer the vehicle to a safe state (SAE levels 4,5).



  ) Ego Motion Fusion: The MPC-TP receives relevant information about the states and parameters of the ego vehicle every 50 ms, along with a with maximum estimation error (marked with a “Δ”):

    
  Spec:EgoFusiona) the relative position of the ego vehicle on the map x,y (Δ x, Δ y),
    
  Spec:EgoFusionb) the ego vehicle velocity v_lon,v_lat (Δ v_lon,Δ v_lat),
    
  Spec:EgoFusionc) the ego vehicle's global heading angle ψ (Δψ),
    
  Spec:EgoFusiond) the ego vehicle's acceleration a_lon,a_lat (Δ a_lon,Δ a_lat),
    
  Spec:EgoFusione) the vehicle's mass m (Δ m),
    
  Spec:EgoFusionf) the vehicle's moment of inertia around the vertical axis I_z (Δ I_z),
    
  Spec:EgoFusiong) the position of the center of gravity relative to the midpoint of the rear axle x_cog, y_cog, z_cog (Δ x_cog, Δ y_cog, Δ z_cog),
    
  Spec:EgoFusionh) the cornering stiffness of the front and the the rear tires C_α,f, C_α,r (Δ C_α,f, Δ C_α,r).



  ) Tracking Controller: Further information about the ego vehicle, in particular about the states of the lower-level actuators, are provided to the MPC-TP through the Tracking Controller every 50 ms:

    
  Spec:TrackControla) the front wheels' steering angles δ_f,l, δ_f,r,
    
  Spec:TrackControlb) the current engine torque T_eng,
    
  Spec:TrackControlc) the total gear ratio i_gear,
    
  Spec:TrackControld) the effective brake torque T_brake.
    

Actuator failures, controller failures or performance degradations are not considered in this paper, as the focus is on SOTIF instead of functional safety.


  ) Environment Fusion: The MPC-TP receives accurate information about the environmental conditions every 50 ms, including

    
  Spec:EnvironmentFusiona) conditions of the road surface, in particular the friction coefficient μ (Δμ) (layer 1),
    
  Spec:EnvironmentFusionb) weather conditions and lighting conditions, in particular the side wind speed v_wind (Δ v_wind) (layer 4).


In reality, environment parameters, such as the friction coefficient or the side wind speed, are variable over space and/or time. For simplicity, however, the architectures presumes the communication of constant values, which characterize the entire drivable area of the ego vehicle over the relevant prediction time, i.e., in this case 6 seconds. The tolerance thus includes the measurement uncertainty as well as the true temporal and/or spacial variations, if applicable.



  ) Tactical Planner: The MPC-TP receives tactical choices from the Tactical Planner every 50 ms, containing

  
  Spec:TacticalPlanninga) the desired target lane (e.g., the current lane, the left or right lane), 
  
  Spec:TacticalPlanningb) a desired reference speed (e.g., because the vehicle is about to leave the highway on the next exit),
  
  Spec:TacticalPlanningc) optional information about the desired lateral position on the target lane (e.g., deviation from the lane center line),
  
  Spec:TacticalPlanningd) optional information about desired distances to other vehicles or objects (e.g., the desired gap to the leading or trailing vehicle).



All requests by the Tactical Planner are facultative. For example, if the Tactical Planner requests a lane change, it is the task of the MPC-TP to evaluate if this lane change is feasible and safe, given the dynamics of the vehicle and the traffic situation (dynamic and static obstacles, traffic rules, etc.). If it is not feasible or not safe, the MPC-TP resorts to a trajectory that keeps the EgoV on its current lane. Similarly, the desired distances in (Spec:TacticalPlanningc,d) are for performance purposes only, and not required for safety.

The tactical choices are generated by the active driving function. For example, for the E^2LCACC the unique command is always to keep the current lane, for the ALCA the lane change commands are issued by the driver, and for the HP the commands are issued by a dedicated software module.





 §.§ Output Interface / Reference Trajectory


The main output of the MPC-TP is a reference trajectory, as shown in the architecture in Figure <ref>. The reference trajectory consists of (i) a trajectory header and (ii) a trajectory body. The trajectory header contains a sequential number, the generation time, a validity time interval, and a checksum.



  ) Trajectory Body: The main part of the reference trajectory is the trajectory body. It is represented by N_p-1 linear segments, as illustrated in Figure <ref>. Each segment, numbered with i=1,…,N_p-1, carries the following data:

    
  Spec:TrajectoryBodya) x_i,y_i: the planar coordinates of the starting point of the segment,
    
  Spec:TrajectoryBodyb) ψ_i: the heading angle of the segment,
    
  Spec:TrajectoryBodyc) v_i: the reference speed of the vehicle along the segment,
    
  Spec:TrajectoryBodyd) δ_i: the reference steering angle along the segment,
    
  Spec:TrajectoryBodye) l_i: the length of the segment.
    
  Spec:TrajectoryBodyf) In addition, the final point of the trajectory is given in terms of its planar coordinates x_N_p,y_N_p.


Note that the trajectory body data are partially redundant. This is mainly to avoid unnecessary and mirrored re-calculations within the MPC-TP.





 §.§ Specification and Design of the MPC-TP


This case study considers a trajectory planner based on Model Predictive Control (MPC). A generic MPC formulation with a mainstream application to highway driving is used, in order to make the SOTIF concept in this paper as generally applicable as possible. It should be remarked, however, that numerous variations of MPC-based planning algorithms have been proposed in the recent literature <cit.>. Moreover, commercial modules for trajectory planning with MPC have been made available, such as the software ProDriver by Embotech <cit.>. It assumes a similar functional architecture as the MPC-TP in this paper, with a sampling time of 100 ms. 

The computations of the MPC-TP are based on an internal prediction model. Following <cit.>, a  dynamic bicycle model with a linear tire model (LDBM) is selected and discretizes with an appropriate numerical integration method (e.g., Runge-Kutta):

    𝐱_t+1=f(𝐱_t,𝐮_t) .

The model is augmented with a suitable patch to circumvent the singularity of the tire forces at the origin. Here t=0,1,2,… denote the discrete time steps, corresponding to multiples of the fundamental sampling time t_s=50 ms. The vector 𝐱_t∈ℝ^6 includes the (model) states at step t, namely

    
  * s_t,e_t: coordinates of the vehicle's reference point in road-aligned (Frenét) coordinates, 
    
  * ψ_t,ω_t: yaw angle and yaw rate of the vehicle,
    
  * v_t,u_t: longitudinal and lateral velocity of the vehicle.

The vector 𝐮_t∈ℝ^2 contains the (control) inputs at step t, namely

    
  * δ_t: the steering angle of the front wheel,
    
  * M_t: the torque of the drivetrain at the wheels, used to accelerate or decelerate the vehicle.

Furthermore, a variable for the reference lane n_t is used to indicate the lane decision for the vehicle at time t. It is assumed that the lanes are numbered from right to left, starting with 1 and counting up to the total number of lanes. The set of available lanes at step t is denoted ℕ_t.

Based on the prediction, the concept of the MPC-TP is to set up and solve the following Optimal Control Problem (OCP) in each time step, using an appropriate numerical optimization solver:

    n_t,𝐮_t,𝐱_tminimize      ℓ_s(s_N_p)+∑_t=1^N_pℓ_e(e_t,n_t)+ℓ_o(s_t,e_t)+ℓ_u(δ_t,M_t) ,
    s.t.      𝐱_t+1=f(𝐱_t,𝐮_t)    ∀ t=0,1,…,N_p-1 ,
       𝐱_0=𝐱̅_0 ,
       𝐧_t∈ℕ_t    ∀ t=1,2,…,N_p ,
       𝐮_t∈𝕌    ∀ t=0,1,…,N_p-1 ,
       𝐱_t∈𝕏^(s)_t ,  𝐱_t∈𝕏^(t)_t    ∀ t=1,2,…,N_p-1 ,
       𝐱_N_p∈𝕏_f .

The objective function (<ref>a) comprises the following cost terms:

    
  * ℓ_s(s_N_p): rewards the progress along the road made by the vehicle over the prediction horizon;
    
  * ℓ_e(e_t,n_t): penalizes the lateral deviation of the vehicle from the center line of the corresponding reference lane;
    
  * ℓ_o(s_t,e_t): repels the vehicle from static and dynamic obstacles, e.g., by means of a potential field;
    
  * ℓ_u(δ_t,M_t): favors a low input usage and smooth changes in the inputs, e.g., for driver comfort and energy efficient driving.

These cost terms are multiplied by individual tuning weights, which are selected during the control design process.

The initial state 𝐱̅_0 of the LDBM (<ref>b,c) is obtained from the module for Ego Motion Fusion. The lane variables n_t over the prediction horizon in (<ref>d) are either a decision variable of the OCP (as for the HP), or they can be pre-determined before solving the OCP (as for E2LCACC and ALCA). Equations (<ref>e,f,g) represent the input constraints, state constraints, and terminal constraints, respectively. The state constraints are split into two sets:

    
  * The safety constraints 𝕏^(s)_t avoid collisions of the EgoV with static and dynamic objects, most importantly other vehicles, called target vehicles (TgtVs). Furthermore, the constraints ensure the validity of the tire model, by restricting the tire forces to remain within the admissible bounds of the linear tire regime (i.e., no slip in the tire-road contact patch).
    
  * The traffic rule and comfort constraints 𝕏^(t)_t represent the constraints from traffic rules (e.g., the EgoV must stay within its lane, the EgoV must respect the speed limit) and passenger comfort (e.g., no excessive acceleration or jerk).

The safety constraints are essential and cannot be relaxed, whereas the traffic rule and comfort constraints can be relaxed in emergency situations. Furthermore, in emergency situations, the LDBM in (<ref>) can be replaced with a nonlinear dynamic bicycle model (NDBM), which uses a Pacejka tire model instead of the linear tire model. By these modifications, the MPC-TP may temporarily operate the vehicle close to its dynamic limits and against the traffic rules, e.g., driving in between two lanes, or without considering passenger comfort.



  ) Further specifications related to the MPC formulation include the following:

    
  Spec:MPCa) The fundamental sampling time is t_s=50 ms.
    
  Spec:MPCb) The planning horizon is N_p=120 steps, which corresponds to a prediction time of T_p=6 s.
    
  Spec:MPCc) The input constraint set 𝕌 represents all limits of the steering, engine and braking systems, as well as the relevant limits of their rate-of-change.
    
  Spec:MPCd) The state constraint sets 𝕏_t are time-dependent and prevent collisions with static and dynamic obstacles, with the exception of vehicles driving behind the ego vehicle on the same lane. Moreover, the state constraints keep the vehicle away from its dynamic limits and ensure satisfaction of the traffic rules, e.g., by preventing forbidden lane changes or violation of the speed limit.
    
  Spec:MPCe) The terminal constraint 𝕏_f corresponds to a safe state of the vehicle in which it can remain indefinitely (e.g., at standstill), or it represents a safe exit strategy (e.g., maximum deceleration until standstill). Its purpose is to ensure the recursive feasibility of the OCP.



The numerical computations of the MPC-TP comprise two main steps:

    
  * setup of the OCP problem,
    
  * solution of the OCP problem.

Step 1 is to parse the OCP into the standard form of a nonlinear programming problem, which is realized by handwritten code. Step 2 is to solve the nonlinear program using a generic optimization solver, for which a generous selection of software packages is available <cit.>. 


    
  ) Besides the optimal trajectory, the optimization solver returns an exit flag q_exit in exactly one of four classes: 

    
    
  q_exit=1: A (local) minimum is found with an acceptable tolerance level.
    
  q_exit=2: A feasible point is found, but without an optimality property (e.g., because the gradient / search direction becomes too small or the solver has reached the maximum number of iterations / solution time).
    
  q_exit=3: The solver detects that no feasible point exists (`certificate of infeasibility').
    
  q_exit=4: A feasible point is not found, but without a certificate of infeasibility (e.g., because the solver has reached the maximum number of iterations / solution time), or an identified or unidentified solver error occurred (e.g., due to insufficient memory).



In reality, most solvers provide further diagnostics of the results, i.e., their exit flag is more detailed than the four classes above. The requirement (Spec:ExitFlag) states that it must be possible to map the true exit flag of the solver uniquely to one of the indicated classes. Moreover, it is assumed that the exit flag correctly characterizes the result of the solver. This can be verified by a separate solution checker. Cases where the exit flag is incorrect are not considered further as part of the SOTIF analysis. They represent software fault and hence belong to the realm of FS.



 §.§ Hazards at the Vehicle Level, Safety Goals, and SOTIF Requirements


For the SOTIF analysis in this paper, the following hazards at the vehicle level are considered:


    
    
  ) The ego vehicle leaves the assigned lane.
    
  ) The ego vehicle collides with other vehicles or obstacles.

These hazards lead directly to the specification of the following safety goals:


    
    
  ) The ego vehicle shall not leave the assigned lane(s).
    
  ) The ego vehicle shall not collide with other vehicles or obstacles.


As part of a top-down safety lifecycle, these safety goals would be broken down recursively into more and more detailed safety requirements until one arrives at safety requirements for the component in question. Safety requirements for an ADS comprise both functional safety requirements and SOTIF-related safety requirements (or SOTIF  requirements in short).
Such a refinement of safety requirements does not take place when developing a SOTIF-EooC. Instead, safety requirements must be assumed for the out-of-context element under consideration. The SOTIF-SEooC's fulfillment of these assumed safety requirements represents its contribution to achieving the original safety goals.

For the MPC-TP, we assume the following SOTIF-related safety requirements: 


    
  ) The MPC-TP shall provide a reference trajectory within t_c=30 ms.[The computation time is less than the sampling time t_s=50 ms, in order to account for additional communication delays.]
    
  ) The reference trajectory provided by the MPC-TP has to be consistent regarding the redundant information in the trajectory body. 
    
  ) The reference trajectory provided by the MPC-TP has to be admissible, in the sense that it must satisfy all relevant actuator constraints:

    
  SR:Admissiblea) The minimum / maximum longitudinal acceleration of the vehicle is a_lon,min=-10 m/s^2, a_lon,max=+3 m/s^2.
    
  SR:Admissibleb) The minimum / maximum velocity of the vehicle is v_min=0 km/h, v_max=130 km/h.[The minimum velocity assumption means that the EgoV may not drive in reverse. A maximum velocity is required by the limited forward view of the vehicle's sensor suite.]
    
  SR:Admissiblec) The maximum front wheel steering angle of the vehicle (in each direction) is δ_max=34^∘.
    
  SR:Admissibled) The maximum front wheel steering rate (in each direction) is γ_max=68^∘ 1/s.
    
  SR:Admissiblee) The maximum lateral acceleration of the vehicle (in each direction) is a_lat,max=5 m/s^2.



Similar to the upstream perception modules, no detailed technical specifications have made for the downstream Tracking Controller (TC). As it is developed independently, responsibilities have to be clearly allocated and distributed between the MPC-TP and the TC. Thus requirements have to be specified for the interface, i.e., in this case, the reference trajectory. The approach proposed in this case study is to require “drivability” based on a nonlinear dynamic double-track model (NDDM):

    
  ) The reference trajectory provided by the MPC-TP shall be drivable, i.e., it must be possible to track it with a NDDM using some arbitrary controller, such that all of the following conditions are satisfied:

    
  SR:Drivablea) The initial condition of the NDDM corresponds to the measured initial condition of the vehicle.
    
  SR:Drivableb) The parameters of the NDDM correspond to those of the vehicle (mass m, moment of inertia I_z, position of the center of gravity x_cog and y_cog, front and rear cornering stiffness C_α,f and C_α,r).
    
  SR:Drivablec) The tires of the NDDM do not saturate, i.e., there is no slip in the tire-road contact patch, for the worst-case friction coeffient μ-Δμ. This means that the tire forces remain inside the linear regime at all times.
    
  SR:Drivabled) The bank angle is between β-Δβ and β+Δβ.
    
  SR:Drivablee) The side wind speed is between v_wind-Δ v_wind and v_wind+Δ v_wind.
    
  SR:Drivablef) All existing actuator constraints are respected: communication delays, actuator dynamics, actuator limits, and limits of the rates of change.
    
  SR:Drivableg) The maximum lateral offset from the reference trajectory is e_max=5 cm.
    
  SR:Drivableh) The length of each reference trajectory is such that it takes at least 5 s to reach the end.



The set of all consistent, admissible and drivable trajectories is denoted as 𝒯. Additionally, the trajectory should be such that it does not lead to collisions of the vehicle.


  ) The trajectory shall be collision-free, i.e., it must stay within the boundaries of the road and it must avoid collisions with all dynamic obstacles. 

    
  SR:CollisionFreea) For lateral collsition avoidance, the MPC-TP may rely on a maximum lateral offset e_max^TC produced by the TC:
    
    
  * for v≤ 30 km/h: e_max^TC=10 cm,
    
  * for v≤ 80 km/h: e_max^TC=15 cm,
    
  * for v≤ 130 km/h: e_max^TC=20 cm.
    
    
  SR:CollisionFreeb) The MPC-TP shall plan its trajectories such that it is always safe for the ego vehicle to stay in its current lane, in particular by maintaining an appropriate safety distance to the leading vehicle.

 
To this end, the MPC-TP may assume that all of the input information it receives is up-to-date and fully correct. 

Note that the scope of the paper is limited with regards to the considered vehicle level hazards and SOTIF requirements. In reality, there may be further hazards and derived requirements, which may involve also other aspects of the lateral motion, a such as acceleration and jerk, as well as the longitudinal motion of the vehicle, e.g., for overtaking maneuvers.



§ SOTIF BACKGROUND




 §.§ Aspects of Safety


The safety of a technical system is commonly defined as the absence of unreasonable risk for the health and life of humans <cit.>. A reasonable level of risk has to account for the fact that most technological systems, including ADSs, can never be entirely safe. Safety risks of E/E systems can be categorized according to the source of potential hazards into

    
  1) hazards that occur due to malfunctioning behavior (Functional Safety), addressed by the ISO 26262 <cit.> standard, and
    
  2) hazards that occur due to functional insufficiencies (Safety of the Intended Functionality, SOTIF), addressed by the ISO 21448 <cit.> standard.

Functional safety and SOTIF are distinct and complementary aspects of safety that both need to be accounted for in the development of an ADS <cit.>. The exclusive focus of this paper, however, is SOTIF.
 



 §.§ SOTIF


The international standard ISO 21448 'Road vehicles - Safety of the intended functionality' published in 2022 defines SOTIF as the absence of unreasonable risk due to hazards resulting from functional insufficiencies of the intended functionality, or its implementation <cit.>. As a basic framework, it defines an iterative lifecycle with several activities that are deemed adequate for developing an ADS that satisfies the SOTIF, as shown in Figure <ref>.
















 §.§ SOTIF-related Element out of Context (SOTIF-EooC)
 

As a consequence of the collaborative development typical for the automotive industry, a supplier company often provides a component that will be integrated by the OEM into a SOTIF-related system. Under this scenario, at the time of developing the component, the the supplier may only know certain aspects of the overarching system. One way of facilitating the achievement of SOTIF in such a context is to treat the component as a SOTIF-related Element out of Context (SOTIF-EooC), according to ISO 21448 <cit.>.   

For a component treated as a SOTIF-EooC, assumptions can be made regarding its use within the whole system and its contribution to the intended functionality. Based on this, assumptions about the SOTIF-related functional insufficiencies (FIs) and their allowed target rate of occurrence can be derived. These assumptions need to be documented. They are used as inputs for the subsequent development of the SOTIF-EooC by the supplier. For the SOTIF-EooC, the identified triggering conditions (TCs) of the component and their resulting FIs are documented together with the assumptions of use for the component (a.k.a. integration requirements).
When the OEM integrates the SOTIF-EooC, the validity of the assumptions or integration requirments is established by  activities in the context of whole vehicle-level functionalities, according to ISO 21448, Clause 4.4.3 <cit.>.

In this paper, the MPC-TP is treated as a SOTIF-EooC. 
In fact, the MPC-TP may be used as part of different driving automation systems, namely E^2LCACC, ALCA, and HP. Assumptions regarding the overall systems, the integration of the MPC-TP and the contributions of the MPC-TP to the intended functionality have been discussed in Sections <ref> and <ref>.
The identification of TCs of the MPC and their resulting FIs are discussed in Section <ref>. Naturally, their analysis leads to an improved architecture and refined specifications, which are presented in Section <ref>.





 §.§ Functional Insufficiencies


r0.55
	

	
    < g r a p h i c s >

	

SOTIF scenario categories.
	



FIs lead to hazards only in specific (driving) scenarios, called the hazardous scenarios, as opposed to non-hazardous scenarios. Conceptually, these scenarios are further sub-divided into known and unknown scenarios. This categorization of scenarios is illustrated in Figure <ref>. Hazardous scenarios may be known or unknown at the start of the  analysis. The overall goal of SOTIF is to improve the specification and the design of the system, or the considered element, with respect to reducing the number of (known and unknown) hazardous scenarios. 









FIs may exist at the system level, e.g., the E2LCACC system, or at the element level, e.g., the 'Sense', 'Plan', or 'Act' parts of the system. Since FIs can lead to hazardous behaviour, their identification and the evaluation of their effects are key activities in the development of a SOTIF related item <cit.>. ISO 21448 subdivides FIs into

    
  * specification insufficiencies, which refer to a (possibly incomplete) specification contributing to hazardous behavior, and
    
  * performance insufficiencies, which refer to limitations of the technical capabilities contributing to hazardous behavior <cit.>.
 



Examples of specification insufficiencies include an incomplete specification of the E2LCACC's headway distance resulting in the ego vehicle not keeping a safe distance to the vehicle in front or the inability of the HP to handle uncommon freeway road signs that were not part of its specification and thus cannot be processed appropriately by the system.
Examples of performance insufficiencies include limitations of technical capabilities of the ALCA such as limited calculation performance, limited perception range of its front radar sensor, or limited actuation capabilities of the foundational steering system  <cit.>.

As this paper is concerned with an MPC-TP for different driving functions, the focus is on FIs at element level, in this case, of a “Plan” component. 





 §.§ Triggering Conditions


Hazardous scenarios can be decomposed into the following elements: (1) A traffic scenario with a triggering condition, (2) which leads to a hazardous behavior of the ego vehicle, (3) which may cause a hazard in combination with the surrounding traffic or environment. Hence a TC is defined as a specific condition of a scenario that initiates a subsequent system reaction contributing to hazardous behavior <cit.>(simplified). 


TCs and FIs are interrelated: A TC of a scenario activates a FI, which results in a subsequent system reaction. Figure <ref> illustrates the activation of FI at the element level by a TC and the resulting hazardous behavior. In general, the SOTIF standard also addresses reasonably foreseeable misuse. This aspect however, remains outside the scope of this paper.



For an example, assume an activated E2LCACC system that uses a radar sensor with an insufficient field of view in cased of curved roads. The upper part of Figure <ref> shows a scenario without a TC. A FI is not being activated. The E2LCACC system works as intended, i.e., the scenario is considered not hazardous.  The lower part of Figure <ref> shows a second, hazardous scenario. Here, the TC is a slower moving vehicle ahead in lane on a curved road. In this case, the assumed FI at the sensor is activated, resulting in a late detection of the leading vehicle. As a result, the system brakes too late, resulting in a rear-end collision with the leading vehicle.   








§ SOTIF ANALYSIS FOR THE MPC-TP




 §.§ Identification and Evaluation of Functional Insufficiencies and Triggering Conditions



ISO 21448 suggests a list of specific methods to analyze potential FIs and TCs <cit.>. Table <ref> shows the list of methods that are selected as appropriate to derive FIs and/or TCs of the MPC-TP, and the ones that are considered further in the remainder of this paper.



For the analysis of the “Plan” component of a system, ISO 21448 <cit.> provides an additional, specific list of methods that can be used for finding FIs and TCs. This list is provided in Table <ref>. The categories considered for the SOTIF analysis in this paper are shown in black, the ones not considered further are shown in gray.



This section describes the application of the selected analysis methods, and the FIs and potential TCs that have been identified. Multiple methods may lead to the same results, in which case the corresponding FIs and TCs are identified with the same number.



 §.§ Analysis of ODD, Use Cases, and Scenarios


This analysis method involves a close examination of the ODD specifications in (Spec:ODD), as well as predictable use cases and scenarios that may lead to potential TCs and FIs of the MPC-TP. Using this approach, the following items have been identified for our case study:  


In congestion situations (TC:AOUCS01), the MPC-TP does not consider the formation of a rescue alley for police or ambulances (FI:AOUCS01). Conversely, the formation of a rescue alley by other vehicles or the presence  of police or ambulances in congestion situations may lead the MPC-TP to perform unusual, illegal, counter-intuitive or even dangerous maneuvers (FI:DangerousManeuver).


In obstacle avoidance scenarios (TC:AOUCS02), the MPC-TP does not allow emergency maneuvers without a proper lane change. For example, it is not possible to avoid an obstacle by driving in between two lanes or by using the shoulder (FI:AOUCS02). Conversely, the emergency obstacle avoidance by other vehicles may lead the MPC-TP to perform counter-intuitive or even dangerous maneuvers (FI:DangerousManeuver).


There is no speed limit on certain sections of the Autobahn. Suppose there is blocking obstacle on the road, such as the rear end of a congestion. If the obstacle is at standstill shortly beyond the prediction horizon of the MPC-TP (TC:AOUCS03) and an obstacle avoidance maneuver is not possible, e.g., by performing a lane change. If the EgoV exceeds a certain speed v_max, this can lead to a rear-end collision because the prediction horizon N_p is not long enough for the EgoV to come to a full stop within it (FI:AOUCS03). 






The design of the current MPC-TP is based on clearly defined traffic rules. It assumes, that surrounding vehicles either have the right-of-way, and are thus considered as obstacles, or they have to yield, and can thus be ignored. However, there are situations where the right-of-way is unclear, such as at the end of two merging lanes (TC:AOUCS04a). Without further measures, this may lead to an overly conservative driving behavior (FI:AOUCS04a), and even a complete deadlock for the EgoV (FI:AOUCS04b).


On freeways, cars typically only drive forwards. However, in exceptional situations, e.g., during congestion or due to police or ambulance operations, it may be necessary to perform reverse maneuvers. Moreover, surrounding vehicles may drive backwards (TC:AOUCS05), in which case the MPC-TP may become infeasible (FI:Infeasible) if the EgoV is at rest or slowly moving forwards.


In the case of a wrong way driver (TC:AOUCS06), the MPC-TP may be unable to compute a feasible trajectory (FI:Infeasible). Or it may lead the MPC-TP to perform counter-intuitive or even dangerous maneuvers (FI:DangerousManeuver).


The MPC-TP shall be able handle situations with an unexpected, sudden change of road conditions. Important examples include a dropping friction coefficient, e.g., due to water / hydroplaning (TC:AOUCS07a), snow / ice (TC:AOUCS07b), or oil (TC:AOUCS07c), or an irregular surface, e.g., due to gravel or stones (TC:AOUCS08a), pot holes (TC:AOUCS08b), or road bumps (TC:AOUCS08c). In such situations, the trajectory generated by the MPC-TP may actually be not drivable, in the sense of (SR:Drivable), e.g., because of the limitations of the LDBM using a linear tire model (FI:Drivable).





 §.§ Analysis of System Architecture (Including Redundancies)


This analysis is based on the system architecture, as depicted in Figure <ref>. The goal is to identify potential FIs and TCs based on the overall behavior of the entire system.

The receding-horizon implementation causes the MPC-TP to act also as a feedback controller for the vehicle, which is a well-known feature of MPC <cit.>. The interoperation of the MPC-TP with the tracking controller (TC) may thus lead to undesired bahavior, such as oscillations (FI:ASA01), or even instability of the EgoV motion. This may be triggered, for example, by external disturbances such as a side wind gust, e.g., on a bridge deck (TC:ASA01), or irregular, non-continuous lane markings, e.g., due to suddenly disappearing (TC:ASA02a), dirty (TC:ASA02b), broken (TC:ASA02c), or covered (TC:ASA02d) lane markings.

Difficulties may also appear in situations where the right-of-way is unclear (TC:AOUCS04), or when other traffic participants do not strictly adhere to the traffic rules (TC:ASA05). Their interaction with the MPC-TP may then lead to an overly conservative behavior of the EgoV (FI:AOUCS04), possibly resulting in a poor progress on the road (FI:AOUCS04a), or even a deadlock (FI:AOUCS04b).

The cut-in of a faster TgtV in front of the EgoV (TC:ASA04) may lead to a violation of the front safety distance, and hence the inability of the MPC-TP to generate a feasible solution (FI:Infeasible). This can be resolved by relaxing the minimum safety distance constraint in case of a leading vehicle with a higher velocity. Even if the safety distance is not violated, the cut-in of a faster TgtV (TC:ASA04) may still lead to an overly conservative braking maneuver on behalf of the EgoV (FI:AOUCS04c).


The MPC-TP does not consider a trailing vehicle driving closely behind the EgoV in the same lane (TC:ASA06). According to (Spec:MPCd), even though it is the responsibility of the trailing vehicle to keep a sufficient safety distance, sudden braking maneuvers may lead to a collision with the trailing vehicle and are thus unsafe (FI:ASA03).




 §.§ Analysis of Functional Dependencies


In the context of this case study, functional dependencies are interpreted as other modules that are actually or potentially connected with the MPC-TP, be it upstream or downstream. The MPC-TP needs to cope with challenges arising from the limitations of these modules. With this interpretation of functional dependencies, TCs and FIs identified with this analysis method may overlap with those identified by analyzing the system architecture (in Section <ref>), in the context of the entire system.


Dynamic obstacles suddenly appearing in the field of view of the system (TC:AFD01), may overwhelm the ability of the MPC-TP to solve the OCP in order to compute a new trajectory in due time (FI:InsufficientTime), violating (SR:Timing), or to find a feasible trajectory at all (FI:Infeasible), violating (SR:CollisionFree). Examples of this group of FIs are a previously shadowed traffic participant (TC:AFD01a), an object dropping from a bridge or a truck (TC:AFD01b), or a large animal (e.g., a wild boar) crossing the freeway (TC:AFD01c).  


At points of sudden traffic rule changes (TC:AFD02), such as the beginning of a speed limit area or a no passing zone, the MPC-TP may also unable to compute a feasible trajectory (FI:Infeasible), or may be unable to solve the OCP within the allotted time (FI:InsufficientTime). 





 §.§ Analysis of Algorithms and Their Outputs or Decisions


In the context of the MPC-TP, this analysis pertains to the formulation of the OCP (<ref>) and the numerical optimization algorithm, the decisions taken by the MPC-TP, as well as the reference trajectory and the exit flag as its outputs.

The OCP is based on the LDBM, whose linear tire model may represent a poor approximation of a real vehicle if essential model assumptions are violated (FI:AAOD01). The validity of the linear tire model is already supported by corresponding constraints 𝕏^(s)_t (see Section  <ref>). The other main assumption of the LDBM is a non-negligible roll motion of the vehicle. It can be violated by an extremely asymmetric load (TC:AAOD01a), a highly dynamic cornering maneuver (TC:AAOD01b), or a high position of the CoG (TC:AAOD01c). 


Potential problems arising from the interoperation of the MPC-TP with the TC and from the lack of interaction-awareness have already been discussed (see Section <ref>). In addition, the MPC-TP may cause a lack of string stability in a chain of vehicles (TC:StringStability). If the leading brakes  abruptly, this can lead to the build-up of an increasing wave of braking maneuvers by all following vehicles. This string instability may be fostered by the MPC-TP in the middle of a chain of human-driven or automated vehicle, which may lead to collisions (FI:StringStability), if the MPC-TP is not properly tuned. 


The outputs of the MPC-TP include the reference trajectory and the solver exit flag (see Figure <ref>). 


Without software or hardware faults, the computed reference trajectory will be consistent, admissible, drivable, and collision-free, if the set 𝒯 of such trajectories is non-empty. However, due to the finite prediction time of T_p=6 s, the MPC-TP may lead the vehicle to an (unsafe) situation that cannot be resolved by a future trajectory, i.e., where 𝒯 becomes empty. In MPC language, this is known as the lack of recursive feasibility (FI:RecursiveFeasibility). A possible TC is the fast approach of a slowly driving TgtV in the front of the EgoV, where the TgtV cannot be avoided by a lateral maneuver (TC:SlowLeadingTgtV).

The solver exit flag itself might provide an indication of a triggered FI. In particular, all exit flag values q_exit≠ 1 should be considered carefully:

    
  * q_exit=2 indicates that the MPC may exceed the allotted computation time (FI:InsufficientTime), but produces a feasible trajectory. This could be triggered by rapidly changing traffic situations (TC:AAOD02) or by suddenly appearing obstacles (TC:AFD01). 
    
  * q_exit=3 means that the MPC-TP is unable to generate a feasible (collision-free) trajectory for the given constraints, because 𝒯 is empty (FI:Infeasible). For example, this can be triggered by (TC:AOUCS05), (TC:AOUCS06), (TC:AOUCS07), (TC:AOUCS08), (TC:ASA04), (TC:AFD01),(TC:AFD02).
    
  * q_exit=4 means that the solver is unable to find a solution (FI:NoSolution), even though 𝒯 may be non-empty. As software and hardware faults are out of scope, this must be due to unpredictable numerical problems (e.g., ill-conditioning, floating point arithmetics). To the best of the authors' knowledge, there are no known specific triggering conditions for this case.






 §.§ Analysis of Boundary Values


For the analysis of boundary values, the MPC-TP is examined in the context of scenarios where relevant parameters take on extreme values. Parameters considered in the analysis include

    
  * ODD and road parameters, i.e., the road bank angle, the friction coefficient μ, the road curvature, the change of curvature, and the wind speed;
    
  * maneuver parameters, i.e., the reference velocity, maximum lateral motion (e.g., due to a sudden lane change or obstacle avoidance), maximum longitudinal motion (e.g., due to emergency braking or maximum acceleration);
    
  * vehicle parameters, i.e., the vehicle mass (m), the moment of inertia (I_z), the position of the CoG (x_cog, y_cog, z_cog), the cornering stiffness of the front and the the rear tires (C_α,f, C_α,r), and all chassis and suspension parameters of the specified vehicle class.


The analysis is based on extensive simulations using a high-fidelity vehicle model, e.g, in CarMaker. The simulations scenarios aim to cover possible combinations of extreme values of the above parameters. The results are checked against the requirements (SR:Timing) - (SR:CollisionFree).

To this end, a database of artificial test scenarios covering corner cases, is generated, based on different incarnations of the maneuver parameters. Specifically for the reference velocity, scenarios are separated into low-speed (v≤ v_low=5 km/h), medium-speed (v≤ v_med=50 km/h), and high-speed (v≤ v_high=130 km/h) categories. For road curvature and change of curvature, only scenarios that (at least partially) involve maximum values are considered. The generation process of corner cases can also be automated, as proposed in Lubiniecki et al. <cit.>.

The database of corner cases is then simulated for extreme values of the ODD and road parameters and the vehicle parameters. Due to the combinatorial nature, the number of simulation scenarios thus becomes intractable. To regain tractability, only pairs of extreme parameter values are included in the boundary value analysis, while all other parameters are set to their default or regular values. For some parameters, such as the height of the CoG z_cog, instead of a default or regular values, it is more reasonable to consider only the maximum value in all simulations. Critical cases are expected to result from certain parameter combinations (in some or all corner cases), such as

    
  * the maximum side wind speed v_wind,max = 8 bft and the maximum bank angle β_max = 8^∘ (TC:CommonTrigger1),
    
  * the minimum friction value μ_min=0.15 and the minimum tire cornering stiffness C_α,f,min=60 kN/rad (TC:CommonTrigger2),
    
  * maximum front tire cornering stiffness C_α,f,max=200 kN/rad and the minimum rear tire cornering stiffness C_α,r,min=60 kN/rad, and vice versa (TC:CommonTrigger3).


These TCs may lead the MPC-TP to provide trajectories that are not drivable by a specific vehicle model or configuration (FI:Drivable). However, these are potential TCs and FIs, and their existence and relevance has to be confirmed by actually performing a large-scale, high-fidelity simulation study.



 §.§ Analysis of Common Triggering Conditions


Generally, common TCs are collected from prior experiences with the development of similar systems. They may be available, for example, by internal reports, industry standards, or other publicly documents. The TCs identified in this way may, of course, overlap with TCs identified using other methods.

The authors are currently not aware of a database or any other source for common TCs for a trajectory planner. However, applicable TCs for other ADSs from the relevant literature have been taken into account in this case study. In particular, selected TCs for a LKA system from <cit.> can analyzed with respect to their ability to challenge the MPC-TP and to trigger potential FIs:

    
  * lane keeping / lane changing scenario with suddenly disappearing lane markings (TC:ASA02a);
    
  * lane keeping / lane changing scenario with dirty, broken, or covered lane markings (TC:ASA02b,c,d);
    
  * lane keeping / lane changing scenario with unrecognizable lane markings due to direct sunlight or change of lighting conditions, e.g., when entering a tunnel (TC:ACTC03a,b);
    
  * insufficient visibility of lane markings due to low visibility, e.g., due to snow / ice or fog (TC:ACTC04a,b);
    
  * infrastructure items that could be misinterpreted as lane markings, e.g., curbs or dividers (TC:ACTC05a,b).

Other common TCs for a trajectory planner that can been identified from the existing literature include

    
  * side wind gusts, e.g., on a bridge deck (TC:ASA01a),
    
  * an uneven road surface, due to potholes (TC:AOUCS08b) or road bumps (TC:AOUCS08c),
    
  * a slippery road surface, e.g., due to a wet road with foilage (TC:ACTC09a) or a snowy / icy road (TC:ACTC09b).


For many of the common TCs from literature it remains unclear however, whether they activate FIs in the MCP-TP or elsewhere in the system. This needs to be verified in simulations or real-world experiments.



 §.§ Environment and Location


The systematic analysis of the environment and location aims to identify relevant freeway scenarios and warning signs that may compromise the safety of the MPC-TP. By means of brainstorming, the following scenarios in the ODD can be identified as potential TCs in this context.

An increase or decrease in the number of lanes on the freeway may present a particularly challenging situation for the MPC-TP, including:

    
  * entering the freeway from an on-ramp (TC:EAL01),
    
  * exiting the freeway via an off-ramp (TC:EAL02),
    
  * lane ending / lane merging situations (TC:AOUCS04a),
    
  * lane adding situations (TC:EAL04).


Similarly, situations with special vehicles or vehicles operating beyond traffic rules may pose particular difficulties for the MPC-TP, such as: 

    
  * a wrong way driver (TC:AOUCS06), 
    
  * a reckless driver or person driving under-the-influence (TC:EAL08),
    
  * an approaching emergency vehicle (TC:EAL05),
    
  * a heavy transport or oversized vehicles (TC:EAL06),
    
  * a vehicle parking on the freeway shoulder (TC:EAL07).



Furthermore, rare situations and uncommon road signs on the German Autobahn can be considered in the context of this analysis, leading to the following list of TCs:

    
  * uneven road surface (road sign 112), e.g., due to ruts or blow-ups (TC:AOUCS08d,e),
    
  * an invisible / indetectable substance is substantially lowering the friction coefficient, e.g., due to an oil spillage (road sign 114) (TC:AOUCS07c),
    
  * the freeway shoulder is not available (TC:EAL09) or the shoulder is to be used as an additional lane (road signs 223.1, 223.2, 223.3) (TC:EAL10),
    
  * end of the freeway (road sign 330.1) (TC:EAL11) or a freeway closure (TC:EAL12),
    
  * an unusually steep slope (road signs 108, 110) (TC:EAL13) or an usually high curvature (road signs 103-10, 103-20, 105-10, 105-20) of the road (TC:EAL14),
    
  * dangerous side wind gusts (road sign 117) (TC:ASA01),
    
    
  * potentially unknown traffic scenarios or traffic signs in other European countries (TC:EAL15).


All of the above TCs may affect the ability of the MPC-TP to solve the OCP and compute a trajectory in due time (FI:InsufficientTime), or to find a feasible (collision-free) trajectory at all (FI:Infeasible). Furthermore, they may cause the MPC-TP to provide trajectories which are not drivable and thus lead the EgoV to leave its assigned lane and/or collide with other vehicles or obstacles (FI:Drivable) or which may cause the vehicle to perform unusual, illegal, counter-intuitive or even dangerous maneuvers (FI:DangerousManeuver).



 §.§ Summary of Identified Triggering Conditions and Functional Insuffciencies


Appendix <ref> provides a table with an overview of all triggering conditions and functional insuffciencies that have been identified for the MPC-TP in this case study.



§ REFINED CONCEPT FOR THE VEHICLE LEVEL CONTROL SYSTEM AND THE MPC-TP


As per the SOTIF lifecycle, the SOTIF-EooC analysis in Section <ref> leads to a revision of the overall concept of the vehicle level control system and the MPC-TP, in particular. The results include a refinement of the MPC-TP specifications and design as well as the ODD, as explained in Sections <ref> and <ref>). As a further consequence, the integration requirements to the vehicle level control system are updated, as detailed in Section <ref>. Some situations that could not be (fully) resolved as part of this case study are described in Sections  and .



 §.§ Refined Specification and Design of the MPC-TP


The refined system architecture based on the SOTIF analysis in Section <ref> is depicted in Figure <ref>. The particular refinements of the specification and the design of the MPC-TP are discussed below.





  §.§.§ Design Improvements















The SOTIF-EooC analysis has led to the following concepts to improve the architecture and algorithmic design of the MPC-TP.

An overly conservative behavior of the EgoV (FI:AOUCS04c) may result, e.g., due to the cut-in of a faster TgtV (TC:ASA04). This potential issue can be mitigated by a proper tuning of the MPC-TP.

A trailing TgtV not keeping a proper safety distance to the EgoV (TC:ASA06) may cause a collision in case of a sudden braking maneuvers of the EgoV (FI:ASA03). 

a*) Even though this lies within the responsibility of the TgtV,  the EgoV shall perform sudden braking maneuvers only when strictly necessary and only to an appropriate extent. This can be accomplished by adding a strong penalty on the usage of high braking inputs in the OCP.

Spec:SuddenBrakingb*) Furthermore, the distance to the trailing vehicle shall aslo be included in the tuning of the MPC-TP.

The interaction with other traffic participants if the right-of-way is unclear (TC:AOUCS04) or if they do not strictly adhere to the traffic rules (TC:ASA05) may lead to an overly conservative behavior of the EgoV (FI:AOUCS04). 

*) This should be properly resolved by using an interaction-aware MPC approach, which is the subject of ongoing research <cit.>. For the given architecture, the interaction-aware MPC also has to be designed in coordination with the Tactical Planner.

Finally, a lack of string stability may be fostered by the MPC-TP in a chain of human-driven or automated vehicles (TC:StringStability), which may potentially lead to collisions (FI:StringStability). The problem, however, is not MPC-specific and has been analyzed in the previous literature, including ACC systems <cit.>. It shall be resolved by a proper tuning of the MPC-TP, which should be verified in extensive simulations and/or on-road tests.



  §.§.§ Adaptations to the Reference Trajectory


The interoperation of the MPC-TP with the lower-level TC may lead to oscillations and even unstable behavior (FI:ASA01). 
Spec:TrajectoryBodyg*) As a solution to this, the first N_r=4 steps of the previously computed trajectory remain unchanged, i.e., only the subsequent inputs are newly computed. N_r is called the reaction horizon. The concept is illustrated in Figure <ref>. The initial condition for the OCP in (<ref>c) thus becomes the predicted state x̂_N_r from the previous OCP solution.



As a negative consequence of this modification, the reaction time of the MPC-TP is increased. This is relevant, for instance, if the EgoV has to react to sudden events in its environment, such as in an emergency braking or obstacle avoidance scenario. However, a reaction time of T_d=N_rt_s=200 ms is still superior to that of an average human driver, which is typically between 600 ms and 1,000 ms, according to recent studies <cit.>. Thus a reaction time of 200 ms is deemed to be sufficient for common road traffic.



  §.§.§ Potential Extension of Computation Time


The regular MPC is designed for a maximum computation time of t_c=30 ms. This is less than the given sampling time, t_s=50 ms, in order to account for additional latencies, e.g., due to communication and actuation delays. Obviously, the choice of t_c should be based on generous computational margins and extensive testing. However, it is generally not possible to guarantee that the solver always provides a definite answer within this allotted computation time, i.e., that it finishes with an exit flag q_exit = 1,2,3 in due time (FI:InsufficientTime). Several TCs have been identified as part of the SOTIF analysis in Section <ref> that may represent a potential cause for an excess computation time, such as a suddenly appearing obstacle (TC:AFD01), sudden changes of traffic rules (TC:AFD02), or rapidly changing traffic situations (TC:AAOD02). A strategy to remedy this issue may look as follows:

Due to the concept of the reaction horizon, introduced in Section <ref>, the computation time can be extended by multiple (in this case: N_r=4) time steps. In occasional peaks, it can reach up to a theoretical maximum time of t̅_c=t_c+(N_r-1)t_s=180 ms. After that, a new trajectory must definitely be provided. If multiple time steps are used to solve the OCP, the new decisions (shown in red in Figure <ref>) comprise a corresponding number of time steps, which are now more than a single one.

Furthermore, the solver settings are adjusted such that the iterations can be halted after t_c, if they have not terminated; or again after each t_c+k t_s, for k=1,2,…,N_r-1. After each instance, the status of the current iterate can be checked. If the current exit flag equals to q_exit = 4, the solver may continue its computations from the current iterate, until finally the ultimate maximum solver time t̅_c is reached.





  §.§.§ Backup MPC


The Backup MPC (MPC-TP2) is a redundant implementation of the Primary MPC (MPC-TP1), which has been discussed thus far. The Backup MPC has the same inputs and outputs as the Primary MPC, as shown in Figure <ref>). It is designed according to the same specifications and requirements, as described in Sections <ref> and <ref>, except for the following differences:

    
  * The Backup MPC is utilized for safety and not passenger comfort. It has the ability to perform any maneuver that is physically possible. This means a different tuning of the weight parameters in the cost function and the traffic rule and comfort constraints 𝕏^(t)_t are removed from the OCP in (<ref>f).
    
  * Instead of a LDBM, which has a linear tire model, the Backup MPC shall include a NDBM with a Pacejka tire model. The NDBM provides a higher fidelity for the dynamics of the vehicle at the limits of handling, even though it may have a longer solution time.
    
  * For additional redundancy, the Backup MPC shall apply a different numerical solver (different algorithm and/or software implementation) and it shall be running on an independent computational platform.


The Backup MPC is constantly running in a shadow mode. During normal operation, when the Primary MPC is active, the solutions of the Backup MPC solutions are simply discarded. The trajectories of the Backup MPC are activated, or subsequently deactivated, according to a new module called Infeasibility Analysis. Its logic is based on the exit flag of the Primary MPC q_exit^(P), the exit flag of the Backup MPC q_exit^(B), and various information from the perception modules; see Section <ref> for further details. In particular, the Backup MPC shall be activated if the Primary MPC is unable to find a feasible solution (FI:Infeasible) under the following circumstances:

    
  * The friction coefficient suddenly drops by a value of μ_d>0 within a specified time interval t_d>0, due to water, snow / ice, or oil (TC:AOUCS07), or an irregular surface, e.g., due to gravel or stones, pot holes, or road bumps (TC:AOUCS08).
    
  * Static or dynamic obstacles suddenly appear close to the vehicle, e.g., due to a previously shadowed traffic participant (TC:AFD01a), a lost cargo or other object dropping from a bridge or a truck (TC:AFD01b), or a large animal (e.g., a wild boar) crossing the freeway (TC:AFD01c).
    
  * There is a sudden change of traffic rules (TC:AFD02), such as the beginning of a speed limit zone or a no passing zone.


The Backup MPC remains active until the Primary MPC produces a optimal solution again (case q_exit=1) and it is safe to switch back. For the latter condition, it is important to carefully avoid problems that may arise from a possible periodic switching behavior between the two planners.



  §.§.§ Infeasibility Analysis


The desirable outcome of the Primary MPC is a (local) minimum with an acceptable tolerance (q_exit = 1). The SOTIF analysis in Section <ref>, however, has revealed multiple scenarios where this may not be the case. Then a new module called the Infeasibility Analysis is activated. Its logic is designed as follows:

If the Primary MPC has not converged to such a minimum and returns only a feasible trajectory (q_exit = 2), the sub-optimal solution is acceptable on an exceptional basis. Because the sub-optimal solution satisfies also the terminal constraints, the recursive feasibility of the trajectory planning is not jeopardized. However, situations where the solver detects the definite infeasibility of the OCP (q_exit = 3) or where it is unable to find a feasible solution (q_exit = 4) require further attention. Here a switching logic, as presented in Table <ref>, is proposed as a basis for the Infeasibility Analysis.



Before even activating the switching logic, however, the Infeasibility Analysis filters a few special situations that will obviously or possibly lead to an infeasibility of the Primary MPC (FI:Infeasible). This includes 

    
  * (TC:AOUCS05) a leading TgtV driving in reverse, while the EgoV is at rest or slowly moving forward; 
    
  * (TC:ASA04) at higher speeds, the cut-in of a faster TgtV in front of the EgoV leading to a violation of the safety distance in the present time step.


For (TC:AOUCS05), a special planning mode shall be designed to check if (a) the EgoV is driving with a speed less than or equal to some threshold v_low (e.g., v_low=5 km/h) and (b) the minimum safety distance to the leading TgtV is already violated at the current time step. If this is the case, a stopping trajectory is automatically issued that transitions the EgoV to, or keeps it in, a complete standstill. When the Primary MPC becomes feasible again, the EgoV resumes its normal operation and moves forward. 

For (TC:ASA04), the situation can be resolved by an additional logic that relaxes the minimum safety distance constraint to a leading TgtV in the Primary MPC, if it has a higher velocity than the EgoV. Note that, even if the safety distance is not violated, the cut-in of a faster TgtV (TC:ASA04) may still lead to an overly conservative braking maneuver on behalf of the EgoV (FI:AOUCS04c). This is a separate issue that has to be resolved by a different measure.

In situations with a slippery road surface (TC:ACTC09) or a sudden drop of the friction value μ within a time interval of t_d for more than μ_d (TC:AOUCS07), in a first step the linear tire regime of the LDBM in (<ref>b) shall be exchanged with a (nonlinear) Pacejka tire model. This measure is taken pre-emptively in order to avoid a lack of drivability of the computed trajectory (FI:Drivablea). The Pacejka tire model is not used as the standard tire model of the Primary MPC because of its additional computational complexity.



According to Table <ref>, if the Primary MPC does not produce a feasible solution (q_exit^(P) = 3,4), the Backup MPC is activated already after 2 steps. It should be remarked that this could be extended, if the reaction horizon were increased beyond N_r=4, which is used in this paper. This may allow for further attempts to modify or update the constraints of the OCP in an acceptable way, similar to the steps described above. Or additional attempts may be taken to modify the algorithm or the solver settings in order to achieve an optimal, or at least a feasible solution, of the OCP.







  §.§.§ MPC-TP Driving Modes


To adequately adapt to specific traffic or environmental situations, the MPC-TP needs to introduce a switchable behavior. *) In particular, (at least)the specialized driving modes
listed in Table <ref> shall be introduced.




The proper driving mode needs to be selected by the overarching system and provided to the primary as well as the back-up MPC-TP as additional mode inputs. The MPC-TPs can then adjust their behaviors considering the detected situation. This goes hand in hand with integration requirements to detect the various driving modes.




 §.§ Refined ODD


Spec:ODDi*) To address issues, where the prediction horizon N_p of the EgoV is not large enough for the EgoV to come to a full stop if needed (cf. FI:AOUCS03 in combination with (TC:AOUCS03), we limit the maximum permissible speed in the ODD: 

    v_max=N_p a_lon,min t_s .

As the prediction horizon N_p is static,  v_max must be chosen in accordance with the maximum deceleration a_lon,min.
a_lon,min≤ 0 is the worst-case (highest) deceleration, depending on current road conditions. It depends on the worst-case values of other ODD parameters, in particular the friction value μ.  


Spec:ODDo*) To cope with (FI005) in combination with (TC015c) the ODD will be limited to freeway sections with a game fence. 

(Spec:ODDf*,h*) To mitigate (FI:Infeasible) in combination with (TC:AOUCS07), in particular with (TC:AOUCS07b), we propose to limit the minimum ambient temperature to +4^∘C  (Spec:ODDf*) and the permissible minimum road friction value (Spec:ODDh*). This will be combined with an integration requirement to conservatively estimate the friction value (IR:FrictionEstimation), cf. Section <ref>.  

The refined ODD is provided in in Table <ref>. 





 §.§ Integration Requirements


A key concept to facilitate the out-of-context development of SOTIF-related elements are assumptions on the encompassing vehicle control system. As these need to be ensured during the integration of the SOTIF-EooC into the ADS, they shall be designated as integration requirements. The new integration requirements of the SOTIF analysis are listed in Table <ref>.







Cases where neither the Primary MPC nor the Back-Up MPC is able to generate a permissible trajectory shall be handled by the encompassing system as follows:

(IR:EmergencyOperationRequest) Situations where the MPC-TP requests an emergency operation shall be handled by the encompassing system utilizing a dedicated emergency planner that is independent of the MPC-TP.

Situations involving irregular road surfaces (TC:AOUCS08) where the MPC-TP would be unable to generate a feasible (collision-free) trajectory (FI:Infeasible) shall be identified and handled by the encompassing system. Here, the Tactical Planner needs to decide what to in such cases, e.g., depending on the size of the stones or potholes and provide an appropriate tactical decision to the MPC-TP or handle the situation without the MPC-TP.
This leads to integration requirement (IR:IrregularRoadSurface): Situations involving irregular road surfaces shall be detected and handled by the encompassing system utilizing the Tactical Planner.





 §.§ Situations Requiring Further Analysis


(Further analysis required) The OCP is based on the LDBM, assuming a linear tire model and no roll motion, which may represent a poor approximation of a real vehicle if these assumptions are violated (FI:AAOD01). The non-negligible roll motion may be violated by an extremely asymmetric load (TC:AAOD01a), a highly dynamic cornering maneuver (TC:AAOD01b), or a high position of the CoG (TC:AAOD01c). A detailed simulation analysis is required to examine whether the LDBM is sufficient for the targeted use cases (especially with respect to SR:Drivable), or whether a two-track model is required instead.

Situations encompassing side wind gusts (TC:ASA01) are deemed as adequately covered by the wind limitation in the ODD (Spec:ODDg).

Potentially unknown traffic scenarios or traffic signs, e.g., due to deviating regulations in other European countries, are potentially problematic as well (TC:EAL15). However, the existing ODD restriction limits the permissible roadway types to the German Autobahn (Spec:ODDa) seems sufficient to mitigate this concern.



 §.§ Interaction-aware MPC


Particular combinations of TCs and FIs cannot be adressed without extending the architecture of the encompassing vehicle, e.g. by adding additional senors and/or capabilities. An example is the combination of (TC:AOUCS04a) and (FI:AOUCS04).
As per our analysis the OCP in (<ref>) may lead to an overly conservative behavior of the EgoV (FI:AOUCS04), possibly leading to poor progress on the road (FI:AOUCS04a), or even a deadlock (FI:AOUCS04b). The only safe way to reduce this conservatively is to internalize the relevant TgtVs in the MPC predictions and to reflect the interactions between the TgtVs and the EgoV by appropriate behavioral models. Interaction-aware MPC approaches, however, are the subject of ongoing and future research <cit.> and are beyond the scope of this paper.



§ CONCLUSION


In this paper, multiple aspects of the general SOTIF framework provided in ISO 21448 have been applied to a generic MPC-based trajectory planner, which is part of an ADS. The generic MPC-TP is treated as an SOTIF-EooC, elaborating this concept that is only sketched in the SOTIF standard. In a first step, the initial concept of a generic MPC-TP has been analyzed in order to identify relevant FIs and corresponding TCs. To this end, the analysis techniques that are described in ISO 21448 only in very general terms have to be interpreted and applied to the case study. In a second step, the resulting comprehensive list of relevant FIs and TCs informed an SOTIF-optimized concept of the MPC-TP. The revised safety architecture includes, in particular, an infeasibility analysis as a component and a back-up trajectory planner, a switchable behavior of the MPC-TP to support special driving modes, and adaptations to the reference trajectory.  

An important aspect of the results is the identification of important research topics on MPC for path planning in the context of automated and autonomous driving, from a safety perspective:


  (i) interaction-awareness with other (human) traffic participants;

  (ii) ability to avoid obstacles and drive safely even at low friction values;

  (iii) design of a recursively feasible MPC-based path planner with a reaction horizon;

  (iv) string stability in a chain of vehicles;

  (v) examination of a safe switching logic between a Primary MPC and a Backup MPC. 


In future work, an extensive simulation study would be beneficial for a more detailed understanding of the potential shortcomings of the MPC approach itself, or the respective vehicle model to be used for its predictions in extreme driving situations.



§ ACKNOWLEDGMENT


The research leading to these results was funded by the German Federal Ministry for Economic Affairs and Climate Action (BMWK) through the project EEMotion.










    10cm(4.5cm,2cm)
        

§ APPENDIX A: OVERVIEW OF TCS AND FIS

    
    [c]|p6cm|p6cm|p7cm|
    TC     Triggered FI     Notes



() traffic congestion
    
() no capability to form a rescue alley for police or ambulances() generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
    
(refined specification/design) introduce switchable behavior of the MPC (driving mode: Rescue Alley)
(integration requirement) congestion situations shall be detected and handled by the encompassing system (a,b)



() obstacle avoidance scenario 
    
() no capability to carry out emergency maneuvers without a proper lane change, driving in between two lanes, or by using the shoulder(FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
    
(refined specification/design) introduce switchable behavior of the MPC (driving mode: Obstacle Avoidance)(integration requirement) obstacle avoidance situations shall be detected and handled by the encompassing system (a,b)



() blocking obstacle shortly beyond the prediction horizon N_p of the MPC-TP and no obstacle avoidance is possible 
    
() prediction horizon N_p of the MPC-TP is too short to come to a full stop within it
    
(ODD restriction) limit permissible max. speed to
v_max=|N_p a_lon,min t_s| (Spec:ODDi*)



(a) right-of-way is unclear: at the end of two merging lanes (TC:AOUCS04b) right-of-way is unclear: special situations during general lane changes 
   
(a) overly conservative behavior of the MPC-TP: inability to ensure sufficient progress on the road(FI:AOUCS04b) overly conservative behavior of the MPC-TP: inability to avoid deadlocks in certain situations (FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
    
(Interaction-aware MPC) interaction-aware MPC approach, in coordination with the Tactical Planner (Spec:InteractionAwareMPC*)



() Leading TgtV driving in reverse, in combination with a stopping or slowly moving EgoV 
    
() inability to generate a feasible (collision-free) trajectory in certain situations 
    
(refined specification/design) infeasibility analysis: selection of stopping trajectory in situations with low speed and safety distance violated



() wrong way driver 
    
(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations (FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
    
(refined specification/design) introduce switchable behavior of the MPC (driving mode: Wrong Way Driver)(integration requirement) if infeasibility still occurs, switch to emergency operation of the encompassing system ()(integration requirement) situations with wrong way drivers shall be detected and handled by dedicated mode of the encompassing system ()



(a) sudden drop of friction coefficient: water (hydroplaning) (TC:AOUCS07b) sudden drop of friction coefficient: snow / ice (TC:AOUCS07c) sudden drop of friction coefficient: oil spillage 
    
(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations 
   
(ODD restriction) limit permissible minimum temperature to exclude snowy/icy conditions (Spec:ODDf*) 
(ODD restriction) limit permissible minimum friction value (Spec:ODDh*)(integration requirement) conservative estimation of the friction value (a)(refined specification/design) avoid small areas filled with water by evasive maneuvers within the lane boundaries




(TC:AOUCS07a) sudden drop of friction coefficient: water (hydroplaning)(TC:AOUCS07b) sudden drop of friction coefficient: snow, ice (TC:AOUCS07c) sudden drop of friction coefficient: oil spillage 
    
(a) inability to generate a drivable trajectory in certain situations: due to linear tire model in LDBM
    (refined specification/design) infeasibility analysis: temporarily relax the linear tire regime if the friction value drops by more than μ_d within t_d



(a) irregular road surface: gravel or stones(TC:AOUCS08b) irregular road surface: potholes(TC:AOUCS08c) irregular road surface: road bumps
     
 (FI:Drivableb) inability to generate a drivable trajectory in certain situations: due to irregular road surface
    
(integration requirement) situations with irregular road surface shall be detected and handled by the encompassing system in combination with the Tactical Planner ()



(a) side wind gusts: bridge deck(TC:ASA02a) irregular lane markings: suddenly disappearing lane markings(b) irregular lane markings: dirty lane markings(TC:ASA02c) irregular lane markings: broken lane markings(TC:ASA02d) irregular lane markings: covered lane markings 
    
() receding-horizon planning of the MPC-TP in combination with tracking controller could lead to oscillations, or even an unstable behavior of the EgoV 
    
(refined specification/design) MPC-TP to keep the first N_r=4 steps (reaction horizon) of the trajectory unchanged (Spec:TrajectoryBodyg*) 



(TC:AOUCS04) right-of-way is unclear() other traffic participants not strictly adhering to the traffic rules 
    (FI:AOUCS04a) overly conservative behavior of the MPC-TP: inability to ensure sufficient progress on the road(FI:AOUCS04b) overly conservative behavior of the MPC-TP: inability to avoid deadlocks in certain situations(FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
   
(Interaction-aware MPC) interaction-aware MPC approach, in coordination with the Tactical Planner (Spec:InteractionAwareMPC*)



26cm() cut-in of a faster TgtV in front of the EgoV
    
(FI:AOUCS04c) overly conservative behavior of the MPC-TP: overly conservative braking in cut-in scenarios 
    
(refined specification/design) tuning of MPC-TP such that it avoids overly conservative braking to a leading TgtV with a positive relative velocity

2-3
    
(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations
    
(refined specification/design) infeasibility analysis: if TgtV cuts into the EgoV's lane and is faster than EgoV, then temporarily relax linear minimum distance to this TgtV



() trailing vehicle driving closely behind the EgoV in the same lane, not keeping a proper safety distance 
    () inability of the MPC-TP to consider trailing traffic in the lane of the EgoV, where a sudden braking maneuver of the EgoV may lead to a collision 
   
(refined specification/design) MPC-TP to brake only when strictly necessary and only to an appropriate extent (Spec:SuddenBrakinga*) (refined specification/design) MPC-TP to account for distance to trailing vehicle by means of tuning the weight of ℓ_o(·,·) in the cost function (Spec:SuddenBrakingb*) 



26cm(a) suddenly appearing obstacle: previously shadowed traffic participant(TC:AFD01b) suddenly appearing obstacle: object dropping from a bridge or truck(TC:AFD01c) suddenly appearing obstacle: large animal (e.g., wild boar) crossing the freeway
    
(FI:Infeasible) inability to compute a feasible (collision-free) trajectory in certain situations () insufficient time to solve the OCP, in order to compute a new trajectory
   
(refined specification/design) infeasibility analysis(refined specification/design) potential extension of computation time to multiple time steps based on the concept of a reaction horizon

2-3
   
(FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
    
(ODD restriction) limit ODD to freeway sections with a game fence (Spec:ODDo*) (refined specification/design) introduce switchable behavior of the MPC (driving mode: Defensive Driving)



(a) sudden traffic rule change: start of a speed limit area(TC:AFD02b) sudden traffic rule change: start of a no passing zone 
    
(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations(FI:InsufficientTime) insufficient time to solve the OCP, in order to compute a new trajectory 
   
(ODD restriction) limit permissible max. speed to
v_max=|N_p a_lon,min t_s| (Spec:ODDi*)(refined specification/design) infeasibility analysis(refined specification/design) potential extension of computation time to multiple time steps based on the concept of a reaction horizon




(a) LDBM assumption violation: asymmetric load(TC:AAOD01b) LDBM assumption violation: dynamic cornering(TC:AAOD01c) LDBM assumption violation: high position of the CoG 
    
()  poor approximation of real vehicle behavior by the LDBM in the OCP
    
(further analysis required) experimental analysis: evaluate if MPC-TP behaves safely in these situations; otherwise consider usage of a more complex model (e.g., a two track model) in the OCP



() chain of vehicles, EgoV in the middle, first vehicle brakes suddenly 
    
() lack of string stability of MPC-TP in a chain of vehicles, potentially leading to collisions
    
(refined specification/design) proper tuning of MPC-TP for string stability, to be verified in simulations and/or on-road tests



() fast approach of a slowly driving TgtV in the front of the EgoV, where the TgtV cannot be avoided by a lateral maneuver

   
(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations() lack of recursive feasibility 
    
(refined specification/design) infeasibility analysis(refined specification/design) if collision is unavoidable, activation of a collision mitigation mode in the encompassing system



() rapidly changing traffic situations 

   
(FI:InsufficientTime) insufficient time to solve the OCP, in order to compute a new trajectory 
   
(refined specification/design) potential extension of computation time to multiple time steps based on the concept of a reaction horizon



no specific TCs known
   
() inability to generate a solution in certain situations at all 
   
(refined specification/design) infeasibility analysis



Pairs of extreme parameter values in some or all corner cases:() max. side wind speed v_wind,max = 8 bft and max. bank angle β_max = 8^∘ () min. friction value μ_min=0.15 and min. tire cornering stiffness C_α,f,min=60 kN/rad(a) max. front tire cornering stiffness C_α,f,max=200 kN/rad × and min. rear tire cornering stiffness C_α,r,min=60 kN/rad(TC:CommonTrigger3b) min. front tire cornering stiffness C_α,f,min=60 kN/rad × and max. rear tire cornering stiffness C_α,r,max=200 kN/rad...
   
(FI:Drivablea) inability to generate a drivable trajectory in certain situations: due to linear tire model in LDBM(FI:AAOD01) poor approximation of real vehicle behavior by the LDBM in the OCP
    
(further analysis needed) experimental analysis for pairs of boundary values, conduct simulations to confirm/discard relevance of LDBM for MPC-TP; otherwise consider usage of a more complex model (e.g., a two track model) in the OCP



(TC:ASA02a) irregular lane markings: suddenly disappearing lane markings (TC:ASA02b) irregular lane markings: dirty lane markings(TC:ASA02c) irregular lane markings: broken lane markings(TC:ASA02d) irregular lane markings: covered lane markings 
    
    
(further analysis needed) common TCs, conduct simulations to confirm/discard relevance for MPC-TP 



(a) unrecognizable lane markings: direct sunlight (TC:ACTC03b) unrecognizable lane markings: change of lighting conditions, e.g., when entering a tunnel 
    
    
(further analysis needed) common TCs, conduct simulations to confirm/discard relevance for MPC-TP 



(a) insufficient visibility of lane markings: snow / ice  (TC:ACTC04b) insufficient visibility of lane markings: fog 
    
    
(further analysis needed) common TCs, conduct simulations to confirm/discard relevance for MPC-TP 



(a) infrastructure items that could be misinterpreted as lane markings: curbs (TC:ACTC05b) infrastructure items that could be misinterpreted as lane markings: dividers 
    
    
(further analysis needed) common TCs, conduct simulations to confirm/discard relevance for MPC-TP 



(TC:AOUCS08b) irregular road surface: potholes(TC:AOUCS08c) irregular road surface: road bumps 
    
    
(further analysis needed) common TCs, conduct simulations to confirm/discard relevance for MPC-TP 



(a) slippery road surface: wet road with foilage(TC:ACTC09b) slippery road surface: snowy / icy road
    
(FI:Drivablea) inability to generate a drivable trajectory in certain situations: due to linear tire model in LDBM
   
(ODD restriction) limit permissible minimum temperature to exclude snowy/icy conditions (Spec:ODDf*)(refined specification/design) infeasibility analysis: temporarily relax the linear tire regime if the friction value drops by more than μ_d within t_d(further analysis needed) common TCs, conduct simulations to confirm/discard relevance for MPC-TP 












26cm() entering the freeway from an on-ramp () exiting the freeway via an off-ramp () lane adding situations
    
(FI:InsufficientTime) insufficient time to solve the OCP, in order to compute a new trajectory(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations
    
(refined specification/design) potential extension of computation time to multiple time steps based on the concept of a reaction horizon(refined specification/design) infeasibility analysis

2-3
    
(FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
    
(further analysis needed) conduct simulations and/or real experiments



26cmsituations with special vehicles or vehicles operating beyond traffic rules:(TC:AOUCS06) wrong way driver () approaching emergency vehicle () heavy transport or oversized vehicles () vehicle parking on the freeway shoulder() reckless driver / person driving under-the-influence
    
(FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
    
(refined specification/design) introduce specialized driving modes for particular situations (Spec:SpecialDrivingModes*)(further analysis needed) conduct simulations and/or real experiments

2-3
    
(FI:InsufficientTime) insufficient time to solve the OCP, in order to compute a new trajectory(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations    
(refined specification/design) potential extension of computation time to multiple time steps based on the concept of a reaction horizon(refined specification/design) infeasibility analysis



26cm(TC:AOUCS08d) irregular road surface: ruts(TC:AOUCS08e) irregular road surface: blow-ups
   
(FI:InsufficientTime) insufficient time to solve the OCP, in order to compute a new trajectory(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations
    
(refined specification/design) potential extension of computation time to multiple time steps based on the concept of a reaction horizon(refined specification/design) infeasibility analysis


2-3
   
(FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
    
(further analysis) extension of perception required




() freeway shoulder not available () freeway shoulder to be used as additional lane 
   
(FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
   
(refined specification/design) introduce specialized driving modes for particular situations (Spec:SpecialDrivingModes*)(further analysis needed) conduct simulations and/or real experiments




() end of freeway () freeway closure
    
 (FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers
   
(further analysis needed) conduct simulations and/or real experiments

() steep road slope () high road curvature 
   
(FI:Drivablea) inability to generate a drivable trajectory in certain situations: due to linear tire model in LDBM(FI:AAOD01) poor approximation of real vehicle behavior by the LDBM in the OCP
   
(further analysis required) experimental analysis: evaluate if MPC-TP behaves safely in these situations; otherwise consider usage of a more complex model (e.g., a two track model) in the OCP



(TC:ASA01) side wind gusts
   
(FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations(FI:InsufficientTime) insufficient time to solve the OCP, in order to compute a new trajectory
    
(no change needed) covered by wind limitation in the ODD (Spec:ODDg)



() potentially unknown traffic scenarios / traffic signs in other European countries
   
(FI:DangerousManeuver) generation of trajectories representing unusual, illegal, counter-intuitive, or dangerous maneuvers(FI:Infeasible) inability to generate a feasible (collision-free) trajectory in certain situations(FI:InsufficientTime) insufficient time to solve the OCP, in order to compute a new trajectory
    
(no change needed) covered by permissible road type restriction in the ODD (Spec:ODDa) 



    
    









 Paper Title: A Qualitative Study on Using ChatGPT for Software Security: Perception vs. Practicality.





















A Qualitative Study on Using ChatGPT for Software Security: Perception vs. Practicality
    
    M. Mehdi Kholoosi12, 
    M. Ali Babar12,
    Roland Croft12
    
    
    1 CREST - The Centre for Research on Engineering Software Technologies, The University of Adelaide, Adelaide, Australia
    2 Cyber Security Cooperative Research Centre, Australia
    
    Emails: {mehdi.kholoosi, ali.babar, roland.croft}@adelaide.edu.au
    August, 2024
==================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Artificial Intelligence (AI) advancements have enabled the development of Large Language Models (LLMs) that can perform a variety of tasks with remarkable semantic understanding and accuracy. ChatGPT is one such LLM that has gained significant attention due to its impressive capabilities for assisting in various knowledge-intensive tasks. Due to the knowledge-intensive nature of engineering secure software, ChatGPT's assistance is expected to be explored for security-related tasks during the development/evolution of software. To gain an understanding of the potential of ChatGPT as an emerging technology for supporting software security, we adopted a two-fold approach. Initially, we performed an empirical study to analyse the perceptions of those who had explored the use of ChatGPT for security tasks and shared their views on Twitter. It was determined that security practitioners view ChatGPT as beneficial for various software security tasks, including vulnerability detection, information retrieval, and penetration testing. Secondly, we designed an experiment aimed at investigating the practicality of this technology when deployed as an oracle in real-world settings. In particular, we focused on vulnerability detection and qualitatively examined ChatGPT outputs for given prompts within this prominent software security task. Based on our analysis, responses from ChatGPT in this task are largely filled with generic security information and may not be appropriate for industry use. To prevent data leakage, we performed this analysis on a vulnerability dataset compiled after the OpenAI data cut-off date from real-world projects covering 40 distinct vulnerability types and 12 programming languages. We assert that the findings from this study would contribute to future research aimed at developing and evaluating LLMs dedicated to software security.







security, software vulnerability, large language model




§ INTRODUCTION



Software security remains one of the most significant challenges facing modern software development. With the rise of agile development practices, a focus is being placed on developer-centric security assurance <cit.>. However, studies have shown that developers fail to keep up with the ever-growing list of required security knowledge and expertise <cit.>. Secure coding alone requires developers to learn and use thousands of security-related patterns, practices, and tools. Some areas of software security, such as Software Vulnerability Management (SVM), need specialised knowledge and expertise for detecting, assessing, patching, and disclosing vulnerabilities <cit.>. Consequently, developers tend to leverage recommended systems and/or social media platforms to gain and use knowledge about software security <cit.>. 

Chatbots, i.e., conversational agents, provide effective interfaces for enhanced information retrieval and task assistance <cit.>. Natural language comprehension offers a potentially game-changing interface facilitating human and computer interaction. Natural language can greatly improve the efficiency in which we perform complicated tasks and use complex technologies by allowing us to operate them with simple natural language queries. 

Like many other domains, such as education and marketing <cit.>, chatbots are gaining significant traction in the software development domain for obtaining knowledge/information related to specific software development tasks, such as API usages <cit.>. However, the use of chatbots for software security is still in its infancy stage <cit.> as current software security chatbots use simple rule-based technologies with limited capabilities <cit.>. This situation is expected to change as the advancements in Large Language Models (LLMs) enable much more powerful and capable chatbots, e.g., ChatGPT. 

ChatGPT is an Artificial Intelligence (AI) chatbot developed by OpenAI[https://openai.com/blog/chatgpt/] created using GPT (Generative Pretrained Transformer)<cit.>, a state-of-the-art LLM. These technical advancements allow ChatGPT to surpass traditional chatbots through its capability to provide answers and solutions to complex queries
<cit.>. 
Training and fine-tuning an LLM with billions of parameters (e.g., 175 billion in GPT-3<cit.> and 1,700 billion in GPT-4<cit.>) presents considerable challenges due to the extensive computational resources required <cit.>. Since ChatGPT was released to the public, this limitation has been addressed, and many users were able to experience LLMs for the first time. Whilst ChatGPT was not specifically developed for the software domain, it still capture intrinsic knowledge that enables software security tasking to a certain degree <cit.>. We believe exploring the capabilities and limits of this intrinsic knowledge is extremely beneficial for end-users as most organisations and practitioners lack the required resources to teach domain-specific knowledge to LLMs. Hence, we intend to investigate ChatGPT's intrinsic knowledge in software security to help enable the use of chatbots in this area in the future.


We conducted a multifaceted exploration of perceptions and practicality to investigate ChatGPT's potential for supporting software security tasks. First, we analysed public discussions of ChatGPT users regarding software security to gain insights into the successes and failures of LLM-based chatbots for secure development. In the subsequent phase of our study, we focused on vulnerability detection as it was the most anticipated security task among practitioners, according to the first step of our analysis. 
Through qualitative analysis of ChatGPT's responses, we obtained an overview of the types and manners in which information (e.g., descriptions and assessments) is provided and examined the responses for potential practicality issues. Using this two-staged comprehensive assessment, we uncovered not only the current capabilities and shortcomings of ChatGPT in the context of software security but also demonstrated how practitioners' expectations differ from the practical application of this technology in the real world. We assert that the findings from this study are expected to guide future endeavours that aim to develop specialised LLMs for security-related tasks.



§ RELATED WORK









 §.§ AI-Based Chatbots for Software Security.

Chatbots have been demonstrated to be suitable for software engineering tasks such as information retrieval <cit.> and API usage <cit.>. However, the use of chatbots for software security is limited. To date, the SKF-Chatbot (Secure Knowledge Framework) is one of the leading software security chatbots, which was developed by the OWASP Foundation[https://owasp.org/] to provide access to Software Vulnerability (SV) information via a chat interface. However, its capabilities are limited due to its lack of contextual understanding <cit.>. Recently, AI-based chatbots have been able to overcome these limitations via the LLMs that power them. ChatGPT has thus far demonstrated remarkable capabilities for semantic comprehension and providing tailored solutions for given tasks <cit.>. 




 §.§ Vulnerability Detection Using Transformer-based Language Models.

Prior studies have explored the use of transformer-based language models to identify software vulnerabilities using a variety of fine-tuning methodologies. For example, Thapa et al. <cit.> evaluated  transformer-based language models (i.e., BERTBase, GPT-2 Base) against recurrent neural network (RNN)-based models (i.e., BiLSTM, BiGRU) for vulnerability detection in software datasets featuring C/C++ source code. According to their findings, transformer-based language models outperformed RNN-based models on all evaluation metrics. In another study, Kalouptsoglou et al. <cit.> fine-tuned various transformer-based models (i.e., BERT variants, GPT-2, BART) and carried out a comparative analysis to determine which of these models is the most appropriate for vulnerability detection.
Furthermore, Chan et al. <cit.> created a vulnerability detection model targeted at detecting vulnerabilities in incomplete code snippets. They leveraged common learning approaches such as fine-tuning on three pre-trained LLMs, namely CodeBERT, code-davinci-002, and text-davinci-003. 



 §.§ Vulnerability Detection Using ChatGPT.

Several research efforts have explored the efficacy of ChatGPT in vulnerability detection. 
Chen et al. <cit.> exclusively probed ChatGPT's capability in identifying smart contract vulnerabilities. Szabó et al. <cit.> focused on identifying vulnerabilities associated with CWE-653 (Improper Isolation or Compartmentalization) using multiple GPT models. Ozturk et al. <cit.> assessed ChatGPT's effectiveness in detecting the top 10 OWASP vulnerability categories in web applications. Zhang et al., <cit.> through comprehensive testing on two Java and C/C++ 
vulnerability datasets, demonstrated that prompt-enhanced approaches could strengthen ChatGPT's vulnerability detection capability.
In a recent study, Fu et al.<cit.> compared the capabilities of ChatGPT (i.e., GPT-3.5 and GPT-4) against three other LLMs (e.g., CodeBERT<cit.>) across various vulnerability tasks. The LLMs used in this study were fine-tuned explicitly for code-related tasks.

There are two main ways in which our study differs from prior studies:


    
  * Prior research has primarily focused on assessing <cit.> and enhancing ChatGPT's vulnerability detection performance through prompt engineering methodologies <cit.> or the integration of supplementary elements <cit.> within the LLM pipeline.
    We instead strive to focus more on insights through qualitative analysis of existing real-world uses and outputs. This shift aims to illuminate the disparity between practitioners' perceived value and the practicality of this technology in real-world settings. We provide further details in Section <ref>.
    
    
    
    
    
    
  * Prior studies that utilised datasets from existing literature <cit.> or open-source projects <cit.> ignored the possibility of data leakage (i.e., overlap between training and test data). This factor could artificially inflate the reported performance of AI models' (e.g., ChatGPT) <cit.>. Consequently, we curated a high-quality vulnerability dataset tailored to our evaluation criteria from unseen data. Section <ref> discusses this in more detail.
    




§ RESEARCH DESIGN


Initially, we present our research questions in Section <ref>. This is followed by a detailed explanation of the data collection methodology in Section <ref>. Lastly, Section <ref> delves into the qualitative analysis processes.






Figure <ref> displays the overall workflow that we used to conduct this study.




 §.§ Research Questions.


The following Research Questions (RQs) motivated our empirical study: 

    
  * RQ1: What are users' perceptions of using ChatGPT for software security? We examine user demographics, topics, and types of discussions to determine how ChatGPT is being applied by users and its potential strengths and weaknesses.

    

    
    
    
    

    
  * RQ2: How practical are ChatGPT's outputs for the most desired software security task? We perform a qualitative analysis of ChatGPT's outputs for the most discussed topic of RQ1 to ascertain the practicality of this technology in real-world settings.



 §.§ Data Collection.




  §.§.§ Collecting Twitter Data and Curating a Sample
 For the first data source, we selected the popular social media site, Twitter[https://twitter.com], for our investigation due to its active use by developers to discuss emerging technologies <cit.> and its use in former research for uncovering public sentiments <cit.>.


We used the Twitter API to collect English language tweets that matched the following keyword search query: . We collected tweets that were posted between December 1, 2022, and February 28, 2023, the first three months of ChatGPT's public release. 

In total, we collected 7716 tweets.











To enable qualitative analysis of our data, we curated a sample of posts for manual examination. To ensure that our data sample was of sufficiently high quality, we selected the 700 most liked tweets (approximately 10% of the collected tweets) from our dataset. Twitter likes are given by users who appreciate or agree with a tweet. Hence, highly liked tweets are influential posts that reflect the viewpoints of a larger audience and represent broader public sentiments. We label these tweets T1-T700.



  §.§.§ Collecting Vulnerability Data and Curating a Sample


To perform the required analysis for RQ2, 
we needed to test ChatGPT's capabilities in real-world settings. This purpose was achieved by using the National Vulnerability Database (NVD) <cit.>, an established repository for SV management data maintained by the National Institute of Standards and Technology (NIST). This database holds an exhaustive array of information on identified SVs in the wild. 

As we planned to do an in-depth manual analysis for RQ2, 
we decided to work with a subset of 70 SVs.
This sample size was of statistical significance, providing 90% confidence and an error margin of 10% <cit.>. Previous researchers who needed to perform manual analysis over SVs also chose a similar sample size <cit.>. 

It is imperative to note that we considered the issue of data leakage (i.e., contamination of training and test data) in our data selection for RQ2. 
It is well known that ChatGPT has been trained using a great deal of publicly available data. Since NVD is a public database, it is possible that ChatGPT encountered NVD data during its training phase. Thus, to prevent data leakage, we only included SVs in our dataset with a published date (i.e., the official date on which a vulnerability is added to the NVD) after Sep 2021 (the knowledge cutoff date for ChatGPT at the time of conducting this study). Any newly discovered SVs after this date have not been seen by ChatGPT during training.

Additionally, three other factors were considered when curating the random sample of 70 SVs. First, as the fixed code was required for validation purposes in our analysis, the SV report on NVD must be directly linked to the GitHub commit that addresses the SV (i.e., fixing commit). The References section of each SV in NVD usually includes a link to the fixing commit. The second consideration is that the fixing commit should be focused and not tangled (i.e., a commit that consists of multiple unrelated changes). 

The last considered factor is that the vulnerability should be contained within a single function. The last two factors are because we planned to perform all analysis in the following steps manually, and tracing complicated commits is an error-prone process.

After incorporating the aforementioned factors into our SV selection criteria, we ended up with a dataset (containing 70 SVs) that covered 40 distinct CWEs (Common Weakness Enumeration) <cit.>, and its affected products (i.e., products and platforms considered vulnerable) were written in 12 different programming languages. 
































By utilising the provided fixing commits, we carefully collected the entire vulnerable and fixed (i.e., patched) versions of functions from the relevant GitHub repositories. The average number of lines of code among our 70 collected vulnerable functions was 32.



  §.§.§ Collecting ChatGPT Outputs
 

In this step, ChatGPT was fed the 70 collected vulnerable functions as input and asked about potential SVs within them. The GPT-4 model has demonstrated superior performance compared to the GPT-3 model in various benchmarks <cit.>. Considering the importance of delivering high-quality and up-to-date research insights in the critical domain of software security, all ChatGPT-related analysis in this paper were performed using the GPT-4 model. 

Prior research has shown that tailored prompts are essential to taking full advantage of LLM's capabilities in various software engineering tasks<cit.>. 
Following their prompt designs, for each of the SVs in our dataset, we utilised the following prompt template in our conversation with ChatGPT and carefully saved its outputs 
for further analysis. As ChatGPT remembers information from previous inputs, each query was made in a new chat. 





[colback=blue!5!white,colframe=blue!75!black,title= Prompt Template]
  Role: You are a software security expert.
  
  Instruction: Please analyze the following code snippet for potential security vulnerabilities. Provide a detailed explanation of the issues you find.
  
  Context: <code-snippet>




 §.§ Qualitative Analysis.




  §.§.§ Qualitative Analysis of Sampled Tweets


To understand user perceptions of ChatGPT for software security (RQ1), we qualitatively analysed our sample of 
700 tweets using thematic analysis <cit.>. This was an iterative and reflective process that we conducted over several phases. 

We started by immersing ourselves in the data by carefully reading each tweet and all surrounding data: other tweets in the thread, replies, attached media, linked articles, and associated user profiles. The first and third authors conducted this pilot data labelling process collaboratively for 30 tweets, and it typically took 15 to 20 minutes per tweet. During this process, we posed potential characteristics of the discussion that address RQ1: user demographics, topics and types of discussion. To determine the values of these characteristics, we performed open coding <cit.> to classify the common discussion points. We 
collaboratively identified keypoints as short summaries for each tweet. We then revised these into themes, which provide a one or two word description of the common topics. We conducted a second round of pilot data labeling on an additional batch of 30 randomly selected tweets to ensure thorough coverage of potential themes. In this iterative process, no new themes emerged, indicating that thematic saturation had been reached. 


Once we were satisfied with the identified themes, two authors independently analysed and categorised the remaining 640 
tweets in the sample. They then compared the annotated data to ensure data coding was conducted consistently. The two raters achieved a Cohen Kappa value of 0.813 <cit.>, which implies a high level of agreement. Any remaining disagreements between the first and third authors were resolved through discussion with the second author. During the labelling process, we excluded any tweets from our sample that were unrelated (434 tweets) to the use of ChatGPT for software security due to falsely matched keywords, e.g., tweets discussing the impact of ChatGPT on job `security'.

Ultimately, 266 tweets were included in the sample.

We also wanted to gain an understanding of general user sentiments toward ChatGPT's use for software security. E.g., whether the author of a tweet had a positive experience using it or was optimistic about its potential applications. We experimented with using the NLTK sentiment analysis tool <cit.> to classify our data automatically. However, we found this tool to be inaccurate from a manual review of 30 sample tweets. This is because security keywords such as vulnerability or security were often falsely interpreted as indicating negative sentiments. Furthermore, a tweet may have an overall positive sentiment but a negative opinion of ChatGPT, and vice versa. For example, “This article does a really good job of highlighting the weakness of ChatGPT.” Hence, we manually classified user sentiments of ChatGPT during qualitative analysis. For each tweet, we labelled the sentiment as either positive, negative, or neutral based on our interpretation of the user's opinion or experience of ChatGPT capabilities for software security. We reserved the neutral sentiment label for tweets that discussed positive and negative points or were ambiguous to identify. The nature of the sentiment of a tweet depends on the type of discussion (e.g., experiences vs. speculation). Hence, we consider the types of discussion whilst performing sentiment analysis, which we elaborate further on in Section <ref>.





  §.§.§ Qualitative Analysis of Sampled Vulnerabilities


To thoroughly investigate the practicality of ChatGPT 
in the vulnerability detection task (RQ2), we first needed to assess ChatGPT's efficacy in this task. This is because we were interested in exploring how characteristics (e.g., tone) of outputs differ when ChatGPT is accurate versus instances of inaccurate outputs.
To 
this end, we manually analysed the output of all 70 queries 
and provided an assessment as to whether the target SV was detected. This manual assessment was performed independently by the first and third authors of this paper who had a cumulative eight years of both academic and industrial software security experience.

For each SV within the dataset, we used the fixing commit as a baseline for our comparison and employed techniques from prior researchers <cit.> to manually analyse the code changes related to a SV. These techniques are explained step by step as follows. In the first step, we concentrated on the SV summary from the NVD Description section. Moreover, we focused on the information available in the source code (i.e., fixing commit). Aside from code changes, the title and message of the fixing commit were also examined, as they usually provide valuable information regarding the functional aspects of the code and the context in which it was altered. Having thoroughly understood the SV, we assessed 
its generated ChatGPT output. 
Accordingly, each output was labelled as either not found or found. Most often, ChatGPT provided a list of potential vulnerabilities within the input code, with the particular SV we were interested in being mentioned in one of the bullet points. These outputs were classified as found. If the output did not contain information about the vulnerability we sought, it would be marked as not found. This is even if the other potential vulnerabilities ChatGPT listed appeared genuine. This decision was because we wanted to maximise the reliability of our manual analysis and we did not have the ground truth (i.e., fixing commit) for comparison for the other detected vulnerabilities.



  §.§.§ Qualitative Analysis of ChatGPT Outputs


In response to RQ2, we conducted a thematic analysis <cit.> of ChatGPT's outputs regarding the vulnerability detection task. In particular, our primary goal was to demonstrate the breadth and depth of information that a conversational agent such as ChatGPT can offer and examine the nuances of its presentation.
For the thematic analysis of this step, we followed a similar approach as explained in Section <ref>. The first and third authors started by collaboratively performing a pilot data labelling of 10 ChatGPT outputs, which constitutes 14% of the 70 outputs in our dataset. During this process, we focused on the types of information ChatGPT provides and the way in which it is presented. After that, the two authors put more than 100 hours each into this qualitative evaluation and independently analysed the remaining 60 outputs in the dataset. In a supervisory role, the second author reviewed subsets of the analysis results every week to ensure their validity and suggested modifications to improve the assessment accuracy where deemed necessary. Implementing such supervision was necessary to mitigate potential biases and discrepancies in the analysis.

This qualitative analysis was performed with NVivo 12.0 software<cit.> to streamline theme finding. 



§ RESULTS





 §.§ RQ1: perceptions of using ChatGPT for software security


We firstly wanted to determine the occupations of users of ChatGPT for software security. Such insights provide essential context for the source of discussions and sentiments in our data. Furthermore, we reveal the invested stakeholders of AI-based chatbot solutions for software security. We analysed user profile descriptions of the tweet authors (n=202) and identified any occupation details that were listed. We then categorised user occupations using open coding as described in Section <ref>. The population of authors was analysed as a set not containing duplicates. This means that a user was only considered once in the population if they authored multiple tweets. We identified five main occupations in our sample data, which we describe below:


    
  * Security Practitioners are individuals (n=78) that conduct software security or cybersecurity jobs or tasks. This includes occupations such as security analysts, hackers, and vulnerability researchers. 

    
  * Security Companies are the Twitter profiles of organisations (n=27) that provide software security solutions and services. 

    
  * Software Practitioners are individuals (n=19) that perform software development or associated tasks for their occupation. 

    
  * Blogs are the profiles for online sites or organisations (n=46) that provide news outlets or blogging services. 
    
    
    
  * Other defines a collection of occupations which had too few samples to form individual categories. These include academics (n=6) and government employees (n=5). User profiles that contained insufficient or vague descriptions of occupation (n=21) were also placed in the Other category.











Table <ref> displays the occupations of Twitter users in our sample. Given that our dataset is comprised of software security discussions, security practitioners are expected to form the largest user demographic (39%). However, this provides evidence that security practitioners are interested in the potential applications of AI chatbots to their domain, which is supported by the noticeable demographic (13%) of commercial security companies that are also discussing the viability of such tools. Furthermore, the high percentage of security practitioners in our data indicates that our Twitter sample contains reliable information from users with background domain knowledge. 

We next wanted to examine the topics of discussion in our sample Twitter data to understand the areas of software security for which ChatGPT was being used. These insights were expected to help us determine the potential strengths and weaknesses of AI chatbots for different software security tasks. From our qualitative analysis of tweets described in Section <ref>, we discovered five main software security categories in which users utilise ChatGPT, which are also displayed in Table <ref> alongside their frequency. We go into detail for each category below. We provide quoted text from the sample tweets that we have analysed. However, many tweets provide associated media to provide further context to the discussion, which we cannot capture within our paper.





































Vulnerability Detection. Vulnerability detection is one of the most important components of SV management. Consequently, it is the most frequent discussion category (29%) within our sample of tweets. Many users were impressed that ChatGPT has even basic capabilities for vulnerability detection due to the difficulty of this task:






    “Experiment time. I fed ChatGPT the Damn Vulnerable Ethereum Smart Contract Migration code in it's entirety. It pointed to four potential vulnerabilities!!!” (T95)



Whilst many users demonstrated that ChatGPT could successfully identify vulnerabilities in simple code snippets, some users were sceptical about the validity of its use in real-world scenarios. 


    “I just went three rounds with a "security researcher" (who was chatGPT in disguise) making reports on our smart contracts. Plausible text and impacts wrapped in magnificent misunderstandings of the basics.” (T49)


ChatGPT produces plausible-sounding answers even for tasks which it cannot necessarily complete successfully. Vulnerability detection requires substantial expertise, so it is hard for many users to verify responses for these tasks. The accuracy of ChatGPT may also be limited as it was not trained to perform these tasks.  


    “Don’t use ChatGPT for security code review. It’s not meant to be used that way, it doesn’t really work (although you might be fooled into thinking it does)” (T80)


However, despite this lack of trust, users still found the vulnerability analysis capabilities helpful in speeding up code review. 


    “I had ChatGPT take code ... and then audit for security. Would I deploy the resulting code? No. But two weeks just became minutes.” (T150)


Vulnerability Exploits. Users have also explored leveraging ChatGPT to develop vulnerability exploits for penetration testing or offensive security purposes. These capabilities help security testers verify vulnerabilities, which exceeds the functionality of many traditional static analysis tools.  


    “Write an exploit code that abuse brute-force vulnerability on [One Time Password]. ChatGPT result is mind blowing. Half of the pen tester I know are not even close to that level of well/detailed reporting skills” (T67)


However, a serious concern is that ChatGPT can be similarly used to enable malicious and unethical hacking. 


    “In one instance, a hacker shared an Android malware code written by ChatGPT, which could steal desired files, compress them, and leak them online.” (T117)



    “...ChatGPT can turn anyone into a ransomware and malware threat actor.” (T163)


Such uses would have severe consequences for the state of software security due to the increased ease of cyber attacks and exploits. We observed that discussions about malicious hacking were primarily contained in the Speculation discussion category. Hence, further investigation is required to validate these fears and claims. 

Information Retrieval. Users have also used ChatGPT for software security information retrieval and education. 


    “Currently using ChatGPT to help me find and research security solutions for my organization. This has been the biggest game changer for my career since Python.” (T131)


However, similar to vulnerability detection, some security experts were sceptical about the quality of information returned by ChatGPT. A substantial concern is that it is challenging to detect false information provided by ChatGPT.


    “People are excited about using ChatGPT for learning. It's often very good. But the danger is that you can't tell when it's wrong unless you already know the answer. I tried some basic information security questions. In most cases the answers sounded plausible but were in fact BS.” (T1)


Code Analysis. ChatGPT can also be used for other secure code analysis tasks, such as understanding security patches, reverse engineering, malware analysis, or decompiling assembly. 


    “...We used ChatGPT to analyse the patch to understand the buffer overflow.” (T38)






Other. Finally, users have also theorised the potential utility of ChatGPT for other security tasks, such as threat modelling, vulnerability disclosure, secure configuration, and implementation of security controls. These topics appeared infrequently which implies that did not receive as much attention or interest, but users were still often positive. 


    “... we've not really explored its use-cases for Threat Modeling, which I think is game-changing...” (T161)



    “I had ChatGPT write the breach notification for... Pretty close to the real thing.” (T94)


We finally aimed to investigate the ways in which users were discussing the use of ChatGPT for software security. Our qualitative analysis uncovered five main types of discussion. We leveraged these types during our sentiment assessment of the tweets. We assessed user sentiments of ChatGPT for the tweets of each category to better understand the public sentiments about this emerging technology.


    
  * Article/Tutorial tweets promote an external article or media source containing information about using ChatGPT for software security. 

    
  * Speculation discusses the theoretical use of ChatGPT for software security tasks and topics, with any example usage or evidence of application. 

    
  * Example tweets present the authors findings when using example queries as proof of concept application for different software security tasks. Evidence of use is described through experiences or even screenshots of the use cases. 

    
  * Applied tweets are similar to Example tweets, except they discuss real-world industrial applications relevant to users' occupations. 
    
    
  * Tool tweets promote a tool that has been constructed using ChatGPT technologies. 




Figure <ref> displays the frequency and user sentiments for each category of discussion.
 Overall sentiments leaned towards positive (54% positive), which may indicate that most users are optimistic about the capabilities of ChatGPT for software security. 

Articles and tutorials demonstrating how ChatGPT could be used for software security was one of the most prominent discussion types. This demonstrates that the security community is interested in the potential use of AI chatbot technology. Furthermore, some users have begun to develop software security products using ChatGPT or related technologies, as shown in the Tool category. 

Sentiments were also primarily positive for Example tweets (77%) which indicates that ChatGPT may perform well when used for simple proof-of-concept applications. Similarly, articles that discuss or demonstrate ChatGPT's potential are also slightly positive (51%). However, applied use has split sentiments, which indicates that ChatGPT may not yet be transferable to real-world software security tasks. 





Furthermore, the sentiment in speculative discussions is generally negative (47%). Hence, many users are sceptical about the actual practical benefits of ChatGPT when used for software security tasks. As the majority of discussion on this topic is either speculation or untested example usage, it is clear that AI chatbots for software security require more investigation and rigorous evaluation. 

[right=1pt, left=1pt, top=1pt, bottom=1pt, colback=azure(web)(azuremist)]
RQ1 Findings: 


    
    
  * Vulnerability detection was the most discussed software security task among practitioners testing ChatGPT.
    
  * The overall sentiment of the discussions among users was positive.
    
    
  * In various security tasks, credibility and practicality of generated information were common concerns.
    
  * Sentiments were primarily positive for simple proof-of-concept use cases. 







 §.§ RQ2: practicality of ChatGPT's outputs when utilised for the most desired software security task











  

  



Using the qualitative analysis described in Section <ref>, we first recorded our observations regarding ChatGPT's vulnerability detection accuracy for each SV. 
A total of 43 out of 70 (61.42%) vulnerabilities were correctly identified by ChatGPT. This efficacy was achieved when ChatGPT was presented with vulnerable code snippets of moderate complexity— specifically, those that are relatively long (average of 32 lines of code per snippet) with focused fixing commits and not overly intricate.



































































We answer RQ2 in the following sections by describing the types of information ChatGPT generated and how that information is presented for the vulnerability detection task. This investigation can provide a basis for future enhancements in the design and functionality of tailored conversational agents for software security. We provide quoted text from the analysed ChatGPT outputs. Figure <ref> displays the overall structure of the provided information in ChatGPT outputs. 




  §.§.§ Assumption
 It has been observed that ChatGPT occasionally makes assumptions about tertiary code when analysing the source code inputted. This kind of information was presented tentatively because they all rest on assumptions.

    “For instance, we're making assumptions about functions like compose_sadb_supported() and pfkey_broadcast() which could have their own set of vulnerabilities.” (Output #30)








  §.§.§ Description
 There was a constant presence of descriptions in the ChatGPT responses. According to our thematic analysis, the descriptions can be divided into three major categories, Supplied Code, Vulnerability, and Mitigation.



    
  * Supplied Code: ChatGPT frequently begins its responses with a description of the supplied code. For instance, the functionality of the code or its programming language.

    “This code snippet is written in Java and seems to be responsible for handling requests to trigger a configuration reload through a security token system.” (Output #70)





    
  * Vulnerability: A large portion of each ChatGPT response is devoted to describing identified vulnerabilities. In addition to classifying these descriptions under the Vulnerability category, we further divided them into two themes for a more granular analysis: “Theoretical and Practical Implications". This is because we sought to distinguish between when ChatGPT delves into abstract security implications and when it explains the actionable aspects of potential exploits.


“Theoretical Implication" example:

    “...
4. Exposure of Sensitive Information: The password recovery link may contain sensitive information (like a reset token). The application must make sure to generate a unique and unpredictable token for each password recovery request. ...
” (Output #14)






“Practical Implication" example:

    “... There are a few places where objects are dereferenced without first ensuring they are not null. This could potentially lead to a segmentation fault and crash the program. ...
” (Output #26)


Based on our observations, out of 43 cases where ChatGPT identified the specific vulnerability of interest, 38 instances (88%) were classified under the “Theoretical Implications” theme, while only 5 instances  (12%) were categorised under “Practical Implications”.

    
  * Mitigation: ChatGPT infrequently provided specific information on how to address the particular vulnerability identified. This category contained two themes: “List of Resolutions" and “Strategy".
    
    With the “List of Resolutions" theme, when ChatGPT detects multiple potential vulnerabilities, it offers a numbered list with brief explanations of how to resolve them.

    “To mitigate these vulnerabilities: 
1. Use parameterized queries or prepared statements to avoid SQL injection. 
2. Perform comprehensive input validation and sanitization on all user inputs.
3. Implement proper access control and authorization checks to prevent unauthorized actions.
4. ...
” (Output #64)








The “Strategy" theme revealed that ChatGPT provides a 
detailed response 
when encountering a 
severe vulnerability. In addition, we noticed that the responses generated in these instances have an imperative tone. For instance, the phrase “you should" is commonly used in this theme.

    “In order to make this script more secure, you should: 
Sanitize and validate all input data to prevent injection attacks. Use a safer file handling technique, which does not allow manipulation of file names. ... ” (Output #51)






























  §.§.§ Security Guideline
 ChatGPT makes a persistent effort to make its responses as comprehensive as possible by including generic security guidelines. This phenomenon was observed in all 70 instances of outputs. We classified these guidelines into three categories: Advise Further Testing, Best Security Practices, and Pointers for Consideration.


    
  * Advise Further Testing: The first and most common category (33 instances) is when ChatGPT demands further security audits to be undertaken via experts and automated tools.


    “... It is always recommended to have your
code reviewed by a security expert and use static and dynamic analysis tools to uncover potential security vulnerabilities.” (Output #31)


    
  * Best Security Practices: In the second category, we identified that ChatGPT relied on best security practices (e.g., input validation) applicable to the particular vulnerability to deliver its response.

    “Misconfigurations in PAM can lead to security vulnerabilities, so ensure that you have properly configured PAM in accordance with best security practices. Always adhere to the principle of least privilege. ” (Output #34)





    
  * Pointers for Consideration: Lastly, the third category contains very basic security considerations.

    “you might want to consider using a pre-made, widely-reviewed authentication library or service, which will likely have these and other security issues resolved, rather than creating your own.” (Output #42)


Considering that these guidelines were all advice, all three security guideline categories were presented tentatively.



  §.§.§ Assessment
 The paramount section of ChatGPT responses pertains to its evaluation of potential vulnerabilities within the submitted source code. Our rigorous analysis identified three assessment categories: Vulnerable Classification, Security Issues and Concerns, and Non-vulnerable Classification.


    
  * Vulnerable Classification: This category emerged as the most prominent among the three assessment categories. Considering that all the codes supplied to ChatGPT were already vulnerable, this prevalence was unsurprising. However, a significant variation in the tone of the responses was discovered, which appears to be related directly to the severity of the vulnerability. To further understand, we separated the detected confidence levels into three distinct themes: “Very High", “High", and “Moderate".
    
Within the “Very High" theme, ChatGPT responses were characterised by an assertive tone. In contrast to the other two themes (High and Moderate), adjectives such as “significant" and “serious" were frequently used. Furthermore, ChatGPT repeatedly hinted at the presence of multiple vulnerabilities instead of identifying one individually.

    “While this code may seem harmless at first glance, it indeed has several serious security vulnerabilities. ...” (Output #9)




The tone is less assertive in the “High" theme compared to the tone in the previous theme. For instance, while ChatGPT continues to identify multiple vulnerabilities, the word “potential" is frequently used to indicate a degree of uncertainty.

    “I've found several potential security vulnerabilities in the code. Here's a detailed analysis: ...” (Output #53)




There is a noticeable drop in certainty at the “Moderate" theme compared to the “High" theme. Apart from the frequent use of the term “potential" ChatGPT recurrently cites an absence of contextual information as a rationale for its implied uncertainty.

    “Based on the provided code snippet, there are a few potential security vulnerabilities to consider: ... Please remember that a complete and accurate analysis would require more context...” (Output #52)




Out of all the correctly identified vulnerabilities (43) under investigation, 25 instances (58%) were classified into this theme. The responses were presented tentatively in this theme compared to the other two themes as ChatGPT was less confident about its outputs.
    
  * Security Issues and Concerns: There have been instances where ChatGPT identified potential security weaknesses but refrained from explicitly labeling them as security vulnerabilities. Alternatively, it has used more subtle terms such as “concerns" and “issues". We have categorised such instances under the Security Issues and Concerns category.

    “However, there are some potential issues to be aware of, more from the perspective of robust programming practices and potential misuse, rather than direct security vulnerabilities: ...” (Output #50)





    
  * Non-vulnerable Classification: ChatGPT made a non-vulnerable assessment in several instances. In these cases, ChatGPT consistently pointed out that the supplied code snippet may not necessarily be secure.

    “At a glance, this code does not seem to contain glaring security vulnerabilities such as buffer overflows, SQL injections, etc., that are commonly associated with insecure code. ...  - an innocent-looking piece of code can become a security hole if used improperly or in an unexpected way.” (Output #21)








  §.§.§ Lack of Context
 

We observed that in exactly %50 of the outputs (35 out of 70), ChatGPT mentioned that more context would be required for a concrete security analysis.

    “While the code snippet does not provide all of the details, we can still make several general observations based on the provided information.” (Output #13)


[right=1pt, left=1pt, top=1pt, bottom=1pt, colback=azure(web)(azuremist)]
RQ2 Findings: 


    
  * Even when ChatGPT correctly pinpoints the vulnerability of interest, it predominantly focuses on abstract rather than practical security implications.
    
  * Although ChatGPT responses appear comprehensive, they are heavily populated with generic security information (e.g., Description, Guideline).
    
  * The severity of the vulnerability appears to be directly related to the tone of the responses.
    
  * ChatGPT generally adopted a tentative tone to present the information.
    
    





§ DISCUSSION


In RQ1, our investigation has shown that software and security practitioners enthusiastically experiment with and share the pros/cons of using ChatGPT for assistance in various software security tasks, such as vulnerability detection and security information retrieval. However, the state of discussions mostly remains in example proof of concept use, whereas only a few cases of applied use have been discussed. Current security practitioners appear to think that ChatGPT is inaccurate as a standalone solution. Hence, ChatGPT may be better suited to secondary tasks such as vulnerability report writing. For instance, users can draft comprehensive and detailed vulnerability reports by providing context-specific prompts and formatting instructions. Nevertheless, security professionals should review and validate the contents to ensure accuracy and completeness.

In the second phase of our study (RQ2), we treated ChatGPT as an oracle for the vulnerability detection task. In our data sample, it achieved a detection accuracy of 61%, which is far from ideal. Furthermore, after qualitatively examining the generated responses, we determined the characteristics (i.e., type and presentation style of information) of this chatbot and found that it is unable to provide reliable information for this task. Our analysis revealed that ChatGPT responses suffer from two specific quality issues. The most prevalent issue we observed was that all responses (70 instances) contained generic security information, as indicated in the “Theoretical Implication” theme and the three themes under the “Security Guideline” information type. Throughout these themes, we found an abundance of unneeded information, resulting in a lack of clarity in the responses. This issue persisted even in cases where ChatGPT correctly identified the vulnerability. Another noteworthy issue we found was that ChatGPT regularly lacks assertiveness when delivering responses. Based on our analysis, the majority of responses were given with a tentative tone, as shown in Figure <ref>, and a consistent request for additional contextual information. 

We believe both of these quality issues may undermine the practicality of ChatGPT for vulnerability detection within industry settings. This is because when a fault localisation tool is not transparent about its limitations, developers may inadvertently spend valuable time and effort verifying potentially inaccurate outputs <cit.>.

However, in contrast to existing weakness detection tools (e.g., static code analyser), conversational agents such as ChatGPT allow users to interface with a tool with profound natural language comprehension abilities. This potentially can have positive effects on novice learners who want to start learning secure coding principles. 


Overall, we advocate the need for further development of domain-relevant LLMs trained using software security data. Although ChatGPT is trained for general use and not intended for software security, our qualitative study shows it can assist and augment processes to a certain degree. Hence, we expect domain-specific LLMs to have potent capabilities for software security in the future. These technologies ultimately need to undergo rigorous industrial evaluation for which the empirical software engineering community can provide leadership in collaboration with AI researchers and practitioners. 
























    
    

    

    
    

    













 §.§ Implications of Our Study


Programmers. Understanding the nuances of responses (RQ2) generated by ChatGPT can offer valuable benefits to programmers who leverage this technology for vulnerability detection. Notably, they may better assess the legitimacy of the generated information and make informed decisions about whether to trust and adopt the security suggestions. Furthermore, by understanding ChatGPT's strengths and limitations, programmers can better integrate this technology into their development workflows, using it as an adjunct to other weakness detection tools. 

AI practitioners. The investigation into ChatGPT's communication of software security information (RQ2) has explicit implications for AI researchers and developers. Insights of this nature may assist them in fine-tuning LLM-based chatbots for specific software security tasks. Specifically, they must strive to create security-tailored LLMs capable of conveying security information to users in a manner that is more straightforward, actionable, and less susceptible to misinterpretation. Such models are likely to perform better and be more practical when applied in real-world security settings.

Security researchers. We have identified some promising research directions for researchers. Our analysis in RQ1 unveiled that there is a lot of speculation (discussions in the Vulnerability Exploits category) regarding the potential misuse of ChatGPT for malicious purposes. Researchers need to identify whether technologies like ChatGPT can enable malicious hacking so that countermeasures can be applied to prevent using these technologies for such purposes. 
Additionally, our analysis in RQ2  demonstrated that ChatGPT alters its language tone when confronted with vulnerabilities of varying severity scores. Further research is required to thoroughly understand the capability of LLMs in estimating vulnerability severity. This capability is imperative in vulnerability prioritisation, where precise severity assessment is essential for effective mitigation strategies.










§ THREATS TO VALIDITY




Construct validity: Our manual analysis may be subject to bias or inaccuracies, as is the case for most qualitative research. We have attempted to minimise such threats to validity by ensuring that two annotators independently performed the required manual analysis.

Internal Validity: We acknowledge that the criteria we used for selecting vulnerability data could have influenced the outcome of our study. As explained in Section <ref>, our intention to manually trace code changes meant we could not trace tangled fixing commits. As a result, we decided to limit the number of line changes within fixing commits to 10 lines. This approach ensured we gathered code snippets that were neither simplistic nor too complex.


External Validity: Qualitative analysis enables us to examine our data in depth, but consequently, we can only engage with a small sample of the total dataset. We cannot generalise beyond the sampled tweets and vulnerabilities due to the effort required to examine the data manually. We obtained our tweet sample based on the most liked and influential tweets, so we expect our sample represents a larger audience. 


Furthermore, public opinions may be inaccurate or biased for emerging technologies, as users may be overly excited or not have had enough time to evaluate their use properly. Despite our desire to extend the three-month period for collecting tweets, we were unable to do so since Twitter's academia API has been deprecated following recent changes at the company.

It is also possible that our findings might not generalise to future ChatGPT versions enriched with updated data. However, we conducted this study using a single ChatGPT version with the same knowledge cut-off date (Sep 2021) for consistency.



§ CONCLUSION






We have conducted an empirical investigation to understand the potential use of ChatGPT for software security. Based on examining public discussions regarding this LLM, we found that practitioners were often optimistic about its utility for various security tasks. 
Our systematic evaluation further demonstrated that ChatGPT can identify vulnerabilities 
with roughly 61% accuracy. Furthermore, we qualitatively demonstrated the versatility and granularity of information that ChatGPT can offer in the vulnerability detection task. The results revealed that ChatGPT outputs in this task are often too generic and ambiguous to be of practical use at the industry level. A tailored security LLM could enhance this capability, enabling practitioners to leverage the abundance of knowledge embedded in LLMs.



§ DATA AVAILABILITY

We have made all our artefacts of this study available via a reproduction package<cit.>.



§ ACKNOWLEDGMENT

This work has been supported by the Cyber Security Cooperative Research Centre Limited whose activities are partially funded by the Australian Government’s Cooperative Research Centre Programme.


IEEEtran





