# Layer Ensembles

###### Abstract

Deep Ensembles, as a type of Bayesian Neural Networks, can be used to estimate
uncertainty on the prediction of multiple neural networks by collecting votes
from each network and computing the difference in those predictions. In this
paper, we introduce a novel method for uncertainty estimation called Layer
Ensembles that considers a set of independent categorical distributions for
each layer of the network, giving many more possible samples with overlapped
layers, than in the regular Deep Ensembles. We further introduce Optimized
Layer Ensembles with an inference procedure that reuses common layer outputs,
achieving up to 19× speed up and quadratically reducing memory usage. We also
show that Layer Ensembles can be further improved by ranking samples,
resulting in models that require less memory and time to run while achieving
higher uncertainty quality than Deep Ensembles.

Index Terms—  Deep Ensembles, Bayesian neural networks, uncertainty
estimation, uncertainty quality

##  1 Introduction

Uncertainty estimation in neural networks is an important task for critical
problems, such as autonomous driving, medical image analysis, or other
problems where silent failures of machine learning systems can lead to high-
cost damages or endanger lives. Bayesian Neural Networks (BNNs) [1, 2, 3]
provide a tool to estimate prediction uncertainty by exploiting a distribution
over the network weights and sampling a set of models with slightly different
predictions for a given input. This difference in the predictions expresses
the uncertainty of the network, while the mean of all predictions is used as
the prediction of the network. The selection of the adopted distribution
affects the computational requirements and statistical quality of the network,
with Gaussian distribution resulting in Bayes By Backpropagation (BBB) [4] and
Hypermodel [5] methods, Bernoulli distribution in Monte Carlo Dropout (MCD)
[6], and Categorical distribution in Deep Ensembles [7].

We introduce Layer Ensembles, which consider a set of weight options for each
layer that are sampled using independent Categorical distributions, resulting
in a high number of models that can have common layer samples. We show that
Layer Ensembles achieve better uncertainty quality than Deep Ensembles for the
same number of parameters, and they allow to dynamically change the number of
samples to keep the best ratio between the uncertainty quality and time cost.

##  2 Related Work

Output uncertainty estimation is usually done by approximating expectation and
covariance of outputs using the Monte Carlo integration with a limited number
of weight samples, which can be simplified to running the network few times
with different samples of random variables and then computing the mean and
variance of the output vectors. Epistemic Neural Networks (ENNs) [8] propose a
framework to estimate an uncertainty quality of a model by generating a
synthetic dataset and training a Neural Network Gaussian Process (NNGP) [9] on
it that represents a true predictive distribution. The model of interest is
then evaluated by the KL-divergence [10] between the true predictive
distribution from NNGP and the predictive distribution of the model of
interest.

Monte Carlo Dropout (MCD) [6], instead of only using Dropout [11] layers as a
form of regularization during training to avoid overtrusting particular
neurons, it also keeps these Dropout layers during inference. This has the
effect of adopting a Bernoulli distribution of weights and sampling different
models from this distribution. Bayes By Backpropagation (BBB) [4] considers a
Gaussian distribution over a network’s weights, which is estimated using the
reparametrization trick [12] that allows to use regular gradient computation.

Variational Neural Networks (VNNs) [13] can be considered in the same group as
MCD and BBB from the Bayesian Model Averaging perspective, where sampled
models can lie in the same loss-basin and be similar, i.e., describing the
problem from the same point of view, as explained in [2]. VNNs consider a
Gaussian distribution over each layer’s predictions, that is parametrized by
the outputs of the corresponding sub-layers. Hypermodels [5] consider an
additional hypermodel $\theta=g_{\nu}(z)$ to generate parameters of a base
model $f_{\theta}(x)$ using a random variable $z\sim\mathcal{N}(0,I)$ as an
input to the hypermodel.

Deep Ensembles [7] have a better uncertainty quality than all other discussed
methods and can be viewed as a BNN with a Categorical distribution over
weights, with the ideal number of weight samples equal to the number of
ensembles. The addition of prior untrained models to Deep Ensembles, as
described in [7], improves the uncertainty quality of the network. Deep Sub-
Ensembles [14] split the neural network into two parts, where the first part
contains only a single trunk network, and the second part is a regular Deep
Ensemble network that operates on features generated by the trunk network.
This reduces the memory and computational load, compared to the Deep
Ensembles, and provides a trade-off between the uncertainty quality and
resource requirements. Batch Ensembles [15] optimize Deep Ensembles by using
all weights in a single matrix operation and using Hadamard product instead of
matrix multiplication that increases inference speed and reduces memory usage.

##  3 Layer Ensembles

We consider a neural network $F(x,w)$ with $N$ layers that takes $x$ as input
and is parametrized by the weights $w$. A Deep Ensemble network is formed by
$K$ identical networks with $N$ layers each, where the corresponding weights
of each network, i.e., $w_{i},\>i\in[1,K]$, are trained independently. We
formulate Layer Ensembles as a stochastic neural network $F(x,w)$ with $N$
layers $\mathrm{LE}_{i}(x,w_{q}^{i}),\>i\in[1,K],\>q\in[1,N]$ and $K$ weight
options for each layer:

$\displaystyle\begin{split}&w_{q}^{i}\sim\mathrm{Categorical}(K).\\\
\end{split}$ (1)

This results in the same memory structure as for Deep Ensembles, with $KN$
weight sets for a network of $N$ layers and $K$ ensembles. However, Layer
Ensembles allow for connections between the layers of different ensembles, by
sampling different layer options to form a network in the ensemble. This
greatly increases the number of possible different weight samples, while those
can contain identical subnetworks. This can be used to speed up the inference
of a set of sampled layers. Fig. 1 illustrates how the same memory structure
of the ensembles is used in Deep Ensembles (Fig. 1(a)) and Layer Ensembles
(Fig. 1(b)).

(a) Deep Ensemble structure

(b) Layer Ensemble structure

(c) Two samples of a Layer Ensemble network with common first two layer
options

Fig. 1: Example structures of (a) Deep Ensembles and (b) Layer Ensembles for a
3-layer network with 3 ensembles ($N=3$, $K=3$). While the memory structure
remains identical, Layer Ensembles have many more options for sampling that
can be optimized considering the common layers in samples. Layer Ensembles
with common layers earlier in the architecture lead to faster inference (c).

Training of Layer Ensembles is done the same way as for Deep Ensembles, by
using a regular loss function and averaging over the predictions of different
weight samples. The number of weight samples for Deep Ensembles is usually
equal to the number of networks $K$, meaning that all the ensembles are used
in the prediction process. The same strategy is not required for Layer
Ensembles, as each layer option can be included in multiple networks. This
means that one can select a few layer options per inference and expect that,
with the sufficient amount of training steps, all the layer weights will be
trained.

Following [7], we consider an output of a prior untrained Layer Ensemble
network added to the output of the trained network, using the same draws from
the random distributions for both untrained and trained networks. Experiments
show that the addition of prior networks improves the uncertainty quality of
Layer Ensembles by a factor of $2$ for each number of ensembles that was
tested.

Layer Ensembles can be used to define Deep Ensembles [7] and Deep Sub-
Ensembles [14] as special cases by sampling specific layers or number of
ensembles to be used. Considering a Layer Ensemble network with $N$ layers and
$K$ ensembles for each layer, we can sample $K^{N}$ possible models. Sampling
$K$ layers with none of the layer options used in two different networks
corresponds to a Deep Ensemble network. By selecting different number of
ensembles per layer, we can achieve Deep Sub-Ensembles by using one ensemble
for the first $T$ layers and $K$ ensembles for the remaining $N-T$ layers,
resulting in a single trunk network and in an ensembled tail network.
Experimenting with the number of ensembles for each layer can result in
interesting new methods for specific analysis problems and is a direction for
future work.

###  3.1 Inference Optimization

Layer Ensembles can reuse outputs of identical subnetworks processing the
input $x$ when they are used in different networks. Fig. 1(c) shows an example
of two Layer Ensembles where the first two layers are identical, and only the
last layer has different weights. Instead of computing
$c_{2}(b_{2}(a_{1}(x)))$ and $c_{1}(b_{2}(a_{1}(x)))$ independently, one can
compute the common layer $V=b_{2}(a_{1}(x))$ first, and then $c_{2}(V)$ and
$c_{1}(V)$.

Algorithm 1 Optimized Layer Ensembles

1:Network $F(x)$, list of sorted samples $S$, layer index $i$, input $x$

2:function OLE($F,S_{i},i,x$)

3: $\mathrm{result}\leftarrow[]$

4: $s_{i+1}\leftarrow[]$

5: if $i=\mathrm{size}(s)$ then return $[x]$ $\triangleright$ Final layer
computed

6: end if

7: $s_{l}\leftarrow S_{i}[0]$

8: $l\leftarrow F[i][s_{l}](x)$ $\triangleright$ First sampled option for
layer $i$

9: for $t\in[0..\mathrm{size}(S_{i})]$ do $\triangleright$ For each sample

10: if $S_{i}[t]\neq s_{l}$ then

11: $\mathrm{result}=\mathrm{result}\cup\mathrm{OLE}(F,s_{i+1},i+1,l)$

12: $s_{l}\leftarrow S_{i}[t]$

13: $\triangleright$ Next sampled option for layer $i$

14: $l\leftarrow F[i][s_{l}](x)$

15: $s_{i+1}\leftarrow[]$

16: end if

17: $\triangleright$ Update sub-samples list for input $l$

18: $s_{i+1}\leftarrow s_{i+1}\cup s_{i}[t][1:]$

19: end for

20: $\mathrm{result}=\mathrm{result}\cup\mathrm{OLE}(F,s_{i+1},i+1,l)$

21: return result

22:end function

23:return OLE$(F,S,0,x)$

Algorithm 1 implements an Optimized Layer Ensembles (OLE) function that
recursively computes the output of a Layer Ensemble network for a set of
sorted layer samples. Layer samples are represented as a set of selected
options for each layer, such as $[1,2,2]$ and $[1,2,1]$ for the model in Fig.
1(c). These samples are sorted in ascending order by the first-most values,
while using later indices in case of identical previous values. This allows to
have the most overlapping samples in a sequence, giving the possibility to
optimize layer executions, as a layer option should be called only once for
the same input and used by all samples that share it. After the current layer
option is used, there is no need to keep its output in memory anymore, as it
will never be used later. The results of the OLE function is an array of
outputs for all runs using this layer, which means that in order to run a full
set of samples the OLE function needs to be called with network function $F$,
samples list $S$, layer index $i=0$, and the input to the network $x$.

##  4 Uncertainty Quality Experiments

Fig. 2: Comparison of mean KL values with 1 STD range for Layer Ensembles with
different number of ensembles and sampled layers, averaged across all
experiment parameters. The best layer samples are selected based on the
validation set and evaluated on the test set.

We implement Layer Ensembles inside the Epistemic Neural Networks (ENNs) [8]
framework to estimate the uncertainty quality of this method and to compare it
with Deep Ensembles. ENNs consider a regression task $y=f(x)+\epsilon$ and
generates a synthetic dataset $D_{T}=\\{(x,y)_{t}$ for $t\in[0,T-1]\\}$, where
$x$ is a $D_{x}$-dimensional input vector, $y$ is an output scalar,
$\epsilon\sim\mathcal{N}(0,\sigma^{2})$ is a random noise, and
$T=D_{x}\lambda$ is a dataset size. A Neural Network Gaussian Process (NNGP)
[9] is trained on the dataset to represent the true model of the data. For
each data point, a model of interest should provide a prediction $\mu$ and an
uncertainty in that prediction $\sigma^{2}$, which is modeled by a one-
dimensional Gaussian distribution $\mathcal{N}(\mu,\sigma^{2})$. Given the
predictions from both the true NNGP model and a model of interest $M$, the
uncertainty quality score $Q(M)$ is computed as:

$\displaystyle\begin{split}&Q(M)=\frac{1}{T}\sum^{T-1}_{t=0}\mathrm{KL}(\mathcal{N}_{M}\parallel\mathcal{N}_{\mathrm{NNGP}}),\\\
&\mathcal{N}_{M}=\mathcal{N}(\mathbb{E}[M(x_{t})],\mathrm{Var}[M(x_{t})]),\\\
&\mathcal{N}_{\mathrm{NNGP}}=\mathcal{N}(\mathbb{E}[\mathrm{NNGP}(x_{t})],\mathrm{Var}[\mathrm{NNGP}(x_{t})]),\end{split}$
(2)

where $M$ and $\mathrm{NNGP}$ are the model of interest and the true NNGP
model, respectively, and $\mathrm{KL}$ is a Kullback–Leibler divergence
function [10].

We implement Layer Ensembles inside the ENNs JAX repository [16] and follow
the same experimental parameters, including $D_{x}\in\\{10,100,1000\\}$,
$\lambda\in\\{1,10,100\\}$, and $\epsilon\in\\{0.01,0.1,1\\}$. The experiments
are repeated with all combinations of $(D_{x},\lambda,\epsilon)$ parameters
and with $10$ different random seeds. The results are then averaged, computing
the mean and variance of uncertainty quality scores for each method. Fig. 4
provides a comparison between Deep Ensembles and Layer Ensembles for different
number of ensembles and number of samples. Layer Ensembles start to achieve
good uncertainty quality with only 3 ensembles and outperform Deep Ensembles
for the same number of ensembles used. This means that the memory footprint is
much lower for Layer Ensembles.

###  4.1 Layer sample ranking

Since there are many possible layer samples for Layer Ensembles, it is not
always feasible to use all possible networks when using Layer Ensembles. One
way to decrease the computational cost is to randomly select fewer layer
samples, resulting in slightly lower quality of uncertainty, as shown in Fig.
4. Another option is to rank layer samples based on uncertainty quality on the
validation set and use the best layer samples when using a particular number
of samples.

Let us consider the full set of layer samples $S=\\{s_{j}|j\in[1,J]\\}$, where
$J$ is a number of all combinations that can be computed by multiplying all
layer-wise numbers of ensembles. To reduce the computational load of layer
sample ranking, we introduce an iterative process of selecting best layer
samples by starting from a single layer sample with the best mean error
$s^{1}$:

$s^{1}=\underset{s_{j}}{\mathrm{argmax}}\>\>Q(M^{\\{s_{j}\\}}),$ (3)

where $Q(\cdot)$ is the uncertainty quality score function, $M^{\\{s_{j}\\}}$
is the Layer Ensemble model applied to a set of layer samples, containing a
single sample $s_{j}$. Given a set of optimal layer samples
$S^{P}=\\{s^{P}_{i}|i\in[1..P]\\}$ of size $P$, the next layer sample set is
created by finding the best addition to the already existing set:

$\displaystyle\begin{split}s^{P+1}&=\underset{s_{j}}{\mathrm{argmax}}\>\>Q(M^{S^{P}\cup
s_{j}}),\\\ S^{P+1}&=S^{P}\cup s^{P+1}.\end{split}$ (4)

Fig. 3: Speed up and memory saved during inference of Optimized Layer
Ensembles, compared to the regular Layer Ensembles for different number of
ensembles of a 4-layer CNN for MNIST classification. This excludes memory used
for the ML framework and model weights.

Fig. 2 illustrates the uncertainty quality results of each best layer sample
set of Layer Ensembles. With a number of ensembles higher than 2, the optimal
uncertainty quality is increased up to a certain number of layer samples and
starts decreasing after that. This means that it is beneficial for both
inference speed and uncertainty quality to not use all the available layer
samples. Layer Ensembles with $5$ ensembles achieves the best uncertainty
quality at $36$ layer samples, which is much lower than the number of $125$
possible layer samples. Even at $20$ layer samples, the uncertainty quality is
just $6\%$ lower, but it is still $2$ times better than the Deep Ensembles
with 30 ensembles, while it uses $6$ times less memory, and it is at least
$1.5\times$ faster, as the speed can be improved by Optimized Layer Ensembles
inference procedure but depends on the overlap between layer samples.

Fig. 4: Comparison of mean KL values with 1 STD range for Deep Ensembles and
Layer Ensembles with random unique layer samples, averaged across all
experiment parameters.

##  5 Conclusions

In this paper, we proposed a novel uncertainty estimation method called Layer
Ensembles, which corresponds to a Bayesian Neural Network with independent
Categorical distribution over weights of each layer. We showed that Layer
Ensembles use parameters more effectively than Deep Ensembles and provide a
flexible way to balance between inference time and model uncertainty quality.
We showed that the inference of Layer Ensembles can be optimized by performing
the same computations once, which increases the inference speed by up to $19$
times and reduces memory usage quadratically. Finally, we proposed a layer
sample ranking system that allows to select best layer samples based on the
combined uncertainty quality, leading to a high increase in uncertainty
quality and reducing both memory and time requirements.

## References

  * [1] David J C Mackay,  “Probable networks and plausible predictions — a review of practical bayesian methods for supervised neural networks,”  Network: Computation in Neural Systems, vol. 6, no. 3, pp. 469–505, 1995.
  * [2] Andrew Gordon Wilson and Pavel Izmailov,  “Bayesian Deep Learning and a Probabilistic Perspective of Generalization,”  in NeurIPS, 2020, vol. 33.
  * [3] Tom Charnock, Laurence Perreault-Levasseur, and François Lanusse,  “Bayesian neural networks,”  arXiv:2006.01490, 2020.
  * [4] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra,  “Weight Uncertainty in Neural Networks,”  in JMLR Workshop and Conference Proceedings, 2015, vol. 37, pp. 1613–1622.
  * [5] Vikranth Dwaracherla, Xiuyuan Lu, Morteza Ibrahimi, Ian Osband, Zheng Wen, and Benjamin Van Roy,  “Hypermodels for exploration,”  in ICLR Proceedings, 2020, vol. 8.
  * [6] Yarin Gal and Zoubin Ghahramani,  “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,”  in JMLR Workshop and Conference Proceedings, 2016, vol. 48, pp. 1050–1059.
  * [7] Ian Osband, John Aslanides, and Albin Cassirer,  “Randomized prior functions for deep reinforcement learning,”  in NeurIPS, 2018, vol. 31, pp. 8626–8638.
  * [8] Ian Osband, Zheng Wen, Mohammad Asghari, Morteza Ibrahimi, Xiyuan Lu, and Benjamin Van Roy,  “Epistemic Neural Networks,”  arXiv:2107.08924, 2021.
  * [9] Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein,  “Deep neural networks as gaussian processes,”  in 6th ICLR Proceedings, 2018.
  * [10] S. Kullback and R. A. Leibler,  “On Information and Sufficiency,”  The Annals of Mathematical Statistics, vol. 22, no. 1, pp. 79 – 86, 1951.
  * [11] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov,  “Dropout: A simple way to prevent neural networks from overfitting,”  Journal of Machine Learning Research, vol. 15, no. 56, pp. 1929–1958, 2014.
  * [12] Diederik P Kingma and Max Welling,  “Auto-encoding variational bayes,”  in ICLR Proceedings, 2014, vol. 2.
  * [13] Illia Oleksiienko, Dat Thanh Tran, and Alexandros Iosifidis,  “Variational neural networks,”  arxiv:2207.01524, 2022.
  * [14] Matias Valdenegro-Toro,  “Deep sub-ensembles for fast uncertainty estimation in image classification,”  arxiv:1910.08168, 2019.
  * [15] Yeming Wen, Dustin Tran, and Jimmy Ba,  “Batchensemble: An alternative approach to efficient ensemble and lifelong learning,”  in ICLR, 2020, vol. 8.
  * [16] Ian Osband, Zheng Wen, Mohammad Asghari, Morteza Ibrahimi, Xiyuan Lu, and Benjamin Van Roy,  “Github code for epistemic neural networks,” 2021.


# Automated DevOps Pipeline Generation for Code Repositories using Large
Language Models

Deep Mehta, Kartik Rawool, Subodh Gujar, Bowen Xu  North Carolina State
UniversityRaleighNorth CarolinaUSA27606 dmmehta2, khrawool, sgujar, bxu22
@ncsu.edu

(2024)

††copyright: acmcopyright††journalyear: 2024††doi: XXXXXXX.XXXXXXX

Automating software development processes through the orchestration of GitHub
Action workflows has revolutionized the efficiency and agility of software
delivery pipelines. This paper presents a detailed investigation into the use
of Large Language Models (LLMs) – specifically, GPT 3.5 and GPT 4 – to
generate and evaluate GitHub Action workflows for DevOps tasks. Our
methodology involves data collection from public GitHub repositories, prompt
engineering for LLM utilization, and evaluation metrics encompassing exact
match scores, BLEU scores, and a novel DevOps Aware score. The research
scrutinizes the proficiency of GPT 3.5 and GPT 4 in generating GitHub
workflows, while assessing the influence of various prompt elements in
constructing the most efficient pipeline. Results indicate substantial
advancements in GPT 4, particularly in DevOps awareness and syntax
correctness. The research introduces a GitHub App built on Probot, empowering
users to automate workflow generation within GitHub ecosystem. This study
contributes insights into the evolving landscape of AI-driven automation in
DevOps practices.

##  1\. Introduction

Automation has become integral to modern software development, and GitHub
Action workflows stand as a pivotal tool in orchestrating streamlined software
delivery pipelines. This paper delves into an in-depth exploration of Large
Language Models (LLMs), specifically GPT 3.5 and GPT 4, to generate and
evaluate GitHub Action workflows tailored for DevOps tasks. Our research
methodology involves the curation of pertinent data from public GitHub
repositories and the engineering of prompts to guide LLMs in workflow
generation. Evaluation metrics, including exact match scores, BLEU scores, and
a newly introduced DevOps Aware score, assess the fidelity of generated
workflows to DevOps best practices.

Moreover, a pivotal contribution of this research is the development of a
dedicated GitHub App. This App capitalizes on the capabilities of LLMs,
allowing seamless workflow generation for any GitHub repository and immediate
commit into the repository itself. This innovative tool empowers developers by
automating the generation and integration of workflows directly into their
projects, streamlining their software development lifecycle.

The outcomes of this research present promising advancements, particularly in
the enhanced capabilities of GPT 4, exhibiting heightened DevOps awareness and
improved syntactic accuracy. The GitHub App’s development further signifies
the practical application of AI-driven models in the realm of DevOps
practices, albeit with identified limitations.

##  2\. Related Work

In the work titled ”Let’s Supercharge the Workflows: An Empirical Study of
GitHub Actions” [1], an in-depth analysis is conducted on the usage patterns
of GitHub Actions. Findings reveal that during the GitHub Actions
configuration, developers are required to set multiple components, each
presenting numerous alternatives. This complexity poses a challenge for less
experienced developers in configuring a suitable GitHub Action Workflow.
Furthermore, the study highlights the dependence of the GitHub Action
configuration process on developers’ personal experience, suggesting a need
for assistive tools to facilitate the configuration of GitHub Actions.

This paper [2] addresses the gap of domain specific language models by
focusing on Ansible-YAML, a widely employed markup language for IT Automation.
The proposed solution, Ansible Wisdom, is a transformer-based model, uniquely
trained on a dataset containing Ansible-YAML. To comprehensively assess its
performance, the paper introduces two novel performance metrics tailored for
YAML and Ansible, aimed at capturing the distinctive characteristics inherent
to this specialized domain.

##  3\. Method and Implementation

In the section, we details the core design (i.e., prompt engineering) in
Section 3.1 and implementation in Section 3.2 of our method DevOps-LLM-Bot.

###  3.1. Prompt Engineering

For generating a YAML file for a given repository, ideally, the more
information about the repository in the prompt, the better performance is
expected to be achieved. However, in practice, due to the token limit and also
the imperfection of the LLM, we integrate our expert knowledge as the feature
of the task into the prompt.

In this work, we customize the prompt with the following 2 parts as our first
attempt. Particularly, for designing _context_ , we consider 2 pieces of task-
specific information, which are _file structure_ and _default branch_ based on
our observation. For example, Figure 1 presents a real-world YAML
file111https://github.com/ajcr/rolling/blob/master/.github/workflows/python-
package.yml. It carries both information about file structure (i.e.
requirements.txt in the install step, ) and branch (i.e. master while defining
workflow triggers, ).

Figure 1. Example GitHub Actions Workflow

###  3.2. Tool

We develop a tool based on GitHub App222https://github.com/apps/devops-llm-
bot. Figure 2 describes the architecture of this app built on top of Probot
framework, which responds to the following events:

  * •

Issue Creation: The app springs into action when a new issue is created with a
title that commences with `@devops`. The context provided during this request
includes the repository file structure and the default branch of the
repository.

  * •

Pull Request Comment: The app is also activated when a comment is made on a
pull request, provided the comment starts with `@devops-llm-bot`. In this
scenario, the GPT is invoked with the context of the existing workflow and all
previous conversations that the user has had within that pull request.

Figure 2. Architecture

The app runs on Azure functions and uses the Octokit client to interact with
GitHub and get information about the repository, such as its structure and
contents. In our work, we develop an app that invoked the GPT 4
(_gpt-4-1106-preview_) chat completion API to generate workflow and raise a
pull request for the developers to review, improvise by conversing with the
bot and merge the workflow file.

###  3.3. Usage

Figure 3. New GitHub Issue

As shown in Figure 3, the bot can automatically generate a workflow for a
GitHub repository when a user creates a new issue with a title that begins
with _@devops_ and a body that specifies any custom requests. The bot replies
to the issue with a message that indicates that it is working on the task.
After the workflow is ready, the bot automatically commits it to a new branch
and transforms the issue into a pull request that merges the new branch with
the default branch of the repository.

Figure 4. GitHub Pull Request Comment

Figure 4 demonstrates the direct interaction between the user and the LLM via
the pull request, facilitating the necessary workflow updates. This
interaction is designed to empower DevOps engineers by enabling them to
generate the workflow without writing any code. It also allows them to verify
the performance of the generated workflow, as it will automatically initiate
upon the creation of the pull request. This ensures that the workflow operates
as anticipated.

##  4\. Evaluation

###  4.1. Research Questions and Methodology

Our experiment setting is designed to answer the following research questions.

  * •

How the performance DevOps-LLM-Bot differ based on different models?

  * •

How do the different components of the prompt contribute to the final
performance of DevOps-LLM-Bot?

  * •

How efficient is DevOps-LLM-Bot?

Figure 5. Framework for experimentation

Figure 5 presents the overview of our evaluation methodology. It demonstrates
how we leverage public GitHub repositories to create a high-quality dataset
for testing LLMs on DevOps tasks. In data collection (Section 4.2), we apply
filter conditions to select relevant repositories. In model evaluation
(Section 4.3), we generate build and test workflow with different settings in
terms of prompt design and LLMs. We compare the generated workflows with the
ground truth and evaluate them using multiple metrics in both automatic and
manual manners.

Automatic Evaluation.

We evaluate the pipeline based on the static similarity measurements between
ground truth and the generated workflow. Specifically, we consider not only
the exact match score, BLEU score, but also we introduce a customized metric
DevOps Aware score (Section 4.3). To mitigate the randomness of the
performance, we run each experiment three times for each setting and calculate
the average scores.

Firstly, the syntax of the workflow is checked which checks if appropriate
keys are used, the file follows workflow syntax and the structure is correct.
Syntax validation of GitHub workflows is performed using actionlint [3] which
is a static checker tool.

For the evaluation, the workflow is checked for syntax if its valid or not and
then the various metrics are calculated on the valid workflows.

Human Evaluation: In addition to automatic evaluation, we also conducted a
manual evaluation to better understand and evaluate the quality of the
generated workflow. Since we do not have a proven evaluation metric for yaml,
a manual evaluation would help invalidate our novel DevOps Aware Score and
also check how semantically our generated workflows are to ground truth.
Hence, we sampled 324 generated workflows, with this sample size, we can
maintain a confidence level of 95% and a 5% margin of error. The workflows
were scored based on a 5-point Likert scale.

###  4.2. Data Preparation

To the best of our knowledge, our work is the first work that focuses on
generating YAML file for a given code repository. Therefore, we construct the
first dataset for the task with two stages, data collection and cleaning
phase.

Data collection phase. We extract repository data from SEART-GHS tool
(https://seart-ghs.si.usi.ch/) using specific criteria based on number of
stars associated with each GitHub repository. The following filtering criteria
is applied:

  1. (1)

With at least 100 Stars. To control the quality of our data, we apply the
common indicator, the number of stars, to filter the low-quality repositories.

  2. (2)

Contain at least 1 .yml or .yaml file within _.github/_ _workflows_ folder. It
indicates the presence of GitHub Actions.

Data cleaning phase. By design, GitHub Actions can be implemented for various
purposes, such as build, test, deployment,
etc333https://github.com/features/actions. Therefore, distinguishing the
purpose of a specific GitHub Actions workflow file is challenging. To address
this, we adopted an exhaustive list of build and test command lines tailored
to specific programming languages.

To streamline our research, we focused on a wide range of programming
languages, they are Java, Python, JavaScript, TypeScript, Kotlin, C#, & C++.
We select 100 data points for each language to construct a manageable dataset.

###  4.3. Evaluation Metrics

In our work, we not only consider the widely-used metrics such as Exact Match
(EM), BLEU [4], and Syntax Correct, but also introduce a new and specialized
metric called _DevOps Aware Score_ to evaluate the generated workflow. The
reason is that those traditional metrics carry certain known limitations. EM
completely ignores the semantically equivalent yet different pairs between the
generated one and the ground truth. BLEU score is another popular metric but
it heavily relies on n-grams and may not capture the overall meaning of the
YAML file. Syntax Correctness is also an important metric but it only focuses
on the syntax-wise.

DevOps Aware is a customized metric defined specifically to compare the
semantic distance between two YAML files. The metric aims to leverage
knowledge of GitHub Action Workflow syntax to assess the effectiveness of code
sections that directly impact the output. It acknowledges that achieving the
same objective can be accomplished through various predefined actions. The
evaluation process involves examining the “jobs” section, which comprises
“steps” specifying actions and commands for execution. We extract the
build/test “jobs” content from both the ground truth and the generated
workflow, parsing through the ”steps” to compare the generated steps with
their corresponding actual steps. The other steps from the ground truth are
overlooked because these workflows may encompass more than just building and
testing.

At present, our assessment is primarily concerned with the Language Learning
Model’s (LLM) ability to generate build and test actions. The comparison
involves checking for keys such as “uses,” “runs,” or “name.” If both the
generated step and the actual step contain ”uses” or ”runs,” an Exact Match
score is calculated. In other cases, the BLEU score is computed. The final
score is the average of all the calculated scores.

Figure 6. DevOps Aware Calculation

###  4.4. Result

Automatic Evaluation: In this comprehensive evaluation of GPT 3.5 and GPT 4 in
generating GitHub Action workflows across multiple programming languages,
several key metrics were employed to assess the models’ performance. Notably,
GPT 4 exhibited a marked improvement over its predecessor, GPT 3.5, in key
areas such as DevOps awareness, BLEU score, and, syntax correctness. The
DevOps Aware Score, a critical metric reflecting the models’ understanding of
DevOps practices in workflow generation, witnessed a substantial increase in
GPT 4 across various languages, with notable improvements in C, C++, and
Python. This suggests that GPT 4 has a more refined understanding of DevOps
best practices, crucial for effective and seamless workflow automation.

Table 1. Comparison of GPT 3.5 and GPT 4 Models | DevOps | BLEU | Exact | Syntax Correct
---|---|---|---|---
| Aware Score | Score | Match | ( Valid / Total )
GPT 3.5 | 0.55 | 0.39 | 0.19 | 77.90 %
GPT 4 | 0.6 | 0.4 | 0.19 | 96.75 %
Table 2. Comparison of valid syntax percentage and DevOps Aware Score Languages | GPT 3.5 | GPT 4 | GPT 3.5 | GPT 4
---|---|---|---|---
| Syntax | Syntax | Score | Score
C# | 87.33 % | 99.33 % | 0.44 | 0.53
C++ | 72.00 % | 95.25 % | 0.3 | 0.37
Java | 66.33 % | 97.6 % | 0.55 | 0.6
Javascript | 88.00 % | 95.67 % | 0.7 | 0.74
Kotlin | 69.33 % | 97.32 % | 0.68 | 0.74
Python | 72.33 % | 95.00 % | 0.45 | 0.49
Typescript | 90.00 % | 96.98 % | 0.7 | 0.73

Manual Evaluation: We have checked for the correlation between the manually
evaluated workflows based on the 5-point likert scale and the DevOps Aware
Score. We choose Pearson correlation for its capacity to quantify linear
associations, enabling nuanced analysis of DevOps and manual score
relationships. The overall correlation of 0.61 across all languages provides
profound insights into the intricate dynamics between automated DevOps
assessments and manual evaluations which validates our novel evaluation metric
for DevOps workflows.

##  5\. How to use?

We’ve developed a GitHub App as part of our research, which you can access
[here]. This app can be installed in any GitHub repository and used to
generate GitHub build and test workflows for that repository.

You’re welcome to explore the bot with this [example PR], or you can try it
out in your own repository by following these steps:

  1. (1)

Install the GitHub app on your repository using the provided link.

  2. (2)

Create a new issue with a title beginning with @devops. The bot will use this
to help generate your workflow. Any custom requests can be included in the
issue description for the bot to consider while generating the workflow.

  3. (3)

You’ll see a comment on your issue from the bot, indicating that your request
is being processed.

  4. (4)

After about a minute, the issue will transform into a pull request and your
workflow file will be generated.

  5. (5)

If you’re satisfied with the workflow, you can merge the pull request.

  6. (6)

If you need modifications, add a comment in the pull request detailing the
changes required. Make sure the comment begins with ‘@devops-llm-bot’ to
trigger the bot to consider your changes and re-generate the workflow.

  7. (7)

Once the workflow is generated, the bot will reply on the PR and add a new
commit. Enjoy exploring the bot!

##  6\. Conclusion

In conclusion, the research outlined in this paper marks a significant
milestone in the fusion of AI-driven capabilities, specifically GPT-4, within
the realm of DevOps methodologies. The meticulous exploration and evaluation
of GPT-4’s capabilities showcased substantial advancements, particularly in
its comprehension of DevOps practices and in generating syntactically accurate
GitHub Action workflows.

The introduction of a dedicated GitHub App, empowered by Large Language
Models, emerges as a significant contribution, allowing developers to automate
workflow generation and seamlessly integrate it into their projects. This
innovation streamlines the software development lifecycle, enabling developers
to craft workflows efficiently and effectively.

Looking ahead, the future holds promising avenues for further enhancements.
Fine-tuning models, experimenting with different models and prompt
combinations, and extending the scope to include deployment steps within
GitHub workflows are potential areas for future work.

## References

  * [1] Tingting Chen, Yang Zhang, Shu Chen, Tao Wang, and Yiwen Wu.  Let’s supercharge the workflows: An empirical study of github actions.  In 2021 IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C), pages 01–10. IEEE, 2021.
  * [2] Saurabh Pujar, Luca Buratti, Xiaojie Guo, Nicolas Dupuis, Burn Lewis, Sahil Suneja, Atin Sood, Ganesh Nalawade, Matt Jones, Alessandro Morari, et al.  Automated code generation for information technology tasks in yaml through large language models.  arXiv preprint arXiv:2305.02783, 2023.
  * [3] Linda_pp (rhysd).  Actionlint - static checker for github actions workflow files. retrieved from https://github.com/rhysd/actionlint.
  * [4] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.


* Keil et al. (2003) Keil, M. T., Raffelt, G. G., & Janka, H.-T. 2003, ApJ, 590, 971
  * Kheswa et al. (2015) Kheswa, B. V., Wiedeking, M., Giacoppo, F., et al. 2015, Physics Letters B, 744, 268
  * Kikuchi et al. (2015) Kikuchi, Y., Hashimoto, M.-. aki ., Ono, M., et al. 2015, Progress of Theoretical and Experimental Physics, 2015, 063E01
  * Ko et al. (2020) Ko, H., Cheoun, M.-K., Ha, E., et al. 2020, ApJ, 891, L24
  * Ko et al. (2019) Ko, H., Cheoun, M.-K., Kusakabe, M., et al. 2019, Acta Physica Polonica B, 50, 385
  * Ko et al. (2020) Ko, H., Jang, D., Kusakabe, M., et al. 2020, ApJ, 894, 99. doi:10.3847/1538-4357/ab84e4
  * Kostelecký & Samuel (1994) Kostelecký, V. A., & Samuel, S. 1994, Phys. Rev. D, 49, 1740
  * Kuo & Pantaleone (1989) Kuo, T. K., & Pantaleone, J. 1989, Phys. Rev. D, 39, 1930
  * Kusakabe et al. (2019) Kusakabe, M., Cheoun, M.-K., Kim, K. S., et al. 2019, ApJ, 872, 164
  * Lahkar et al. (2017) Lahkar, N., Kalita, S., Duorah, H. L., et al. 2017, Journal of Astrophysics and Astronomy, 38, 8
  * Liu et al. (2010) Liu, M.-C., Nittler, L. R., Alexander, C. M. O., et al. 2010, ApJ, 719, L99
  * Lodders et al. (2009) Lodders, K., Palme, H., & Gail, H.-P. 2009, Landolt Börnstein, 4B, 712
  * Malatji et al. (2019) Malatji, K. L., Wiedeking, M., Goriely, S., et al. 2019, Physics Letters B, 791, 403
  * Mathews et al. (2011) Mathews, G. J., Pehlivan, Y., Kajino, T., et al. 2011, ApJ, 727, 10
  * Mathews et al. (2012) Mathews, G. J., Kajino, T., Aoki, W., et al. 2012, Phys. Rev. D, 85, 105023
  * Mikheev & Smirnov (1986) Mikheev, S. P., & Smirnov, A. I. 1986, Nuovo Cimento C Geophysics Space Physics C, 9, 17
  * Mirizzi et al. (2016) Mirizzi, A., Tamborra, I., Janka, H.-T., et al. 2016, Nuovo Cimento Rivista Serie, 39, 1
  * Münker et al. (2000) Münker, C., Weyer, S., Mezger, K., et al. 2000, Science, 289, 1538
  * Nakamura et al. (2010) Nakamura, K., Yoshida, T., Shigeyama, T., et al. 2010, ApJ, 718, L137
  * Nötzold & Raffelt (1988) Nötzold, D., & Raffelt, G. 1988, Nuclear Physics B, 307, 924
  * O’Connor et al. (2018) O’Connor, E., Bollig, R., Burrows, A., et al. 2018, Journal of Physics G Nuclear Physics, 45, 104001
  * Olive & Particle Data Group (2014) Olive, K. A., & Particle Data Group 2014, Chinese Physics C, 38, 090001
  * Olive & Vangioni (2019) Olive, K. A. & Vangioni, E. 2019, MNRAS, 490, 4307
  * Pantaleone (1992) Pantaleone, J. 1992, Physics Letters B, 287, 128
  * Pehlivan et al. (2011) Pehlivan, Y., Balantekin, A. B., Kajino, T., et al. 2011, Phys. Rev. D, 84, 065008
  * Pignatari et al. (2010) Pignatari, M., Gallino, R., Heil, M., et al. 2010, ApJ, 710, 1557
  * Prantzos (2012) Prantzos, N. 2012, A&A, 542, A67
  * Qian & Fuller (1995) Qian, Y.-Z., & Fuller, G. M. 1995, Phys. Rev. D, 51, 1479
  * Raffelt (2001) Raffelt, G. G. 2001, ApJ, 561, 890
  * Raffelt (2012) Raffelt, G. 2012, arXiv e-prints, arXiv:1201.1637
  * Raffelt et al. (2013) Raffelt, G., Sarikas, S., & Seixas, D. de S. 2013, Phys. Rev. Lett., 111, 091101
  * Rauscher et al. (2013) Rauscher, T., Dauphas, N., Dillmann, I., et al. 2013, Reports on Progress in Physics, 76, 066201
  * Samuel (1993) Samuel, S. 1993, Phys. Rev. D, 48, 1462
  * Sanloup et al. (2000) Sanloup, C., Blichert-Toft, J., Télouk, P., et al. 2000, Earth and Planetary Science Letters, 184, 75
  * Sasaki et al. (2017) Sasaki, H., Kajino, T., Takiwaki, T., et al. 2017, Phys. Rev. D, 96, 043013
  * Schaeffer et al. (1987) Schaeffer, R., Declais, Y., & Jullian, S. 1987, Nature, 330, 142
  * Shigeyama & Nomoto (1990) Shigeyama, T., & Nomoto, K. 1990, ApJ, 360, 242
  * Sieverding et al. (2018) Sieverding, A., Martínez-Pinedo, G., Huther, L., et al. 2018, ApJ, 865, 143
  * Sieverding et al. (2019) Sieverding, A., Langanke, K., Martínez-Pinedo, G., et al. 2019, ApJ, 876, 151
  * Sigl (2017) Sigl, G. 2017, Astroparticle Physics: Theory and Phenomenology, by Günter Sigl. Atlantis Studies in Astroparticle Physics and Cosmology, 2017, Atlantis Press
  * Sigl & Raffelt (1993) Sigl, G., & Raffelt, G. 1993, Nuclear Physics B, 406, 423
  * Silberberg & Tsao (1990) Silberberg, R. & Tsao, C. H. 1990, Phys. Rep., 191, 351
  * Smirnov (2005) Smirnov, A. Y. 2005, Physica Scripta Volume T, 121, 57
  * Suzuki & Kajino (2013) Suzuki, T., & Kajino, T. 2013, Journal of Physics G Nuclear Physics, 40, 083101
  * Tamborra et al. (2017) Tamborra, I., Hüdepohl, L., Raffelt, G. G., et al. 2017, ApJ, 839, 132
  * Volpe et al. (2013) Volpe, C., Väänänen, D., and Espinoza, C. 2013, Phys. Rev. D, 87, 113010
  * Woosley (1988) Woosley, S. E. 1988, ApJ, 330, 218
  * Woosley et al. (1990) Woosley, S. E., Hartmann, D. H., Hoffman, R. D., et al. 1990, ApJ, 356, 272
  * Wu et al. (2015) Wu, M.-R., Qian, Y.-Z., Martínez-Pinedo, G., et al. 2015, Phys. Rev. D, 91, 065016
  * Yoshida et al. (2005) Yoshida, T., Kajino, T., & Hartmann, D. H. 2005, Phys. Rev. Lett., 94, 231101
  * Yoshida et al. (2006) Yoshida, T., Kajino, T., Yokomakura, H., et al. 2006, Phys. Rev. Lett., 96, 091101
  * Yoshida et al. (2008) Yoshida, T., Suzuki, T., Chiba, S., et al. 2008, ApJ, 686, 448
  * Yoshida et al. (2004) Yoshida, T., Terasawa, M., Kajino, T., et al. 2004, ApJ, 600, 204
  * Ydrefors & Suhonen (2013) Ydrefors, E. & Suhonen, J. 2013, Phys. Rev. C, 87, 034314