# On Higgs inflation in non-minimally coupled models of gravity

N. Mohammedi
Institut Denis Poisson (CNRS - UMR 7013),
Université de Tours,
Faculté des Sciences et Techniques,
Parc de Grandmont, F-37200 Tours, France. e-mail: noureddine.mohammedi@univ-
tours.fr

###### Abstract

Models of inflation in which the Higgs field is non-minimally coupled to
gravity lead, after a recaling of the metric, to a complicated expression for
the potential of the inflaton field. Nevertheless, this potential produces the
desired features, both theoretical and experimental, after some approximations
are made. In this note, we also allow for a modification of the Higgs kinetic
term in such a way that the resulting potential for the inflaton field is,
without any approximation, of the simplest form after the rescaling of the
metric.

##  1 Introduction

The Standard Model of particles physics has proved to be a robust theory in
describing the observed interactions between particles at the level of
particle accelerators. However, this successful theory does not seem to answer
questions related to the cosmological evolution of the Universe. In
particular, the nature of dark matter or dark energy as well as the early
inflationary regime of the Universe remain a mystery. In this note we will be
mostly concerned with the problem of inflation. Introductory reviews of cosmic
inflation can be found in [1, 2, 3, 4, 5].

It is widely accepted now that inflation might be driven by a scalar field
moving slowly in a certain potential. The discovery of the Higgs field has
completed the Standard Model and opened a new window in cosmology. It is
therefore tempting to identify the Higgs field with the scalar field needed
for inflation.

The issue now is how to couple the Standard Model to gravity in order to
account for inflation through the Higgs field. Various models can be
constructed depending on what kind of coupling one chooses. Accounts of the
Higgs field in cosmology can be found in [6, 7, 8]. Here we will mention only
what is called the non-minimal Higgs inflation model as it is relevant to our
study. In this theory, the coupling of the Higgs field to gravity is given by
the action [9, 10, 11]

$\displaystyle S=\int
d^{4}x\sqrt{-{g}}\left[-\frac{1}{2}\left(M^{2}+{2\xi}\,H^{\dagger}H\right)R+{\cal
L}_{\text{SM}}\right]\,\,\,.$ (1.1)

The quantity ${\cal L}_{\text{SM}}$ is the full Standard Model Lagrangian
written in curved space-time and $H$ is the Higgs doublet. We will return to
this model of inflation in some details in section 3. It suffices to say for
the moment that under a certain approximation (see later), this theory gives a
viable explanation of inflation.

In order to get the standard term $\int d^{4}x\sqrt{-{g}}R$ for the
gravitational sector, the authors of refs.[9, 10, 11] carried out a conformal
rescaling of the metric field. This operation induced a complicated expression
for the potential of the Higgs field (or equivalently, gave a non-canonical
kinetic term for the Higgs field). Nevertheless, this complication disappears
under the above cited approximation [10, 11] .

The aim of this note is to provide a model of Higgs inflation where no
approximation is needed. We have simply modified the Higgs kinetic term in the
Standard Model non-minimally coupled to gravity. More specifically, we propose
the model described by the action

$\displaystyle S=\int
d^{4}x\sqrt{-{g}}\left[-\frac{1}{2}\left(M^{2}+{2\xi}\,H^{\dagger}H\right)R+F\left(H^{\dagger}H\right)\left(D_{\mu}H\right)^{\dagger}\left(D^{\mu}H\right)+{\cal
L}_{\text{SM}}^{\prime}\right]\,\,\,$ (1.2)

to account for inflation of the early Universe. Here ${\cal
L}_{\text{SM}}^{\prime}$ is the Standard Model Lagrangian without the gauge
covariant kinetic term and the function $F\left(H^{\dagger}H\right)$ is
determined in such a way that the resulting kinetic term of the inflaton
field, after a rescaling of the metric, is in the usual canonical form. This
theory could be seen as an improvement of the proposition of ref.[10]. It is
also crucial to mention that the two models coincide for a large Higgs field
(which is precisely the approximation made in ref.[10]).

The article is organised as follows: In the next section we review the
Starobinsky model of inflation [12] as there are similarities between this
model and non-minimal Higgs inflation [13]. Some details of the non-minimal
coupling of the Higgs field to gravity are given in section 3. Our
modification of the non-minimally coupled Higgs field is presented in section
4. The last section is dedicated to some conclusions.

##  2 The Starobinsky model of inflation : An inspiration

Various models dealing with the issue of the rapid expansion of the early
Univers have been inspired by the Starobinsky model of inflation [13]. This is
described by the action [12]

$\displaystyle S_{0}=\int
d^{4}x\sqrt{-{g}}\left[-\frac{M^{2}_{\text{p}}}{2}\,{R}+\frac{1}{12M^{2}_{\text{S}}}\,{R}^{2}\right]\,\,\,.$
(2.1)

As it is well-known, the quadratic term in the scalar curvature can be traded
for a scalar field [14]. One starts by writing the equivalent action

$\displaystyle S_{1}=\int
d^{4}x\sqrt{-{g}}\left[-\frac{M^{2}_{\text{p}}}{2}\left(1+\frac{\varphi}{3M^{2}_{\text{p}}M^{2}_{\text{S}}}\right){R}-\frac{\varphi^{2}}{12M^{2}_{\text{S}}}\right]\,\,\,,$
(2.2)

where $\varphi$ is an auxiliary field whose equation of motion is
$\varphi=-R$. Upon replacing this in the last action we recover the
Starobinsky model (2.1).

The action (2.2) can be cast in a more familiar form by going to the Einstein
frame through the field redefinition

$\displaystyle g_{\mu\nu}\longrightarrow e^{\omega
X(x)}\,{g}_{\mu\nu}\,\,\,\,\,\,\,\,\,\,\,\,{\text{with}}\,\,\,\,\,\,\,\,\,\,\,\,\left(1+\frac{\varphi}{3M^{2}_{\text{p}}M^{2}_{\text{S}}}\right)\,e^{\omega
X(x)}=1\,\,\,\,\,\,,\,\,\,\,\,\,\,\omega=-\,\sqrt{\frac{2}{3}}\,\frac{1}{M_{\text{p}}}\,\,\,\,.$
(2.3)

With this conformal rescaling of the metric, the action (2.2) yields (up to a
total derivative)

$\displaystyle S_{2}=\int
d^{4}x\sqrt{-{g}}\left[-\frac{M^{2}_{\text{p}}}{2}\,{R}+\frac{1}{2}\,{g}^{\mu\nu}\partial_{\mu}X\,\partial_{\nu}X-\frac{3}{4}\,M^{4}_{\text{p}}M^{2}_{\text{S}}\left(1-\,e^{\omega
X}\right)^{2}\right]\,\,\,.$ (2.4)

The two actions (2.1) and (2.4) could be thought of as dual to each other.

We see that the field $X(x)$ is moving in a potential which is almost flat for
very large values of $X(x)$. This is the most sought feature which realises
the slow-roll condition needed for inflationary models. In this case the
kinetic term of the field $X(x)$ is negligeable and the cosmological evolution
of the Universe is dominated by the constant term
$\frac{3}{4}\,M^{4}_{\text{p}}M^{2}_{\text{S}}$ (playing the role of a
cosmological constant). The Starobinsky model is compatible with cosmological
measurements for $M_{\text{S}}\approx 10^{-5}$.

It is essentially the theory in (2.4) which was reached by the non-minimally
coupled Higgs field of ref.[10]. It was also shown in [13] that other models
of inflation could be thought of, under certain assumptions, as descendants of
the Starobinsky model.

##  3 The Higgs field non-minimally coupled to gravity

Here and in the rest of this note, we will assume that the Higgs doublet is in
the unitary gauge

$H\left(x\right)=\frac{1}{\sqrt{2}}\left(\begin{array}[]{l}0\\\
h\left(x\right)\end{array}\right)\,\,\,.$ (3.1)

The Higgs field non-minimally coupled to gravity is described, in the Jordan
frame, by the action [9, 10, 11]

$\displaystyle S_{\text{J}}=\int
d^{4}x\sqrt{-{g}}\left[-\frac{1}{2}\left(M^{2}+{\xi}\,h^{2}\right)R+\frac{1}{2}\partial_{\mu}h\partial^{\mu}h-\frac{\lambda}{4}\left(h^{2}-v^{2}\right)^{2}\right]\,\,\,.$
(3.2)

We have written down only the terms relevant to our study. The constant $M$
and the reduced Planck mass ${M_{\text{p}}}=\frac{1}{\sqrt{8\pi
G}}=2.4335\times 10^{18}$ GeV are related by

${M_{\text{p}}}^{2}=M^{2}+\xi\,v^{2}$ (3.3)

with the Higgs field expectation value $v=246$ GeV. We are assuming that
$\hbar=c=1$.

This theory, however, is best studied in the Einstein frame by performing the
conformal transformation

$\displaystyle g_{\mu\nu}\longrightarrow
e^{\omega\varphi}\,{g}_{\mu\nu}\,\,\,\,\,\,\,\,\,\,\,\,{\text{with}}\,\,\,\,\,\,\,\,\,\,\,\,\omega=-\,\sqrt{\frac{2}{{3}}}\,\frac{1}{M_{\text{p}}}\,\,\,\,.$
(3.4)

The fields $\varphi$ and ${h}$ are related by

$\displaystyle\left(M^{2}+{\xi}{{h}^{2}}\right)e^{\omega\varphi}={M^{2}_{\text{p}}}\,\,\,\,.$
(3.5)

Therefore one can use either one of the two fields to explore the cosmological
properties of the theory. We find it more convenient to keep the field
$\varphi$ instead of the field $h$ as it makes the various approximations more
transparent.

In terms of the field $\varphi$ and up to a total derivative, the action in
the Einstein frame is given by

$\displaystyle S_{\text{E}}$ $\displaystyle=$ $\displaystyle\int
d^{4}x\sqrt{-{g}}\bigg{\\{}-\frac{M^{2}_{\text{p}}}{2}\,{R}+\frac{3}{4}M^{2}_{\text{p}}\omega^{2}\left[{1}+\frac{1}{6\xi}\frac{1}{\left(1-\frac{M^{2}}{M^{2}_{\text{p}}}\,e^{\omega\varphi}\right)}\right]{g}^{\mu\nu}\partial_{\mu}\varphi\partial_{\nu}\varphi$
(3.6) $\displaystyle-$
$\displaystyle\frac{\lambda}{4}\left(\frac{M^{2}_{\text{p}}}{\xi}\right)^{2}\bigg{(}1-e^{\omega\varphi}\bigg{)}^{2}\bigg{\\}}\,\,\,.$

We have used ${M_{\text{p}}}^{2}=M^{2}+\xi\,v^{2}$ in the last term. We notice
that the potentiel terms is readily in the much sought exponentiel form (as in
the Starobinsky model). This is precisely what one needs for inflation.
However, the kinetic term for the scalar field $\varphi$ is not standard.
Nevertheless, in the case when

$\frac{M^{2}}{M^{2}_{\text{p}}}\,e^{\omega\varphi}<<1\,\,\,\,\,\,\,\,\,,\,\,\,\,\,\,\,\,\,\xi>>1$
(3.7)

the second term in the kinetic energy can be neglected and one arrives at the
action111The approximation
$\frac{M^{2}}{M^{2}_{\text{p}}}\,e^{\omega\varphi}<<1$ is the slow-roll
condition and corresponds to $h^{2}>>\frac{M^{2}}{\xi}$ as can be seen from
(3.5).

$\displaystyle S_{\text{E}}$ $\displaystyle\simeq$ $\displaystyle\int
d^{4}x\sqrt{-{g}}\bigg{\\{}-\frac{M^{2}_{\text{p}}}{2}\,{R}+\frac{1}{2}\,{g}^{\mu\nu}\partial_{\mu}\varphi\partial_{\nu}\varphi-\frac{\lambda}{4}\left(\frac{M^{2}_{\text{p}}}{\xi}\right)^{2}\bigg{(}1-e^{\omega\varphi}\bigg{)}^{2}\bigg{\\}}\,\,\,.$
(3.8)

This theory looks then very much like the Starobinsky model (2.4) with the
identification $3M^{2}_{\text{S}}=\frac{\lambda}{\xi^{2}}$. The observed
cosmological data requires $\xi=1.8\times 10^{4}$ for a Higgs self-
coupling222The constant $\lambda$ is related to the Higgs mass $m_{\text{H}}$
and the Higgs expectation value $v$ by the relation
$\lambda=\frac{m^{2}_{\text{H}}}{2v^{2}}\simeq\frac{(126)^{2}}{2\times(246)^{2}}\simeq
0.13$. $\lambda=15\times 10^{-2}$ (see [10, 11]). These values are also
compatible with $M_{\text{S}}\approx 10^{-5}$.

The authors of refs.[9, 10, 11] have adopted another route in studying the
model in (3.2). There, a standard kinetic term was obtained by introducing a
new field $\chi$ through

$\displaystyle\left(\frac{d\chi}{d\varphi}\right)^{2}=\frac{3}{2}M^{2}_{\text{p}}\omega^{2}\left[{1}+\frac{1}{6\xi}\frac{1}{\left(1-\frac{M^{2}}{M^{2}_{\text{p}}}\,e^{\omega\varphi}\right)}\right]\,\,\,.$
(3.9)

The action (3.6) becomes then

$\displaystyle S_{\text{E}}=\int
d^{4}x\sqrt{-{g}}\left\\{-\frac{M^{2}_{\text{p}}}{2}\,{R}+\frac{1}{2}{g}^{\mu\nu}\partial_{\mu}\chi\partial_{\nu}\chi-U\left(\chi\right)\right\\}\,\,\,.$
(3.10)

The potential term $U\left(\chi\right)$ for the new field $\chi(x)$ is
extracted from

$U\left(\chi\right)=\frac{\lambda}{4}\left(\frac{M^{2}_{\text{p}}}{\xi}\right)^{2}\bigg{(}1-e^{\omega\varphi}\bigg{)}^{2}$
(3.11)

by first integrating (3.9) and then expressing $\varphi$ in terms of $\chi$.
This results in a quite involved expression for $U\left(\chi\right)$. However,
when $\frac{M^{2}}{M^{2}_{\text{p}}}\,e^{\omega\varphi}<<1$ and $\xi<<1$ one
sees from (3.9) that
${\omega\varphi}\simeq-\sqrt{\frac{2}{3}}\,\frac{1}{M_{\text{p}}}\,\chi$. In
this case the actions (3.10) and (3.8) are the same after the exchange
$\varphi\leftrightarrow\chi$.

##  4 A scalar-tensor gravity with the Higgs field

In this section, we propose another modification of the coupling of the
Standard Model to gravity. The Higgs field is still non-minimally coupled to
gravity. However, the Higgs kinetic term is in a non-standard form. In this
way, we obtain (in the Einstein frame) a scalar field with a canonical kinetic
term moving in a potential having the desired exponential behaviour. The
important feature we should emphasise here is that no approximation is needed.

We start from the action

$\displaystyle S_{\text{J}}=\int
d^{4}x\sqrt{-{g}}\left[-\frac{1}{2}\left(M^{2}+{\xi}\,h^{2}\right)R+\frac{1}{2}\,\frac{\xi\,h^{2}}{\left(M^{2}+{\xi}\,h^{2}\right)}\,\partial_{\mu}h\partial^{\mu}h-\frac{\lambda}{4}\left(h^{2}-v^{2}\right)^{2}\right]\,\,\,.$
(4.1)

This means that the gauged kinetic term for the Higgs fields is modified as

$\displaystyle\frac{\xi\,\left(H^{\dagger}H\right)}{\left[M^{2}+{2\xi}\,\left(H^{\dagger}H\right)\right]}\left(D_{\mu}H\right)^{\dagger}\left(D^{\mu}H\right)\,\,\,\,.$
(4.2)

Here $D_{\mu}$ is the usual $SU\left(2\right)\times U\left(1\right)$ gauge
covariant derivative. It is important to notice that in the limit
$h^{2}>>\frac{M^{2}}{\xi}$, the two models (4.1) and (3.2) coincide.

We reach the Einstein frame by rescaling the metric as

$\displaystyle g_{\mu\nu}\longrightarrow
e^{\omega\varphi}\,{g}_{\mu\nu}\,\,\,\,\,\,\,\,\,\,\,\,\text{with}\,\,\,\,\,\,\,\,\,\,\,\,\left(M^{2}+{\xi}{{h}^{2}}\right)e^{\omega\varphi}$
$\displaystyle=$ $\displaystyle{M^{2}_{\text{p}}}\,\,\,\,.$ (4.3)

Keeping as our variable the scalar field $\varphi$, we arrive at the action
(up to a total derivative)

$\displaystyle S_{\text{E}}$ $\displaystyle=$ $\displaystyle\int
d^{4}x\sqrt{-{g}}\bigg{\\{}-\frac{M^{2}_{\text{p}}}{2}\,{R}+\frac{1}{2}{g}^{\mu\nu}\partial_{\mu}\varphi\partial_{\nu}\varphi-\frac{\lambda}{4}\left(\frac{M^{2}_{\text{p}}}{\xi}\right)^{2}\bigg{(}1-e^{\omega\varphi}\bigg{)}^{2}\bigg{\\}}\,\,\,.$
(4.4)

The constant $\omega$ is fixed by the relation

$\displaystyle\omega=-\,\frac{2}{M_{\text{p}}}\sqrt{\frac{\xi}{\left(1+6\xi\right)}}\simeq-\,\sqrt{\frac{2}{3}}\,\frac{1}{M_{\text{p}}}\,\,\,(\text{for}\,\,\,\,\,\,\xi>>1)\,\,\,.$
(4.5)

Again, the relation ${M_{\text{p}}}^{2}=M^{2}+\xi\,v^{2}$ has been used. The
potential of the scalar field $\varphi$ varies slowly (the slow-roll
condition) when $e^{\omega\varphi}<<1$. That is, inflation takes place for a
large Higgs field $h$ as deduced from
$\left(M^{2}+{\xi}{{h}^{2}}\right)e^{\omega\varphi}={M^{2}_{\text{p}}}$.

The theory defined by the action (4.4) has been obtained without any
approximation. On the contrary, for the inflationary model defined by (3.8)
the approximations $\frac{M^{2}}{M^{2}_{\text{p}}}\,e^{\omega\varphi}<<1$ and
$\xi>>1$ are assumed [10, 11].

As usual, the first and second slow-roll parameters are defined by [1, 2, 3,
4, 5]

$\displaystyle\epsilon=\frac{M_{\text{p}}^{2}}{2}\,\left(\frac{V^{\prime}}{V}\right)^{2}\,\,\,\,\,\,\,\,\,\,\,,\,\,\,\,\,\,\,\,\,\,\,\eta={M_{\text{p}}^{2}}\,\frac{V^{\prime\prime}}{V}\,\,\,\,,$
(4.6)

where a prime stands for the derivative with respect to $\varphi$ of the
potential

$V\left(\varphi\right)=\frac{\lambda}{4}\left(\frac{M^{2}_{\text{p}}}{\xi}\right)^{2}\bigg{(}1-e^{\omega\varphi}\bigg{)}^{2}\,\,\,\,.$
(4.7)

The inflationary observables are given by

$A_{s}=\frac{1}{24\pi^{2}M_{\text{p}}^{4}}\,\frac{V}{\epsilon}\,\,\,\,\,,\,\,\,\,\,\,\,n_{s}=1+2\eta-6\epsilon\,\,\,\,\,,\,\,\,\,\,\,\,r=16\epsilon\,\,\,\,.$
(4.8)

Explicitly, we have

$\displaystyle A_{s}$ $\displaystyle=$
$\displaystyle\frac{1}{192\pi^{2}M_{\text{p}}^{2}\omega^{2}}\,\frac{\lambda}{\xi^{2}}\,\frac{\left(1-e^{\omega\varphi}\right)^{4}}{e^{2\omega\varphi}}\,\,\,,$
$\displaystyle n_{s}$ $\displaystyle=$ $\displaystyle
1-{4M_{\text{p}}^{2}\omega^{2}}\,\frac{e^{\omega\varphi}\left(1+e^{\omega\varphi}\right)}{\left(1-e^{\omega\varphi}\right)^{2}}\,\,\,,$
$\displaystyle r$ $\displaystyle=$
$\displaystyle{32M_{\text{p}}^{2}\omega^{2}}\,\frac{e^{2\omega\varphi}}{\left(1-e^{\omega\varphi}\right)^{2}}\,\,\,.$
(4.9)

The quantities $A_{s}$, $r$ and $n_{s}$ are evaluated at $\varphi=\varphi_{*}$
defined through

$\displaystyle
N_{*}=\frac{1}{M_{\text{p}}}\,\int_{\varphi_{\text{end}}}^{\varphi_{*}}\frac{d\varphi}{\sqrt{2\epsilon}}=\frac{1}{2M_{\text{p}}^{2}\omega^{2}}\bigg{[}\left(e^{-\omega\varphi_{*}}+\omega\varphi_{*}\right)-\left(e^{-\omega\varphi_{\text{end}}}+\omega\varphi_{\text{end}}\right)\bigg{]}\,\,\,.$
(4.10)

Here $N_{*}$ is the number of e-folds and ${\varphi_{\text{end}}}$ is the
value of the field at which inflation ends. This is determined by the
condition $\epsilon\left({\varphi_{\text{end}}}\right)=1$ and is found to be
given by

${\varphi_{\text{end}}}=-\frac{1}{\omega}\ln\left(1-\sqrt{2}{M_{\text{p}}}\omega\right)\,\,\,.$
(4.11)

If one uses the approximation
${M_{\text{p}}}\omega\simeq-\,\sqrt{\frac{2}{3}}$, as given in (4.5), then
${\varphi_{\text{end}}}\simeq 0.9402{M_{\text{p}}}$.

The computation of ${\varphi_{*}}$ follows from (4.10) and we found

$\displaystyle{\omega\varphi_{*}}$ $\displaystyle=$ $\displaystyle
W\left(-e^{-c}\right)+c\,\,\,,$ $\displaystyle c$ $\displaystyle\equiv$
$\displaystyle{2M_{\text{p}}^{2}\omega^{2}}\,N_{*}+\left(e^{-\omega\varphi_{\text{end}}}+\omega\varphi_{\text{end}}\right)\simeq\frac{4}{3}N_{*}+1.3870\,\,\,.$
(4.12)

Here $W$ is the Lambert functions satisfying $W(x)e^{W(x)}=x$. The constant
$c\simeq 81.3870$ for $N_{*}=60$.

Since in our case the constant $c$ is real and we have $-e^{-1}<-e^{-c}<0$
then $W_{-1}\left(-e^{-c}\right)$ could be either the principal branch $W_{0}$
or the branch $W_{-1}$. Using the defining relation of $W$ one gets

$\displaystyle e^{\omega\varphi_{*}}$ $\displaystyle=$
$\displaystyle-\frac{1}{W\left(-e^{-c}\right)}\,\,\,.$ (4.13)

This is what one needs for the determination of the parameters $\epsilon$,
$\eta$ and $A_{s}$ given in (4.9). See also [7] for a semilar treatment.

As $-e^{-c}$ is of the order of $-10^{-36}$, the two branches of the Lambert
function $W\left(-e^{-c}\right)$ have, around zero, the expansions [15, 16,
17]

$\displaystyle W_{0}\left(-e^{-c}\right)$ $\displaystyle=$
$\displaystyle-e^{-c}-\left(-e^{-c}\right)^{2}+\dots\,\,\,\,,$ $\displaystyle
W_{-1}\left(-e^{-c}\right)$ $\displaystyle=$
$\displaystyle{-c}-\ln\left(c\right)+\dots\,\,\,\,.$ (4.14)

Choosing the branch $W_{0}$ leads to an extremely big value for
$e^{\omega\varphi_{*}}$. Therefore, we take
$e^{\omega\varphi_{*}}=-1/W_{-1}\left(-e^{-c}\right)$ and to first order we
have $e^{\omega\varphi_{*}}\simeq 1/c$.

Substituting $e^{\omega\varphi}$ by approximately $1/c$ in (4.9) and assuming
that $c>>1$, we find that

$\displaystyle A_{s}$ $\displaystyle\simeq$
$\displaystyle\frac{1}{192\pi^{2}M_{\text{p}}^{2}\omega^{2}}\,\frac{\lambda}{\xi^{2}}\,c^{2}\simeq\frac{1}{72\pi^{2}}\,\frac{\lambda}{\xi^{2}}\,N_{*}^{2}\,\,\,\,.$
(4.15)

In the last expression we have taken $c\simeq\frac{4}{3}N_{*}$. The
experimental value for $A_{s}$ is around [18]

$\displaystyle A_{s}\simeq 10^{-10}\,e^{3.094}\,\,\,\,.$ (4.16)

This fixes the value of the parameter $\xi$ as

$\xi\simeq 600\,c\,\sqrt{\lambda}\simeq 800N_{*}\sqrt{\lambda}\,\,\,.$ (4.17)

Taking $N_{*}\simeq 60$ [18] and assuming that the Higgs self-coupling is
$\lambda=0.15$, we get the value

$\displaystyle\xi\simeq 1.86\times 10^{4}\,\,\,.$ (4.18)

In the same way and for $N_{*}=60$, the quantities $n_{s}$ and $r$ are
approximately given by

$\displaystyle n_{s}$ $\displaystyle\simeq$ $\displaystyle
1-\frac{2}{N_{*}}\simeq 0.9667\,\,\,\,,$ $\displaystyle r$
$\displaystyle\simeq$ $\displaystyle\frac{12}{N_{*}^{2}}\simeq
0.0033\,\,\,\,.$ (4.19)

These nicely agree with the measured values [18].

Before leaving this section, let us explore how does the field $\varphi$
couple to the other fields of the Standard Model. In the unitary gauge and
according to (4.2), a generic coupling between the Higgs field and any gauge
field is of the form333When the Standard Model is coupled to gravity, all the
fermionic kinetic terms as well as the gauge kinetic terms are invariant under
the local rescalings
$g_{\mu\nu}\,\longrightarrow\,e^{\omega\varphi}\,{g}_{\mu\nu}$,
$A_{\mu}\,\longrightarrow\,A_{\mu}$,
$\psi_{L,R}\,\longrightarrow\,e^{-\frac{3}{4}\omega\varphi}\,\psi_{L,R}$. The
spacetime dependent Dirac $\gamma^{\mu}$ matrices are also rescaled as
$\gamma^{\mu}\,\longrightarrow\,e^{-\frac{1}{2}\omega\varphi}\,\gamma^{\mu}$.

$\displaystyle\sqrt{-g}\,\frac{\xi\,h^{2}}{\left(M^{2}+{\xi}\,h^{2}\right)}\,h^{2}A_{\mu}A_{\nu}g^{\mu\nu}\,\,\,\,.$
(4.20)

Under the rescalings
$g_{\mu\nu}\,\longrightarrow\,e^{\omega\varphi}\,{g}_{\mu\nu}$,
$A_{\mu}\,\longrightarrow\,A_{\mu}$ with
$\left(M^{2}+{\xi}{{h}^{2}}\right)e^{\omega\varphi}={M^{2}_{\text{p}}}$, this
coupling term becomes

$\displaystyle\sqrt{-g}\,\frac{\xi\,h^{2}}{\left(M^{2}+{\xi}\,h^{2}\right)}\,h^{2}A_{\mu}A_{\nu}g^{\mu\nu}\,\longrightarrow\,\frac{M^{2}_{\text{p}}}{\xi}\left(1-\frac{M^{2}}{M^{2}_{\text{p}}}\,e^{\omega\varphi}\right)^{2}\sqrt{-{g}}\,A_{\mu}A_{\nu}{g}^{\mu\nu}\,\,\,\,.$
(4.21)

We see that in the inflationary phase when
$\frac{M^{2}}{M^{2}_{\text{p}}}\,e^{\omega\varphi}<<1$ a decoupling between
the fields $\varphi$ and $A_{\mu}$ takes place.

The Yukawa terms have not been modified in this model. A generic mass
generating Yukawa interaction is still given by

$\displaystyle\sqrt{-g}\,h\left(\bar{\psi}_{L}\,\chi_{R}+\bar{\chi}_{R}\psi_{L}\right)\,\,\,\,,$
(4.22)

where $\psi_{L}$ is a left-handed fermion and $\chi_{R}$ is a right-handed
fermion. Under the rescalings
$g_{\mu\nu}\,\longrightarrow\,e^{\omega\varphi}\,{g}_{\mu\nu}$,
$(\psi_{L},\chi_{R})\,\longrightarrow\,e^{-\frac{3}{4}\omega\varphi}(\psi_{L},\chi_{R})$
together with
$\left(M^{2}+{\xi}{{h}^{2}}\right)e^{\omega\varphi}={M^{2}_{\text{p}}}$, this
fermionic coupling term transforms into

$\displaystyle\sqrt{-g}\,h\left(\bar{\psi}_{L}\,\chi_{R}+\bar{\chi}_{R}\psi_{L}\right)\,\longrightarrow\,\frac{M_{\text{p}}}{\sqrt{\xi}}\left(1-\frac{M^{2}}{M^{2}_{\text{p}}}\,e^{\omega\varphi}\right)^{\frac{1}{2}}\sqrt{-\hat{g}}\left(\bar{\psi}_{L}\,\chi_{R}+\bar{\chi}_{R}\psi_{L}\right)\,\,\,\,.$
(4.23)

The interaction between the field $\varphi$ and the fermions is highly
suppressed during inflation.

##  5 Conclusions

There is a freedom when it comes to coupling the Standard Model of particle
physics to gravity. Since the inclusion of gravity leads to non-
renormalisability of the theory, the Standard Model coupled to gravity might
therefore be viewed as an effective theory. In this note we have exploited
this freedom and proposed a viable effective theory for describing the
inflationary regime of the early Universe. We have modified the model,
advocated in ref.[10], where the Higgs field is non-minimaly coupled to
gravity. The guiding principle to this modification is that it results in $i)$
a canonical kinetic term for the inflaton field and $ii)$ a simple slow-roll
potential for this field. We have indeed succeeded in meeting these two
requirements without any approximation. This is achieved by modifying the
Higgs kinetic term. Our model is therefore a completion of the effective
theory proposed in [10]. It totally agrees with its cosmological findings.

Our study could be extended to the Higgs-dilaton models of gravity [19, 20,
21, 22]. For this, we consider the action

$\displaystyle S_{\text{J}}$ $\displaystyle=$ $\displaystyle\int
d^{4}x\sqrt{-{g}}\left[-\frac{1}{2}\left(\tau\,\sigma^{2}+{\xi}\,h^{2}\right)R+\frac{A}{2}\,\partial_{\mu}\sigma\partial^{\mu}\sigma+\frac{B}{2}\,\partial_{\mu}h\partial^{\mu}h+{C}\,\,\partial_{\mu}\sigma\partial^{\mu}h\right.$
(5.1) $\displaystyle-$
$\displaystyle\left.\frac{\lambda}{4}\left(h^{2}-\alpha\sigma^{2}\right)^{2}-\beta\sigma^{4}\right]\,\,\,.$

The quantities $A$, $B$ and $C$ are functions of the two scalar fields
$\sigma$ and $h$. The Higgs-dilaton model correspond to $A=B=1$ and $C=0$ [19,
20, 21, 22].

As usual, we rescal the metric as

$\displaystyle g_{\mu\nu}\longrightarrow
e^{\omega\varphi}\,{g}_{\mu\nu}\,\,\,\,\,\,\,\,\,\,\,\,\text{with}\,\,\,\,\,\,\,\,\,\,\,\,\left(\tau\,\sigma^{2}+{\xi}{{h}^{2}}\right)e^{\omega\varphi}$
$\displaystyle=$ $\displaystyle{M^{2}_{\text{p}}}\,\,\,\,.$ (5.2)

The aim now is to find the functions $A$, $B$ and $C$ such that two of the
scalar fields have canonical kinetic terms and their corresponding potential
is simple. As the three fields $\sigma$, $h$ and $\varphi$ are related by the
last relation, we choose to keep the two fields $\sigma$ and $\varphi$. The
functions $A$, $B$ and $C$ that fulfill these requirements are

$\displaystyle A$ $\displaystyle=$
$\displaystyle\frac{\tau\,\sigma^{2}+{\xi}\,h^{2}}{{M^{2}_{\text{p}}}}+\frac{4\tau^{2}\left(1-\frac{2}{3}\omega^{2}{M^{2}_{\text{p}}}\right)}{\omega^{2}{M^{2}_{\text{p}}}}\frac{\sigma^{2}}{\tau\,\sigma^{2}+{\xi}\,h^{2}}\,\,\,\,,$
$\displaystyle B$ $\displaystyle=$
$\displaystyle\frac{4\xi^{2}\left(1-\frac{2}{3}\omega^{2}{M^{2}_{\text{p}}}\right)}{\omega^{2}{M^{2}_{\text{p}}}}\frac{h^{2}}{\tau\,\sigma^{2}+{\xi}\,h^{2}}\,\,\,\,,$
$\displaystyle C$ $\displaystyle=$
$\displaystyle\frac{4\tau\,\xi\left(1-\frac{2}{3}\omega^{2}{M^{2}_{\text{p}}}\right)}{\omega^{2}{M^{2}_{\text{p}}}}\frac{\sigma\,h}{\tau\,\sigma^{2}+{\xi}\,h^{2}}\,\,\,\,.$
(5.3)

With these expression for $A$, $B$ and $C$, the above rescaling of the metric
yields the action

$\displaystyle S_{\text{E}}$ $\displaystyle=$ $\displaystyle\int
d^{4}x\sqrt{-{g}}\left[-\frac{{M^{2}_{\text{p}}}}{2}\,R+\frac{1}{2}\,\partial_{\mu}\sigma\partial^{\mu}\sigma+\frac{1}{2}\,\partial_{\mu}\varphi\partial^{\mu}\varphi\right.$
(5.4) $\displaystyle-$
$\displaystyle\left.\frac{\lambda}{4}\left(\frac{{M^{2}_{\text{p}}}}{\xi}\right)^{2}\left[1-\frac{\xi}{{M^{2}_{\text{p}}}}\left(\alpha+\frac{\tau}{\xi}\right)\sigma^{2}\,e^{\omega\varphi}\right]^{2}-\beta\,\left(\sigma^{2}\,e^{\omega\varphi}\right)^{2}\right]\,\,\,.$

Although the first term in the expression of $A$ breaks the global scale
invariance, the resulting action in the Einstein frame is simple. It is
appealing to investigate the cosmological consequences of this theory along
the lines of ref.[23]. We will report on this project elsewhere. It is also
worth treating the case $A=B=1$ and $C={\text{constant}}$, which preserves the
global scale invariance.

## References

  * [1] Andrew R. Liddle, An introduction to cosmological inflation, arXiv:astro-ph/9901124.
  * [2] Shinji Tsujikawa, Introductory review of cosmic inflation, (2003), arXiv:hep-ph/0304257.
  * [3] Leonardo Senatore, Lectures on Inflation, arXiv:1609.00716 [hep-th].
  * [4] Muhammad Zahid Mughal, Iftikhar Ahmad and Juan Luis García Guirao, Relativistic Cosmology with an Introduction to Inflation, Universe 2021, 7, 276. https://doi.org/10.3390/universe7080276.
  * [5] Julien Lesgourgues, Inflationary cosmology, (2006),
https://lesgourg.github.io/courses.html.

  * [6] Christian F. Steinwachs, Higgs field in cosmology, arXiv:1909.10528 [hep-ph].
  * [7] Javier Rubio, Higgs inflation, Front. Astron. Space Sci. 5:50 (2019), arXiv:1807.02376 [hep-ph].
  * [8] Bart Horn, The Higgs Field and Early Universe Cosmology: A (Brief) Review, Physics 2020, 2(3), 503-520, arXiv:2007.10377 [hep-ph].
  * [9] David Kaiser, Primordial Spectral Indices from Generalized Einstein Theories, Phys. Rev. D 52 (1995) 4295-4306, arXiv:astro-ph/9408044.
  * [10] Fedor L. Bezrukov and Mikhail Shaposhnikov, The Standard Model Higgs boson as the inflaton, Phys. Lett. B 659 (2008) 703, arXiv:0710.3755 [hep-th].
  * [11] Fedor Bezrukov, The Higgs field as an inflaton, Class. Quantum Grav. 30 (2013) 214001, arXiv:1307.0708 [hep-ph].
  * [12] Alexei A. Starobinsky, A New Type of Isotropic Cosmological Models Without Singularity, Phys. Lett. B 91 (1980) 99.
  * [13] Alex Kehagias, Azadeh Moradinezhad Dizgaha and Antonio Riottoa, Comments on the Starobinsky Model of Inflation and its Descendants, Phys. Rev. D 89 (2014) 043527, arXiv:1312.1155 [hep-th].
  * [14] Brian Whitt, Fourth Order Gravity as General Relativity Plus Matter, Phys. Lett. B 145 (1984) 176-178.
  * [15] Table of Integrals, Series, and Products, I. S. Gradshteyn and I. M. Ryzhik, (Alan Jeffrey, Editor), Academic Press, Fifth Edition (1994).
  * [16] Handbook of Mathematical Functions, Edited by M. Abramowitz and I. A. Stegun, Dover Publications, New York, Ninth Printing (1970).
  * [17] NIST Handbook of Mathematical Functions, Edited by F. W. J. Olver, D. W. Lozier, R. F. Boisvert and C. W. Clark, Cambridge University Press, New York, First Published (2010).
  * [18] Planck Collaboration: Y. Akrami et al., Planck 2018 results. X. Constraints on inflation, Astronomy $\&$ Astrophysics 641, A10 (2020), arXiv:1807.06211 [astro-ph.CO].
  * [19] Mikhail Shaposhnikov, Daniel Zenhäusern, Scale invariance, unimodular gravity and dark energy, Phys. Lett. B 671  (2009) 187-192, arXiv:0809.3395 [hep-th].
  * [20] Juan García-Bellido, Javier Rubio, Mikhail Shaposhnikov and Daniel Zenhäusern, Higgs-Dilaton Cosmology: From the Early to the Late Universe, Phys. Rev. D 84 (2011) 123504, arXiv:1107.2163 [hep-ph].
  * [21] Diego Blas, Mikhail Shaposhnikov and Daniel Zenhäusern, Scale-invariant alternatives to general relativity, Phys. Rev. D 84 (2011) 044001, arXiv:1104.1392 [hep-th].
  * [22] Fedor Bezrukov, Georgios K. Karananas, Javier Rubio and Mikhail Shaposhnikov, Higgs-Dilaton Cosmology: an effective field theory approach, Phys. Rev. D 87 (2013) 096001, arXiv:1212.4148 [hep-ph].
  * [23] Santiago Casas, Martin Pauly and Javier Rubio, Higgs-Dilaton Cosmology: An inflation - dark energy connection and forecasts for future galaxy surveys, Phys. Rev. D 97 (2018) 043520, arXiv:1712.04956 [astro-ph.CO].


# HumanEval on LLMs Revisted in Late 2023

1st Daniel Li  Vanderbilt University
Nashville, USA
<EMAIL_ADDRESS>2nd Lincoln Murr  Vanderbilt University
Nashville, US
<EMAIL_ADDRESS>
###### Abstract

In 2023, we are using the latest models of GPT-4 to advance program synthesis.
The large language models have significantly improved the state-of-the-art for
this purpose. To make these advancements more accessible, we have created a
repository that connects these models to Huamn Eval. This dataset was initally
developed to be used with a language model called CODEGEN on natural and
programming language data. The utility of these trained models is showcased by
demonstrating their competitive performance in zero-shot Python code
generation on HumanEval tasks compared to previous state-of-the-art solutions.
Additionally, this gives way to developing more multi-step paradigm synthesis.
This benchmark features 160 diverse problem sets factorized into multistep
prompts that our analysis shows significantly improves program synthesis over
single-turn inputs.

All code is open source at https://github.com/daniel442li/gpt-human-eval.

##  I Introduction

###  I-A Background on LLMs

A promising prospect lies in merging artificial intelligence techniques with
software engineering processes [10]. The advent of large language models such
as GPT [11], with their capabilities to generate human-like text, including
software code, is a notable advancement in this context . Over the past
several years, significant progress has been made in the development of large
language models, including GPT-3, Codex, ChatGPT, and impressive GPT-4 [7].
These models exhibit increasingly human-like capabilities, such as powerful
conversation, in-context learning, and code generation across a wide range of
open-domain tasks . However, in order to fully leverage the potential of large
language models for code generation and ensure accurate results, it is crucial
to assess their ability to generate code accurately. One method commonly used
for this assessment is the pass@1 and pass@10 evaluation approach. The 1@k and
10@k evaluation approach involves evaluating the quality of code generated by
the large language models by quantitatively measuring the accuracy of the
generated code against a set of known correct solutions, with 1@k passing
representing the likelihood of passing the test with one generation. Likewise
10@k represents the likelihood of passing the test with ten generations. By
utilizing the 1@k and 10@k evaluation approach, we can effectively measure the
ability of large language models to generate code accurately.

###  I-B Prompt Engineering

Prompt engineering is a crucial aspect in harnessing the full potential of
large language models for code generation. It involves crafting precise and
informative prompts or instructions that guide the model to generate code
snippets that accurately fulfill the given function or task. There have been
various techniques developed and studied for prompt engineering, including
chain of thought prompting, few-shot learning, and latent prompting. The
current challenge is to find the most effective and efficient prompt
engineering methods that can produce high-quality code generation results. The
highest performing prompt engineering strategy on Human Eval as of now for
pass@1 is called Language Agent Tree Search [15]. Below are some better known
prompt engineering techniques and their reasoning.

  1. 1.

Chain-of-thought (CoT) Prompting [13] This method was developed for complex
scenarios where a direct mapping from the input to the output isn’t
straightforward, like in mathematical problems or challenging questions. It
breaks down the thought process into intermediate steps, like stepping stones,
from the input to the output. The LLM generates a sequence of thoughts, each
building upon the previous, which helps in reaching the final answer.

  2. 2.

Tree-of-thought (ToT) Prompting [14] ToT prompting is an extension of CoT,
which utilizes multiple reasoning paths and structures the process as a search
through a tree where each node is a thought. This method is like searching for
a solution over a tree, where paths are generated by a CoT-like process. It
systematically explores the tree using algorithms such as breadth-first or
depth-first search.

  3. 3.

Reasoning via Planning (RAP) [5] RAP is similar to ToT but uses Monte Carlo
Tree Search (MCTS). Here, heuristics from the LM are used to inform the search
strategy, enhancing the decision-making process by evaluating the likelihood
or confidence of actions.

###  I-C Research Goal: Prompt Engineering vs Native Abilities

Using Human Eval, we start exploring the effectiveness of different prompt
engineering techniques in improving the code generation abilities of large
language models. We also explore the performance across models and we theorize
on the potential of GPT-4 to exhibit emergent, foundational AGI behaviors in
the context of code generation without the use of prompt engineering. Whether
the path way to AGI behaviors in code generation comes directly from the model
or from prompt engineering is a question that needs further investigation.
Some argue that prompt engineering limits the true capabilities of generative
language models [12]. They believe that prompt engineering restricts the
models’ ability to generate truly creative and innovative outputs, as it
constrains them to predefined templates and structures. In the case of code
generation, AGI capabilities may stem from repeated generation on the LLM’s
inherent understanding of programming concepts, rather than relying solely on
prompt engineering techniques. This may look like the use of reward signals
which are used to encourage the model to learn from its mistakes and improve
its code generation abilities over time. This would mean the evaluations that
are solely focused on prompt engineering may not fully capture the true
potential and capabilities of LLMs in code generation and that developing
accurate reward signals for LLMs may be a more effective approach for
enhancing their code generation abilities. However, proponents of prompt
engineering argue that it is a necessary tool to ensure accurate and valuable
outputs from generative language models. Prompt engineering still plays a
critical role in leveraging the capabilities of LLMs, allowing for explicit
instruction, chain-of-thought prompting, and zero-shot/few-shot learning [6].

###  I-D Significance of the Study

Prompt engineering and the evaluation of LLM performance in complex tasks are
crucial areas of research, especially as we move towards more advanced
language models like GPT-4 [3]. LLMs have shown tremendous potential in
various domains, including code generation. In 2023, it is important to
evaluate the effectiveness of prompt engineering techniques in enhancing the
code generation abilities of large language models. This evaluation will
provide insights into the strengths and limitations of prompt engineering and
the potential for LLMs like GPT-4 to exhibit foundational AGI behaviors in
code generation.

###  I-E Objectives of the Study

The objectives of this study are threefold:

  * •

To evaluate the performance of GPT-4, a large language model, in code
generation tasks.

  * •

To assess the effectiveness of prompt engineering strategies in guiding
GPT-4’s code generation abilities.

  * •

To compare the use of $pass@1$ and $pass@10$ evaluation methods in determining
the ability of GPT-4 to generate accurate and efficient code.

###  I-F Key Contributions

This study aims to contribute to the existing literature on prompt engineering
and LLM performance evaluation by specifically evaluating GPT-4 . By focusing
on code generation tasks, this study will provide insights into the
effectiveness of prompt engineering in enhancing the code generation abilities
of LLMs. Furthermore, by comparing the use of pass@1 and pass@10 evaluation
methods, this study will shed light on the most appropriate approach to
evaluate and measure the code generation capabilities of LLMs like GPT-4. We
release an Open Source Code and Dataset, allowing for further research and
replication of the findings presented in this study

##  II Methodology

###  II-A Goals

This study was designed with specific goals in mind:

  1. 1.

To articulate sub-research questions that delve deeper into the overarching
research questions.

  2. 2.

To describe the experimental design in detail, including instructions,
challenge scenarios, and evaluation criteria.

  3. 3.

To provide an overview of the testing results and describe possible
applications of the findings.

###  II-B Sub-Research Questions

To address the main research question, we propose the following sub-research
questions:

  1. 1.

How does the performance of GPT-4 compare to previous versions of large
language models in code generation tasks?

  2. 2.

What impact does prompt engineering have on the code generation abilities of
GPT-4?

  3. 3.

Will future LLMs demonstrate increasingly human-like capabilities in code
generation tasks and surpass the necessity for prompt engineering?

###  II-C Evaluation Set

The Python coding problems for this study were sourced from the OpenAI
HumanEval dataset, which contains 164 problems complete with prompts, example
cases, and tests, all human-written [4]. Each problem had a variable number of
unit tests, tailored to their specific requirements.

###  II-D Experiment Design

####  II-D1 Ability for Models to Pass HumanEval

##### Method

We will evaluate different LLM models performance on HumanEval for both pass@1
passes and pass@10 passes.

  * •

Specifications:

We utilize the OpenAI API to access models

We create a framework that creates a json file with all the answers that can
interchange models

We utilize multiple threads to receive multiple iteration

We can then use the OpenAI grading framework to run the testing script

##### Evaluation Criteria

The LLM’s performance will be assessed in terms of:

  * •

Accuracy:

In this experiment, we are solely investigating potential increases in
performance. This score out of 100 is given by the OpenAI HumanEval framework.

  * •

Example Question:

⬇

1def generate_integers(a, b):

2 """

3 Given two positive integers a and b, return the even digits between a

4 and b, in ascending order.

5

6 For example:

7 generate_integers(2, 8) => [2, 4, 6, 8]

8 generate_integers(8, 2) => [2, 4, 6, 8]

9 generate_integers(10, 14) => []

10 """

11 lower = max(2, min(a, b))

12 upper = min(8, max(a, b))

13

14 return [i for i in range(lower, upper+1) if i % 2 == 0]

15

16def check(candidate):

17 # Check some simple cases

18 assert candidate(2, 10) == [2, 4, 6, 8], "Test 1"

19 assert candidate(10, 2) == [2, 4, 6, 8], "Test 2"

20 assert candidate(132, 2) == [2, 4, 6, 8], "Test 3"

21 assert candidate(17,89) == [], "Test 4"

22

23 # Check some edge cases that are easy to work out by hand.

24 assert True, "This prints if this assert fails 2 (also good for
debugging!)"

  * •

Example Answer:

⬇

1def generate_integers(a, b):

2 # ensure a <= b

3 if a > b:

4 a, b = b, a

5

6 list_even_numbers = [number for number in range(a, b+1) if number % 2 == 0]

7

8 return list_even_numbers

##  III Results

Model | pass@k [%]
---|---
| k = 1 | k = 10 | k = 100
CODEX 300M | 13.17 | 20.37 | 36.27
CODEX 2.5B | 21.36 | 35.42 | 59.50
CODEX 12B | 28.81 | 46.81 | 72.31
CodeGEN-Mono 350M | 12.76 | 23.11 | 35.19
CodeGEN-Mono 2.7B | 23.70 | 36.64 | 57.01
CodeGEN-Mono 6.1B | 26.13 | 42.29 | 65.82
CODE-DAVINCI-002 | 48.17 | 74.9 | 92.1
TEXT-DAVINCI-002 | 30.48 | NA | NA
TEXT-DAVINCI-003 | 59.14 | NA | NA
GPT-3.5-TURBO-0301 (CHATGPT) | 72.19 | 89.02 | NA
GPT-4-0613 (CHATGPT) | 82.68 | 95.73 | NA
GPT-4-1106-PREVIEW (CHATGPT) | 85.73 | 98.17 | NA
TABLE I: Model performance comparison

###  III-A Research Question 1: How does the performance of GPT-4 compare to
previous versions of large language models in code generation tasks?

In order to answer this research question, the code generation performance of
GPT-4 will be compared to previous versions of large language models. Codex
models were the original models used for code generation, followed by CodeGen,
GPT-2 and GPT-3. The evaluation of GPT-4’s performance in code generation
tasks compared to previous versions of large language models revealed
noteworthy findings. The latest GPT models showed significant improvements in
code generation capabilities compared to their predecessors. Even the newest
model of GPT4 showed improvements over the old GPT4. Notably, with an pass@1
score of 85.73 and a pass@10 of 98.17. This an incredible performance boost
compared to the original models which were unable to break even 50 percent
using a pass@10.

###  III-B Research Question 2: What impact does prompt engineering have on
the code generation abilities of GPT-4?

Prompt engineering plays a crucial role in improving the code generation
abilities of GPT-4. Techniques such as LATS or Reflexion boast incredible
pass@1 performance metrics at 94.4 and 91.0 respectively. However, the main
drawbacks of using these techniques are that they require additional human
effort and expertise in designing and refining prompts. They also take a
considerable amount of time to implement, optimize, and as a result, may
increase the overall cost of using GPT-4 for code generation tasks. Whether or
not these techniques will be relevant in the accelerating landscape of LLMs is
something to be explored.

##  IV Discussion

###  IV-A Research Question 1: How does the performance of GPT-4 compare to
previous versions of large language models in code generation tasks?

The evaluated performance of GPT-4 in code generation tasks surpasses that of
previous versions of large language models. With performance this performant,
it is clear that GPT-4 has made significant advancements in its code
generation abilities. Looking into the future, there are several potential
implications. Firstly, it is evident that GPT-4 showcases increasingly human-
like capabilities in code generation tasks. This raises the possibility of
using LLMs like GPT-4 as powerful tools for automating code generation
processes, potentially reducing the need for human intervention in certain
programming tasks. It also shows the ability for GPT to showcase AGI without
the use of prompt engineering. With very little human intervention and prompt
engineering required, GPT-4 demonstrates the potential to revolutionize code
generation practices in the software development industry.

The exceptional score achieved suggests that the base models are approaching
extraordinary levels of performance. Moreover, the comparison with previous
versions of large language models demonstrates a clear advancement in code
generation capabilities. The noticeable improvements in performance from GPT-2
to GPT-3, and now to GPT-4, affirm the trajectory of progress in the field of
language models.

Moving forward, it will be essential to delve into the impact of prompt
engineering on the code generation abilities of GPT-4, as well as the
potential for future LLMs to exhibit increasingly human-like capabilities and
potentially surpass the necessity for prompt engineering. This will provide
valuable insights into the evolving landscape of language models and their
applications in code generation tasks.

###  IV-B Research Question 2: What impact does prompt engineering have on the
code generation abilities of GPT-4?

With the advent of prompt engineering, there are some common techniques that
arise that increase performance. Among these, self-reflection and advanced
search algorithms are notable for their impact on performance.

Self-reflection, as an aspect of machine learning, involves an algorithm’s
capacity to evaluate and adapt its processes based on its prior performance or
decisions. This capability enables the algorithm to continuously learn from
its experiences, refine its decision-making, and adjust to new or evolving
conditions. The integration of self-reflection into learning algorithms, as
seen in frameworks like LATS, has been demonstrated to be beneficial,
contributing to a more refined data processing and understanding [15].

In parallel, advanced search algorithms are pivotal in optimizing machine
learning system efficiency. A prime example of such an algorithm is the Monte
Carlo Tree Search (MCTS). MCTS, a heuristic search algorithm, is particularly
effective in complex and uncertain decision-making environments. Unlike
simpler algorithms like A* or Depth-First Search (DFS), MCTS adopts a more
methodical and explorative approach. It constructs a search tree where each
node signifies a potential state in the decision space, and utilizes random
sampling to assess and weigh the potential outcomes of various choices. This
approach enables a more thorough and strategic exploration of possibilities,
leading to improved decision-making in intricate scenarios [8].

###  IV-C Research Question 3: Will future LLMs demonstrate increasingly
human-like capabilities in code generation tasks annd surpass the necessity
for prompt engineering?

Future LLMs have the potential to exhibit increasingly human-like capabilities
in code generation tasks and may eventually surpass the necessity for prompt
engineering. We have seen promising progress with LLM agents. Such as GPT-4,
which has demonstrated significant advancements in code generation tasks.
However, whether future LLMs will completely eliminate the necessity for
prompt engineering remains a topic for further exploration.

The methodology employed in this study to evaluate the code generation
performance of GPT-4 involved the use of different evaluation methods such as
pass@1 and pass@10. The results of these evaluations provided valuable
insights into the model’s ability to generate accurate and efficient code.
Additionally, the study delved into the impact of prompt engineering
techniques on the code generation abilities of GPT-4, shedding light on the
benefits and potential drawbacks associated with these strategies.

The findings from this study not only contribute to the existing literature on
prompt engineering and LLM performance evaluation but also have broader
implications for the future development and application of large language
models. The insights gained from this research are instrumental in
understanding the potential of LLMs, such as GPT-4, to achieve foundational
AGI behaviors in code generation.

Moving forward, it is essential to continue exploring the evolving
capabilities of LLMs and their potential to exhibit increasingly human-like
behaviors in code generation tasks. The research should also focus on
identifying the limitations and challenges associated with prompt engineering,
as well as evaluating the feasibility of incorporating few-shot examples for
in-context learning to enhance the code generation abilities of LLMs.

Furthermore, the release of an open-source code and dataset from this study is
a significant contribution that will enable further research and replication
of the findings. The availability of this valuable resource fosters
collaboration and encourages the exploration of novel approaches to prompt
engineering and LLM performance evaluation.

##  V Limitations & Threats to Validity

It is important to acknowledge the limitations of using only one evaluation
set when assessing the performance of GPT-4 in code generation tasks. Relying
solely on a single evaluation set may not capture the entire spectrum of the
model’s capabilities and limitations. Different evaluation sets may present
varied challenges and nuances that a singular set might not encompass.
Therefore, incorporating multiple evaluation sets with diverse characteristics
and complexities would provide a more comprehensive understanding of the
model’s performance in code generation tasks.

Additionally, the generalization of findings from the evaluation of GPT-4
solely based on code generation tasks needs to be approached with caution.
While the model’s proficiency in code generation tasks is a significant
aspect, its performance in other domains and tasks should also be evaluated to
gain a holistic understanding of its capabilities and limitations.

Furthermore, the findings of this study may be influenced by the specific
prompt engineering techniques and few-shot examples utilized in the
evaluation. It is essential to recognize that different prompt engineering
strategies and in-context learning examples may yield varying impacts on the
model’s performance. Exploring a wider range of prompt engineering methods and
in-context learning scenarios can provide a more robust analysis of the
effects of these techniques on the code generation abilities of GPT-4.

Moreover, the study’s reliance on GPT-4 as the sole representative of large
language models may limit the generalizability of the findings to other LLMs.
Different models may exhibit unique strengths and weaknesses, and their
individual performances in code generation tasks should be separately examined
to draw accurate comparisons and conclusions.

##  VI Related Work

###  VI-A Benchmarks for Program Synthesis

Recent advancements in the field of program synthesis have seen a variety of
novel approaches and benchmarks introduced. For instance, a paper titled
”Inductive Program Synthesis via Iterative Forward-Backward Abstract
Interpretation” [14] introduces a method based on iterative forward-backward
abstract interpretation to synthesize programs. This approach provides a
structured synthesis algorithm that is sound and complete, meaning that if a
solution exists for a given program synthesis instance, the algorithm can find
it.

Another paper, ”xCodeEval [2]: A Large Scale Multilingual Multitask Benchmark
for Code Understanding, Generation, Translation and Retrieval,” presents a new
benchmarking framework called xCodeEval. This framework is aimed at evaluating
multilingual multitask capabilities, including program synthesis, and
introduces a new execution benchmarking environment known as ExecEval to
support this.

Moreover, the paper ”Hierarchical Neural Program Synthesis” [1] proposes a
framework called Hierarchical Neural Program Synthesizer (HNPS), which focuses
on synthesizing longer programs by composing shorter programs. The method
involves learning a task embedding space and creating datasets specifically
designed to train models in program composition.

Additionally, the work presented in ”Enhancing Robot Program Synthesis Through
Environmental Context” [9] focuses on synthesizing robot programs by
leveraging partially observed environments. This approach aims to rectify
potentially erroneous code segments by learning an environment embedding space
and employing a graph structure to provide program rectification guidance.

These papers reflect the current direction in program synthesis research,
focusing on more complex synthesis processes, multilingual and multitask
benchmarks, as well as incorporating environmental context into the synthesis
of robotic programs. These advancements represent a significant step forward
in the program synthesis domain.

###  VI-B Concluding Remarks & Lessons Learned

The results of this study shed light on the immense potential of GPT-4 and
LLMs in code generation tasks, providing valuable insights into the impact of
prompt engineering, as well as the future trajectory of language models. The
findings not only demonstrate the significant advancements made by GPT-4 in
code generation abilities but also hint at the increasing human-like
capabilities exhibited by LLMs.

The implications of these findings are far-reaching. They indicate the
potential for LLMs like GPT-4 to serve as powerful tools for automating code
generation processes, potentially reducing the need for significant human
intervention in certain programming tasks. Furthermore, the emergence of LLMs
with AGI behaviors, such as GPT-4, opens up new possibilities for
revolutionizing code generation practices in the software development
industry.

As this research evolves, it is essential to continue exploring the evolving
capabilities of LLMs and their potential integration into diverse domains
beyond code generation tasks. Continued evaluation of prompt engineering
techniques and the feasibility of incorporating few-shot examples for in-
context learning will provide further insights into enhancing the code
generation abilities of LLMs.

In conclusion, the findings of this study contribute significantly to the
existing literature on prompt engineering and LLM performance evaluation, and
they provide a foundation for the future development and application of large
language models. The potential for LLMs like GPT-4 to exhibit increasingly
human-like behaviors in code generation tasks marks a pivotal moment in the
advancement of language models and their potential to reshape programming
practices.

###  VI-C Acknowledgements

We wish to acknowledge the instrumental role of AI systems in our research,
specifically ChatGPT Perplexity AI, and Jenni AI. These AI tools have been
indispensable in assisting with various aspects of this paper, ranging from
initial brainstorming and literature review to the refinement of our
methodology and the drafting of certain sections. Their advanced natural
language processing capabilities significantly contributed to the efficiency
and depth of our research process. The collaborative integration of these AI
technologies has demonstrated the immense potential of human-AI collaboration
in academic research.

## References

  * [1] Hierarchical neural program synthesis.  arXiv preprint arXiv:2303.06018, 2023.
  * [2] xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval.  arXiv preprint arXiv:2303.03004, 2023.
  * [3] Steven Bryant.  Assessing gpt-4’s role as a co-collaborator in scientific research: A case study analyzing einstein’s special theory of relativity.  04 2023.
  * [4] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code, 2021.
  * [5] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu.  Reasoning with language model is planning with world model, 2023.
  * [6] David C. S. Li and Weng-Fai Wong.  Iaso: Enhancing syntax error correction using deep learning techniques.  07 2023.
  * [7] OpenAI.  Gpt-4 technical report, 2023.
  * [8] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.  Reflexion: Language agents with verbal reinforcement learning, 2023.
  * [9] Author Unknown.  Enhancing robot program synthesis through environmental context.  OpenReview, 2023.  [Online]. Available: https://openreview.net/forum?id=pZ2Ww45GkL.
  * [10] Waqas Uzair.  Six-tier architecture for ai-generated software development: A large language models approach.  06 2023.
  * [11] Ashish Vaswani et al.  Attention is all you need.  Conference Name or Journal, 2017.  Introduction of the Transformer model.
  * [12] Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon.  Can chatgpt write a good boolean query for systematic review literature search?  01 2023.
  * [13] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.  Chain-of-thought prompting elicits reasoning in large language models, 2023.
  * [14] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan.  Tree of thoughts: Deliberate problem solving with large language models, 2023.
  * [15] Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang.  Language agent tree search unifies reasoning acting and planning in language models, 2023.

##  VII Appendix

All code is open source: https://github.com/daniel442li/gpt-human-eval/