# Blue quantum emitter in hexagonal boron nitride and carbon chain tetramer:
proposition of identification

Marek Maciaszek<EMAIL_ADDRESS>Faculty of Physics, Warsaw
University of Technology, Koszykowa 75, 00–662 Warsaw, Poland  Center for
Physical Sciences and Technology (FTMC), Vilnius LT–10257, Lithuania  Lukas
Razinkovas  Center for Physical Sciences and Technology (FTMC), Vilnius
LT–10257, Lithuania  Department of Physics/Centre for Materials Science and
Nanotechnology, University of Oslo, P.O. Box 1048, Blindern, Oslo N-0316,
Norway

###### Abstract

Single photon emitters in hexagonal boron nitride offer a gateway to the
future of quantum technologies, yet their identification remains challenging
and subject to ongoing debate. We demonstrate through ab initio calculations
that the optical properties of a carbon chain tetramer are in excellent
agreement with the characteristics of a blue quantum emitter in hexagonal
boron nitride emitting at 435 nm. Its calculated zero-phonon line energy (2.77
eV) and radiative lifetime (1.6 ns) perfectly align with experimental
observations. The relatively weak electron–phonon coupling (Huang–Rhys factor
of 1.5) indicates intense emission at the zero-phonon line. Despite the
absence of an inversion center in the carbon tetramer, we demonstrate that it
exhibits a negligible linear Stark effect, consistent with experimental
findings. Additionally, our hypothesis explains the experimental observation
that the formation of blue emitters is only possible in samples containing
numerous ultraviolet emitters, which are likely identical to carbon dimers.

##  I Introduction

Hexagonal boron nitride (h-BN) has long been employed in nanotechnology, yet
recent years have seen a notable surge in attention toward this material,
spurred by the discovery of numerous single-photon emitters within h-BN [1].
Its wide bandgap enables it to accommodate single-photon emitters from the
near-infrared to ultraviolet spectrum. Notably, many emitters demonstrate high
brightness and stability even at room temperature, positioning them as
promising candidates for applications in emerging quantum technologies [2, 3,
4].

A blue quantum emitter with a zero-phonon line (ZPL) of 2.85 eV (435 nm) in
hexagonal boron nitride was observed in 2019 by Shevitski et al. [5].
Subsequently, its optical properties have been extensively explored in
numerous studies [6, 7, 8, 5, 9]. Its excited state lifetime was determined as
1.8–2.1 ns [9, 8]. The quantum efficiency of the blue emitters was estimated
to be at least 20%, making them bright while exhibiting excellent
photostability. It was concluded that the defect, being the blue quantum
emitter, has no metastable state, and the suggested electronic structure
comprises two levels within the bandgap: one fully occupied in the lower half
and an empty one in the upper half [6]. Moreover, it was demonstrated that the
emitter exhibits a very weak Stark effect induced by out-of-plane fields and a
much stronger but almost purely quadratic for in-plane fields [7].

Blue emitters can be created at chosen locations by electron irradiation
treatment [8], and this procedure is efficient only in samples exhibiting a
high concentration of ultraviolet (featuring a ZPL of 4.1 eV) emitters [9].
These blue emitters remain stable under annealing treatments up to
800$\mathrm{\SIUnitSymbolCelsius}$ [10]. Nonetheless, annealing at
1000$\mathrm{\SIUnitSymbolCelsius}$ leads to the disappearance of the blue
emitters and the emergence of other emitting centers. The ZPLs of these new
emitters are observed across a spectral range from 420 to 480 nm (2.58 to 2.95
eV).

A minimal linear Stark effect has prompted the hypothesis of nitrogen or
carbon interstitial defects being responsible for blue emission [7]. This idea
was further elaborated with the theory that the negatively charged nitrogen
split interstitial defect is the source of the blue emission [11]. Although
the calculated properties of the nitrogen interstitial largely align with
experimental data, there are two significant issues with this assignment: (1)
the computed radiative lifetime is 54 ns, and (2) the observed correlation
between blue and UV emitters, typically attributed to carbon dimers, cannot be
simply explained.

In the following, we demonstrate that all these experimental observations can
be well explained by the properties of the carbon chain tetramer
${(\mathrm{C}_{\mathrm{B}}\mathrm{C}_{\mathrm{N}})}_{2}$, where four carbon
atoms substitute on four neighbouring sites within the h-BN lattice (see Fig.
1). Our reasoning is based on four arguments: (1) the simultaneous presence of
both blue and UV emitters becomes understandable when identifying the 435 nm
emitter as a carbon-based complex, (2) the ZPL of the carbon tetramer,
calculated to be 2.77 eV, closely matches the experimental value, (3) the
computed radiative lifetime of 1.6 ns shows excellent agreement with
experimental findings, and (4) the linear Stark effect is predicted to be
negligible.

The carbon chain tetramer can exist in two geometric configurations: cis or
trans (see Fig. 1). While calculated results favor the trans conformer, which
exhibits better agreement with experimental findings, both conformers will be
analyzed and discussed.

Figure 1: Cis and _trans_ configurations of carbon tetramers
${(\mathrm{C}_{\mathrm{B}}\mathrm{C}_{\mathrm{N}})}_{2}$ in h-BN.

##  II Methodology

Ab initio calculations were performed using spin-polarized density functional
theory (DFT) as implemented in VASP code [12]. The projector augmented wave
(PAW) method was employed with a plane wave energy cutoff set to 500 eV [13].
In calculations, we used a 288-atom hexagonal supercell (equivalent to a
$6\times 6\times 2$ arrangement of the primitive cell), sampling the Brillouin
zone at the $\Gamma$ point only. Ionic relaxation was conducted until forces
fell below 0.01 eV/Å.

The formation energies of the defect (refer to Sec. III.1) were calculated
using the Heyd–Scuseria–Ernzerhof (HSE) hybrid density functional [14]. To
accurately capture the material’s band gap, the mixing parameter $\alpha$ was
adjusted from the default value of 0.25 to 0.31, achieving a band gap of 5.95
eV. This computational setup yielded lattice constants $a=2.490$ Å and
$c=6.558$ Å, which show good agreement with experimental measurements
($a=2.506$ Å and $c=6.603$ Å [15]). Furthermore, the formation enthalpy of the
compound was calculated to be $-2.89$ eV, aligning well with the experimental
value of $-2.60$ eV [16].

The formation energy $E_{f}$ of a carbon chain tetramer is expressed as [17,
18]

$\displaystyle E_{f}(q)=$ $\displaystyle
E_{D}(q)-E_{H}-4\mu_{\mathrm{C}}+2(\mu_{\mathrm{B}}+\mu_{\mathrm{N}})$
$\displaystyle+q(E_{V}+E_{F})+E_{\text{corr}}$

where $E_{D}$ and $E_{H}$ represent the total energies of the supercell with
and without a tetramer, respectively. Additionally, $\mu_{\mathrm{B}}$,
$\mu_{\mathrm{N}}$, and $\mu_{\mathrm{C}}$ denote the chemical potentials of
boron, nitrogen, and carbon, respectively. When calculating the formation
energy, we assumed the chemical potential of carbon to be equal to that in
bulk graphite. This represents the maximum possible value of
$\mu_{\mathrm{C}}$, indicating the highest carbon availability during growth.
It is worth noting that
$\mu_{\mathrm{B}}+\mu_{\mathrm{N}}=\mu_{\mathrm{BN}}=E_{H}/n$, where $n$
represents the number of formula units in the supercell ($n=144$ in our case).
Therefore, $E_{f}$ does not depend on the individual chemical potentials of B
or N. $q$ is the charge state of the tetramer, $E_{V}$ stands for the energy
of the valence band maximum (VBM), and $E_{\text{corr}}$ is a correction that
describes the interaction between charged defects in periodic supercells.
$E_{\text{corr}}$ value was calculated using the Freysoldt–Neugebauer–Van de
Walle methodology [19]. $E_{F}$ is the Fermi level with respect to $E_{V}$.
The charge-state transition level $\varepsilon(q/q^{\prime})$ is defined as
the value of $E_{F}$ for which the formation energies of two charge states,
$q$ and $q^{\prime}$, are equal.

In modeling electronic properties(Sec. III.2), we utilized the meta-GGA r2SCAN
exchange–correlation functional [20, 21], which provided a band gap of 4.83
eV. The optimized lattice constants, $a=2.502$ Å and $c=6.515$ Å, along with
the formation enthalpy of $-2.89$ eV (equal to the HSE result), exhibit good
alignment with experimental findings, demonstrating the effectiveness of
r2SCAN in simulating bulk properties despite underestimation of the band gap.

Our decision to utilize the r2SCAN functional stemmed from the HSE
functional’s tendency to overestimate ZPL energies. While r2SCAN tends to
slightly underestimate ZPL energies, its performance in the context of diamond
defects, for instance, yields mean absolute errors for ZPL similar to those
obtained using HSE with a standard mixing parameter ($\alpha=0.25$) [22].
Elevating $\alpha$ to match the experimental band gap implies a potential
widening of HSE’s deviation in ZPL energy predictions. Furthermore, such
adjustment of $\alpha$ raises concerns regarding the accuracy of calculations,
as this approach may not satisfy Koopman’s theorem [11]. However, for a
comprehensive evaluation, all calculations were also performed using HSE with
$\alpha=0.31$, and the comparison of results obtained with both methodologies
is provided in the Supplementary Material.

Van der Waals interactions were taken into account by employing the Grimme-D3
method in HSE calculations [23] and the nonlocal rVV10 method [24, 25, 26, 27]
in r2SCAN calculations.

##  III Results and discussion

###  III.1 Formation and temperature stability

Figure 2: Formation energy of
trans-${(\mathrm{C}_{\mathrm{B}}\mathrm{C}_{\mathrm{N}})}_{2}$ as a function
of the Fermi level.

The formation energy of the neutral carbon chain tetramer (trans conformer) is
3.30 eV (Fig. 2). Its neutral charge state is stable in a wide range of the
Fermi level: charge-state transition levels are 1.24 and 5.16 eV above the VBM
for $(+/0)$ and $(0/-)$, respectively. The $2+$ and $2-$ charge states were
found not to be stable.

Regarding the cis conformer, its formation energy is lower by 0.06 eV than
that of the trans conformer, and its charge-state transition levels are
$\varepsilon(+/0)=1.07$ eV and $\varepsilon(0/-)=5.23$ eV above the VBM. Since
further analysis of the formation and temperature stability focuses on the
neutral charge state, where the formation energies are nearly identical for
both conformers, the conclusions are applicable to the cis conformer as well.
Quantitative results pertain to the trans conformer.

Although the formation energy of carbon tetramers is not particularly high,
blue emitters are not formed under equilibrium conditions but through electron
irradiation treatment, and the number of fabricated emitters depends on the
electron dose [8, 9]. It was demonstrated that the formation of blue emitters
is most efficient in samples exhibiting a high concentration of 4.1 eV
emitters [9]. Since the 4.1 eV emitters are commonly identified with carbon
dimers $\mathrm{C}_{\mathrm{B}}\mathrm{C}_{\mathrm{N}}$ [28], this
experimental observation suggests that blue emitters are carbon related
defects. Furthermore, it was speculated that blue emitters are formed through
the restructuring of carbon dimers induced by the electron beam, which breaks
the bonds and enhances the drift of ionized atoms [9]. This aligns well with
our hypothesis, as a carbon tetramer can be formed from two dimers.
Alternatively, one could posit that carbon tetramers are not formed by merging
two dimers but rather by the fragmentation of larger carbon clusters induced
by the electron beam. The presence of such clusters should correlate with the
large concentration of carbon dimers, as both phenomena are caused by high
carbon density in the h-BN sample.

The blue emitters remain stable up to 800$\mathrm{\SIUnitSymbolCelsius}$.
However, after annealing at 1000$\mathrm{\SIUnitSymbolCelsius}$ for 1 hour, it
was demonstrated that the blue emitters disappear, and other emitters emerge,
exhibiting various ZPLs, mostly distributed between 2.58 and 2.95 eV [10]. In
the following, we discuss the results of this experiment in the context of the
tetramer model.

Experimental observations suggest that annealing at
1000$\mathrm{\SIUnitSymbolCelsius}$ sufficiently enhances diffusion to restore
thermodynamic equilibrium within the timeframe of the experiment. The
existence of tetramers under thermodynamic equilibrium at
1000$\mathrm{\SIUnitSymbolCelsius}$ is unlikely due to their high formation
energy. Even assuming the maximal value of the carbon chemical potential, the
density of tetramers at 1000$\mathrm{\SIUnitSymbolCelsius}$ is $3\times
10^{10}~{}\text{cm}^{-3}$. Assuming a flake thickness of 50 nm, this density
implies one tetramer per square with a side length of 25 $\mu$m, exceeding the
area examined in [10]. We propose that the disappearance of blue emitters is
caused by the dissociation of tetramers, which can be conceptualized as
spontaneous fragmentation induced by high temperature. The presence of smaller
carbon defects is more likely due to their lower formation energies.

When one of the bonds in a tetramer breaks, smaller carbon complexes such as
dimers, trimers, or monomers are formed. The following are three possible
scenarios of the dissociation of a tetramer:

$\displaystyle{(\mathrm{C}_{\mathrm{B}}\mathrm{C}_{\mathrm{N}})}_{2}$
$\displaystyle\rightarrow 2(\mathrm{C}_{\mathrm{B}}\mathrm{C}_{\mathrm{N}})$
$\displaystyle\quad(\Delta E=0.99~{}\text{eV}),$
$\displaystyle{(\mathrm{C}_{\mathrm{B}}\mathrm{C}_{\mathrm{N}})}_{2}$
$\displaystyle\rightarrow(\mathrm{C}_{2}\mathrm{C}_{\mathrm{N}})^{-}+\mathrm{C}_{\mathrm{B}}^{+}$
$\displaystyle\quad(\Delta E=2.71~{}\text{eV}),$
$\displaystyle{(\mathrm{C}_{\mathrm{B}}\mathrm{C}_{\mathrm{N}})}_{2}$
$\displaystyle\rightarrow(\mathrm{C}_{2}\mathrm{C}_{\mathrm{B}})^{+}+\mathrm{C}_{\mathrm{N}}^{-}$
$\displaystyle\quad(\Delta E=2.87~{}\text{eV}).$

Here, $\Delta E$ describes the energetic cost of such a reaction, calculated
as the difference in formation energies between substrates and products. These
values were estimated using theoretical data provided in [29]. According to
the $\Delta$E values, the most probable scenario appears to be the
dissociation into two dimers. However, the $\Delta$E values for the latter two
reactions correspond to two isolated products (i.e., a monomer and a trimer at
a large distance). In actual reactions, the monomer and trimer are in close
proximity to each other just after dissociation. Consequently, strong
electrostatic attraction reduces $\Delta$E, and this reduction can be
estimated to be as significant as 1 eV, or even slightly higher, for certain
configurations. Thus, the formation of monomers and trimers is also likely to
occur.

The observed diversity in ZPLs can also be attributed to differences in
distances between charged carbon defects, such as monomers and trimers,
resulting in variations in electrostatic interactions. ZPLs of $C_{B}^{+}$ and
$C_{N}^{-}$ pairs exhibit a strong dependence on the distance between carbon
atoms, as demonstrated in [30]. The similar phenomenon of the emergence of
other emitters after annealing was also noted for carbon-based visible (red)
emitters, and the optical properties of the produced emitter families are
similar in both cases [10]. This observation aligns well with our
identification.

The annealing temperature allows for the estimation of the activation energy
of the rate-limiting process. Assuming a typical phonon frequency in h-BN of
$10^{14}~{}\text{s}^{-1}$ [31] and a process rate on the order of
$1~{}\text{s}^{-1}$, we can approximate its value to be between 3.0 and 3.5
eV. This value is similar to the migration barrier of boron vacancies in h-BN,
which is 3.09 eV for $V_{\mathrm{B}}^{-}$ [32]. It can be expected that
electron irradiation creates vacancies at a high density. If these vacancies
are present close to the tetramer, they can mediate carbon diffusion. However,
due to their high formation energies, boron vacancies eventually vanish. It
can be hypothesized that since boron vacancies are produced in the vicinity of
irradiated spots and thus close to blue emitters, they are able to mediate
carbon diffusion; however, after some time, they are annealed.

###  III.2 Electronic structure and optical properties

Figure 3: Kohn–Sham electronic energy level diagram and corresponding orbital
isosurfaces for the trans conformer.

In the analysis that follows, we focus on the neutral charge state of the
tetramer, whose electronic structure features two distinct levels within the
bandgap: a lower-lying, fully occupied level, and an upper-lying, vacant
level, as shown in Fig. 3. The electronic structure of the blue emitter
proposed in [6], deduced from the photostability of the emission, closely
resembles the structure of the tetramer presented here.

Our computations demonstrate that both conformers of the carbon chain tetramer
present $C_{s}$ symmetry. The localized states within the band gap belong to
the irreducible representation $a^{\prime\prime}$, with the lower-energy state
denoted as $a^{\prime\prime}(1)$ and the higher-energy state as
$a^{\prime\prime}(2)$ (see Fig. 3). This configuration indicates that all
bound electronic states of neutral charge state exhibit $A^{\prime}$ symmetry,
implying that both permanent (diagonal) and transition (off-diagonal) dipole
moments align within the two-dimensional plane of the h-BN lattice.

In the molecular orbital picture, the ground state is a closed-shell singlet
represented by wavefunction
$|S_{0}\rangle=|a^{\prime\prime}(1)\bar{a}^{\prime\prime}(1)\rangle$, where
the notation without and with a bar distinguishes spin-up from spin-down
molecular orbitals. The transition to singly excited singlet $|S_{1}\rangle$
and triplet $|T\rangle$ states involves excitation from $a^{\prime\prime}(1)$
to $a^{\prime\prime}(2)$. For $m_{s}=1$ projection, the triplet state is
depicted by a single determinant wavefunction
$|T\rangle=|a^{\prime\prime}(1)a^{\prime\prime}(2)\rangle$. The excited
singlet state, $|S_{1}\rangle$, on the other hand, emerges as a strongly-
correlated, multi-determinant state:
$|S_{1}\rangle=1/\sqrt{2}(|a^{\prime\prime}(1)\bar{a}^{\prime\prime}(2)\rangle-|a^{\prime\prime}(2)\bar{a}^{\prime\prime}(1)\rangle)$.
Due to its multi-determinant nature, traditional DFT cannot directly model
$|S_{1}\rangle$. Thus, we determine its energy using an approximate spin-
purification approach [33, 34]. It involves two distinct energy computations:
one for the single-determinant triplet state and another for a mixed spin
determinant,
$|M\rangle={|a^{\prime\prime}(1)\bar{a}^{\prime\prime}(2)\rangle}$. The energy
of the excited singlet state is approximately given by [28]:

$E(S_{1})=2E(M)-E(T).$ (1)

We employed the delta-self-consistent-field ($\Delta$SCF) methodology,
constraining occupations of Kohn–Sham orbitals, for computing single-
determinant energy terms $E(M)$ and $E(T)$. Concerning the geometric
structure, given that the excited singlet and triplet states possess identical
orbital configurations and differ only due to exchange splitting, it indicates
that the geometry of the excited state $S_{1}$ is close to that of the triplet
state. Therefore, we approximated the geometry of $S_{1}$ by the relaxed
geometry of the $T$ state.

The ZPL energies have been calculated as 2.77 eV for the trans conformer and
3.10 eV for the cis conformer. Since the r2SCAN method tends to slightly
underestimate ZPL energies, the calculated value for the trans conformer
closely matches the experimental result of 2.85 eV. Notably, using the relaxed
geometry of the mixed spin state ($M$ state) to represent the singlet excited
state $S_{1}$ leads to ZPL energies lower by about 0.04 eV (all results are
detailed in the Supplemental Material). This observation is consistent with
the above assumption that the geometry of the triplet state closely
approximates that of the excited singlet state.

Electronic relaxation from the singlet excited state $S_{1}$ can proceed
through two pathways: a radiative transition to the singlet ground state or
intersystem crossing to the triplet state $T$, located 0.35 eV below $S_{1}$.
Given the relatively large exchange splitting of 0.35 eV between the singlet
and triplet states, coupled with minor differences in potential energy
surfaces (leading to small Huang–Rhys factor), it is expected that the non-
radiative lifetime substantially exceeds the radiative lifetime, thereby
making radiative decay the predominant mechanism.

The radiative lifetime, $\tau_{\text{rad}}$, can be estimated using equation:

$\frac{1}{\tau_{\text{rad}}}=\frac{n_{r}E_{\text{ZPL}}^{3}\mu^{2}}{3\pi\varepsilon_{0}\hbar^{4}c^{3}},$
(2)

where $n_{r}$ is the refractive index of h-BN, $E_{\text{ZPL}}$ is the ZPL
energy, $\mu$ is the optical transition dipole moment, and $\varepsilon_{0}$
is the vacuum permittivity. The transition dipole moment, $\mu$, is estimated
utilizing the expressions derived for molecular orbital states:

$\mu=\langle S_{0}|\sum_{i}\mathbf{r}_{i}|S_{1}\rangle=\sqrt{2}\langle
a^{\prime\prime}(1)|\mathbf{r}|a^{\prime\prime}(2)\rangle,$ (3)

where the final single-particle expression is obtained by applying the Condon
rules for determinantal states. We then calculate $\langle
a^{\prime\prime}(1)|\mathbf{r}|a^{\prime\prime}(2)\rangle$ using single-
particle Kohn–Sham orbitals determined from the triplet electronic and
geometric configuration, a common practice in DFT calculations.

With the ZPL energy set to an experimental value of 2.85 eV and using
$n_{r}=2.25$ [35, 36], the radiative lifetime for the trans conformer is
calculated to be 1.6 ns. This finding aligns closely with experimental
lifetimes reported between 1.8 and 2.1 ns [8, 9]. The calculated transition
dipole moment is 1.77 eÅ. For the cis conformer, $\tau_{\text{rad}}$ is 2.5
ns, with a transition dipole moment of 1.42 eÅ.

To quantify the electron-phonon coupling strength, we calculated the
Huang–Rhys factor $S$ within the effective mode approximation, following the
methodology outlined in Ref. [37]. The Huang–Rhys factor, $S$, characterizes
the mean number of phonons emitted during electronic transitions. The energy
released due to lattice relaxation after transitioning to the ground state was
calculated to be 0.20 eV. We determined the effective phonon frequency to be
$\hbar\omega=134~{}\text{meV}$, resulting in a Huang–Rhys factor of $S=1.50$.
This factor is pivotal for estimating of the luminescence intensity fraction
within the zero-phonon line (ZPL), referred to as the Debye–Waller factor
(DWF). Employing the single-mode approximation allows for an estimation of
$\mathrm{DWF}\approx e^{-S}$, an approximation applicable in the limit
$\hbar\omega/E_{\text{ZPL}}\ll 1$. This yields a theoretical Debye–Waller
factor of $\mathrm{DWF}=0.22$. Using a more rigorous formula, as explained in
the Supplementary Material, results in a marginally increased value of 0.28.
The experimental measurements of the DWF, which were reported as 0.26 in Ref.
[5] and 0.40 in Ref. [8], place our calculated value in agreement with the
experimental findings. Regarding the cis conformer, the calculated values are
$S=1.91$ and $\hbar\omega=122~{}\text{meV}$.

HSE results confirm the general conclusions drawn from r2SCAN calculations.
HSE with $\alpha=0.31$ yields higher values of the ZPL, 3.29 eV and 3.58 eV
for the trans and cis conformers, respectively. Since HSE typically
overestimates ZPL, especially with an assumed $\alpha$ larger than its
standard value of 0.25, HSE values can be considered upper limits for the ZPL.
Consequently, r2SCAN provides lower limits. The experimental value of 2.85 eV
falls within this estimated range for the trans conformer. Concerning the
radiative lifetime, its values are slightly smaller when calculated using HSE
(1.4 ns and 2.2 ns for trans and cis, respectively). All results obtained with
both functionals are presented in the Supplementary Material. Additionally,
all data are calculated for both approximations of the geometry of the singlet
excited state: by the geometry of the triplet ($T$) and the mixed spin ($M$)
states.

Finally, we examine results related to the Stark effect. Experiments revealed
that an out-of-plane electric field causes a negligible linear Stark shift,
estimated at 0.04 meV/V nm-1, and no quadratic dependency was detected in this
direction. Conversely, a significant quadratic shift in the ZPL was noted
under in-plane electric fields, with the linear component remaining very small
[7].

It is often assumed that the absence of a linear Stark effect implies the
defect has a center of inversion symmetry. However, this assertion should be
correctly understood in reverse: the presence of an inversion symmetry center
in a defect guarantees the absence of a linear Stark effect. Nevertheless,
there are instances where a defect lacking inversion symmetry does not exhibit
a linear Stark effect or demonstrates a very faint one. For instance, this
phenomenon may occur when the electric field induces identical shifts in both
optically active energy levels [38].

As previously mentioned, theory predicts no linear Stark shift for an out-of-
plane field according to group theory. The experimentally observed minimal
linear Stark shift in the out-of-plane direction could be attributed to sample
inhomogeneity.

For the in-plane electric field, we introduce a straightforward ab initio
methodology to evaluate the linear Stark shift. Within the first-order
perturbation theory, the ZPL shift is given by:

$\Delta E_{\text{ZPL}}=-\mathbf{E}e\left(\langle
S_{1}|\sum_{i}\mathbf{r}_{i}|S_{1}\rangle-\langle
S_{0}|\sum_{i}\mathbf{r}_{i}|S_{0}\rangle\right),$ (4)

where $\mathbf{E}$ denotes the applied electric field. Leveraging molecular
orbital wavefunctions, this shift can be expressed using single-particle
matrix elements:

$\Delta E_{\text{ZPL}}=-\mathbf{E}e\left(\langle
a^{\prime\prime}(2)|\mathbf{r}|a^{\prime\prime}(2)\rangle-\langle
a^{\prime\prime}(1)|\mathbf{r}|a^{\prime\prime}(1)\rangle\right),$ (5)

which can be calculated from Kohn–Sham single-particle states. The amplitudes
of the permanent dipole moments are calculated to be smaller than
$10^{-3}~{}e$Å for both $a^{\prime\prime}(1)$ and $a^{\prime\prime}(2)$.
Insignificant values of amplitudes can also be inferred from the symmetry of
the spatial distribution of the charge density (see Fig. 3). The obtained
results demonstrate that the linear Stark shift for the trans tetramer under
in-plane electric fields should be negligibly small, which is consistent with
experimental observations. A similar understanding can be applied to the cis
conformer, for which the vanishing of the linear Stark shift is also expected.

##  IV Conclusions

In summary, we presented the results of ab initio modeling of the carbon chain
tetramer in hexagonal boron nitride. Its calculated optical properties match
very well with experimental characteristics of the blue emitter. The
calculated ZPL of 2.77 eV closely aligns with the experimental value of 2.85
eV, and the radiative lifetime of 1.6 ns is in good agreement with the
measured values of 1.8–2.1 ns. Moreover, we demonstrated that the electron-
phonon coupling is relatively weak ($S=1.5$), and the calculated Debye–Waller
factor is 0.28, in good accordance with the experimental results (0.26 and
0.40). The linear Stark effect should be absent or extremely weak for both in-
plane and out-of-plane external fields, consistent with experimental
observations.

Additionally, we discussed experimental studies of the formation and annealing
of blue emitters, proposing that the carbon chain tetramer provides a coherent
explanation for their observations. Specifically, our identification of the
blue emitter as the carbon tetramer elucidates why the fabrication of blue
emitters is efficient only in h-BN samples containing numerous UV emitters.
Furthermore, the postulated dissociation of the tetramer with temperature
provides an explanation for the observed formation of other emitters with
similar ZPLs after annealing at 1000$\mathrm{\SIUnitSymbolCelsius}$,
attributing it to the emergence of carbon monomers, dimers and trimers at
different distances.

## Acknowledgments

Research was funded by the Warsaw University of Technology within the
Excellence Initiative: Research University (IDUB) programme. Computational
resources were provided by the Interdisciplinary Center for Mathematical and
Computational Modelling (ICM), University of Warsaw (Grant No. GB81-6), and
the High Performance Computing center “HPC Saulėtekis” in the Faculty of
Physics, Vilnius University.

## References

  * [1] Caldwell J D, Aharonovich I, Cassabois G, Edgar J H, Gil B and Basov D N 2019 Nature Reviews Materials 4 552–567 ISSN 2058-8437 URL https://doi.org/10.1038/s41578-019-0124-1
  * [2] Tran T T, Bray K, Ford M J, Toth M and Aharonovich I 2016 Nature Nanotechnology 11 37–41 ISSN 1748-3395 URL https://doi.org/10.1038/nnano.2015.242
  * [3] Sajid A, Ford M J and Reimers J R 2020 Reports on Progress in Physics 83 044501 URL https://dx.doi.org/10.1088/1361-6633/ab6310
  * [4] Aharonovich I, Tetienne J P and Toth M 2022 Nano Letters 22 9227–9235 pMID: 36413674 (Preprint eprint https://doi.org/10.1021/acs.nanolett.2c03743) URL https://doi.org/10.1021/acs.nanolett.2c03743
  * [5] Shevitski B, Gilbert S M, Chen C T, Kastl C, Barnard E S, Wong E, Ogletree D F, Watanabe K, Taniguchi T, Zettl A and Aloni S 2019 Phys. Rev. B 100(15) 155419 URL https://link.aps.org/doi/10.1103/PhysRevB.100.155419
  * [6] Zhigulin I, Yamamura K, Ivády V, Gale A, Horder J, Lobo C J, Kianinia M, Toth M and Aharonovich I 2023 Materials for Quantum Technology 3 015002 URL https://dx.doi.org/10.1088/2633-4356/acb87f
  * [7] Zhigulin I, Horder J, Ivády V, White S J, Gale A, Li C, Lobo C J, Toth M, Aharonovich I and Kianinia M 2023 Phys. Rev. Appl. 19(4) 044011 URL https://link.aps.org/doi/10.1103/PhysRevApplied.19.044011
  * [8] Fournier C, Plaud A, Roux S, Pierret A, Rosticher M, Watanabe K, Taniguchi T, Buil S, Quélin X, Barjon J, Hermier J P and Delteil A 2021 Nature Communications 12 3779 ISSN 2041-1723 URL https://doi.org/10.1038/s41467-021-24019-6
  * [9] Gale A, Li C, Chen Y, Watanabe K, Taniguchi T, Aharonovich I and Toth M 2022 ACS Photonics 9 2170–2177 (Preprint eprint https://doi.org/10.1021/acsphotonics.2c00631) URL https://doi.org/10.1021/acsphotonics.2c00631
  * [10] Chen Y, Gale A, Yamamura K, Horder J, Condos A, Watanabe K, Taniguchi T, Toth M and Aharonovich I 2023 Applied Physics Letters 123 041902 ISSN 0003-6951 (Preprint eprint https://pubs.aip.org/aip/apl/article-pdf/doi/10.1063/5.0155311/18056302/041902_1_5.0155311.pdf) URL https://doi.org/10.1063/5.0155311
  * [11] Ganyecz Á, Babar R, Benedek Z, Aharonovich I, Barcza G and Ivády V 2024 Nanoscale 16(8) 4125–4139 URL http://dx.doi.org/10.1039/D3NR05811E
  * [12] Kresse G and Furthmüller J 1996 Phys. Rev. B 54(16) 11169–11186 URL https://link.aps.org/doi/10.1103/PhysRevB.54.11169
  * [13] Blöchl P E 1994 Phys. Rev. B 50(24) 17953–17979 URL https://link.aps.org/doi/10.1103/PhysRevB.50.17953
  * [14] Heyd J, Scuseria G E and Ernzerhof M 2003 The Journal of Chemical Physics 118 8207–8215 ISSN 0021-9606 (Preprint eprint https://pubs.aip.org/aip/jcp/article-pdf/118/18/8207/19093575/8207_1_online.pdf) URL https://doi.org/10.1063/1.1564060
  * [15] Paszkowicz W, Pelka J B, Knapp M, Szyszko T and Podsiadlo S 2002 Applied Physics A 75 431–435 ISSN 1432-0630 URL https://doi.org/10.1007/s003390100999
  * [16] Tomaszkiewicz I 2002 Pol. J. Chem. 76 891–899
  * [17] Zhang S B and Northrup J E 1991 Phys. Rev. Lett. 67(17) 2339–2342 URL https://link.aps.org/doi/10.1103/PhysRevLett.67.2339
  * [18] Freysoldt C, Grabowski B, Hickel T, Neugebauer J, Kresse G, Janotti A and Van de Walle C G 2014 Rev. Mod. Phys. 86(1) 253–305 URL https://link.aps.org/doi/10.1103/RevModPhys.86.253
  * [19] Freysoldt C, Neugebauer J and Van de Walle C G 2009 Phys. Rev. Lett. 102(1) 016402 URL https://link.aps.org/doi/10.1103/PhysRevLett.102.016402
  * [20] Sun J, Ruzsinszky A and Perdew J P 2015 Phys. Rev. Lett. 115(3) 036402 URL https://link.aps.org/doi/10.1103/PhysRevLett.115.036402
  * [21] Furness J W, Kaplan A D, Ning J, Perdew J P and Sun J 2020 The Journal of Physical Chemistry Letters 11 8208–8215 pMID: 32876454 (Preprint eprint https://doi.org/10.1021/acs.jpclett.0c02405) URL https://doi.org/10.1021/acs.jpclett.0c02405
  * [22] Maciaszek M, Žalandauskas V, Silkinis R, Alkauskas A and Razinkovas L 2023 The Journal of Chemical Physics 159 084708 (Preprint eprint https://pubs.aip.org/aip/jcp/article-pdf/doi/10.1063/5.0154319/18102318/084708_1_5.0154319.pdf) URL https://doi.org/10.1063/5.0154319
  * [23] Grimme S 2006 Journal of Computational Chemistry 27 1787–1799 (Preprint eprint https://onlinelibrary.wiley.com/doi/pdf/10.1002/jcc.20495) URL https://onlinelibrary.wiley.com/doi/abs/10.1002/jcc.20495
  * [24] Dion M, Rydberg H, Schröder E, Langreth D C and Lundqvist B I 2004 Phys. Rev. Lett. 92(24) 246401 URL https://link.aps.org/doi/10.1103/PhysRevLett.92.246401
  * [25] Román-Pérez G and Soler J M 2009 Phys. Rev. Lett. 103(9) 096102 URL https://link.aps.org/doi/10.1103/PhysRevLett.103.096102
  * [26] Klimeš J c v, Bowler D R and Michaelides A 2011 Phys. Rev. B 83(19) 195131 URL https://link.aps.org/doi/10.1103/PhysRevB.83.195131
  * [27] Peng H, Yang Z H, Perdew J P and Sun J 2016 Phys. Rev. X 6(4) 041005 URL https://link.aps.org/doi/10.1103/PhysRevX.6.041005
  * [28] Mackoit-Sinkevičienė M, Maciaszek M, Van de Walle C G and Alkauskas A 2019 Applied Physics Letters 115 212101 ISSN 0003-6951 (Preprint eprint https://pubs.aip.org/aip/apl/article-pdf/doi/10.1063/1.5124153/13286483/212101_1_online.pdf) URL https://doi.org/10.1063/1.5124153
  * [29] Maciaszek M, Razinkovas L and Alkauskas A 2022 Phys. Rev. Mater. 6(1) 014005 URL https://link.aps.org/doi/10.1103/PhysRevMaterials.6.014005
  * [30] Auburger P and Gali A 2021 Phys. Rev. B 104(7) 075410 URL https://link.aps.org/doi/10.1103/PhysRevB.104.075410
  * [31] Geick R, Perry C H and Rupprecht G 1966 Phys. Rev. 146(2) 543–547 URL https://link.aps.org/doi/10.1103/PhysRev.146.543
  * [32] Weston L, Wickramaratne D, Mackoit M, Alkauskas A and Van de Walle C G 2018 Phys. Rev. B 97(21) 214104 URL https://link.aps.org/doi/10.1103/PhysRevB.97.214104
  * [33] Ziegler T, Rauk A and Baerends E J 1977 Theoretica chimica acta 43 261–271
  * [34] von Barth U 1979 Phys. Rev. A 20(4) 1693–1703 URL https://link.aps.org/doi/10.1103/PhysRevA.20.1693
  * [35] Lee S Y, Jeong T Y, Jung S and Yee K J 2019 physica status solidi (b) 256 1800417 (Preprint eprint https://onlinelibrary.wiley.com/doi/pdf/10.1002/pssb.201800417) URL https://onlinelibrary.wiley.com/doi/abs/10.1002/pssb.201800417
  * [36] Grudinin D V, Ermolaev G A, Baranov D G, Toksumakov A N, Voronin K V, Slavich A S, Vyshnevyy A A, Mazitov A B, Kruglov I A, Ghazaryan D A, Arsenin A V, Novoselov K S and Volkov V S 2023 Mater. Horiz. 10(7) 2427–2435 URL http://dx.doi.org/10.1039/D3MH00215B
  * [37] Alkauskas A, Lyons J L, Steiauf D and Van de Walle C G 2012 Phys. Rev. Lett. 109(26) 267401 URL https://link.aps.org/doi/10.1103/PhysRevLett.109.267401
  * [38] Udvarhelyi P, Nagy R, Kaiser F, Lee S Y, Wrachtrup J and Gali A 2019 Phys. Rev. Appl. 11(4) 044022 URL https://link.aps.org/doi/10.1103/PhysRevApplied.11.044022


11institutetext: University of Arizona, Department of Computer Science,
Tucson, AZ, USA 22institutetext: University of Arizona, Department of
Astronomy, Tucson, AZ, USA

# Large Language Models As MOOCs Graders

Shahriar Golchin 11 Nikhil Garuda 22 Christopher Impey 22 Matthew Wenger 22

###### Abstract

Massive open online courses (MOOCs) unlock the doors to free education for
anyone around the globe with access to a computer and the internet. Despite
this democratization of learning, the massive enrollment in these courses
means it is almost impossible for one instructor to assess every student’s
writing assignment. As a result, peer grading, often guided by a
straightforward rubric, is the method of choice. While convenient, peer
grading often falls short in terms of reliability and validity. In this study,
using 18 distinct settings, we explore the feasibility of leveraging large
language models (LLMs) to replace peer grading in MOOCs. Specifically, we
focus on two state-of-the-art LLMs: GPT-4 and GPT-3.5, across three distinct
courses: Introductory Astronomy, Astrobiology, and the History and Philosophy
of Astronomy. To instruct LLMs, we use three different prompts based on a
variant of the zero-shot chain-of-thought (Zero-shot-CoT) prompting technique:
Zero-shot-CoT combined with instructor-provided correct answers; Zero-shot-CoT
in conjunction with both instructor-formulated answers and rubrics; and Zero-
shot-CoT with instructor-offered correct answers and LLM-generated rubrics.
Our results show that Zero-shot-CoT, when integrated with instructor-provided
answers and rubrics, produces grades that are more aligned with those assigned
by instructors compared to peer grading. However, the History and Philosophy
of Astronomy course proves to be more challenging in terms of grading as
opposed to other courses. Finally, our study reveals a promising direction for
automating grading systems for MOOCs, especially in subjects with well-defined
rubrics.

###### Keywords:

Large Language Models (LLMs) Massive Open Online Courses (MOOCs) Grading
Automation.

##  1 Introduction

Massive open online courses (MOOCs)111https://www.mooc.org/ have
revolutionized access to education by offering a wide array of free courses to
anyone with an internet connection. This platform has democratized education
and made it globally accessible, accommodating an unlimited number of
participants [17, 18]. However, the resulting increase in participant numbers,
though beneficial to a broader audience, introduces challenges in delivering
personalized and constructive feedback on student performance, including the
task of grading their assignments. To address this issue, peer grading has
become a common practice. While peer grading enhances student engagement and
motivation, its reliability and validity are often questioned [8].

From the application of statistical methods [27, 13, 29, inter alia] to the
recent emergence of large language models (LLMs) in the realm of Natural
Language Processing, this field has significantly fertilized various other
disciplines [4, 2, 28, 30, inter alia] and has also paved the way for
addressing more complex tasks [3]. In this study, we investigate the potential
of LLMs in handling the task of grading assignments in MOOCs and the viability
of replacing the current peer grading system with LLMs on this platform,
aiming to establish a more streamlined and automated grading mechanism. This
potentially reduces human intervention in the grading process, thereby serving
more students with more accurate feedback and enhancing the overall learning
experience.

In this study, we employ three distinct prompting strategies to assess the
grading abilities of two state-of-the-art LLMs—GPT-4 [23] and GPT-3.5 [24]—in
three MOOC subjects: Introductory Astronomy, Astrobiology, and the History and
Philosophy of Astronomy [14, 15, 16]. Our methodology is primarily grounded on
utilizing a modified version of the zero-shot chain-of-thought prompting
(Zero-shot-CoT) technique [20], but we tweak this prompt by providing it with
a more in-context understanding of the given task. In particular, we augment
this prompt in a threefold manner: (1) we fortify Zero-shot-CoT with
instructor-provided answers; (2) we boost Zero-shot-CoT by integrating both
instructor-guided answers and grading rubrics; and (3) we enrich Zero-shot-CoT
with correct answers offered by the instructor but supplemented with LLM-
generated rubrics based on the correct answers. Our findings suggest that
GPT-4 using Zero-shot-CoT prompting with both instructor-provided answers and
rubrics outperforms GPT-3.5 and peer grading in terms of score alignment with
those given by instructors. However, grading courses that require imaginative
or speculative thinking, such as History and Philosophy of Astronomy, is found
to be a challenge for both LLMs and peer grading. Despite this, GPT-4 still
excels over peer grading by producing scores more closely aligned with those
given by the instructors in such scenarios.

The main contributions of our study are:

(1) We explore the potential of replacing the peer grading mechanism with the
use of LLMs in MOOCs, aiming for an automated and more accurate grading
process that minimizes human involvement, particularly in scenarios needing
grading feedback for a large number of participants.

(2) Our experiments encompass 18 distinct scenarios, applying two LLMs (GPT-4
and GPT-3.5) with three prompting techniques (Zero-shot-CoT plus correct
answers, Zero-shot-CoT complemented with correct answers and rubrics, and
Zero-shot-CoT with correct answers and LLM-presented rubrics) to real-world
data from three MOOC topics (Introductory Astronomy, Astrobiology, and the
History and Philosophy of Astronomy).

(3) When integrating Zero-shot-CoT with instructor-crafted correct answers and
rubrics, together with GPT-4, it outperforms all other settings, including
peer grading, in generating grades more aligned with those allocated by
instructors. However, our findings demonstrate that this alignment is more
pronounced in courses that require less creative and imaginative thinking.

##  2 Related Work

The existing foundation for this research stems from preceding studies that
explored peer review grading in MOOCs [8, 10]. These studies showed a
connection between peer review participation and elements such as student
engagement and course completion rates. They also revealed issues related to
inconsistency in grades and problems concerning reliability and validity
within this process.

Regarding the exploration of LLMs in educational settings, the primary focus
has been on their utilization by instructors and students to generate and
respond to educational content [19, 1]. One such exemplary application is at
Khan Academy, GPT-4 is employed as a learning assistance tool, aiding in
student learning via content-specific question-answering [22].

Despite its relevance, there is a noticeable gap in research that directly
parallels our study. Several studies [21, 9, 26] employed the classic encoder-
only component of Transformer network [5, 31], examining the grading of
students’ assignments within short question-answering contexts. Despite their
value, these works are confined to short question-answering settings and
necessitate the training of individual, subject-specific models, all the while
involving human interaction. The research bearing the closest resemblance to
our current endeavor is [25], wherein GPT-3.5 (integrated within the framework
of ChatGPT) is used in a corporate context, providing feedback on employee
performance metrics.

Our work, while centers around grading students’ assignments, is substantially
different from others in the sense that we are the first to use GPT-4 in
addition GPT-3.5 to study the possibility of replacing peer grading system in
MOOCs, eliminating the necessity for human involvement in the grading process.

##  3 Approach

###  3.1 Prompts

To guide LLMs in grading assignments, we formulate three distinct prompts
based on a variation of Zero-shot-CoT.222We refer to this as a variation of
Zero-shot-CoT since this version contains more information compared to the
original Zero-shot-CoT, presented in [20]. After conducting a series of
preliminary experiments, we found that the Zero-shot-CoT method yields scores
more closely aligned with those given by instructors in contrast to vanilla
zero-shot prompting. As a result, we chose to employ this prompting approach
throughout our study. Further, this methodology enables us to observe the
underlying reasoning for each score generated by the LLM, as the LLM is tasked
with speaking out its thoughts, thereby proactively mitigating potential
hallucinations during the grade generation process by the LLMs. We develop
three prompts, which incorporate Zero-shot-CoT in combination with: (1)
instructor-provided correct answers, (2) instructor-presented answers and
rubrics, and (3) correct answers furnished by the instructor, along with the
LLM-generated rubric. In the following, we provide a detailed explanation of
each prompt.

Zero-shot-CoT with Instructor-provided Answers. Initially, we implement Zero-
shot-CoT by incorporating the correct answers, supplied by course instructors,
as in-context information when prompting LLMs to generate grades for
assignments. Further, each student’s assignment is embedded into the input
prompt, and the LLM is tasked with grading based on the instructor-provided
answer for each question. It is important to note that within this prompt,
there is no guidance on how the grading should be carried out, and the only
information about the grading is the total points available for each question.
Finally, the grading is conducted for each student and question separately,
ensuring that there is no influence from other questions or the grades of
other students on the downstream grades generated by the LLM. Figure 1
illustrates the Zero-shot-CoT prompt with correct answers integrated.

Instruction: You are a fair and knowledgeable instructor whose task is to
evaluate the student’s assignments in accordance with the correct answers to
each of the questions that are presented in the section that follows. Make
sure you speak your thoughts aloud so that the students can understand the
rationale for any points deducted. Let’s grade assignments step by step. \- -
- Questions/Answers: QUESTION 1 Clearly identify the detection methods used to
gather data for each exoplanet. Briefly explain how each detection method
works. Correctly identify both detection methods. Clearly explain how each
detection method works. Correctly identify which physical characteristics can
be learned from each set of data, and explain why. Clearly identify physical
characteristics for both exoplanets. Clearly identify one exoplanet as Earth-
like. ANSWER 1 The radial velocity technique was used to gather data to create
the radial velocity versus time graphs for Star A and Star B. The radial
velocity technique measures the Doppler shift of the star’s absorption lines
that are caused by the exoplanet tugging on the star. Using the transit
technique would allow you to obtain a stellar brightness versus time graph,
like the graph for Star A. Transits take place when the exoplanet passes in
front of its star. There is a noticeable dip or drop in the brightness of the
star, caused by the exoplanet blocking a bit of the star’s light. Using the
radial velocity curves, we can determine both the orbital period of exoplanets
around their stars as well as the mass of the exoplanet. For Star A, the
period is about 200 days. For Star B, the period is about 6 days. As the
planet tugs the star toward and away from the observer, we can tell how long
it takes to go around the star and how hard it pulls on the star. We can infer
from the periods that Exoplanet B is closer to its star than Exoplanet A is to
its star. The larger radial velocity means that Exoplanet B is massive and
very close to Star B. Exoplanet B is likely a Hot Jupiter based on this data.
Using the transit data, you can estimate the radius or size of the exoplanet.
The ratio of exoplanet radius to star radius is about 0.014. This means one
exoplanet radius is about 0.014 of a star radius. The Earth is about 0.009 of
the Sun’s radius, creating a comparable ratio between Exoplanet A-Star A and
the Earth-Sun. Based on the data, Star A is the more likely candidate for an
Earth-like and habitable exoplanet. It has a period shorter than Earth’s, a
distance shorter than the Earth-Sun distance, and has a larger planet to star
ratio than the Earth to the Sun. In contrast, Star B has a dramatically
smaller period, and Exoplanet B is much closer to Star B, meaning it is not
likely to harbor life. Grade: 10/10 \- - - Student’s Answers: The methods used
to collect data from the exoplanets were the transit and radial velocity. The
transit method works through the monitoring of the starlight of a particular
star for periodic dimming since that is an indication that an exoplanet is
blocking out a portion of that starlight as it orbits, to apply this method
the orbital plane needs to be tilted towards the observer. The radial velocity
method measures the reflex motion of stars to detect exoplanets, because the
absorbed light is periodically red and then blue-shifted recurring each orbit,
the data can be added together. This measurement is done by taking several
spectra of the star and comparing them. […] {The answer is partially cut due
to space constraints.} Grade:  \- - - 9/10  The student has correctly
identified the detection methods used to gather data for each exoplanet and
explained how identified which physical characteristics can be learned from
each detection method  works. The student has also correctly each set of data
and explained why. However, the student made a small mistake in the analysis
of the radial velocity graphs. The student stated that the radial velocity  of
Star A lasts for nearly 275 days and Star B lasts for 11 days, which is
incorrect. The correct interpretation should be that the orbital period of
Exoplanet A is about 275 days and Exoplanet B is about 11 days. The student
correctly identified Exoplanet A as more Earth-like. Overall, the student
demonstrated a good understanding of the topic but made a minor error in
interpreting the data.

Figure 1: An illustration of the Zero-shot-CoT prompt along with answers
provided by the course instructor to grade students’ assignments. Each
question is assessed individually for every student. We repeat this process
for all questions and students, incorporating their answers into the prompt,
and tasking the LLM to grade their assignments.

Instruction: You are a fair and knowledgeable instructor whose task is to
evaluate the student’s assignments in accordance with the correct answers and
provided rubric to each of the questions that are presented in the section
that follows.  Make sure that the score deduction follows the provided rubric.
Let’s grade assignments step by step and explain the reasons behind the point
deduction given the rubric. \- - - Rubric: QUESTION 1 Correctly identify which
physical characteristics can be learned from each set of data, and explain
why. 2 points: The writer clearly identifies physical characteristics for both
exoplanets 1 point: The writer identifies physical characteristics of only one
exoplanet 0 points: The writer does not clearly identify physical
characteristics of either exoplanet Clearly identify the detection methods
used to gather data for each exoplanet. Briefly explain how each detection
method works. 4 points: The writer correctly identifies both detection
methods; clear explanation of how each detection method works 3 points: The
writer correctly identifies both detection methods; explanation for one or
both methods is unclear or incomplete 2 points: The writer correctly
identifies and explains one detection method; does not identify other
detection method, no explanation 1 point: The writer correctly identifies one
detection method; explanation of detection method unclear or incomplete 0
points: The writer does not identify or explain either detection method
Clearly identify one exoplanet as Earth-like 1 point: Writer clearly
identifies one exoplanet as Earth-like 0 points: Writer does not clearly
identify one exoplanet as Earth-like Student writes with sufficient clarity
and detail to communicate their points effectively 2 points: Writer presents
arguments in clear, logical way that demonstrates understanding of concepts;
correct use of scientific terms/language. Connections between concepts are
well developed. 1 point: Arguments attempt to address key concepts, some gaps
in logic or comprehension. Some use of scientific terms, majority used
correctly. 0 points: Arguments are difficult to follow, multiple flaws in
logic. Incorrect or no use of key terms. Connections between concepts not
present. The student uses data from the graphs to support their explanations.
1 point: Yes, the writer uses data to support their explanation. 0 points: No,
the writer does not use sufficient data to support their explanation.
Questions/Answers: {As Figure 1.} Grade: 10/10 \- - - Student’s Answers: {As
Figure 1.} Grade: \- - - 10/10 The student correctly identified both detection
methods and provided clear explanations of how each method works, earning 4
points. They also correctly identified physical characteristics for both
exoplanets, earning 2 points. The student clearly identified exoplanet A as
Earth-like, earning 1 point. The student’s arguments were presented in a
clear, logical way that demonstrated understanding of concepts, earning 2
points. Lastly, the student used data from the graphs to support their
explanations, earning 1 point. No points were deducted as the student met all
the criteria in the rubric.

Figure 2: A representation of the Zero-shot-CoT prompt that incorporates both
instructor-provided correct answers and grading rubrics for grading students’
assignments. The grading process utilized in the Zero-shot-CoT with correct
answers only (Figure 1) is also applied here.

Instruction: Your task is to design a rubric that addresses the following
questions/answers. The followings are the homework assignments for the
Astrobiology course. This course is designed for undergraduate students
majoring in Astronomy. The rubric must be unbiased and adaptable, capable of
fairly evaluating any kind of student writing assignment. The scoring
breakdown for each question should be as follows: the score for Question 1 is
10, the score for Question 2 is 10, and the score for Question 3 is 10. Your
thoughtful consideration in creating this rubric will ensure all students’
work is evaluated equitably and consistently. Make sure that the rubric will
provide points that are whole numbered. Each rubric should be dedicated to
each question separately. \- - - Questions/Answers: QUESTION 1 {As Figure 1.}
ANSWER 1 {As Figure 1.} Full Score: 10/10 QUESTION 2 Discuss how habitable
zone range and spectral type are related. […] {The question is partially cut
due to space constraints.} ANSWER 2 The habitable zone is defined as the range
of distances from a star within which liquid water can exist on a planet. […]
{The answer is partially cut due to space constraints.} Full Score: 10/10
QUESTION 3 Discuss why/whether it is plausible that exobiology exists at all,
in terms of exoplanet type, spectral type and orbital distance. […] {The
question is partially cut due to space constraints.} ANSWER 3 The star around
HD-127688 has a K spectral type. […] {The answer is partially cut due to space
constraints.} Full Score: 10/10 \- - - Rubric for Question 1: \-
Identification of detection methods (2 points): The student correctly
identifies the detection methods used for each exoplanet. \- Explanation of
detection methods (2 points): The student provides a clear and accurate
explanation of how each detection method works. \- Identification of physical
characteristics (2 points): The student correctly identifies the physical
characteristics that can be learned from each set of data. \- Explanation of
physical characteristics (2 points): The student provides a clear and accurate
explanation of why these physical characteristics can be learned from the
data. \- Identification of Earth-like exoplanet (2 points): The student
correctly identifies one exoplanet as Earth-like. Rubric for Question 2: \-
Discussion of habitable zone and spectral type (2 points): The student
provides a clear and accurate explanation of how the habitable zone range and
spectral type are related. \- Response to Yousef’s statement (2 points): The
student clearly states whether they agree or disagree with Yousef’s statement
and provides evidence and data to support their answer. \- Response to Lora’s
statement (2 points): The student clearly states whether they agree or
disagree with Lora’s statement and provides evidence and data to support their
answer. \- Use of data (2 points): The student effectively uses data to
support their answers. \- Clarity of explanation (2 points): The student’s
explanations are clear and logical. […] {The answer is partially cut due to
space constraints.}

Figure 3: A depiction of the prompt employed to generate a rubric using GPT-4
for the Astrobiology course. We repeat this process for all the courses under
study by replacing the course name, correct answer, total grade, and the
question. The generated rubric is then embedded into the prompt containing
Zero-shot-CoT and the correct answer for grading students’ writing
assignments. Specifically, the prompt displayed in Figure 2 is used for
grading, where the instructor-supplied rubric is substituted with an LLM-
produced rubric.

Zero-shot-CoT with Instructor-provided Answers and Rubrics. For our second
prompting technique, we build upon the previously mentioned prompt by
incorporating the rubrics presented by the instructors for each question. This
integration seeks to guide the LLM towards the grading methodology typically
employed by instructors, thereby achieving a higher extent of alignment with
the grades allocated by the instructors. Retaining all components from the
prior prompting technique, we include instructor-provided rubrics in the
prompt and follow the same grading procedure as in the previous stage. Figure
2 illustrates the enhancement of the Zero-shot-CoT prompt with the inclusion
of the correct answers and rubrics.

Zero-shot-CoT with Instructor-provided Answers and LLM-generated Rubrics.
Within this approach, we diverge from the other two previous methods where the
reference information in the prompt is centered around the human (instructor)
and instead, include a portion of information generated by the LLM. This idea
is inspired by the fact that an LLM can generate better rubrics compared to
those provided by a human, which can better align with human preferences. The
reasoning behind this is that LLMs are trained on an extensive amount of data,
and thus, possess comprehensive interdisciplinary knowledge, such as knowledge
from both education and astronomy in our case, that allows them to produce
improved rubrics. To implement this, we prompt GPT-4 (see Section 4 for more
details) and utilize the template prompt shown in Figure 3 to generate rubrics
based on the provided correct answer, total grade, and the question itself.
Once the rubric is generated, it replaces the rubric used in the previous
stage with the instructor-provided rubric (Figure 2), and the same process is
repeated to grade all students’ assignments.

###  3.2 Bootstrap Resampling

As previously discussed, MOOCs cater to a large number of participants for
each course, and our study aims to evaluate the effectiveness of LLMs in
grading assignments for a substantial student population. However, due to
budget limitations of our project when using proprietary LLMs, such as GPT-4,
as well as the need for careful analysis of the grades assigned by LLMs to
individual questions, we conduct the grading process using LLMs on a more
manageable subset of the entire collection of students’ assignments (refer to
Section 4 for further details). On the other hand, it remains crucial to
establish the performance of LLMs in grading a significant volume of
assignments, representative of the regular situations in MOOCs. To address
this requirement based on the manageable subset of assignments we examine, we
use the bootstrap resampling technique [6, 7]. Ultimately, we evaluate the
performance of LLMs in assigning grades compared to those given by the
instructors by analyzing the average grades generated per question in each
course.

###  3.3 Baseline

We analyze average scores produced by the LLMs for each question across all
courses, comparing them to the peer grading as the baseline and the
instructor-assigned grades as the ground truth. By evaluating the average
scores with respect to the ground truth, our goal is to identify the most
effective type of prompt and its corresponding LLM. In this context, a prompt,
along with its associated LLM, is deemed superior if it demonstrates greater
alignment with the instructor-assigned grades compared to those obtained
through peer grading.

##  4 Experimental Setup

Data. This study employs data gathered from three MOOCs: Introductory
Astronomy, Astrobiology, and the History and Philosophy of Astronomy [14, 15,
16]. The process of obtaining the data followed a strict ethical roadmap,
guided by an Institutional Review Board from our university. The participants
willingly filled out a survey and formally consented to allow their course
data to be used for research. All data related to the students was de-
identified and purposefully sampled to represent the full range of assignment
grades.

Although we did our best to avoid bias in the sampling process, we are aware
that our courses might not be universally representative of the global
population, due to the inherent limitations in subsampling from the entire
population. They are, however, representative of the audience on Coursera as a
whole. The majority of students in these MOOCs are between the ages of 25 and
44. A small majority of students are male in all enrolled courses with the
smallest difference between gender in the Astrobiology course (45% female
versus 54% male) and the greatest difference in gender in the History and
Philosophy course (34% female versus 65% male). Geographically, the United
States and India are the homes of most students. Finally, since all data are
fully de-identified, the assignment selection is minimally influenced by
gender or geography.

Regarding the attributes of the assignments, students are instructed to
compose their answers using no more than 500 words. Nevertheless, this
constraint on the submission length is not strictly imposed. Specifically, the
Introductory Astronomy course comprises five assignments, each worth 9 points
except for the first question, which carries 6 points. The Astrobiology course
features three questions, each valued at 10 points, while the Philosophy and
History of Astronomy course contains four questions, each with 4 points.

In all of our experiments within the paper, we select a subset consisting of
10 writing assignments submitted by students for every course. Moreover, for
implementing bootstrap resampling, we resample 10,000 samples. It is worth
noting that the data used in this research was sourced from a proprietary
repository, automatically factoring in the issue of data contamination in LLMs
[12, 11].

Setting. In all of our experiments involving GPT-4 and GPT-3.5, we utilize the
gpt-4-0613 and gpt-3.5-turbo-0613 snapshots, which are accessed through the
OpenAI API. We set the temperature to zero and the maximum token limit to
2048, tasking LLMs with generating grades for each question across all three
courses. All mentioned hyperparameters are maintained consistently throughout
the evaluation of all prompts and across all LLMs in our experiments.

Instructor Grading. In our study, we consider the grades assigned by
instructors (university professors) to be our ground truth. These professors
are subject matter experts in the fields relevant to each course, and
meticulously evaluate the assignments under consideration in this study. The
grading process by the instructors is conducted in a double-blind manner,
i.e., the instructors do not hold any knowledge regarding peer grading or the
students’ personal information.

Peer Grading. For each writing assignment, three to four fellow classmates are
randomly selected to evaluate the work using a grading rubric provided by the
course instructor. This rubric is identical to the one utilized by the
instructor for grading. Although the grading process is conducted anonymously
to ensure impartiality, the final grade for each assignment is determined by
calculating the median of the scores received from the peer graders.

LLM-generated Rubrics. We utilize GPT-4 to generate rubrics for questions
(assignments) in each course using the template prompt shown in Figure 3.
These rubrics are generated based on the specific questions for each course,
the respective correct answers supplied by the instructors, and the total
score for each question. After the rubrics are generated, they are integrated
into the template prompt containing Zer-shot-CoT and correct answers for
grading, as displayed in Figure 2. It is worth mentioning that all the rubrics
produced by GPT-4 are entirely independent of any rubrics provided by the
instructors.

Table 1: Average grades from grading 10 students’ writing assignments using
the three proposed prompting techniques: Zero-shot-CoT with instructor-
provided answers (Instr. Ans.), Zero-shot-CoT combined with both instructor-
supplied answers and rubrics (Instr. Ans. & Instr. Rub.), and Zero-shot-CoT
incorporating the instructor-provided answers and LLM-generated rubrics
(Instr. Ans. & LLM Rub.). The results encompass three MOOC subjects:
Introductory Astronomy (Astr.), Astrobiology (A. Bio.), and the History and
Philosophy of Astronomy (H.P.A.). For each prompt, we employ both GPT-3.5 and
GPT-4 to generate grades. The generated grades are compared to instructor-
assigned grades (instr. grades) and peer grades.

GPT-3.5 GPT-4 Courses Q Instr. Grades Peer Grades Instr. Ans. Instr. Ans. &
Instr. Rub. Instr. Ans. & LLM Rub. Instr. Ans. Instr. Ans. & Instr. Rub.
Instr. Ans. & LLM Rub. Astr. Q1 3.90 5.15  4.50 2.90 4.10  4.75 4.40 4.40 Q2
8.20 7.55  7.60 8.30 8.40  8.65 8.30 8.50 Q3 7.50 7.40  6.80 7.20 6.60  7.60
7.30 7.60 Q4 7.40 7.45  6.80 7.10 6.80  7.50 6.90 7.05 Q5 5.50 7.40  6.40 7.40
6.40  6.20 5.90 6.35 A. Bio. Q1 6.80 7.50  7.40 6.70 6.58  7.50 7.10 7.10 Q2
6.70 7.45  7.00 7.30 5.52  7.90 7.40 7.10 Q3 7.90 9.05  6.70 6.50 5.02  8.10
7.50 7.50 H.P.A. Q1 3.50 3.60  2.70 2.00 2.85  3.50 3.20 3.20 Q2 2.40 3.70
2.90 1.80 2.56  3.25 3.10 2.95 Q3 2.70 3.40  2.70 1.20 1.70  3.65 3.20 3.20 Q4
2.20 3.80  3.00 1.10 2.20  3.25 2.70 2.95

##  5 Results and Discussion

Detailed in Table 1 and Table 2 are the outcomes of our evaluation of
students’ writing assignments using two LLMs—GPT-4 and GPT-3.5—based on a
suite of three unique prompts crafted within this study. Table 1 displays the
grades directly obtained from grading the assignments of 10 students included
in our research, while Table 2 showcases the results after the implementation
of bootstrap resampling. Our examination unfolds several key insights:

(1) Both Table 1 and Table 2 reveal that GPT-4 overall outperforms GPT-3.5 by
generating grades that are more closely aligned with the grades provided by
instructors, especially in the context of the History and Philosophy of
Astronomy course. This remains constant across all three prompts assessed.

(2) Focusing on the prompts, ones that integrate both instructor-supplied
rubrics and LLM-produced rubrics give rise to grades that are better aligned
with grades given by instructors. This is true for both LLMs: GPT-4 and
GPT-3.5.

(3) In almost all scenarios, GPT-4, when prompted with instructor-provided
answers and rubrics, generates grades that are either superior to or on par
with peer grades. This even includes the History and Philosophy of Astronomy
course, which requires speculative/imaginative thinking abilities.

(4) As expected, the LLM-produced rubrics, on average, as displayed in Table
2, can perform on the same level as the instructor-provided rubrics. This
finding is interesting as it suggests that LLMs could be used for rubrics
generation in educational settings with the purpose of automating the entire
grading process, only requiring the provision of correct answers.

(5) From the results listed in both Table 1 and Table 2, it is clear that the
most difficult course to grade is the History and Philosophy of Astronomy.
Both LLMs and peer graders find it challenging to produce grades that are in
close alignment with those assigned by instructors. However, even in these
cases, the superior performance of GPT-4 remains apparent as its grades are
closer to grades assigned by instructors compared to those allocated by peer
grading.

Table 2: Average grades from grading 10 students’ writing assignments based on
bootstrap resampling with 10,000 resampled samples. Each average grade
includes a subscripted standard deviation. Similar to Table 1, we employ three
different prompting strategies: Zero-shot-CoT with instructor-provided answers
(Instr. Ans.), Zero-shot-CoT combined with both instructor-presented answers
and rubrics (Instr. Ans. & Instr. Rub.), and Zero-shot-CoT incorporating the
instructor-provided answers and LLM-generated rubrics (Instr. Ans. & LLM
Rub.), across three MOOC topics: Introductory Astronomy (Astr.), Astrobiology
(A. Bio.), and the History and Philosophy of Astronomy (H.P.A.). For each
experiment with each prompt, we utilize both GPT-3.5 and GPT-4. The generated
grades are contrasted with the instructor-provided grades (instr. grades) and
peer grades.

GPT-3.5 GPT-4 Courses Q Instr. Grades Peer Grades Instr. Ans. Instr. Ans. &
Instr. Rub. Instr. Ans. & LLM Rub. Instr. Ans. Instr. Ans. & Instr. Rub.
Instr. Ans. & LLM Rub. Astr. Q1 $3.90_{\pm 0.54}$ $5.15_{\pm 0.27}$ $4.50_{\pm
0.16}$ $2.90_{\pm 0.61}$ $4.10_{\pm 0.61}$ $4.75_{\pm 0.41}$ $4.40_{\pm 0.41}$
$4.40_{\pm 0.48}$ Q2 $8.20_{\pm 0.37}$ $7.55_{\pm 0.75}$ $7.60_{\pm 0.29}$
$8.30_{\pm 0.25}$ $8.40_{\pm 0.21}$ $8.65_{\pm 0.20}$ $8.30_{\pm 0.28}$
$8.50_{\pm 0.27}$ Q3 $7.51_{\pm 0.92}$ $7.41_{\pm 0.82}$ $6.81_{\pm 0.75}$
$7.21_{\pm 0.85}$ $6.61_{\pm 0.78}$ $7.61_{\pm 0.87}$ $7.31_{\pm 0.86}$
$7.61_{\pm 0.87}$ Q4 $7.41_{\pm 0.86}$ $7.45_{\pm 0.46}$ $6.81_{\pm 0.78}$
$7.11_{\pm 0.91}$ $6.81_{\pm 0.94}$ $7.51_{\pm 0.77}$ $6.91_{\pm 0.90}$
$7.06_{\pm 0.89}$ Q5 $5.51_{\pm 0.94}$ $7.40_{\pm 0.83}$ $6.41_{\pm 0.78}$
$7.40_{\pm 0.91}$ $6.41_{\pm 1.10}$ $6.21_{\pm 0.90}$ $5.91_{\pm 1.06}$
$6.36_{\pm 1.10}$ A. Bio. Q1 $6.81_{\pm 1.09}$ $7.50_{\pm 0.79}$ $7.41_{\pm
0.84}$ $6.71_{\pm 1.18}$ $6.60_{\pm 0.94}$ $7.50_{\pm 0.70}$ $7.11_{\pm 0.95}$
$7.11_{\pm 0.85}$ Q2 $6.71_{\pm 0.83}$ $7.46_{\pm 1.02}$ $7.01_{\pm 0.76}$
$7.31_{\pm 0.83}$ $5.53_{\pm 1.30}$ $7.91_{\pm 0.68}$ $7.41_{\pm 0.83}$
$7.11_{\pm 0.98}$ Q3 $7.89_{\pm 0.85}$ $9.04_{\pm 0.52}$ $6.70_{\pm 0.71}$
$6.49_{\pm 0.85}$ $5.02_{\pm 1.35}$ $8.10_{\pm 0.33}$ $7.50_{\pm 0.41}$
$7.50_{\pm 0.29}$ H.P.A. Q1 $3.50_{\pm 0.21}$ $3.60_{\pm 0.21}$ $2.70_{\pm
0.14}$ $2.00_{\pm 0.00}$ $2.85_{\pm 0.20}$ $3.50_{\pm 0.21}$ $3.20_{\pm 0.31}$
$3.20_{\pm 0.22}$ Q2 $2.39_{\pm 0.29}$ $3.69_{\pm 0.20}$ $2.90_{\pm 0.22}$
$1.80_{\pm 0.31}$ $2.56_{\pm 0.25}$ $3.25_{\pm 0.23}$ $3.10_{\pm 0.17}$
$2.95_{\pm 0.23}$ Q3 $2.70_{\pm 0.20}$ $3.40_{\pm 0.38}$ $2.69_{\pm 0.32}$
$1.20_{\pm 0.42}$ $1.70_{\pm 0.38}$ $3.65_{\pm 0.14}$ $3.20_{\pm 0.19}$
$3.20_{\pm 0.18}$ Q4 $2.20_{\pm 0.28}$ $3.80_{\pm 0.19}$ $3.00_{\pm 0.14}$
$1.10_{\pm 0.33}$ $2.20_{\pm 0.19}$ $3.25_{\pm 0.24}$ $2.70_{\pm 0.28}$
$2.95_{\pm 0.23}$

##  6 Conclusion

We introduced three distinct prompts based on the zero-shot chain-of-thought
(Zero-shot-CoT) prompting technique for utilizing LLMs to automate the grading
of students’ assignments in educational platforms with substantial student
populations, such as MOOCs. Specifically, our method consists of Zero-shot-CoT
with instructor-supplied correct answers, Zero-shot-CoT augmented with
instructor-offered answers and rubrics, and Zero-shot-CoT combined with
correct answers provided by the instructors and LLM-generated rubrics.

We examined the performance of each proposed prompting strategy using two
state-of-the-art LLMs—GPT-4 and GPT-3.5—by grading assignments from three
MOOCs: Introductory Astronomy, Astrobiology, and the History and Philosophy of
Astronomy. Our results suggested that the integration of Zero-shot-CoT with
instructor-presented correct answers and rubrics provided by either the
instructor or LLM, when using GPT-4 as the underlying foundation model,
surpasses peer grading across all subjects. We also discovered that grading in
courses demanding reasoning and imaginative thinking skills, such as the
History and Philosophy of Astronomy, is more difficult in comparison to
others.

In summary, we found LLMs more effective than peer grading for deriving grades
that closely align with the ones assigned by instructors on high-traffic
educational platforms.

## References

  * [1] Alseddiqi, M., AL-Mofleh, A., Albalooshi, L., Najam, O.: Revolutionizing online learning: The potential of chatgpt in massive open online courses. European Journal of Education and Pedagogy (2023). https://doi.org/10.24018/ejedu.2023.4.4.686
  * [2] Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., Do, Q.V., Xu, Y., Fung, P.: A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. CoRR abs/2302.04023 (2023). https://doi.org/10.48550/ARXIV.2302.04023, https://doi.org/10.48550/arXiv.2302.04023
  * [3] Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y.T., Li, Y., Lundberg, S., et al.: Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023)
  * [4] Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y.T., Li, Y., Lundberg, S.M., Nori, H., Palangi, H., Ribeiro, M.T., Zhang, Y.: Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR abs/2303.12712 (2023). https://doi.org/10.48550/ARXIV.2303.12712, https://doi.org/10.48550/arXiv.2303.12712
  * [5] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [6] Efron, B.: Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics 7(1), 1 – 26 (1979). https://doi.org/10.1214/aos/1176344552, https://doi.org/10.1214/aos/1176344552
  * [7] Efron, B., Tibshirani, R.J.: An Introduction to the Bootstrap. No. 57 in Monographs on Statistics and Applied Probability, Chapman & Hall/CRC, Boca Raton, Florida, USA (1993)
  * [8] Formanek, M., Wenger, M.C., Buxner, S.R., Impey, C.D., Sonam, T.: Insights about large-scale online peer assessment from an analysis of an astronomy mooc. Computers & Education 113, 243–262 (2017). https://doi.org/https://doi.org/10.1016/j.compedu.2017.05.019, https://www.sciencedirect.com/science/article/pii/S0360131517301240
  * [9] Funayama, H., Sato, T., Matsubayashi, Y., Mizumoto, T., Suzuki, J., Inui, K.: Balancing cost and quality: An exploration of human-in-the-loop frameworks for automated short answer scoring. Cornell University - arXiv (2022). https://doi.org/10.48550/arxiv.2206.08288
  * [10] Gamage, D., Gamage, D., Staubitz, T., Staubitz, T., Whiting, M.E., Whiting, M.E.: Peer assessment in moocs: Systematic literature review. Distance Education (2021). https://doi.org/10.1080/01587919.2021.1911626
  * [11] Golchin, S., Surdeanu, M.: Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models. arXiv e-prints arXiv:2311.06233 (Nov 2023). https://doi.org/10.48550/arXiv.2311.06233
  * [12] Golchin, S., Surdeanu, M.: Time travel in llms: Tracing data contamination in large language models. In: The Twelfth International Conference on Learning Representations (2023)
  * [13] Hassan, R., Bandi, C., Tsai, M.T., Golchin, S., P D, S.M., Rafatirad, S., Salehi, S.: Automated supervised topic modeling framework for hardware weaknesses. In: 2023 24th International Symposium on Quality Electronic Design (ISQED). pp. 1–8 (2023). https://doi.org/10.1109/ISQED57927.2023.10129378
  * [14] Impey, C.: Astrobiology: Exploring other worlds, https://www.coursera.org/learn/astrobiology-exploring-other-worlds
  * [15] Impey, C.: Astronomy: Exploring time and space, https://www.coursera.org/learn/astro
  * [16] Impey, C.: Knowing the universe: History and philosophy of astronomy, https://www.coursera.org/learn/knowing-the-universe
  * [17] Impey, C., Wenger, M., Formanek, M., Buxner, S.: Bringing the universe to the world: lessons learned from a massive open online class on astronomy. Communicating Astronomy with the Public 21, 20–30 (2016)
  * [18] Impey, C.D., Wenger, M.C., Austin, C.L.: Astronomy for astronomical numbers: A worldwide massive open online class. The International Review of Research in Open and Distributed Learning 16(1) (Feb 2015). https://doi.org/10.19173/irrodl.v16i1.1983, https://www.irrodl.org/index.php/irrodl/article/view/1983
  * [19] Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., Stadler, M., Weller, J., Kühn, J., Kasneci, G.: Chatgpt for good? on opportunities and challenges of large language models for education. Learning and Individual Differences (2023). https://doi.org/10.1016/j.lindif.2023.102274
  * [20] Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners. In: Advances in Neural Information Processing Systems. vol. 35, pp. 22199–22213 (2022)
  * [21] Morris, W., Crossley, S.A., Holmes, L., Trumbore, A.: Using transformer language models to validate peer-assigned essay scores in massive open online courses (moocs). International Conference on Learning Analytics and Knowledge (2023). https://doi.org/10.1145/3576050.3576098
  * [22] OpenAI: (Mar 2023), https://openai.com/customer-stories/khan-academy
  * [23] OpenAI: GPT-4 technical report. CoRR abs/2303.08774 (2023). https://doi.org/10.48550/ARXIV.2303.08774, https://doi.org/10.48550/arXiv.2303.08774
  * [24] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P.F., Leike, J., Lowe, R.: Training language models to follow instructions with human feedback. In: NeurIPS (2022), http://papers.nips.cc/paper\\_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html
  * [25] Pinto, G., Cardoso-Pereira, I., Ribeiro, D.B., Lucena, D., Souza, A., Gama, K.: Large language models for education: Grading open-ended questions using chatgpt. Brazilian Symposium on Software Engineering (2023). https://doi.org/10.1145/3613372.3614197
  * [26] Riordan, B., Riordan, B., Horbach, A., Horbach, A., Cahill, A., Cahill, A., Zesch, T., Zesch, T., Lee, C.M., Lee, C.M.: Investigating neural architectures for short answer scoring. BEA@EMNLP (2017). https://doi.org/10.18653/v1/w17-5017
  * [27] Shamshiri, A., Ryu, K.R., McCullough, S., Park, J.Y.: Constory: Automatic story investigator of public perception on the mega urban infrastructure project: poster abstract. In: Proceedings of the 9th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation. p. 293–294. BuildSys ’22, Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3563357.3567751, https://doi.org/10.1145/3563357.3567751
  * [28] Shamshiri, A., Ryu, K.R., Park, J.Y.: Text mining and natural language processing in construction. Automation in Construction 158, 105200 (2024). https://doi.org/https://doi.org/10.1016/j.autcon.2023.105200, https://www.sciencedirect.com/science/article/pii/S0926580523004600
  * [29] Tavabi, N., Pruneski, J., Golchin, S., Singh, M., Sanborn, R., Heyworth, B., Kimia, A., Kiapour, A.: Building large-scale registries from unstructured clinical notes using a low-resource natural language processing pipeline. medRxiv (2022)
  * [30] Tavabi, N., Raza, M., Singh, M., Golchin, S., Singh, H., Hogue, G.D., Kiapour, A.M.: Disparities in cannabis use and documentation in electronic health records among children and young adults. npj Digit. Medicine 6 (2023). https://doi.org/10.1038/S41746-023-00885-W, https://doi.org/10.1038/s41746-023-00885-w
  * [31] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)