# Planar defects as a way to account for explicit anharmonicity in high
temperature thermodynamic properties of silicon.

M. V. Kondrin<EMAIL_ADDRESS>Institute for High Pressure Physics
RAS, 108840 Troitsk, Moscow, Russia  Y.B. Lebed  Institute for Nuclear
Research RAS, Moscow, Russia  V.V. Brazhkin  Institute for High Pressure
Physics RAS, 108840 Troitsk, Moscow, Russia

###### Abstract

Silicon is indispensable in semiconductor industry. Understanding its high-
temperature thermodynamic properties is essential both for theory and
applications. However, first-principle description of high-temperature
thermodynamic properties of silicon (thermal expansion coefficient and
specific heat) is still incomplete. Strong deviation of its specific heat at
high temperatures from the Dulong-Petit law suggests substantial contribution
of anharmonicity effects. We demonstrate, that anharmonicity is mostly due to
two transverse phonon modes, propagating in (111) and (100) directions, and
can be quantitatively described with formation of the certain type of
nanostructured planar defects of the crystal structure. Calculation of these
defects’ formation energy enabled us to determine their input into the
specific heat and thermal expansion coefficient. This contribution turns out
to be significantly greater than the one calculated in quasi-harmonic
approximation.

##  I Introduction

Silicon is a widely used semiconductor with immense impact in industry.
Understanding its high-temperature thermodynamic properties is necessary both
for theory and applications. Also, due to its archetypal diamond structure,
silicon is among the touchstones for testing new theories. Still, there are
black spots in its first principle description, in particular, high-
temperature thermodynamic properties. In large extent this is due to the fact
that high-temperature properties are connected to anharmonicity effects, which
are difficult to describe by ab-initio methods alone.

In fully harmonic approximation both isochoric specific heat $c_{V}$ and
isobaric thermal expansion coefficient $\alpha_{V}$ in solids at the limit of
high temperatures both should converge to constant value Landau  _et al._
(1980). For $c_{V}$ this is a result of the classical Dulong-Petit law. In the
latter case the constant value of $\alpha_{V}$ at high temperature is the
result of quasiharmonic approximation where free energy taken as the sum of
potential energy of atoms assumed to be confined by the harmonic potential and
phonons is minimized at certain temperature with respect to volume. This
results in equation Landau  _et al._ (1980):

$\alpha_{V}=\frac{\gamma c_{V}}{B_{0}V}$ (1)

Here $\gamma=-\frac{V}{\Theta}\frac{d\Theta}{dV}$ is the Grüneisen parameter
which in zeroth approximation is a constant independent of temperature, $V$ is
specific volume and $B_{0}=-V\frac{dP}{dV}$ is a bulk modulus at zero
pressure. $\Theta$ in the definition of the Grüneisen parameter can be taken
as characteristic frequency of phonon spectrum like, for example, the Debye
temperature. Still, this is an oversimplification, because $\gamma$ can be
different for different phonon modes (even negative for transverse acoustic
modes in silicon and germanium which lead to negative thermal expansion in
these crystals at low enough temperature Swenson (1983)), therefore parameter
$\gamma$ acquires dependence on temperature. Nonetheless, at temperatures
above the frequency of the highest phonon mode $\gamma$ tends to constant
value and, so, the expansion coefficient does the same.

In quasiharmonic approximation $c_{V}$ at high temperature does approach 3$R$
level and the specific heat at constant pressure $c_{P}$ (which in contrast to
$c_{V}$ is usually measured in experiments) is connected to $c_{V}$ by
thermodynamic identity:

$c_{P}-c_{V}=\alpha^{2}_{V}B_{0}VT$ (2)

, where $\alpha_{V}$ – volume thermal expansion coefficient, $V$ – specific
volume , $T$ \- temperature.

In solids, due to comparatively low value of $\alpha_{V}$, the difference
between $c_{P}$ and $c_{V}$ at melting temperature usually does not exceed few
percents and therefore can be neglected. However, in real solid materials both
high temperature expansion coefficients and isochoric heat capacity deviate
from constants, as a result of explicit anharmonicity effectsBorn and Brody
(1921); Wallace (1965); Cowley (1968). Precise values of this deviations in
silicon, prescribed by quasiharmonic approximation, are not strictly known,
because in various measurements of high temperature specific heat Gerlich  _et
al._ (1965); Desai (1986); Yamaguchi and Itagaki (2002) and expansion
coefficients Swenson (1983); Maissel (1960); Watanabe  _et al._ (2004); Dutta
(1962); Okada and Tokumaru (1984); Roberts (1981) differ from one another by
about 10 %, but all of them are definitely not constant. Possible origin of
this scatter we will discuss below.

Ab-initio computation of anharmonic effects at finite temperatures requires
either resource-demanding ab-initio molecular dynamics simulations (see e.g.
the computation of thermodynamic parameters of aluminum Grabowski  _et al._
(2009)) or, under some simplifying assumptions which let to sample potential
energy surface of material in vicinity of equilibrium point, by strictly DFT
methods at zero temperature (so called stochastic-initialized temperature-
dependent effective potential/s-TDEP method Kim  _et al._ (2018)). The latter
approach was recently applied to describe thermal expansion coefficient of
silicon Kim  _et al._ (2018). Still, in this case we do not agree with authors
remark that “a simple physical model for the anomalous thermal expansion of
silicon is unlikely because different effects contribute to the thermal
expansion. In particular, anharmonicity and nuclear quantum effects are
difficult to formulate as a simple 3D model”Kim  _et al._ (2018). In fact, we
are able to demonstrate, that major part of high temperature anharmonicity in
silicon is connected to its’ few (namely two) anharmonic modes, which lead to
formation of metastable state (planar defect) with relatively low energy.

This analogy between defect and anharmonic phonon oscillation was proposed in
the number of our recent works Kondrin  _et al._ (2020, 2021, 2022) and was
successfully used to resolve discrepancies in experimental thermodynamic
properties of graphite near the melting temperature Savvatimskii and Onufriev
(2020); Savvatimskiy  _et al._ (2016); Kondratyev and Rakhel (2019); Kondrin
_et al._ (2021). In the nutshell, this analogy relates strong anharmonicity to
finite displacement of atom layers caused by shear phonon modes, and
subsequent reconstruction of interatomic bonds. It is this reconstruction
which leads to drastic lowering of potential energy at certain values of atom
displacement, and can be effectively described as anharmonicity of interatomic
potential. On the other hand, reconstruction of interatomic bonds can be
regarded as extended planar defects, and its thermodynamic properties can be
calculated using formation energy value of this defect. This approach has its
strong sides because it can be developed by zero temperature ab-initio theory
and requires knowledge of only few lowest energy defects which mostly
contribute to thermodynamic properties of material. We will demonstrate, that
in silicon only two phonon modes have to be taken into account to successfully
describe its high-temperature thermodynamic properties (both thermal expansion
and specific heat).

##  II Explicit anharmonicity in solids

It should be noted, that in general, the expansion coefficient of silicon is
regarded to be small at the temperatures below 1000 K, so silicon was proposed
as the expansion coefficient standard Swenson (1983); Watanabe  _et al._
(2004). Nonetheless, at high temperatures it increases due to explicit
anharmonicity Born and Brody (1921); Wallace (1965); Cowley (1968).
Considering anharmonicity effects it is appropriate to discern two types of
anharmonicity – a uniform one (which we call bulk anharmonicity), and the
transverse one, which is due to bond reconstruction, caused by transverse
phonon modes mentioned above. Bulk anharmonicity is caused by softening of
elastic constant at high temperatures. Although, in some works this
contribution is called quasiharmonic, but it heavily depends on anharmonicity
of elastic modulus on pressure(volume) change, so it includes purely
anharmonic input. This effect in silicon is evident from experimental
measurements of Young’s moduli along different directions Vanhellemont  _et
al._ (2014) and speed of sound at various directions Goncharova  _et al._
(1983). Still, in quasiharmonic approximation this effect can be described by
substitution of $B_{0}V$ term in Eq. (1) by the temperature dependent
analogue. So, bulk modulus at finite temperatures can be replaced by the
relation $B=B_{0}+B^{\prime}\Delta P=B_{0}-B^{\prime}B_{0}\Delta
V/V=B_{0}-B^{\prime}B_{0}\int\limits_{0}^{T}\alpha_{V}dT$. Here, $B^{\prime}$
is the pressure derivative of bulk modulus which can also be obtained by zero
temperature ab-initio calculations. Obviously, in this approximation $V$
should be replaced by $V+V\int\limits_{0}^{T}\alpha_{v}^{0}(T)dT$. So, the
contribution to thermal expansion coefficient due to isotropic softening of
bulk modulus can be handled by iterative procedure, that is the next iteration
of expansion coefficient can be written as ($\alpha_{V}^{0}$ is just the
quasiharmonic approximation described by Eq. 1):

$\alpha_{V}^{1}=\alpha_{V}^{0}(1+(B^{\prime}-1)\int\limits_{0}^{T}\alpha_{v}^{0}(T)dT)$
(3)

The difference between bulk and transverse anharmonicity can be related to the
difference between quasiharmonic and explicit anharmonicity. The latter is
supposed to lead to the thermal dependence of phonon modes at constant volume.
Still we should point out that the transverse anharmonicity leads also to
another manifestation namely to the random splitting of phonon modes, leading
to broadening of phonon dispersion curves. This effect is observed in
experiment Kim  _et al._ (2015).

Figure 1: Various theoretical and experimental data of linear expansion
coefficient of silicon. Red dashed curve – quasiharmonic approximation uniform
expansion coefficient given by Eq: 3. $\times$ – experimental data by Okada
et. al. Okada and Tokumaru (1984) measured by X-Ray technique. $\circ$ –
experimental data of Maissel Maissel (1960) measured by optical technique.
Magenta dotted curve – theoretical contribution from (111) planar defects.
Green solid curve – sum of two inputs to linear expansion coefficient from
uniform dilatation and expansion caused by (111) planar defects. Note that the
last curve is not a fit of experimental data because it contains no fitting
parameters bat rather a comparison of theoretical and experimental results.

If we take the value of the temperature dependent Grüneisen parameter from ab-
initio calculation for pure silicon done before Wei  _et al._ (1994), and
values of $B^{\prime}\approx 4$ and $B_{0}\approx 90$ Wang  _et al._ (2015)
the value of $\alpha^{1}_{V}(T)$ will not deviate much from constant at high
temperature up to the melting point (see Fig. 1). Its value is even lower than
the expansion coefficient obtained by X-Ray measurements by Okada et al. Okada
and Tokumaru (1984). Therefore, we should seek for another contribution to the
expansion coefficient of silicon. We argue, that this previously unaccounted
for contribution to thermal expansion coefficient of silicon can be described
by taking into consideration only two planar defects.

Figure 2: Silicon (111) (panel a) and (100) (panel b) defect structures. Atoms
of pristine silicon slabs are shown by solid green, and defect layer atoms -
by translucent color. For (111) structure the two different interface layers
are shown by different color.

\begin{overpic}[width=212.47617pt]{si-111.png}
\put(10.0,90.0){\fcolorbox{black}{white}{\Large a)}} \end{overpic}

\begin{overpic}[width=212.47617pt]{si-100.png}
\put(10.0,90.0){\fcolorbox{black}{white}{\Large b)}} \end{overpic}

These two proposed earlier defect structures Kondrin  _et al._ (2020) are
depicted in Fig. 2. They can be regarded as two “crumpled” monoatomic layers,
aligned along (111) and (100) direction (so they were named (111) and (100)
defect structures respectively), separated by slabs of pristine crystal
structures. They are related to M-carbon Oganov and Glass (2006); Li  _et al._
(2009) and S-carbon He  _et al._ (2012). The last (100) structure of planar
defect in diamond was considered before Goss  _et al._ (2006). Computer
simulation of amorphous carbon and its subsequent deconvolution into “basic”
crystal structures demonstrated, that combination of crumpled layers with 5-,
7- membered cycles (similar to defects considered here) and pristine
diamond/lonsdeleite diamond slabs has the lowest energy Deringer  _et al._
(2017). In our consideration it is important to keep in mind, that the defect
itself (crystal layers located in between two interface sheets shown in Fig. 2
by translucent color) may be shifted with respect to the surrounding crystal.
On the other hand, pristine crystal slabs separated by the defect should not
shift relative to each other, so their position remains the same as in initial
pure crystal. In other words, thin planar defect (consisting of few atomic
layers) can be created by finite transverse shift in the bulk of infinitely
thick crystal by not perturbing the relative position of atomic layers in the
pristine crystal. This requirement allows to create infinitesimal number of
thin defects in the limit of low temperatures, when the defect concentration
is low. Another important note is, that the two interface sheets are identical
in the case of (100) defect and different in case of (111) defect. This
observation is related to different crystal symmetry of the corresponding
defect structures – it is monoclinic in case of (111) defect and orthorhombic
in case of (100) one.

As it was demonstrated in Ref. Kondrin  _et al._ (2022), the concentration of
interface sheets is described by the formula:

$x=\left(\exp\left(\frac{\Delta E_{1}+\Delta
E_{2}}{k_{B}T}\right)+1\right)^{-1}$ (4)

where $\Delta E_{i}$ is a formation energy of one of interface sheets per the
number of atoms in the sheet. Note that concentration of interface sheets in
the limit of infinitely high temperature converges to 0.5, that is, to evenly
mixed layers of pristine crystal structure, separated by interface atomic
sheets.

From concentration of the defects, we can calculate their contribution to
various thermodynamic properties of the defected crystal:

$\Delta c_{P}=(\Delta E_{1}+\Delta E_{2})\frac{dx}{dT}$ (5)

and

$\Delta\alpha_{V}=\frac{\Delta V_{1}+\Delta V_{2}}{V}\frac{dx}{dT}$ (6)

For this purpose we have to know only the formation energy of the defect
$\Delta E_{i}$ and its effect on the specific volume of the crystal $\Delta
V_{i}$. We should stress one more time, that these planar defects are purely
virtual and can be regarded as another way to treat explicit anharmonicity of
the crystal. In this way, anharmonic contribution to the thermodynamic
properties acquires obvious meaning as a change of crystal energy and volume
caused by the defects. Note, that calculation of these properties does require
nothing more except the ab-initio calculations at zero temperature.

It is interesting to note, that beforehand we don’t know, which type of
anharmonicity (bulk or transverse one) produces higher effect on thermodynamic
properties of particular crystal. It seems that bulk input dominates in
metals. See e.g. Ref. Grabowski  _et al._ (2009), where it was demonstrated,
that high-temperature specific heat and expansion coefficient in aluminum are
mainly described by bulk anharmonicity. It seems, that in covalently bonded
materials the situation is opposite, and the large part of anharmonicity
present in them is due to transverse anharmonicity (like in germanium Kondrin
_et al._ (2022)). More than half a century ago, there was a discussion about
“microscopic and macroscopic” expansion of solids measured by different
methods (by macroscopic dilatometry and microscopic X-Ray diffraction, see
e.g. Dutta (1962)). Involuntarily, we are about to provide another
(thermodynamic) dimension for this discussion , as the difference between bulk
and transverse anharmonicity.

##  III Methods

To calculate the formation energy of defect silicon structure we used
QuantumESPRESSO software packageGiannozzi  _et al._ (2017). For preparation of
input files we heavily used `cif2cell` program Björkman (2011).

For the density functional calculation of formation energy, we employed the
Perdew-Burke-Ernzerhof exchange correlation method with norm-conserving
pseudopotentials with energy cutoff 70 Ry. The planar defects were simulated
in approximately 48-atom supercell. For integration over Brillouin zone,
unshifted $4\times 8\times 1$ Monkhorst-Pack grid was used (with the number of
nodes being roughly inversely proportional to the length in a corresponding
direction). Crystal lattices and atom positions at fixed external pressure
were fully optimized, until residual force on every atom did not exceed 0.001
Ry/bohr and additional stress – 0.5 kbar. The estimated convergence error does
not exceed 2 meV/atom. The formation energy of the defect was calculated by
the energy difference between defected structure ($E$) and defect-free silicon
lattice ($E_{host}$) of similar dimensions:

$\Delta E=E-E_{host}n/n_{host}$ (7)

where $n$ and $n_{host}$ are the number of atoms in defected and pristine
silicon. It should be noted, that for planar defects, not the formation energy
defined this way, but rather its normalized value (by the layer interface area
or the number of atoms, comprising the interface layers) has physical meaning.
We will call it a specific formation energy.

##  IV Results

The specific formation energy $\Delta E_{1}+\Delta E_{2}$ is equal to 190
meV/atom (29 meV//Å2)) for (111) defect and 355 meV/atom (48 meV/Å2) for (100)
defect. Interesting to note that despite this strong difference between
respective energies per interface atom due to strong asymmetry of (111) defect
the “upper limit on the melting temperature” (the energy barrier which has to
be overcame by shear displacement of atomic layers with respect to one another
Kondrin  _et al._ (2020)) set by both types of defects is about the same.
Since the one of the interface sheets of (111) defect consists of purely
lonsdeleite type of interatomic bonds (see Fig. 2), whose energy is close to
that of pristine silicon. Consequently, almost all of additional energy in
this defect is introduced by exactly half of interface atoms. So, this upper
limit (190 meV/atom $\approx$ 2190 K/atom) is only slightly larger than
experimentally observed melting temperature (1690 K).

The additional volume, introduced by defects, requires slightly subtler
calculations. The defects with concentration of interface sheets equal to
$x=1/6$ that depicted in Fig. 2 have relative volume expansion $\frac{\Delta
V}{xV}$ (which has to be substituted in Eq. 6) equal to 0.03642 (for (111)
defect) and 0.08439 (for (100) defect). However, in this calculation it was
implicitly assumed, that all defects are aligned along the single
crystallographic direction, and two types of defects are formed independently
of each other. Therefore, we have to divide relative volume expansion by the
number of equivalent crystallographic direction for respective defect type (4
for (111) defect and 3 for (100) defect). To take into account the difference
between defects, we assume, that they independently occupy the crystal volume,
so, the volume is equally distributed by the two types of defects (
corresponding “occupation factor” is equal to 0.5 for both defects). For
simplicity, we also assume that change of volume is caused by the change of
lengths in the direction, normal to interface sheets. This can be explained by
that in the low temperature limit (low defect concentrations) the area along
the defect planes is determined by the area of rather thick layers of pristine
silicon, which more or less retains geometry of pristine silicon crystal, and
all changes in volume are restricted to the direction, normal to planar
defects. This observation is corroborated by inspection of relative crystal
lattice change in (111) defect with concentration 1/6. Taking into account all
these considerations, we can write down the final equation for linear thermal
expansion along the single (111) direction (the value 1100 in the formula
$\approx 2196/2$):

$\Delta l/l(111)=0.5\cdot 0.036\cdot 1100/(2T^{2}\cosh(1100/T)^{2})/4$ (8)

The similar equation with obvious corrections can be applied to linear
expansion along (100) direction.

Figure 3: Xray diffraction profiles for pristine silicon (red $+$) and silicon
with (111) planar defects with concentration $x=1/6$ (blue $\times$). X-ray
wavelength was assumed to be KCuα.

Eq. 8 is depicted in Fig. 1 as thick green solid curve. For drawing this line
we also include uniform bulk contribution from the softening of interatomic
bonds (shown as red dashed curve in Fig. 1). We have compared it with
experimental data of Maissel Maissel (1960) , where expansion coefficient
along (111) direction of single crystal of silicon was measured by optical
methods. We can see good coincidence of amplitude of predicted and measured
expansion coefficient, and especially, correspondence of position of maximum
on the expansion coefficient (at $\approx$ 900 K) observed for measured and
calculated curves. Note also the difference between the values of linear
expansion, measured by optical dilatometer and X-ray method. We argue, that
X-ray method measures only uniform dilatation, caused by bulk anharmonicity
although by the way it somewhat underestimates it. On the other hand, direct
optical methods measure “true” dilatation, which also includes contribution
from transverse anharmonic modes. The difference between the two methods is
evident if one draws X-Ray diffraction patterns from pristine silicon and
silicon with (111) planar defects with concentration $x=1/6$ (see Fig. 3). It
can be easily concluded from inspection of his Figure, that accumulation of
planar defects leads to splitting of X-ray reflections and eventually their
broadening, rather than shift. Note, that exact values of splitting and
relative amplitudes of individual peaks are rather spurious issue because they
heavily depend on exact position of interface layers, so only the relative
broadening of reflections which takes into account the related groups of peaks
is meaningful in this case. Because of incoherent character of diffraction
from planar defects randomly distributed along the bulk of the crystal, they
can be compared to stacking faults and twins Balogh  _et al._ (2006) leading
to broadening not the whole Bragg peak but rather of its “pedestal”. Over this
base the comparatively narrow coherent reflection is superimposed. We should
also remind, that significant broadening of optical phonon dispersion branches
measured by inelastic neutron scattering Kim  _et al._ (2015) as well as
optically active Raman modes Hart  _et al._ (1970) were previously registered
in experiments. So, we insist that it is the relative broadening of X-Ray
peaks rather than change of their position, that enables one to figure out the
thermal expansion due to anharmonicity of transverse phonon modes. It should
be taken into account , that concentration of interface layers along one
direction used for simulation of X-ray diffraction pattern in Fig. 3
corresponds to very high temperature above the experimental melting
temperature of silicon.

Figure 4: Various theoretical and experimental data of linear expansion
coefficient of silicon. Red dashed curve – quasiharmonic approximation uniform
expansion coefficient given by Eq: 3. $\circ$ – experimental data of Watanabe
et al. Watanabe  _et al._ (2004) measured along (100) direction by optical
technique. Magenta dotted curve – theoretical contribution from (100) planar
defects. Green solid curve – sum of two inputs to linear expansion coefficient
from uniform dilatation and expansion caused by (100) planar defects. Note
that the last curve is not a fit of experimental data because it contains no
fitting parameters bat rather a comparison of theoretical and experimental
results.

In similar way can be handled thermal expansion along (100) direction (see
Fig. 4). In this case for comparison we have chosen the most recent optical
data of Watanabe et al. Watanabe  _et al._ (2004). It is clearly seen, that
our approximation significantly exceeds experimental expansion coefficient,
especially in the high temperature region. Still, for making the final
decision about validity of our theory and its compliance to experiment, even
more high temperature experimental data above 1000 K is needed. Up to now, to
the best of our knowledge, no optical measurements of expansion coefficient of
silicon to the temperatures up to the melting point exist.

Figure 5: Isobaric heat capacity of silicon. $\ast$ – recommended experimental
data of Desai Desai (1986). Red curve – theoretical result including Debye
contribution (with TD=640 K and anharmonic input from two types of planar
defects (111) and (100). Dashed black line – calculation with Morse potential
by Trivedi et al. Trivedi  _et al._ (1977). Note that the last two curves are
not fits of experimental data because they contain no fitting parameters bat
rather a comparison of theoretical and experimental results.

Using relative concentration of (111) and (100) defects and their respective
specific formation energy we can also evaluate isobaric specific heat (see
Fig. 5). Beside anharmonic contribution we added harmonic input described by
Debye equation with TD=640 K. For comparison we used the recommended
experimental data of Desai Desai (1986). Experimental dependence is more or
less reproduced by theoretical law with exception of high-temperature range
where calculated data are below the experimental ones. The difference between
high temperature data can be well explained by presence of small concentration
of even more high-energy planar defects namely (110) one Kondrin  _et al._
(2020) with specific formation energy 1.5 times higher, than that of (100)
defect. We can also mention, that newer data of specific heat obtained by drop
calorimeter methods Yamaguchi and Itagaki (2002) produce substantially lower
specific heat values. Earlier we have explained this underestimation of high
temperature specific heat measured by drop calorimeters by the effect of
planar defects quenching during cooling Kondrin  _et al._ (2022). The most
reliable method of determination of specific heat would be “one-shot
measurement” like pulse-heating one Savvatimskii and Onufriev (2020);
Savvatimskiy  _et al._ (2016); Kondratyev and Rakhel (2019).

##  V Discussion

We would like to briefly discuss the similarities and differences between our
approach and s-TDEP method used earlier for determination of thermal expansion
of silicon Kim  _et al._ (2018). Both methods are fully ab-initio zero
temperature ones. The difference between them is that s-TDEP method used
stochastic sampling of 3$N$ coordinated (where $N$ is the number of atoms in
calculation supercell) potential surface and subsequent fitting of obtained
PES by third-order polynomial. We believe, that although this method by itself
is sound, there are some implicit assumptions in its fitting procedure. This
procedure takes into account only two-phonon interactions and neglects higher
order ones. Still, from this fitting and subsequent perturbative methods, it
is very difficult to obtain anharmonic contribution to specific heat.
Moreover, as stated in Ref. Cowley (1968), the lowest order and highest order
contributions to specific heat have large amplitude but different signs. Thus,
the series are slow to converge.

As a side remark¿ it is interesting to recall rather old research of Trivedi
et al. Trivedi  _et al._ (1977). They calculate high temperature anharmonic
contribution to specific heat of elements with diamond structure, taking into
account only terms up to fourth order, but with artificially chosen
interatomic potential (namely the Morse potential). The result of their
calculation is shown in Fig. 5. It is clear, that their calculations heavily
underestimate the trend, observed in experiments. We believe, that this
discrepancy is due to the fact, that the Morse potential significantly
underestimates anharmonicity present in real silicon.

On the other hand, our approach is not perturbative and takes into account
only the modes, producing metastable local minima at finite displacement with
quite low energies. In other words, we assume, that the strongest input into
anharmonic thermodynamic properties is due to “topologically” different modes
(having different number of maxima/minima from harmonic potential). In some
detail it was discussed in Ref. Kondrin  _et al._ (2022). Despite not very
strict formulation of the method, these local minima can be found as the
energy lowering, produced by bond reconstruction. Taking into account the
great success recently achieved by various methods of ab-initio search for new
crystal structures Oganov and Glass (2006), this approach (as we believe) can
be easily computerized. Example of Ref. Deringer  _et al._ (2017), where two
of planar defects in diamonds similar to the one considered here was found by
purely computer procedure, convinces us in eligibility of such an approach.

In some extent, this approach can be compared to the one taken earlier Hart
_et al._ (1970); Menéndez and Cardona (1984); Debernardi  _et al._ (1995),
where temperature dependence of Raman mode linewidth in silicon was
interpreted in terms of anharmonic decay of $\Gamma$ optical phonon into two
acoustical phonons with opposite wavevectors. It was shown earlier by
measuring second order Raman under high temperatures Klotz  _et al._ (1997);
Brazhkin  _et al._ (2000) and ultrasonic measurements of softening of phonon
modes under high pressure Goncharova  _et al._ (1983), that it is the
transverse acoustic phonon mode at the edge of Brillouine zone in X point,
which is most influenced by anharmonic effects. This anharmonicity is in some
way transferred to optical phonons, which are prone to decay into this
(transverse acoustical) channel.

From more philosophical standing point, duality anharmonicity-defect can be
regarded as an extension of de Broigle’s duality wave-particle to phonons. As
with fundamental particles, the particle-like properties of phonons are
realized, then there is an inter-phonon coupling that is anharmonicity, which
in its turn leads to formation of crystal defects. It is like an electron,
which acquires definite coordinate only when interacting with detector
material . Although, we can not provide direct conclusions from this duality,
but it may be interesting to dig into this analogy deeper.

##  VI Conclusions

Using analogy between phonon anharmonicity and crystal defects and the result
of rather inexpensive zero temperature ab-initio calculations, we are able to
describe high-temperature thermal expansion coefficient and isobaric specific
heat of silicon. We have demonstrated, that anharmonicity of only two
transverse phonon modes (propagating along (111) and (100) directions)
contributes to these thermodynamic properties at high-temperatures. This
calculations accounts for more than 10% deviation of experimentally measured
thermodynamic properties in the vicinity of melting point of silicon from the
values obtained by quasi-harmonic approximation.

## Acknowledgements

This work is supported by the Russian Science Foundation (grant No.
19-12-00111).

## References

  * Landau  _et al._ (1980) L. Landau, L. Pitaevskii, and E. Lifshitz,  _Statistical Physics_ , Course of theoretical physics (Pergamon Press, Oxford, 1980).
  * Swenson (1983) C. A. Swenson, Journal of Physical and Chemical Reference Data 12, 179 (1983).
  * Born and Brody (1921) M. Born and E. Brody, Zeitschrift für Physik 6, 132 (1921).
  * Wallace (1965) D. C. Wallace, Phys. Rev. 139, A877 (1965).
  * Cowley (1968) R. A. Cowley, Reports on Progress in Physics 31, 123 (1968).
  * Gerlich  _et al._ (1965) D. Gerlich, B. Abeles, and R. E. Miller, Journal of Applied Physics 36, 76 (1965).
  * Desai (1986) P. D. Desai, Journal of Physical and Chemical Reference Data 15, 967 (1986).
  * Yamaguchi and Itagaki (2002) K. Yamaguchi and K. Itagaki, Journal of Thermal Analysis and Calorimetry 69, 1059 (2002).
  * Maissel (1960) L. Maissel, Journal of Applied Physics 31, 211 (1960).
  * Watanabe  _et al._ (2004) H. Watanabe, N. Yamada, and M. Okaji, International Journal of Thermophysics 25, 221 (2004).
  * Dutta (1962) B. N. Dutta, physica status solidi (b) 2, 984 (1962).
  * Okada and Tokumaru (1984) Y. Okada and Y. Tokumaru, Journal of Applied Physics 56, 314 (1984).
  * Roberts (1981) R. B. Roberts, Journal of Physics D: Applied Physics 14, L163 (1981).
  * Grabowski  _et al._ (2009) B. Grabowski, L. Ismer, T. Hickel, and J. Neugebauer, Phys. Rev. B 79, 134106 (2009).
  * Kim  _et al._ (2018) D. S. Kim, O. Hellman, J. Herriman, H. L. Smith, J. Y. Y. Lin, N. Shulumba, J. L. Niedziela, C. W. Li, D. L. Abernathy, and B. Fultz, Proceedings of the National Academy of Sciences 115, 1992 (2018).
  * Kondrin  _et al._ (2020) M. Kondrin, Y. Lebed, and V. Brazhkin, Diamond and Related Materials 110, 108114 (2020).
  * Kondrin  _et al._ (2021) M. V. Kondrin, Y. B. Lebed, and V. V. Brazhkin, Phys. Rev. Lett. 126, 165501 (2021).
  * Kondrin  _et al._ (2022) M. Kondrin, Y. Lebed, and V. Brazhkin, physica status solidi (b) 259, 2100463 (2022).
  * Savvatimskii and Onufriev (2020) A. I. Savvatimskii and S. V. Onufriev, Phys. Usp. 63, 1015 (2020).
  * Savvatimskiy  _et al._ (2016) A. Savvatimskiy, S. Onufriev, and A. Kondratyev, Carbon 98, 534 (2016).
  * Kondratyev and Rakhel (2019) A. M. Kondratyev and A. D. Rakhel, Phys. Rev. Lett. 122, 175702 (2019).
  * Vanhellemont  _et al._ (2014) J. Vanhellemont, A. K. Swarnakar, and O. V. der Biest, ECS Transactions 64, 283 (2014).
  * Goncharova  _et al._ (1983) V. A. Goncharova, E. V. Chernysheva, and F. F. Voronov, Fizika Tverdogo Tela 25, 3680 (1983), (in Russian).
  * Kim  _et al._ (2015) D. S. Kim, H. L. Smith, J. L. Niedziela, C. W. Li, D. L. Abernathy, and B. Fultz, Phys. Rev. B 91, 014307 (2015).
  * Wei  _et al._ (1994) S. Wei, C. Li, and M. Y. Chou, Phys. Rev. B 50, 14587 (1994).
  * Wang  _et al._ (2015) C. Wang, J. Gu, X. Kuang, and S. Xiang, Zeitschrift für Naturforschung A 70 (2015), 10.1515/zna-2015-0027, https://dx.doi.org/10.1515/zna-2015-0027.
  * Oganov and Glass (2006) A. R. Oganov and C. W. Glass, The Journal of Chemical Physics 124, 244704 (2006).
  * Li  _et al._ (2009) Q. Li, Y. Ma, A. R. Oganov, H. Wang, H. Wang, Y. Xu, T. Cui, H.-K. Mao, and G. Zou, Phys. Rev. Lett. 102, 175506 (2009).
  * He  _et al._ (2012) C. He, L. Sun, C. Zhang, X. Peng, K. Zhang, and J. Zhong, Solid State Communications 152, 1560 (2012).
  * Goss  _et al._ (2006) J. P. Goss, P. R. Briddon, R. Jones, and M. I. Heggie, Phys. Rev. B 73, 115204 (2006).
  * Deringer  _et al._ (2017) V. L. Deringer, G. Csányi, and D. M. Proserpio, ChemPhysChem 18, 873 (2017).
  * Giannozzi  _et al._ (2017) P. Giannozzi, O. Andreussi, T. Brumme, O. Bunau, M. B. Nardelli, M. Calandra, R. Car, C. Cavazzoni, D. Ceresoli, M. Cococcioni, and et. al., Journal of Physics: Condensed Matter 29, 465901 (2017).
  * Björkman (2011) T. Björkman, Computer Physics Communications 182, 1183 (2011).
  * Balogh  _et al._ (2006) L. Balogh, G. Ribárik, and T. Ungár, Journal of Applied Physics 100, 023512 (2006).
  * Hart  _et al._ (1970) T. R. Hart, R. L. Aggarwal, and B. Lax, Phys. Rev. B 1, 638 (1970).
  * Trivedi  _et al._ (1977) P. C. Trivedi, H. O. Sharma, and L. S. Kothari, Journal of Physics C: Solid State Physics 10, 3487 (1977).
  * Menéndez and Cardona (1984) J. Menéndez and M. Cardona, Phys. Rev. B 29, 2051 (1984).
  * Debernardi  _et al._ (1995) A. Debernardi, S. Baroni, and E. Molinari, Phys. Rev. Lett. 75, 1819 (1995).
  * Klotz  _et al._ (1997) S. Klotz, J. M. Besson, M. Braden, K. Karch, P. Pavone, D. Strauch, and W. G. Marshall, Phys. Rev. Lett. 79, 1313 (1997).
  * Brazhkin  _et al._ (2000) V. V. Brazhkin, S. G. Lyapin, I. A. Trojan, R. N. Voloshin, A. G. Lyapin, and N. N. Mel’nik, JETP Letters 72, 195 (2000).


Figure 1. Three methods to connect user interface components to large language
models. 1) static prompts are predefined prompts that can be selected directly
from the UI, 2) template-based prompts generate prompts based on selected
options in the UI, 3) free-form prompts provide a direct way of interacting
with prompts.

figure description

# Generating Diverse Explanations of Code Snippets using GPT-3

Stephen MacNeil  Temple University1801 N Broad StPhiladelphiaPAUSA19122
<EMAIL_ADDRESS>,  Andrew Tran  Temple University1801 N Broad
StPhiladelphiaPAUSA19122<EMAIL_ADDRESS>,  Joanne Kim  Temple
University1801 N Broad StPhiladelphiaPAUSA19122<EMAIL_ADDRESS>,
Ziheng Huang  University of California—San Diego9500 Gilman DriveLa
JollaCAUSA92093<EMAIL_ADDRESS>,  Seth Bernstein  Temple University1801 N
Broad StPhiladelphiaPAUSA19122<EMAIL_ADDRESS>and  Dan Mogil
Temple University1801 N Broad StPhiladelphiaPAUSA19122<EMAIL_ADDRESS>
(2022)

# Prompt Templates: Generating Prompts for Large Language Model based on UI
Affordances

Stephen MacNeil  Temple University1801 N Broad StPhiladelphiaPAUSA19122
<EMAIL_ADDRESS>,  Andrew Tran  Temple University1801 N Broad
StPhiladelphiaPAUSA19122<EMAIL_ADDRESS>,  Joanne Kim  Temple
University1801 N Broad StPhiladelphiaPAUSA19122<EMAIL_ADDRESS>,
Ziheng Huang  University of California—San Diego9500 Gilman DriveLa
JollaCAUSA92093<EMAIL_ADDRESS>,  Seth Bernstein  Temple University1801 N
Broad StPhiladelphiaPAUSA19122<EMAIL_ADDRESS>and  Dan Mogil
Temple University1801 N Broad StPhiladelphiaPAUSA19122<EMAIL_ADDRESS>
(2022)

# Prompt Middleware: Generating Prompts for Large Language Models based on UI
Affordances

Stephen MacNeil  Temple University1801 N Broad StPhiladelphiaPAUSA19122
<EMAIL_ADDRESS>,  Andrew Tran  Temple University1801 N Broad
StPhiladelphiaPAUSA19122<EMAIL_ADDRESS>,  Joanne Kim  Temple
University1801 N Broad StPhiladelphiaPAUSA19122<EMAIL_ADDRESS>,
Ziheng Huang  University of California—San Diego9500 Gilman DriveLa
JollaCAUSA92093<EMAIL_ADDRESS>,  Seth Bernstein  Temple University1801 N
Broad StPhiladelphiaPAUSA19122<EMAIL_ADDRESS>and  Dan Mogil
Temple University1801 N Broad StPhiladelphiaPAUSA19122<EMAIL_ADDRESS>
(2022)

# Prompt Middleware: Mapping Prompts for Large Language Models to UI
Affordances

Stephen MacNeil  Temple University1801 N Broad StPhiladelphiaPAUSA19122
<EMAIL_ADDRESS>,  Andrew Tran  Temple University1801 N Broad
StPhiladelphiaPAUSA19122<EMAIL_ADDRESS>,  Joanne Kim  Temple
University1801 N Broad StPhiladelphiaPAUSA19122<EMAIL_ADDRESS>,
Ziheng Huang  University of California—San Diego9500 Gilman DriveLa
JollaCAUSA92093<EMAIL_ADDRESS>,  Seth Bernstein  Temple University1801 N
Broad StPhiladelphiaPAUSA19122<EMAIL_ADDRESS>and  Dan Mogil
Temple University1801 N Broad StPhiladelphiaPAUSA19122<EMAIL_ADDRESS>
(2022)

large language models, prompt middleware, prompt programming

††conference: Make sure to enter the correct conference title from your rights
confirmation email; Oct 20–Nov 2, 2022; Bend, Oregon††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††journalyear: 2022††copyright:
rightsretained††conference: …; Oct 20–Nov 2, 2022; Bend, Oregon††booktitle: …
(UIST 2022), August 7–11, 2022, Bend, Oregon††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Abstract

To help users do complex work, researchers have developed techniques to
integrate AI and human intelligence into user interfaces (UIs). With the
recent introduction of large language models (LLMs), which can generate text
in response to a natural language prompt, there are new opportunities to
consider how to integrate LLMs into UIs. We present Prompt Middleware, a
framework for generating prompts for LLMs based on UI affordances. These
include prompts that are predefined by experts (static prompts), generated
from templates with fill-in options in the UI (template-based prompts), or
created from scratch (free-form prompts). We demonstrate this framework with
FeedbackBuffet, a writing assistant that automatically generates feedback
based on a user’s text input. Inspired by prior research showing how templates
can help non-experts perform more like experts, FeedbackBuffet leverages
template-based prompt middleware to enable feedback seekers to specify the
types of feedback they want to receive as options in a UI. These options are
composed using a template to form a feedback request prompt to GPT-3. We
conclude with a discussion about how Prompt Middleware can help developers
integrate LLMs into UIs.

##  2\. Introduction

Previous research has demonstrated ways that intelligence can be integrated
into UI (Bernstein et al., 2010; Dahlbäck et al., 1993; Maulsby et al., 1993;
Kulkarni et al., 2012; Kittur et al., 2012). In ‘Wizard of Oz’ systems, an
expert manually controls UI features to simulate an intelligent user interface
(Dahlbäck et al., 1993; Maulsby et al., 1993). Similarly, crowdsourcing
systems, such as Soylent (Bernstein et al., 2010), integrate crowdworkers to
power UIs through crowd workflows (Kulkarni et al., 2012; Kittur et al.,
2012). Finally, specialized machine learning models have also been trained for
a specific task and then embedded into systems and interfaces (Völkel et al.,
2020). Across these systems, rules, heuristics, workflows, and specialized
models guide the ways that interface affordances can be enhanced with
intelligence.

Recent advances in natural language processing have resulted in large language
models (LLMs), such as GPT-3 (Brown et al., 2020), which have the ability to
understand natural language prompts and generate relevant text responses.
These models are already being used to facilitate creative work (Huang et al.,
2023; Ding et al., 2023; Di Fede et al., 2022; Yuan et al., 2022; Mirowski et
al., 2023). However, it is not yet clear how to best integrate LLMs into
existing UI. In this paper, we explore three methods for integrating LLMs into
UI using prompts that are predefined by experts (static prompts), generated
from templates with fill-in options in the UI (template-based prompts), or
created from scratch (free-form prompts). These three techniques for
integrating LLMs into UIs, which we call Prompt Middleware, provide varying
amounts of control and guidance to users over the underlying prompt generation
process. We demonstrate the concept of Prompt Middleware by developing
FeedbackBuffet, a writing assistant that generates feedback for users by
guiding them through a menu of feedback options allowing them to determine the
type of feedback they would like to receive. FeedbackBuffet implements the
‘template prompt’ middleware to package these feedback options into a prompt
for GPT-3.

##  3\. Prompt Middleware: Connecting UI Affordances to LLMs

Crafting high-quality prompts is challenging (Jiang et al., 2022; Reynolds and
McDonell, 2021). To help people create high-quality prompts, PromptMaker
guides users to create their own prompts with templates and procedural
guidance (Jiang et al., 2022). Another approach called AI Chaining simplifies
a complex prompting process by splitting a request into smaller requests which
are individually prompted and then stitched back together (Wu et al., 2022).
This approach was shown to improve performance and transparency.

Where previous work has focused on making prompt engineering easier, these
approaches have not yet addressed two crucial aspects: 1) techniques to
scaffold domain expertise into the prompting process, and 2) directly
integrating LLMs into user interfaces. We propose Prompt Middleware as a
framework for achieving these two goal by mapping options in the UI to
generate prompts for an LLM. Prompt Middleware acts as a middle layer between
the LLM and the UI, while also embedding domain expertise into the prompting
process. The UI abstracts away the complexity of the prompts and separates
concerns between a user completing their tasks and the prompts that might
guide LLMs to help them in those tasks. Summarized in Figure 1, the following
sections introduce three Prompt Middleware types: static, template-based, and
free-form.

###  3.1. Static prompts leverage best practices

Prompt engineering and few-shot learning are common techniques to improve the
quality of responses from LLMs (Reynolds and McDonell, 2021). We propose the
concept of static prompts as a method for making these best practices
available to users in a UI. A static prompt is a predefined prompt generated
by experts through prompt engineering to achieve high-quality responses from
an LLM. As shown in Figure 1, static prompts can be hidden behind a button in
a UI to send a predefined prompt to an LLM on behalf of the user. This allows
users to tap into best practices with minimal effort but at the cost of giving
up control of prompt generation.

###  3.2. Template-based prompts provide flexibility

Previous researchers have shown how expertise and best practices can be
directly embedded into templates to guide crowdworkers (Kim et al., 2015),
non-experts (MacNeil et al., 2021, 2023; Hui et al., 2018; Yuan et al., 2016),
and even experts (Gawande, 2009) to do better work. For example, Motif, a
video storytelling application, leverages storytelling patterns to guide
users’ story creation (Kim et al., 2015). Inspired by how templates can guide
people, we explore how expert templates might similarly guide LLMs. We propose
template-based prompts as a method for generating prompts by filling in a pre-
made template with options from a user interface. The template and user
interface can integrate expertise and best practices while giving users more
control through options in the UI.

###  3.3. Free-form prompts provide full control

Previous research shows that developing free-form prompts can be challenging
(Jiang et al., 2022). However, experts can generate high-quality prompts
through the process of prompt engineering. Providing users with full control
of the prompting process may be desired in some cases. Free-form prompts
provide full access to users as they design their prompt from scratch.

Figure 2. FeedbackBuffet enables users to insert writing samples (1) and
select from a set of predefined options for the type of feedback they want to
receive (2). Using a template, these options are combined to form a prompt (3)
which is sent to GPT-3 using OpenAI’s public API (4). GPT-3 then generates
feedback, which is displayed in a text box for the user to review (5).

figure description

##  4\. Case Study

The case study methodology is a technique for illustrating an idea through the
use of examples (Yin et al., 2018). To engage more deeply with the concept of
Prompt Middleware, we developed the FeedbackBuffet prototype which implements
the template-based prompting design pattern. This case study illustrates what
template-based prompting might look like when implemented in a user interface.

###  4.1. FeedbackBuffet System

FeedbackBuffet is a writing assistant that allows users to request automated
feedback for any writing sample, such as an essay, email, or statement of
purpose, based on UI options. As shown in Figure 2, UI options offer users
relevant feedback options which are combined using a template to form a prompt
for GPT-3. The template integrates best practices of feedback design and cues
the feedback seeker to consider qualities of good feedback. FeedbackBuffet
implements the template-based prompt middleware to integrate intelligence into
the interface.

####  4.1.1. System Implementation

The system is implemented as a ReactJS web app. The prompts are generated
through template literals (i.e.: string interpolation) where each selected
option from the UI is injected into the template to form a string that is sent
as a prompt to OpenAI via API calls using zero-shot learning.

####  4.1.2. Integrate Best Practices in Feedback Design

There are principles and best practices for feedback design, such as asking a
clarifying question and then making a statement (Lekschas et al., 2021),
sandwiching criticism between two positive comments (Docheff, 1990), and
making feedback actionable (Krause et al., 2017). The feedback template used
by FeedbackBuffet is based on a feedback framework that includes valence,
level of abstraction, and feedback type (Alvarez et al., 2012), summarized in
Figure 2. We present examples of the feedback generated by GPT-3 using our
template in Figure 3.

###  4.2. Use Case: Requesting Design Feedback

To illustrate how FeedbackBuffet operates, we present the following use case
about Sasha, a CS student who is taking career preparatory course to work on
his statement of purpose. Sasha completes a first draft of his statement and
he receives feedback from his instructor that critiques the structure—Sasha
did not start with a strong motivation. He focused too much on the graduate
program before motivating the reader. After adding motivation based on his
journey into computers, he uses FeedbackBuffet to get more feedback before his
next class. He pastes his statement into the input area and selects options to
request feedback about the content of this draft. These options along with his
draft are packaged as a prompt and then sent to GPT-3. He receives the
feedback shown in Figure 2. Based on this feedback, Sasha edits his statement
to add more specific details about how he learned to code by forming an
informal group with his friends. He continues to iterate on his statement of
purpose, periodically referring back to FeedbackBuffet, and he is excited to
show his progress to his instructor.

##  5\. Discussion

In this paper, we build on existing research (MacNeil et al., 2021; Hui et
al., 2018; MacNeil et al., 2023; Bernstein et al., 2010; Pandey et al., 2018)
for integrating expertise and intelligence into UIs. We introduce the Prompt
Middleware Framework to guide the process of integrating LLMs into a UI. We
demonstrate this vision with FeedbackBuffet, a intelligent writing assistant
that automatically generates feedback based on text input. Given that previous
attempts at integrating intelligence may require effort to acquire
intelligence sources or be costly, FeedbackBuffet offers a lightweight method
for integrating intelligence and best practices into a UI. FeedbackBuffet’s UI
acts as a facade around the LLM, abstracting away the complexity of
interacting with LLMs. While FeedbackBuffet currently focuses on template-
based prompts, we could include static prompts as well. For example, with a
button titled ‘Pros and Cons’, which would send the prompt shown in Figure 1
to an LLM.

Researchers have identified several challenges individuals face when
interacting with general purpose AI, such as a lack of awareness about the
AI’s capabilities which can lead them to request overly complicated on non-
existent tasks from the AI agent (Yang et al., 2020). Researchers are still
developing an understanding of the capabilities of LLMs, but in this paper we
show it is possible to convey known possibilities afforded by LLMs through a
UI. Through static prompts, users can use prompts that have been engineered by
experts to be effective. Through template-based prompts, they can choose from
a list of menu options to generate prompts that have been previously tested by
experts. This ability to communicate the capabilities afforded by LLMs has the
potential to make them more accessible for non-experts.

As future work, we plan to evaluate three systems, including FeedbackBuffet,
that embody these three types of prompt middleware to understand how best to
integrate LLMs into existing UI. Through this evaluation, we also hope to
develop a better understanding of how much control is desired when interacting
with LLMs through UI. While complete control in the form of free-form prompts
might be desired in some contexts, it likely depends. For example, a feedback
system based on static prompts, which provide less control, may simplify the
feedback request process.

##  6\. Conclusion

In this paper, we present FeedbackBuffet, a writing assistant that generates
feedback on writing samples using GPT-3. The user can choose from a set of
feedback options that are combined using a template to form a prompt for
GPT-3. This system demonstrates how templates can serve as middleware to map
affordances in a user interface to prompt a large language model. This work
serves as an initial step toward developing a prompt middleware that can
bridge the gap between users and large language models.

## References

  * (1)
  * Alvarez et al. (2012) Ibis Alvarez, Anna Espasa, and Teresa Guasch. 2012\.  The value of feedback in improving collaborative writing assignments in an online learning environment.  _Studies in Higher Education_ 37, 4 (2012), 387–400.
  * Bernstein et al. (2010) Michael S Bernstein, Greg Little, Robert C Miller, Björn Hartmann, Mark S Ackerman, David R Karger, David Crowell, and Katrina Panovich. 2010.  Soylent: a word processor with a crowd inside. In _Proceedings of the 23nd annual ACM symposium on User interface software and technology_. 313–322.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in Neural Information Processing Systems_ 33 (2020), 1877–1901.
  * Dahlbäck et al. (1993) Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993\.  Wizard of Oz studies—why and how.  _Knowledge-based systems_ 6, 4 (1993), 258–266.
  * Di Fede et al. (2022) Giulia Di Fede, Davide Rocchesso, Steven P. Dow, and Salvatore Andolina. 2022.  The Idea Machine: LLM-Based Expansion, Rewriting, Combination, and Suggestion of Ideas. In _Proceedings of the 14th Conference on Creativity and Cognition_ (Venice, Italy) _(C &C ’22)_. Association for Computing Machinery, New York, NY, USA, 623–627.  https://doi.org/10.1145/3527927.3535197
  * Ding et al. (2023) Zijian Ding, Arvind Srinivasan, Stephen Macneil, and Joel Chan. 2023.  Fluid Transformers and Creative Analogies: Exploring Large Language Models’ Capacity for Augmenting Cross-Domain Analogical Creativity. In _Proceedings of the 15th Conference on Creativity and Cognition_ (Virtual Event, USA) _(C &C ’23)_. Association for Computing Machinery, New York, NY, USA, 489–505.  https://doi.org/10.1145/3591196.3593516
  * Docheff (1990) Dennis M Docheff. 1990\.  The feedback sandwich.  _Journal of Physical Education, Recreation & Dance_ 61, 9 (1990), 17–18.
  * Gawande (2009) Atul Gawande. 2009\.  _The Checklist Manifesto: How to Get Things Right_.  Metropolitan Books.
  * Huang et al. (2023) Ziheng Huang, Kexin Quan, Joel Chan, and Stephen MacNeil. 2023\.  CausalMapper: Challenging Designers to Think in Systems with Causal Maps and Large Language Model. In _Proceedings of the 15th Conference on Creativity and Cognition_ (Virtual Event, USA) _(C &C ’23)_. Association for Computing Machinery, New York, NY, USA, 325–329.  https://doi.org/10.1145/3591196.3596818
  * Hui et al. (2018) Julie S Hui, Darren Gergle, and Elizabeth M Gerber. 2018\.  Introassist: A tool to support writing introductory help requests. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_. 1–13.
  * Jiang et al. (2022) Ellen Jiang, Kristen Olson, Edwin Toh, Alejandra Molina, Aaron Donsbach, Michael Terry, and Carrie J Cai. 2022.  PromptMaker: Prompt-based Prototyping with Large Language Models. In _CHI Conference on Human Factors in Computing Systems Extended Abstracts_. 1–8.
  * Kim et al. (2015) Joy Kim, Mira Dontcheva, Wilmot Li, Michael S. Bernstein, and Daniela Steinsapir. 2015.  Motif: Supporting Novice Creativity through Expert Patterns. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ (Seoul, Republic of Korea) _(CHI ’15)_. Association for Computing Machinery, New York, NY, USA, 1211–1220.  https://doi.org/10.1145/2702123.2702507
  * Kittur et al. (2012) Aniket Kittur, Susheel Khamkar, Paul André, and Robert Kraut. 2012.  CrowdWeaver: visually managing complex crowd work. In _Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work_. 1033–1036.
  * Krause et al. (2017) Markus Krause, Tom Garncarz, JiaoJiao Song, Elizabeth M. Gerber, Brian P. Bailey, and Steven P. Dow. 2017.  Critique Style Guide: Improving Crowdsourced Design Feedback with a Natural Language Model. In _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_ (Denver, Colorado, USA) _(CHI ’17)_. Association for Computing Machinery, New York, NY, USA, 4627–4639.  https://doi.org/10.1145/3025453.3025883
  * Kulkarni et al. (2012) Anand Kulkarni, Matthew Can, and Björn Hartmann. 2012\.  Collaboratively crowdsourcing workflows with turkomatic. In _Proceedings of the acm 2012 conference on computer supported cooperative work_. 1003–1012.
  * Lekschas et al. (2021) Fritz Lekschas, Spyridon Ampanavos, Pao Siangliulue, Hanspeter Pfister, and Krzysztof Z. Gajos. 2021\.  Ask Me or Tell Me? Enhancing the Effectiveness of Crowdsourced Design Feedback. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (Yokohama, Japan) _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, Article 564, 12 pages.  https://doi.org/10.1145/3411764.3445507
  * MacNeil et al. (2021) Stephen MacNeil, Zijian Ding, Kexin Quan, Thomas j Parashos, Yajie Sun, and Steven P. Dow. 2021\.  Framing Creative Work: Helping Novices Frame Better Problems through Interactive Scaffolding. In _Creativity and Cognition_ (Virtual Event, Italy) _(C &C ’21)_. Association for Computing Machinery, New York, NY, USA, Article 30, 10 pages.  https://doi.org/10.1145/3450741.3465261
  * MacNeil et al. (2023) Stephen MacNeil, Ziheng Huang, Kenneth Chen, Zijian Ding, Alexander Yu, Kendall Nakai, and Steven P. Dow. 2023.  Combining Freeform Curation with Structured Templates. In _Creativity and Cognition_ (Gathertown) _(C &C ’19)_. ACM, New York, NY, USA, 11 pages.  https://doi.org/10.1145/3591196.3593337
  * Maulsby et al. (1993) David Maulsby, Saul Greenberg, and Richard Mander. 1993\.  Prototyping an Intelligent Agent through Wizard of Oz. In _Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems_ (Amsterdam, The Netherlands) _(CHI ’93)_. Association for Computing Machinery, New York, NY, USA, 277–284.  https://doi.org/10.1145/169059.169215
  * Mirowski et al. (2023) Piotr Mirowski, Kory W Mathewson, Jaylen Pittman, and Richard Evans. 2023.  Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–34.
  * Pandey et al. (2018) Vineet Pandey, Justine Debelius, Embriette R Hyde, Tomasz Kosciolek, Rob Knight, and Scott Klemmer. 2018.  Docent: transforming personal intuitions to scientific hypotheses through content learning and process training. In _Proceedings of the Fifth Annual ACM Conference on Learning at Scale_. 1–10.
  * Reynolds and McDonell (2021) Laria Reynolds and Kyle McDonell. 2021.  Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_ (Yokohama, Japan) _(CHI EA ’21)_. Association for Computing Machinery, New York, NY, USA, Article 314, 7 pages.  https://doi.org/10.1145/3411763.3451760
  * Völkel et al. (2020) Sarah Theres Völkel, Christina Schneegass, Malin Eiband, and Daniel Buschek. 2020.  What is ”Intelligent” in Intelligent User Interfaces? A Meta-Analysis of 25 Years of IUI. In _Proceedings of the 25th International Conference on Intelligent User Interfaces_ (Cagliari, Italy) _(IUI ’20)_. Association for Computing Machinery, New York, NY, USA, 477–487.  https://doi.org/10.1145/3377325.3377500
  * Wu et al. (2022) Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022\.  AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 385, 22 pages.  https://doi.org/10.1145/3491102.3517582
  * Yang et al. (2020) Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020.  _Re-Examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design_.  Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3313831.3376301
  * Yin et al. (2018) Robert K Yin et al. 2018\.  Case study research and applications: Design and methods.  _Los Angeles, UK: Sage_ (2018).
  * Yuan et al. (2022) Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022\.  Wordcraft: story writing with large language models. In _27th International Conference on Intelligent User Interfaces_. 841–852.
  * Yuan et al. (2016) Alvin Yuan, Kurt Luther, Markus Krause, Sophie Isabel Vennix, Steven P Dow, and Bjorn Hartmann. 2016\.  Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques. In _Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing_ (San Francisco, California, USA) _(CSCW ’16)_. Association for Computing Machinery, New York, NY, USA, 1005–1017.  https://doi.org/10.1145/2818048.2819953

Figure 3. Examples of feedback that GPT-3 can provide by combining different
options within our feedback template. These results can be compared across two
example contexts—an email and a short statement of purpose.