# Phase Pattern Formation in Grain Boundaries

I. S. Winter<EMAIL_ADDRESS>Sandia National Laboratories, Albuquerque,
NM, 87123, USA  T. Frolov<EMAIL_ADDRESS>Lawrence Livermore National
Laboratory, Livermore, CA, 94550, USA

###### Abstract

In this work we derive conditions that predict the existence of two-phase
periodic-pattern grain boundary structures that are stable against coarsening.
While previous research has established that elastic effects can lead to phase
pattern formation on crystal surfaces, the possibility of stable grain
boundary structures composed of alternating grain boundary phases has not been
previously analyzed. Our theory identifies the specific combination of grain
boundary and materials properties that enable the emergence of patterned grain
boundary states and shows that the dislocation content of grain boundary phase
junctions, absent in surface phenomena, weakens the stability of the patterned
structures. The predictions of the theory are tested using a model copper
grain boundary that exhibits multiple phases and two-phase pattern formation.
We discuss how, similarly to surfaces, elastic effects associated with grain
boundary phase junctions have profound implications for how grain boundary
phases transform.

Elastic effects arising from interactions between junctions of surface phases
are known to have a profound effect on phase transformations in 2D Marchenko
(1981); Hannon  _et al._ (2001). Line forces, also known as force monopoles,
exist at surface phase junctions due to the difference in surface stress
Marchenko and Parshin (1980); Alerhand  _et al._ (1988); Pehlke and Tersoff
(1991); Marchenko (1981); Srolovitz and Hirth (1991). A seminal study by
Marchenko showed that the interaction of these surface line defects can in
some cases reduce the surface energy of the system relative to a coarse two-
phase state. This stabilizes a new surface structure, which can be described
as a periodic pattern composed of alternating surface phases Marchenko and
Parshin (1980). The characteristic dimensions of the pattern depend on the
temperature, composition, and elastic properties of the system.

Conventional thermodynamics, which does not consider long-range interactions,
predicts surface phase diagrams with well-defined transition boundaries that
correspond to the equilibrium coexistence of the different phases. In reality,
elastic interactions can smear the coexistence lines defined by the equality
of free energies, creating a new domain represented by the patterned state.
Patterned surface states with a well-defined pattern wavelength and no well-
defined transition temperature have been experimentally observed and confirmed
for surface phases of silicon Hannon  _et al._ (2001).

Grain boundaries are internal interfaces inside a material that, just like
surfaces, exhibit different phases Cantwell  _et al._ (2014); Krause  _et al._
(2019); Cantwell  _et al._ (2020). However, the possibility of metastable
pattern formation at GBs is currently not known. Just as for surfaces, the
junctions between different GB phases give rise to line forces, because
different GB phases have different GB stresses Winter  _et al._ (2022a). In
this regard, GB phases are equivalent to surface phases and have the potential
for forming patterned states. However, one crucial distinction is that GB
phase junctions also contain dislocations Frolov  _et al._ (2021); Winter  _et
al._ (2022a, b). This additional dislocation-dislocation interaction energy
contribution can counteract the elastic energy of line forces, prohibiting the
formation of patterned GB states. In this work, we derive equations describing
the energy of the patterned states at GBs that include both line forces and
dislocation interactions and demonstrate conditions under which patterned GB
states can exist. Using the predictions of our model we identify a copper
model GB system that exhibits pattern behavior. Our simulations demonstrate
that the structure composed of alternating GB phases is stable against
coarsening at finite temperatures.

In a manner analagous to Hamilton et al.’s study of GB facets Hamilton  _et
al._ (2003), let us consider a periodic GB phase pattern consisting of two GB
phases as depicted in Fig. 1. We will term the two GB phases to be the
$\alpha$ and $\beta$ phases. The repeat distance of the pattern is of length
$d$, and the lengths of the $\alpha$ and $\beta$ phases are $l$ and $l-d$
respectively. Both a line force and dislocation can exist at the GB phase
junction Winter  _et al._ (2022a). The line force arises due to the difference
in GB stresses:
$\bm{f}=\left(\tau_{11}^{\alpha}-\tau_{11}^{\beta}\right)\hat{\bm{e}}_{1}$.
The dislocation is a result of the difference in excess volume, excess shear,
and the density of atoms at the GB for the two GB phases Frolov  _et al._
(2021). The energy per unit length of such a two-phase pattern depicted in
Fig. 1 is expressed in terms of $\chi=l/d$ as

$E(\chi)=\Delta\gamma^{\alpha\beta}\chi+\frac{A}{d}\ln\left(\frac{d\sin(\pi\chi)}{\pi\rho}\right)+\frac{2\Gamma^{\alpha\beta}+B}{d},$
(1)

where $\Delta\gamma^{\alpha\beta}=\gamma^{\alpha}-\gamma^{\beta}$ is the
difference in GB energy between the $\alpha$ and $\beta$ GB phases. $A$ and
$B$ are elastic terms related to $\bm{f}$, $\bm{b}$, and the bulk elastic
constants of the material. $\rho$ and $\Gamma^{\alpha\beta}$ are the core
radius and core energy per unit-length of the GBPJ. A derivation of Eq. (1) is
given in the Supplemental Materials.

Figure 1: Panel a depicts a schematic of a periodic GB phase pattern, with the
$\alpha$ and $\beta$ GB phases represented by green and orange regions. The
repeat length of the pattern is $d$ and the length of the $\alpha$ phase is
$l$. Panel b depicts the corresponding periodic pattern of GBPJs represented
as line forces and dislocations.

From Eq. (1) it can be shown that a local maximum or minimum energy exists for
the dual-phase system if $\frac{dE(\chi)}{d\chi}=0,\chi\in(0,1)$. For pattern
formation to be at least metastable, this extreme point, $\chi_{c}$, must be a
local minimum:$\frac{d^{2}E(\chi)}{d\chi^{2}}|_{\chi=\chi_{c}}>0$. As
$\frac{d^{2}E(\chi)}{d\chi^{2}}=-\frac{\pi^{2}A}{d\sin^{2}(\pi\chi)}$, the
possibility for pattern formation to occur is entirely dependent on the sign
of the elastic coefficient, $A$. If $A<0$, a pattern can exist in a metastable
state.

To better understand the conditions necessary for pattern formation to occur,
we consider the case of an elastically isotropic system, as this allows us to
write an analytical expression for $A$ Winter  _et al._ (2022a):

$\displaystyle A$ $\displaystyle=A^{dd}+A^{pp}+A^{dp},$ (2a) $\displaystyle
A^{dd}$
$\displaystyle=\mu\Big{[}(1-\nu)b_{2}^{2}+(b_{1}^{2}+b_{3}^{2})^{2}\Big{]}/2\pi(1-\nu),$
(2b) $\displaystyle A^{pp}$ $\displaystyle=-f_{1}^{2}(3-4\nu)/8\pi\mu(1-\nu),$
(2c) $\displaystyle A^{dp}$ $\displaystyle=-f_{1}b_{3}(1-2\nu)/2\pi(1-\nu),$
(2d)

where $\mu$ is the shear modulus, $\nu$ Poisson’s ratio and $b_{i}$ the
components of the Burgers vector. The first term in $A$, the dislocation-
dislocation interaction energy, must be positive if the material is to be
elastically stable. The second term, the line force-line force interaction
must be negative if the material is to be elastically stable. The line force-
dislocation interaction can be positive or negative depending on the signs of
$f_{1}$ and $b_{3}$.

Let us now try to find what values of $f_{1}$ allow for $A<0$. $A$ is a
quadratic equation with respect to $f_{1}$, so finding the roots of $f_{1}$ is
instructive:

$\displaystyle f^{0,low}_{1}$
$\displaystyle=-\frac{2\mu}{3-4\nu}\left(b_{3}(1-2\nu)+\sqrt{b_{1}^{2}(3-4\nu)+(1-\nu)\Big{[}4b_{3}^{2}(1-\nu)+b_{2}^{2}(3-4\nu)\Big{]}}\right),$
(3a) $\displaystyle f^{0,high}_{1}$
$\displaystyle=-\frac{2\mu}{3-4\nu}\left(b_{3}(1-2\nu)-\sqrt{b_{1}^{2}(3-4\nu)+(1-\nu)\Big{[}4b_{3}^{2}(1-\nu)+b_{2}^{2}(3-4\nu)\Big{]}}\right).$
(3b)

Mechanical stability requires that $-1<\nu<1/2$, so the roots of $f_{1}$ must
be real for an elastically stable material. A necessary condition for pattern
formation, $A<0$, is met if $f_{1}>f^{0,high}_{1}$ or $f_{1}<f^{0,low}$. Thus
the theory allows us to formulate the following screening criteria for GBs
capable of pattern formation. GB phases should have large differences in GB
stresses, a small Burgers vector at the GBPJ, and a small shear modulus of the
material.

Following the criteria identified by the theory we consider the possibility of
pattern formation in a face-centered-cubic Cu $\Sigma 5(210)[001]$ GB composed
of repeating units of split-kite and filled-kite GB phases Frolov  _et al._
(2013). These two GB phases have a large difference in $\tau_{11}$ as shown in
TABLE 1 resulting in a large line force, $f_{1}=-4.66$ J$/$m2, while having a
relatively small Burgers vector, $b<0.5\ \mathrm{\AA}$ Frolov  _et al._
(2021). To match the boundary dimensions along the tilt axis in our
coexistence simulations we selected one of the metastable, but essentially
energy degenerate compared to the ground state, split-kite structure Zhu  _et
al._ (2018).

Table 1:  Properties of the two grain boundary phases considered in this work. GB phase | Split kite | Filled kite
---|---|---
GB Energy (J$/$m2) | 0.942 | 0.953
$\tau_{11}$ (J$/$m2) | -2.36 | 2.30
Structural Unit Area ($a_{0}^{2}$) | $\sqrt{5}\times 2$ | $7\sqrt{5}\times 2$
Fraction of Plane | 1/2 | 6/7

Following the work of Barnett and Lothe Barnett and Lothe (1974), we applied
anisotropic linear elasticity theory to calculate the elastic coefficient in
Eq. (1), $A$, using an embedded-atom method potential of Cu Mishin  _et al._
(2001) within the molecular dynamics simulation code, LAMMPS Thompson  _et
al._ (2022). Eq. (1) remains valid in the anisotropic case, but in general the
elastic coefficient $A$ does not necessarily have an analytical form. At 0 K,
$A$ was calculated from the elastic constants of the system, $\tau_{11}$ for
each GB phase (given in TABLE 1), and the Burgers vector of the GBPJ. Using
the approach outlined in Ref. Frolov  _et al._ (2021), we found the Burgers
vector of the GBPJ to be $\bm{b}=0.368\hat{\bm{e}}_{3}$. From these inputs we
find $A=-31$ meV$/$Å, and as a result, predict pattern formation to occur.

Figure 2: Panel a depicts the double-period GB-phase-patterned state. Panel b
shows a comparison of the energy of a double-period structure using molecular
statics simulations and elasticity theory. The increasing energy shows that
the patterned state is stable against coarsening.

To test if the magnitude of $A$ is large enough to maintain pattern stability
we first consider 0 K molecular statics simulations of the effect of pattern
morphology on system energy. Since the two different grain boundary phases are
composed of a different number of atoms (TABLE 1) and the total number of
atoms in the simulation box is fixed, a heterogeneous state composed of just
two phases is always stable in a wide range of temperatures. As a result, to
investigate the stability against coarsening, we have to consider a double-
period system consisting of four GB phases repeating over a length $D$, two
split kites and two filled kites as shown in Fig. 2a. The double-period system
is constructed such that the total length of split- and filled-kite islands is
constant, and as a result the number of atoms in the system too: if one split-
kite island grows in length, the other shrinks. If we constrain the larger
(smaller) split-kite island to have the same length as the larger (smaller)
filled-kite island, we can describe the system by one parameter, $\Delta l$:
the difference in length between the two islands of a given GB phase junction,
shown in Fig. 2. The elastic energy from such a double-period structure can be
written as

$\Delta E^{DP}\left(\Delta l\right)=A\ln\left(\cos\left(\frac{2\pi\Delta
l}{D}\right)\right).$ (4)

The derivation of Eq. (4) is detailed in the Supplemental Materials. In Fig.
2, Eq. (4) is compared to molecular statics simulations of the double-period
system as a function of $\Delta l$, which are generated using the method
described in Ref. Winter  _et al._ (2022a). As the simulations contained
approximately $500,000$ atoms, the molecular statics simulations of the
double-period structure were performed by annealing the structures at 300 K
for 20 ns and then relaxing the structure using the FIRE optimization
algorithm Bitzek  _et al._ (2006). While the grain boundary structures were
already in their lowest energy states from the beginning, the structure of the
GBPJ cores required additional optimization which occurred during the
annealing stage. The GB energy of all systems converged to $10^{-8}$
eV$/\mathrm{\AA}^{2}$. However, due to the large system size and complex GB
microstructure, the scatter of the 0 K energies is still seen in Fig. 2.
Overall, however, we see an excellent agreement between the analytical
prediction of equation (4) and direct GB energy calculations of the patterned
state using molecular statics at 0K. Both calculations suggest that the
regular patterned state should be stable against coarsening.

To investigate the effect of temperature on the stability of the patterned
state, we calculated $A$ in the temperature interval from 0 to 1000 K. This
required the calculation of the elastic constants, $\tau_{11}$, and thermal
expansion as a function of temperature, the details of which are given in the
Supplemental Materials. The calculated elastic coefficient $A(T)$ is shown in
Fig. 3. It remains negative, predicting the stability of the patterned state
at finite temperature. The temperature however has a destabilizing effect as
$A$ gradually decreases in magnitude and reaches a near zero value at 1000 K,
marking the limit of the patterned state’s stability.

Figure 3: The elastic coefficient, $A$, plotted as a function of temperature
confirms the stability of the patterned state at finite temperatures. Its
decreasing magnitude suggests that the temperature decreases the stability of
the patterned state. Error bars are smaller than the size of the data points.
Figure 4: Panels a and b depict the initial and final island configurations.
Blue atoms correspond to split-kites islands, red atoms correspond to filled-
kite islands, and yellow atoms correspond to GB phase junctions. Panel c shows
the evolution of islands over time.

Finally, we tested the predictions of our theory and calculations by directly
simulating the evolution of the double-period patterned state at high
temperature using molecular dynamics in the NVT ensemble. The patterned state
containing four GB phase islands shown in Fig. 4 should be stable against
coarsening into a two-phase state. For our MD simulations, we selected 800 K
as the optimal testing temperature. At higher temperatures, the stabilizing
force becomes vanishingly small and comparable to fluctuations, so we expect
the pattern to go away. At lower temperatures, the island size fluctuations
become too infrequent to provide convincing proof of stability. The initial
patterned state with all four islands of equal length, $l=D/4$, is shown in
Fig. 4a. We ran the simulation for a total of 2.3 $\upmu$s.

We estimated the lengths of the four islands, shown in Fig. 4c, by calculating
the contour of the GB (and the step heights between GB phase junctions). The
contour was estimated by applying polyhedral template matching Larsen  _et
al._ (2016), as implemented in OVITO Stukowski (2010), with non-FCC atoms
considered to be part of the GB. Hyperbolic tangent functions were fit to the
GB contour to calculate island lengths, with more details of the simulation
given in the Supplemental Materials. The GB pattern showed remarkable
stability during the first 0.75 $\upmu$s, during which the length of all
islands fluctuated around the equal size state with $l=D/4$, as shown in Fig.
4c. We note that states with different $l$ are visited during this time
interval as the junctions migrate by several GB units (see movie in the
Supplemental Materials).

At approximately t=0.75 $\upmu$s we observe a large fluctuation towards a
coarser state when the length of one of the split-kite islands increases
significantly at the expense of the other split-kite island. However, at
approximately 1.75 $\upmu$s, the system returns to the patterned state,
confirming its stability. The state of the system after 2.3 $\upmu$s is shown
in Fig. 4b.

The observed large fluctuation of the island size away from the even size
state at 800 K is indicative of an already very small stabilizing contribution
from the elastic interactions compared to thermal fluctuations. We expect that
at higher temperatures thermal fluctuations will dominate the evolution of
such a microstructure eventually leading to coarsening. At lower temperatures
on the other hand we expect the patterned state to be stable.

There is an increasing recognition that just like in 3D materials, complex
defect microstructures exist inside interfaces and greatly influence the
properties of polycrystalline materials. GBs may contain ordered or random
networks of disconnections and/or GB phase junctions. These defected GB
microstructures could form during non-equilibrium processes such as plastic
deformation, grain growth, exposure to fluxes of point defects during
radiation damage, and other processes that involve changes in temperature and
chemical composition.

Our understanding of the dynamic evolution and stability of these GB defect
microstructures is currently severely limited by the lack of techniques that
enable the identification and visualization of these defects in the GB plane
both experimentally and in simulations. Only recently algorithms capable of
robust identification of disconnection and GB phase junctions have been
proposed Winter  _et al._ (2022b); Deka  _et al._ (2023). Despite these
limitations, there is growing interest in understanding the collective
behavior of disconnections and GBPJs. For example, recent theoretical studies
predicted GB topological transitions due to disconnections Chen  _et al._
(2020), which could result in dramatic changes in GB mobility Olmsted  _et
al._ (2007).

Complex dual-phase patterns composed of different GB phases have recently been
investigated by high-resolution transmission electron microscopy (HRTEM) in Cu
GBs Meiners  _et al._ (2020); Langenohl  _et al._ (2022), with dual-phase
patterned GB microstructures having been directly observed by HRTEM in
$\langle 111\rangle$ tilt GBs in Cu Langenohl  _et al._ (2022). To explain the
pattern metastability the authors of the study postulated that an array of
pre-existing disconnections inherited from processing could exist at the
boundary and these disconnections would effectively repel GB phase junctions
and make this dual-phase pattern structure metastable. Using our analysis, we
can estimate the elastic coefficient to be $A=40$ meV$/$Å, which means that
this particular Cu GB (without additional pre-existing disconnections) should
not form a metastable patterned state. We conclude that the observed state is
either a result of an incomplete transformation or indeed the presence of a
preexisting array of additional disconnections.

Inspired by well-known theoretical and experimental results for surface
phases, here we investigated the stability of patterned states of GBs composed
of different GB phases separated by line defects containing both line forces
and dislocation defects. We derived equations describing the energy of such
states and identified conditions where the pattern state becomes metastable,
providing a barrier to coarsening into a more common two-phase structure. We
show that the Burgers content present at GB phase junctions has a
destabilizing effect on the pattern state. Using our theoretical model we
identified a model system that should exhibit pattern behavior and verified it
by direct atomistic simulations at finite temperature as well as energy
calculations at 0 K.

###### Acknowledgements.

This work was performed under the auspices of the U.S. Department of Energy
(DOE) by Lawrence Livermore National Laboratory under contract DE-
AC52-07NA27344. TF was supported by the U.S. DOE, Office of Science under an
Office of Fusion Energy Sciences Early Career Award. Computing support for
this work came from the Lawrence Livermore National Laboratory Institutional
Computing Grand Challenge program. Sandia National Laboratories is a multi-
mission laboratory managed and operated by National Technology & Engineering
Solutions of Sandia, LLC (NTESS), a wholly owned subsidiary of Honeywell
International Inc., for the U.S. Department of Energy’s National Nuclear
Security Administration (DOE/NNSA) under contract DE-NA0003525. TF is grateful
to Mark Asta for stimulating discussions.

## References

  * Marchenko (1981) V. Marchenko, JETP Lett 33 (1981).
  * Hannon  _et al._ (2001) J. B. Hannon, F. J. Meyer zu Heringdorf, J. Tersoff, and R. M. Tromp, Phys. Rev. Lett. 86, 4871 (2001).
  * Marchenko and Parshin (1980) V. Marchenko and A. Y. Parshin, Sov. Phys. JETP 52, 129 (1980).
  * Alerhand  _et al._ (1988) O. L. Alerhand, D. Vanderbilt, R. D. Meade, and J. D. Joannopoulos, Phys. Rev. Lett. 61, 1973 (1988).
  * Pehlke and Tersoff (1991) E. Pehlke and J. Tersoff, Phys. Rev. Lett. 67, 465 (1991).
  * Srolovitz and Hirth (1991) D. Srolovitz and J. Hirth, Surface Science 255, 111 (1991).
  * Cantwell  _et al._ (2014) P. R. Cantwell, M. Tang, S. J. Dillon, J. Luo, G. S. Rohrer, and M. P. Harmer, Acta Materialia 62, 1 (2014).
  * Krause  _et al._ (2019) A. R. Krause, P. R. Cantwell, C. J. Marvel, C. Compson, J. M. Rickman, and M. P. Harmer, Journal of the American Ceramic Society 102, 778 (2019).
  * Cantwell  _et al._ (2020) P. R. Cantwell, T. Frolov, T. J. Rupert, A. R. Krause, C. J. Marvel, G. S. Rohrer, J. M. Rickman, and M. P. Harmer, Annual Review of Materials Research 50, 465 (2020).
  * Winter  _et al._ (2022a) I. S. Winter, R. E. Rudd, T. Oppelstrup, and T. Frolov, Phys. Rev. Lett. 128, 035701 (2022a).
  * Frolov  _et al._ (2021) T. Frolov, D. L. Medlin, and M. Asta, Phys. Rev. B 103, 184108 (2021).
  * Winter  _et al._ (2022b) I. Winter, T. Oppelstrup, T. Frolov, and R. Rudd, Acta Materialia 237, 118067 (2022b).
  * Hamilton  _et al._ (2003) J. C. Hamilton, D. J. Siegel, I. Daruka, and F. Léonard, Phys. Rev. Lett. 90, 246102 (2003).
  * Frolov  _et al._ (2013) T. Frolov, D. L. Olmsted, M. Asta, and Y. Mishin, Nature Communications 4, 1899 (2013).
  * Zhu  _et al._ (2018) Q. Zhu, A. Samanta, B. Li, R. E. Rudd, and T. Frolov, Nature Communications 9, 467 (2018).
  * Barnett and Lothe (1974) D. M. Barnett and J. Lothe, Journal of Physics F: Metal Physics 4, 1618 (1974).
  * Mishin  _et al._ (2001) Y. Mishin, M. J. Mehl, D. A. Papaconstantopoulos, A. F. Voter, and J. D. Kress, Phys. Rev. B 63, 224106 (2001).
  * Thompson  _et al._ (2022) A. P. Thompson, H. M. Aktulga, R. Berger, D. S. Bolintineanu, W. M. Brown, P. S. Crozier, P. J. in ’t Veld, A. Kohlmeyer, S. G. Moore, T. D. Nguyen, R. Shan, M. J. Stevens, J. Tranchida, C. Trott, and S. J. Plimpton, Comp. Phys. Comm. 271, 108171 (2022).
  * Bitzek  _et al._ (2006) E. Bitzek, P. Koskinen, F. Gähler, M. Moseler, and P. Gumbsch, Phys. Rev. Lett. 97, 170201 (2006).
  * Larsen  _et al._ (2016) P. M. Larsen, S. Schmidt, and J. Schiøtz, Modelling and Simulation in Materials Science and Engineering 24, 055007 (2016).
  * Stukowski (2010) A. Stukowski, MODELLING AND SIMULATION IN MATERIALS SCIENCE AND ENGINEERING 18 (2010), 10.1088/0965-0393/18/1/015012.
  * Deka  _et al._ (2023) N. Deka, A. Stukowski, and R. B. Sills, Acta Materialia 256, 119096 (2023).
  * Chen  _et al._ (2020) K. Chen, D. J. Srolovitz, and J. Han, Proceedings of the National Academy of Sciences 117, 33077 (2020).
  * Olmsted  _et al._ (2007) D. L. Olmsted, S. M. Foiles, and E. A. Holm, Scripta Materialia 57, 1161 (2007).
  * Meiners  _et al._ (2020) T. Meiners, T. Frolov, R. E. Rudd, G. Dehm, and C. H. Liebscher, Nature 579, 375 (2020).
  * Langenohl  _et al._ (2022) L. Langenohl, T. Brink, R. Freitas, T. Frolov, G. Dehm, and C. H. Liebscher, Nature Communications 13, 3331 (2022).


11institutetext: STCA Search & Distribution Group, Microsoft, China
11email<EMAIL_ADDRESS>
# Large Language Models are Diverse Role-Players
for Summarization Evaluation

Ning Wu  Ming Gong Corresponding author. Linjun Shou  Shining Liang  Daxin
Jiang

###### Abstract

Text summarization has a wide range of applications in many scenarios. The
evaluation of the quality of the generated text is a complex problem. A big
challenge to language evaluation is that there is a clear divergence between
existing metrics and human evaluation. A document summary’s quality can be
assessed by human annotators on various criteria, both objective ones like
grammar and correctness, and subjective ones like informativeness,
succinctness, and appeal. Most of the automatic evaluation methods like
BLUE/ROUGE may be not able to adequately capture the above dimensions. In this
paper, we propose a new evaluation framework based on LLMs, which provides a
comprehensive evaluation framework by comparing generated text and reference
text from both objective and subjective aspects. First, we propose to model
objective and subjective dimensions of generated text based on roleplayers
prompting mechanism. Furthermore, we introduce a context-based prompting
mechanism that is able to generate dynamic roleplayer profiles based on input
context. Finally, we design a multi-roleplayer prompting technology based on
batch prompting and integrate multiple outputs into the final evaluation
results. Experimental results on three real datasets for summarization show
that our model is highly competitive and has a very high consistency with
human annotators.

###### Keywords:

Large Language Model Summarization Evaluation Role Player.

##  1 Introduction

Text summarization has wide applications in various research and application
fields. Recently, some works found that there is a clear gap between existed
metrics like BLEU [13], ROUGE, BertScore [20] and human annotations [5, 19].
Although typical overlap-based and model-based metrics can capture lexicon
level or semantic level similarity between generated text and reference text,
specific dimensions like coherence, grammar, and interestingness still can’t
be captured. As depicted in Figure 1, the summarization task reveals the
inadequacy of traditional metrics such as BLUE/ROUGE: they are unable to
reflect the true quality of the text after reaching a certain level. To
achieve consistency between human evaluation and automatic metrics, we
encounter two main challenges: 1) How to model objective criteria of
evaluation such as coherence and grammar. 2) How to model subjective criteria
of evaluation such as interestingness [4, 7], comprehensiveness, and
usefulness from the standpoint of users. Natural language has various modes of
expression for the same concept, so assessing its quality based on a few
static criteria is hard.

Figure 1: Two summarizations of CNN News, they are generated by two models
(GPT3 [2], T0 [16]), and have similar BLEU and ROUGE metrics, but the second
summary is obviously worse than the first one on two more complicated
dimensions.

Motivated by the ability of LLMs to handle multi-domains, we investigate how
to leverage LLMs for measurement in this paper. Since it is difficult to make
LLMs provide a consistent and fair score for the generated text [17], we
propose a comparison-based evaluation method to quantify the quality of the
generated text, namely DRPE, which stands for Diverse Role-Player for
Summarization Generation Evaluation. In particular, we devise a roleplayer-
based prompting strategy in this system for objective and subjective dimension
measurement. Our method comprises two parts: 1) Static roles construction and
dynamic roles prompts generation. 2) A multi-roleplayer framework to conduct a
comprehensive evaluation.

For a given generation task, its measurement can be broken down into several
dimensions. Typical objective metrics such as coherence and grammar are
relatively easy to be agreed upon by most people, so we manually created
static roles for each objective dimension of the task. It is expressed as
<_Judger types_ , _Judger description_ >. With a static role, we prompt LLM by
asking it to impersonate a real judger based on the judger type and
description and then vote for the better option. Furthermore, a comprehensive
measurement is usually complex and dynamic. Depending on different cases in a
summarization task, different aspects need to be taken into account.
Therefore, we propose to dynamically generate some potential users based on
the content and let LLMs conduct subjective measurements on behalf of these
users. The dynamic roles can be expressed as <_User types_ , _User
description_ >. Lastly, we design a multi-roleplayer framework to eliminate
redundant roleplayers and integrate the vote results of multiple roleplayers.
Moreover, the multi-roleplayer framework can also enhance the stability of the
LLM-based measurement system with relatively low inference costs. Experimental
results show that our method significantly surpasses zero-shot LLMs and
existing metrics on three typical summarization datasets with human
annotations.

##  2 Related Works

###  2.1 Large Language Model

Large language model has been found to be capable of few-shot learning [2].
Chain-of-Thought [18] is proposed to empower model reasoning capability for
complex tasks. ZeroShot-Cot [8] still shows relatively strong reasoning
ability without any examples. Least-to-Most [22] LLM decomposes a complex
question into several sub-questions and solves these sub-questions in sequence
and finally gives a complete answer to the original question. Recent work [5]
discovered that both reference-based and reference-free automatic metrics
cannot reliably evaluate zero-shot summaries. In this paper, we mainly explore
the capability of LLM to compare the generated text and reference text.

###  2.2 Existed Metrics

The most widely used metric in machine translation is BLEU [13], which
includes several modifications to Exact-$P_{n}$. A smoothed variant, SENTBLEU
is computed at the sentence level. METEOR [1] computes Exact-$P_{1}$ and
Exact-$R_{1}$ while allowing backing-off from exact unigram matching to
matching word stems, synonyms, and paraphrases. ROUGE [9] is a commonly used
metric for summarization evaluation. ROUGE-n computes Exact-$R_{n}$ (usually n
= 1, 2), while ROUGE-L is a variant of Exact-$R_{1}$ with the numerator
replaced by the length of the longest common subsequence. BERTScore [20]
computes a similarity score for each token in the candidate sentence with each
token in the reference sentence. MoverScore [21] investigates the
effectiveness of existing contextualized representations and Earth Mover’s
Distance [15] for comparing system predictions and reference texts, leading to
a new automated evaluation metric that achieves high correlation with human
judgments of text quality.

Figure 2: The overall framework of DRPE. Objective roleplayers are curated
manually based on task setting, and subjective roleplayers are dynamically
generated by LLMs. After diversity clustering, similar roles are eliminated,
and all roles are played by LLMs to compare two candidates in batch prompting.
Finally, results from multiple roles are aggregated.

##  3 Methodology

As discussed in Section 1, currently, it forms a gap with human evaluation
that automatic metrics for text generation stop at surface similarity (lexicon
level or semantic level) which leads to biased perception and evaluation of
the text generation capability of LLMs. In this section, we elaborate our
proposed measurement framework for text generation primarily includes
diversified roleplayers generation and roleplayers-based evaluation.

###  3.1 Diversified RolePlayers Generation

To build a novel framework differing from existing calculation-based automatic
metrics, we decompose this task into objective and subjective dimensions and
propose an LLM-based measurement method. In our framework, the LLMs act as a
judge with a distinctive role to evaluate the text quality in a corresponding
dimension and generate its evaluation results. Thus, we need to generate
diversified roleplayers for objective and subjective dimensions at first.

Figure 3: Three different static roles for summarization task. For different
generation tasks, different aspects need to be taken into consideration. A
good summary usually requires fluent writing, proper wording and capturing key
points of raw article.

####  3.1.1 Objective RolePlayers

Overlap-based metrics such as BLUE/ROUGE measure lexicon-level consistency
with n-grams. Model-based metrics like BERTScore can capture subtle syntax or
semantic changes with embeddings similarity, but they have a limitation in
evaluating high-quality text from LLMs. The parameter scale of BERT is much
smaller than that of LLMs, so it cannot represent LLMs’ rich semantics.
Consequently in this paper, we take some advanced quality dimensions like
fluency, consistency, grammar and coherence into consideration, which were
rarely adopted before as they are difficult to be measured accurately. Since
different tasks usually require different objective dimensions, and these
dimensions are relatively easy to be agreed on by most people, hence we
manually curated static objective dimensions for the summarization task and
make sure all these dimensions are fundamental and objective. The static
objective roles schema is presented below:<_Judger type_ , _Judger
description_ >. where each _Judger_ works on one or multiple specific
objective dimensions and _Judger description_ breaks down and exposit what
specifically the Judger would focus on when evaluating. As shown in Figure 3,
three different objective roles are designed for summarization tasks.

Figure 4: Coarse-grained and fine-grained prompting mechanism for
comprehensive user profiles generation.

####  3.1.2 Subjective RolePlayers

Text generation, unlike understanding tasks, does not have a perfect golden
standard. Human-written material can only offer a high-quality example.
Different readers may judge the text according to their own perspectives. For
instance, consider the sports news about a renowned athlete in a game,

  * •

For the writing style, ordinary readers expect that it’s concise and easy to
understand, while journalists would pay attention to its structure and choices
of words.

  * •

For the content, causal fans like comprehensive data of the sports player and
horizontal comparison with others, while die-hard fans are more eager for in-
depth analysis of the sports player through data.

Therefore, we propose to collect subjective evaluations of model-generated
text from diverse user perspectives, including whether they think the text is
interesting, useful, etc. These dimensions are more abstract and difficult to
quantify than objective dimensions which few studies have touched on and
addressed to our knowledge. Specifically, we take each generated text as the
context and prompt the LLM to generate its potential readers dynamically
following the below schema: <_User type_ , _User description_ >. Here we
design two user role generation prompts. As shown in Figure 4, the former
requires the LLM to consider the most common occupations with most people in
the world which is coarse-grained and the latter aims to categorize people
based on their familiarity with the text topics which is fine-grained. We
merge objective judgers and subjective users generated by two kinds of
prompting mechanisms as multi-role players for the next process. Considering
that there may exist duplicate or similar users, we propose to conduct
diversity clustering to improve measurement performance and reduce inference
costs. First, each roleplayer type and its description are concatenated as the
input of Sentence-BERT [14] to obtain the representation. Next, we use the
$k$-means algorithm to cluster roleplayers, and those closest to each cluster
center are kept. Finally, the chosen role players will be leveraged for text
evaluation.

###  3.2 RolePlayer-based Evaluation

To mitigate the discrepancy between the human evaluation and automatic
metrics, we propose to leverage the roleplayers as crowdsourcing voters to
compare the summaries from multi-dimensions. Besides the static roleplayers
that scrutinize the objective dimensions including grammar, fluency,
coherence, etc, the dynamic roleplayers are generated according to the current
article and simulate the psychology and behavior of the article readers
(roles) to convey their subjective feelings. It’s expected that our method
could achieve higher consistency with human evaluation than existing automatic
metrics focusing on surface similarity.

####  3.2.1 Evaluation of RolePlayer

Given the article $A$, we posit the reference summary generated by humans as
$S$, and the candidate summary generated by models as $\hat{S}$, respectively.
To evaluate, all the roleplayers perform pair-wise comparison as Figure 5,
since point-wise comparisons are inconsistent across different samples and
list-wise comparisons are not stable. By our prompting, the LLMs play a
specific role and output its analysis of two summaries, finally voting which
summary is of better quality. In more detail, we parse and quantify the
comparison result as $\hat{a}$:

$\hat{a}=\left\\{\begin{aligned} &1&&{\text{If voting is candidate summary
$\hat{S}$ }}\\\ &0&&{\text{If voting is reference summary $S$ }}\\\
\end{aligned}\right.$ (1)

Figure 5: Compare generated summary and reference summary by multiple
roleplayers with batch prompting.

Assuming the DRPE score for generated text $\hat{S}$ is
$\text{DRPE}(\hat{S}|A,S)$, it could be obtained by modeling the joint
probability of reason and voting result as below:

$\displaystyle\text{DRPE}(\hat{S}|A,S)=\mathds{1}(\hat{a}=1)P(\hat{a},\bm{r}|\bm{p},A,S,\hat{S},R)$
(2)

where $\bm{r}$ is the comparison reason, $\bm{p}$ represents the prompt used
here, $\hat{a}$ is voting result from LLMs, $A$ is raw article and $R$ is the
role. To compute $P(\hat{a},\bm{a})$, similar to [17], we leverage the
confidence when roleplayer generates its voting normalized by the output
length [2],

$\displaystyle P(\hat{a},\bm{r}|\bm{p},A,S,\hat{S},R)$
$\displaystyle=\exp^{\frac{1}{K}\sum_{k=1}^{K}\log
P(t_{k}|\bm{p},A,S,\hat{S},R,t_{1},\dots,t_{k-1})}$ (3)

where $\log P(t_{k}|\bm{p},A,S,\hat{S},R,t_{1},\dots,t_{k-1})$ is the log
probability of $k$-th token $t_{k}$ in $\bm{r}$ and $K$ is number of tokens.

####  3.2.2 Batch Prompting

To efficiently get comparison results for the summary pair ($S$, $\hat{S}$)
from multiple roleplayers, we design a multi-roleplayer framework based on
batch prompting to measure by both objective and subjective metrics in one-off
inference. The different metrics are reflected by the objective and subjective
roleplayers generated and clustered in Section 3.1. As shown in Figure 5,
first, all the roleplayers are prompted to give votes for ($S$, $\hat{S}$)
with respect to $A$ in their point of view, i.e., which summary could better
describe the article, and tell us the reasons. Then we aggregate the results
to parse $\hat{a}=\\{\hat{a}_{j}\\}_{j=1}^{N}$ where $N$ is the number of
roleplayers. According to Equation 2, the multi-roleplayer DRPE score by batch
prompting can be formulated as below:

$\displaystyle\text{DRPE}(\hat{S}|A,S)=P(\hat{a}|\bm{p},A,S,\hat{S})$
$\displaystyle=\sum_{j=1}^{N}\mathds{1}(\hat{a}_{j}=1)P(\hat{a}_{j},\bm{r}|\bm{p},A,S,\hat{S},R_{j})$
(4)

where $R_{j}$ denotes $j$-th roles. Compared with Self-Consistency CoT [17],
our framework decouples the answer (comparison result) and reasoning path
(comparison reason), and brings higher inference efficiency as Self-
Consistency CoT needs to generate each candidating answer separately while our
method generates all voting results with once inference.

##  4 Experiments

###  4.1 Experiments Setting

####  4.1.1 Datasets.

•CNN2022 [11, 6]: contains reference summaries that are approximately 3-4
sentences long. Summaries in this dataset are highly extractive and lead-
biased. We use human study data on 100 recent articles from CNN, collected
between March 1, 2022, and June 31, 2022. Each article is annotated by at
least three judgers, and they are asked to choose the best and worst summaries
from three candidates. To reduce noises, we only keep the best summary with at
least 2 out of 3 votes and worst summary with at least 2 out of 3 votes.
Finally, we obtain 81 best and worst summaries as candidate summaries with a
piece of corresponding news. Finally, we use GPT-3 of the text-DaVinci-003 to
generate reference summarization, and finally use our method to compare the
candidate summary.

•BBC2022 [12]: contains 1 sentence summaries of BBC news articles. In this
dataset, references summaries, and consequently generated summaries from fine-
tuned models are highly abstractive. We use human study data on 100 recent
articles from BBC, collected between March 1, 2022, and June 31, 2022. We also
take a similar preprocessing strategy like CNN2022 on this dataset.

•SummEval [3]: contains a large and diverse collection of human judgments of
model-generated summaries on the CNN/Daily Mail dataset annotated by both
expert judges and crowd-source workers. For each news, we select two worst
summaries and two best summaries according to their average scores on four
dimensions (coherence, consistency, fluency and relevance) labeled by experts.
Finally, regarding one hundred news and corresponding reference summary in
SummEval, we obtain 400 candidate summaries.

####  4.1.2 Metrics.

To measure the consistency between various metrics and human annotations, we
follow the WMT18 [10] standard practice and use absolute Pearson correlation
$|\rho|$ to evaluate metric quality.

####  4.1.3 Baselines.

Automatic metrics proposed for summarization evaluation can be broadly divided
into two categories: (1) overlap-based metrics, specifically ROUGE METEOR and
BLEU, and (2) similarity-based metrics that compute the similarity between
embeddings representations of generated and reference summaries. Specifically,
we report BERTScore and MoverScore. For LLMScore, we carefully design prompts
for LLM, and directly use it to predict a better passage.

####  4.1.4 Implementation.

We use the public GPT-3 [2] of the text-DaVinci-003 version with 175B
parameters from OpenAI for the LLMs implementation and use greedy decoding for
inference with the temperature set to 0. We select this LLM because it has
relatively good capability among public LLMs. Especially, we use three roles
General Public, Critic, and News Author which are described in Figure 3 as
objective roleplayers in our DRPE, and we prompt the model to generate 4
dynamic roles for each case.

###  4.2 Results

Table 1: Pearson correlation between several automatic metrics and human
annotation. We bold the highest numbers for each dataset, and use AVG to
denote the average scores on the three datasets. Results of GPT-D3 and DRPE
are averaged over five runs with slight changing in prompts.

Type | Method | CNN2022 | SummEval | BBC2022 | AVG
---|---|---|---|---|---
Overlap | ROUGE-1 | 0.466 | 0.431 | 0.469 | 0.461
ROUGE-2 | 0.437 | 0.354 | 0.443 | 0.411
ROUGE-L | 0.422 | 0.322 | 0.436 | 0.393
BLEU | 0.475 | 0.372 | 0.502 | 0.450
METEOR | 0.514 | 0.473 | 0.561 | 0.516
Similarity | BERTScore | 0.554 | 0.455 | 0.568 | 0.526
MoverScore | 0.456 | 0.385 | 0.442 | 0.428
LLM | GPT-D3 | 0.713 | 0.503 | 0.692 | 0.636
DRPE | 0.816 | 0.683 | 0.784 | 0.761

####  4.2.1 Comparison with Existed Metrics

Tables 1 shows Pearson correlation to human judgments. We observe that typical
overlap-based metrics generally performed badly and relevance-based metrics
also underperformed. The simplest LLM-based method has a consistently better
performance than BERTScore. Two types of LLM-based methods, GPT-D3, and
DRPEScore have a clear gap between the other two methods. Especially,
DRPEScore consistently performs better than GPT-D3.

Table 2: Pearson correlation between several automatic metrics and human
annotation. AVG denotes the average scores on the three datasets.

Method | CNN2022 | SummEval | BBC2022 | Avg
---|---|---|---|---
DRPE | 0.816 | 0.683 | 0.784 | 0.761
w/o Batch Inferring | 0.822 | 0.672 | 0.766 | 0.753
w/o Clustering | 0.782 | 0.665 | 0.751 | 0.733
w/o Dynamic Roles | 0.742 | 0.669 | 0.703 | 0.705
w/o Static Roles | 0.734 | 0.604 | 0.711 | 0.683

####  4.2.2 Ablation Study

We conducted an ablation study on DRPE to gain insights into the detailed
method design. We prepare four variants of our method: (1) w/o Batch Inferring
denotes without the batch prompting, each role is inferred alone; (2) w/o
Clustering denotes without clustering mechansim; (3) w/o Dynamic Roles denotes
without dynamic roles generation; (4) w/o Static Roles denotes without the
human designed static roles. Table 2 presents all comparison results of the
four variants. As we can see, the performance rank on three datasets can be
given as: w/o Static Roles < w/o Dynamic Roles < w/o Clustering < w/o Batch
Inferring < DRPE. These results indicate that all components are essential to
improve performance. And we can also find that batch inferring is able to save
lots of inference tokens without losing performance.

(a) Roles number k w.r.t correlation on CNN2022 dataset.

(b) Roles number k w.r.t correlation on BBC2022 dataset.

Figure 6: Effect of role number on model performance.

####  4.2.3 Effects of Hyperparameters

We test DRPEScore and w/o Static Roles with different subjective role numbers
$k$ in [0, 2, 4, 6] on two datasets. Figure 6 shows the experimental results.
When $k$ increases from 0 to 6, the experimental indicators first increase and
then slightly decrease on the CNN2022 dataset, when $k$=4, the correlation to
human evaluation achieves a peak. On the BBC2022 dataset, experimental results
are similar, and more roles (6) don’t bring improvements compared to fewer
roles (4).

Figure 7: Evaluation procedure for two summaries given a piece of news. We use
the green font to represent the content generated by the model. With suitable
prompting, the LLM is able to generate relevant roles and generate judgment
for these roles.

####  4.2.4 Qualitative Analysis

We have previously demonstrated the effectiveness of our model on two
summarization tasks. In this section, we conduct a qualitative analysis to
explain why DRPEScore can achieve good performance. Figure 7 shows an example
of our model’s evaluation process. Given a news article about the legal
dispute between Amber Heard and Johnny Depp, our model has to select a better
summary from two candidates. First, we generate several subjective roles based
on the news content, such as _Amber Heard Fans User_ , _Johnny Depp Fans User_
, _Celebrity Gossip User_ , and _Legal System User_. These roles are
representative of different perspectives and preferences that can be captured
by LLMs. Second, we use LLMs to simulate each user and judger and compare the
two summaries. We employ a batch prompting mechanism to accelerate the
inference procedure. Notably, LLMs predict that the _Johnny Depp Fans User_ ,
who might have a negative attitude towards _Amber Heard Fans User_ , will
favor summary 1.

##  5 Conclusion

We propose DRPE, a new comparison-based method for evaluating generated text
against gold standard references. Our DRPE is designed to be simple and task-
agnostic. Our experiments illustrate that DRPE could provide a human-like
ability to conduct a comprehensive evaluation, especially on challenging long
text generation like summarization tasks. In future work, we look forward to
exploring the capabilities of LLMs as judgers on more text evaluation tasks
and reducing the computation cost.

## References

  * [1] Banerjee, S., Lavie, A.: Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. pp. 65–72 (2005)
  * [2] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)
  * [3] Fabbri, A.R., Kryściński, W., McCann, B., Xiong, C., Socher, R., Radev, D.: Summeval: Re-evaluating summarization evaluation. TACL 9, 391–409 (2021)
  * [4] Gao, J., Pantel, P., Gamon, M., He, X., Deng, L.: Modeling interestingness with deep neural networks. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pp. 2–13 (2014)
  * [5] Goyal, T., Li, J.J., Durrett, G.: News summarization and evaluation in the era of gpt-3. arXiv preprint arXiv:2209.12356 (2022)
  * [6] Hermann, K.M., Kocisky, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., Blunsom, P.: Teaching machines to read and comprehend. NIPS 28 (2015)
  * [7] Hidi, S., Baird, W.: Interestingness—a neglected variable in discourse processing. Cognitive science 10(2), 179–194 (1986)
  * [8] Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 (2022)
  * [9] Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text summarization branches out. pp. 74–81 (2004)
  * [10] Ma, Q., Bojar, O., Graham, Y.: Results of the wmt18 metrics shared task: Both characters and embeddings achieve good performance. In: Proceedings of the third conference on machine translation: shared task papers. pp. 671–688 (2018)
  * [11] Nallapati, R., Zhou, B., Gulcehre, C., Xiang, B., et al.: Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023 (2016)
  * [12] Narayan, S., Cohen, S.B., Lapata, M.: Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745 (2018)
  * [13] Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: ACL. pp. 311–318 (2002)
  * [14] Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. In: EMNLP. pp. 3982–3992 (2019)
  * [15] Rubner, Y., Tomasi, C., Guibas, L.J.: The earth mover’s distance as a metric for image retrieval. International journal of computer vision 40(2), 99 (2000)
  * [16] Sanh, V., Webson, A., Raffel, C., Bach, S.H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T.L., Raja, A., et al.: Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207 (2021)
  * [17] Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Zhou, D.: Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)
  * [18] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., Zhou, D.: Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022)
  * [19] Yuan, X., Wang, T., Wang, Y.H., Fine, E., Abdelghani, R., Lucas, P., Sauzéon, H., Oudeyer, P.Y.: Selecting better samples from pre-trained llms: A case study on question generation. arXiv preprint arXiv:2209.11000 (2022)
  * [20] Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019)
  * [21] Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C.M., Eger, S.: Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. arXiv preprint arXiv:1909.02622 (2019)
  * [22] Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Bousquet, O., Le, Q., Chi, E.: Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625 (2022)


{https://doi.org/10.1088/0954-3899/39/4/043101} {\bibfield  {journal}
{\bibinfo  {journal} {Journal of Physics G}\ }\textbf {\bibinfo {volume}
{39}},\ \bibinfo {pages} {043101} (\bibinfo {year} {2012})}\BibitemShut
\bibitem [{\citenamefont {Hagen}\ \emph {et~al.}(2014)\citenamefont {Hagen},
\citenamefont {Papenbrock}, \citenamefont {Hjorth-Jensen},\ and\
\citenamefont {Dean}}]{Hagen2014}%
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {G.}~\bibnamefont
{Hagen}}, \bibinfo {author} {\bibfnamefont {T.}~\bibnamefont {Papenbrock}},
\bibinfo {author} {\bibfnamefont {M.}~\bibnamefont {Hjorth-Jensen}},\ and\
\bibinfo {author} {\bibfnamefont {D.~J.}\ \bibnamefont {Dean}},\ }\bibfield
{title} {\bibinfo {title} {Coupled-cluster computations of atomic nuclei},\
}\href@noop {} {\bibfield  {journal} {\bibinfo  {journal} {Rep. Progr.
Phys.}\ }\textbf {\bibinfo {volume} {77}},\ \bibinfo {pages} {096302}
(\bibinfo {year} {2014})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Morris}\ \emph {et~al.}(2018)\citenamefont {Morris},
\citenamefont {Simonis}, \citenamefont {Stroberg}, \citenamefont {Stumpf},
\citenamefont {Hagen}, \citenamefont {Holt}, \citenamefont {Jansen},
\citenamefont {Papenbrock}, \citenamefont {Roth},\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {T.~D.}\ \bibnamefont
{Morris}}, \bibinfo {author} {\bibfnamefont {J.}~\bibnamefont {Simonis}},
\bibinfo {author} {\bibfnamefont {S.~R.}\ \bibnamefont {Stroberg}}, \bibinfo
{author} {\bibfnamefont {C.}~\bibnamefont {Stumpf}}, \bibinfo {author}
{\bibfnamefont {G.}~\bibnamefont {Hagen}}, \bibinfo {author} {\bibfnamefont
{J.~D.}\ \bibnamefont {Holt}}, \bibinfo {author} {\bibfnamefont {G.~R.}\
\bibnamefont {Jansen}}, \bibinfo {author} {\bibfnamefont {T.}~\bibnamefont
{Papenbrock}}, \bibinfo {author} {\bibfnamefont {R.}~\bibnamefont {Roth}},\
and\ \bibinfo {author} {\bibfnamefont {A.}~\bibnamefont {Schwenk}},\
}\bibfield  {title} {\bibinfo {title} {Structure of the lightest tin
isotopes},\ }\href {https://doi.org/10.1103/PhysRevLett.120.152503}
{\bibfield  {journal} {\bibinfo  {journal} {Phys. Rev. Lett.}\ }\textbf
{\bibinfo {volume} {120}},\ \bibinfo {pages} {152503} (\bibinfo {year}
{2018})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Lenske}\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {H.}~\bibnamefont
{Lenske}}\ and\ \bibinfo {author} {\bibfnamefont {N.}~\bibnamefont
{Tsoneva}},\ }\bibfield  {title} {\bibinfo {title} {{Dissolution of shell
structures and the polarizability of dripline nuclei}},\ }\href
{https://doi.org/10.1140/epja/i2019-12811-6} {\bibfield  {journal} {\bibinfo
{journal} {Eur. Phys. J. A}\ }\textbf {\bibinfo {volume} {55}},\ \bibinfo
{pages} {238} (\bibinfo {year} {2019})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Sun}\ \emph {et~al.}(2021)\citenamefont {Sun},
\citenamefont {Hagen}, \citenamefont {Jansen},\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {Z.~H.}\ \bibnamefont
{Sun}}, \bibinfo {author} {\bibfnamefont {G.}~\bibnamefont {Hagen}}, \bibinfo
{author} {\bibfnamefont {G.~R.}\ \bibnamefont {Jansen}},\ and\ \bibinfo
{author} {\bibfnamefont {T.}~\bibnamefont {Papenbrock}},\ }\bibfield  {title}
{\bibinfo {title} {Effective shell-model interaction for nuclei ``southeast''
of $^{100}\mathrm{Sn}$},\ }\href
{https://doi.org/10.1103/PhysRevC.104.064310} {\bibfield  {journal} {\bibinfo
{journal} {Phys. Rev. C}\ }\textbf {\bibinfo {volume} {104}},\ \bibinfo
{pages} {064310} (\bibinfo {year} {2021})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Preskill}(2018)}]{preskill2018quantum}%
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {J.}~\bibnamefont
{Preskill}},\ }\bibfield  {title} {\bibinfo {title} {Quantum computing in the
nisq era and beyond},\ }\href@noop {} {\bibfield  {journal} {\bibinfo
{journal} {Quantum}\ }\textbf {\bibinfo {volume} {2}},\ \bibinfo {pages} {79}
(\bibinfo {year} {2018})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Ollitrault}\ \emph {et~al.}(2020)\citenamefont
{Ollitrault}, \citenamefont {Kandala}, \citenamefont {Chen}, \citenamefont
{Barkoutsos}, \citenamefont {Mezzacapo}, \citenamefont {Pistoia},
\citenamefont {Sheldon}, \citenamefont {Woerner}, \citenamefont {Gambetta},\
and\ \citenamefont {Tavernelli}}]{Ollitrault_2020}%
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {P.~J.}\ \bibnamefont
{Ollitrault}}, \bibinfo {author} {\bibfnamefont {A.}~\bibnamefont {Kandala}},
\bibinfo {author} {\bibfnamefont {C.-F.}\ \bibnamefont {Chen}}, \bibinfo
{author} {\bibfnamefont {P.~K.}\ \bibnamefont {Barkoutsos}}, \bibinfo
{author} {\bibfnamefont {A.}~\bibnamefont {Mezzacapo}}, \bibinfo {author}
{\bibfnamefont {M.}~\bibnamefont {Pistoia}}, \bibinfo {author} {\bibfnamefont
{S.}~\bibnamefont {Sheldon}}, \bibinfo {author} {\bibfnamefont
{S.}~\bibnamefont {Woerner}}, \bibinfo {author} {\bibfnamefont {J.~M.}\
\bibnamefont {Gambetta}},\ and\ \bibinfo {author} {\bibfnamefont
{I.}~\bibnamefont {Tavernelli}},\ }\bibfield  {title} {\bibinfo {title}
{Quantum equation of motion for computing molecular excitation energies on a
noisy quantum processor},\ }\href@noop {} {\bibfield  {journal} {\bibinfo
{journal} {Phys. Rev. Res.}\ }\textbf {\bibinfo {volume} {2}},\ \bibinfo
{pages} {043140} (\bibinfo {year} {2020})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Lipkin}\ \emph {et~al.}(1965)\citenamefont {Lipkin},
\citenamefont {Meshkov},\ and\ \citenamefont {Glick}}]{lipkin1965validity}%
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {H.~J.}\ \bibnamefont
{Lipkin}}, \bibinfo {author} {\bibfnamefont {N.}~\bibnamefont {Meshkov}},\
and\ \bibinfo {author} {\bibfnamefont {A.}~\bibnamefont {Glick}},\ }\bibfield
{title} {\bibinfo {title} {Validity of many-body approximation methods for a
solvable model:(i). exact solutions and perturbation theory},\ }\href@noop {}
{\bibfield  {journal} {\bibinfo  {journal} {Nucl. Phys. B}\ }\textbf
{\bibinfo {volume} {62}},\ \bibinfo {pages} {188} (\bibinfo {year}
{1965})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Cervia}\ \emph {et~al.}(2021)\citenamefont {Cervia},
\citenamefont {Balantekin}, \citenamefont {Coppersmith}, \citenamefont
{Johnson}, \citenamefont {Love}, \citenamefont {Poole}, \citenamefont
{Robbins},\ and\ \citenamefont {Saffman}}]{cervia2021lipkin}%
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {M.~J.}\ \bibnamefont
{Cervia}}, \bibinfo {author} {\bibfnamefont {A.}~\bibnamefont {Balantekin}},
\bibinfo {author} {\bibfnamefont {S.}~\bibnamefont {Coppersmith}}, \bibinfo
{author} {\bibfnamefont {C.~W.}\ \bibnamefont {Johnson}}, \bibinfo {author}
{\bibfnamefont {P.~J.}\ \bibnamefont {Love}}, \bibinfo {author}
{\bibfnamefont {C.}~\bibnamefont {Poole}}, \bibinfo {author} {\bibfnamefont
{K.}~\bibnamefont {Robbins}},\ and\ \bibinfo {author} {\bibfnamefont
{M.}~\bibnamefont {Saffman}},\ }\bibfield  {title} {\bibinfo {title} {Lipkin
model on a quantum computer},\ }\href@noop {} {\bibfield  {journal} {\bibinfo
{journal} {Phys. Rev. C}\ }\textbf {\bibinfo {volume} {104}},\ \bibinfo
{pages} {024305} (\bibinfo {year} {2021})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Rowe}(1968)}]{rowe1968equations}%
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {D.}~\bibnamefont
{Rowe}},\ }\bibfield  {title} {\bibinfo {title} {Equations-of-motion method
and the extended shell model},\ }\href@noop {} {\bibfield  {journal}
{\bibinfo  {journal} {Rev. Mod. Phys.}\ }\textbf {\bibinfo {volume} {40}},\
\bibinfo {pages} {153} (\bibinfo {year} {1968})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Ring}\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {P.}~\bibnamefont
{Ring}}\ and\ \bibinfo {author} {\bibfnamefont {P.}~\bibnamefont {Schuck}},\
}\href@noop {} {\emph {\bibinfo {title} {The nuclear many-body problem}}}\
(\bibinfo  {publisher} {Springer Science \& Business Media},\ \bibinfo {year}
{2004})\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Gambacurta}\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {D.}~\bibnamefont
{Gambacurta}}\ and\ \bibinfo {author} {\bibfnamefont {M.}~\bibnamefont
{Grasso}},\ }\bibfield  {title} {\bibinfo {title} {Second rpa calculations
with the skyrme and gogny interactions},\ }\href@noop {} {\bibfield
{journal} {\bibinfo  {journal} {Eur. Phys. J. A}\ }\textbf {\bibinfo {volume}
{52}},\ \bibinfo {pages} {1} (\bibinfo {year} {2016})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Cao}\ \emph {et~al.}(2009)\citenamefont {Cao},
\citenamefont {Colo}, \citenamefont {Sagawa}, \citenamefont {Bortignon},
\citenamefont {Sciacchitano} \emph {et~al.}}]{cao2009effects}%
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {L.-G.}\ \bibnamefont
{Cao}}, \bibinfo {author} {\bibfnamefont {G.}~\bibnamefont {Colo}}, \bibinfo
{author} {\bibfnamefont {H.}~\bibnamefont {Sagawa}}, \bibinfo {author}
{\bibfnamefont {P.}~\bibnamefont {Bortignon}}, \bibinfo {author}
{\bibfnamefont {L.}~\bibnamefont {Sciacchitano}}, \emph {et~al.},\ }\bibfield
{title} {\bibinfo {title} {Effects of the tensor force on the multipole
response in finite nuclei},\ }\href@noop {} {\bibfield  {journal} {\bibinfo
{journal} {Phys. Rev. C}\ }\textbf {\bibinfo {volume} {80}},\ \bibinfo
{pages} {064304} (\bibinfo {year} {2009})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Drozdz}\ \emph {et~al.}(1990)\citenamefont {Drozdz},
\citenamefont {Nishizaki}, \citenamefont {Speth},\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {S.}~\bibnamefont
{Drozdz}}, \bibinfo {author} {\bibfnamefont {S.}~\bibnamefont {Nishizaki}},
\bibinfo {author} {\bibfnamefont {J.}~\bibnamefont {Speth}},\ and\ \bibinfo
{author} {\bibfnamefont {J.}~\bibnamefont {Wambach}},\ }\bibfield  {title}
{\bibinfo {title} {The nuclear response within extended rpa theories},\
}\href@noop {} {\bibfield  {journal} {\bibinfo  {journal} {Phys. Rep.}\
}\textbf {\bibinfo {volume} {197}},\ \bibinfo {pages} {1} (\bibinfo {year}
{1990})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Peruzzo}\ \emph {et~al.}(2014)\citenamefont
{Peruzzo}, \citenamefont {McClean}, \citenamefont {Shadbolt}, \citenamefont
{Yung}, \citenamefont {Zhou}, \citenamefont {Love}, \citenamefont
{Aspuru-Guzik},\ and\ \citenamefont {O’brien}}]{peruzzo2014variational}%
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {A.}~\bibnamefont
{Peruzzo}}, \bibinfo {author} {\bibfnamefont {J.}~\bibnamefont {McClean}},
\bibinfo {author} {\bibfnamefont {P.}~\bibnamefont {Shadbolt}}, \bibinfo
{author} {\bibfnamefont {M.-H.}\ \bibnamefont {Yung}}, \bibinfo {author}
{\bibfnamefont {X.-Q.}\ \bibnamefont {Zhou}}, \bibinfo {author}
{\bibfnamefont {P.~J.}\ \bibnamefont {Love}}, \bibinfo {author}
{\bibfnamefont {A.}~\bibnamefont {Aspuru-Guzik}},\ and\ \bibinfo {author}
{\bibfnamefont {J.~L.}\ \bibnamefont {O’brien}},\ }\bibfield  {title}
{\bibinfo {title} {A variational eigenvalue solver on a photonic quantum
processor},\ }\href@noop {} {\bibfield  {journal} {\bibinfo  {journal} {Nat.
Commun.}\ }\textbf {\bibinfo {volume} {5}},\ \bibinfo {pages} {1} (\bibinfo
{year} {2014})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Rizzo}\ \emph {et~al.}(2022)\citenamefont {Rizzo},
\citenamefont {Libbi}, \citenamefont {Tacchino}, \citenamefont {Ollitrault},
\citenamefont {Marzari},\ and\ \citenamefont {Tavernelli}}]{rizzo2022one}%
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {J.}~\bibnamefont
{Rizzo}}, \bibinfo {author} {\bibfnamefont {F.}~\bibnamefont {Libbi}},
\bibinfo {author} {\bibfnamefont {F.}~\bibnamefont {Tacchino}}, \bibinfo
{author} {\bibfnamefont {P.~J.}\ \bibnamefont {Ollitrault}}, \bibinfo
{author} {\bibfnamefont {N.}~\bibnamefont {Marzari}},\ and\ \bibinfo {author}
{\bibfnamefont {I.}~\bibnamefont {Tavernelli}},\ }\bibfield  {title}
{\bibinfo {title} {One-particle green's functions from the quantum equation
of motion algorithm},\ }\href@noop {} {\bibfield  {journal} {\bibinfo
{journal} {arXiv:2201.01826}\ } (\bibinfo {year} {2022})}\BibitemShut
\bibitem [{\citenamefont {Meshkov}\ \emph {et~al.}(1965)\citenamefont
{Meshkov}, \citenamefont {Glick},\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {N.}~\bibnamefont
{Meshkov}}, \bibinfo {author} {\bibfnamefont {A.}~\bibnamefont {Glick}},\
and\ \bibinfo {author} {\bibfnamefont {H.}~\bibnamefont {Lipkin}},\
}\bibfield  {title} {\bibinfo {title} {Validity of many-body approximation
methods for a solvable model:(ii). linearization procedures},\ }\href@noop {}
{\bibfield  {journal} {\bibinfo  {journal} {Nucl. Phys. B}\ }\textbf
{\bibinfo {volume} {62}},\ \bibinfo {pages} {199} (\bibinfo {year}
{1965})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Glick}\ \emph {et~al.}(1965)\citenamefont {Glick},
\citenamefont {Lipkin},\ and\ \citenamefont {Meshkov}}]{glick1965validity}%
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {A.}~\bibnamefont
{Glick}}, \bibinfo {author} {\bibfnamefont {H.}~\bibnamefont {Lipkin}},\ and\
\bibinfo {author} {\bibfnamefont {N.}~\bibnamefont {Meshkov}},\ }\bibfield
{title} {\bibinfo {title} {Validity of many-body approximation methods for a
solvable model:(iii). diagram summations},\ }\href@noop {} {\bibfield
{journal} {\bibinfo  {journal} {Nucl. Phys. B}\ }\textbf {\bibinfo {volume}
{62}},\ \bibinfo {pages} {211} (\bibinfo {year} {1965})}\BibitemShut
\bibitem [{\citenamefont {Fallieros}\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {S.}~\bibnamefont
{Fallieros}}\ and\ \bibinfo {author} {\bibfnamefont {R.}~\bibnamefont
{Ferrell}},\ }\bibfield  {title} {\bibinfo {title} {Collective enhancement of
e 2 matrix elements in light nuclei},\ }\href@noop {} {\bibfield  {journal}
{\bibinfo  {journal} {Phys. Rev.}\ }\textbf {\bibinfo {volume} {116}},\
\bibinfo {pages} {660} (\bibinfo {year} {1959})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Bohm}\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {D.}~\bibnamefont
{Bohm}}\ and\ \bibinfo {author} {\bibfnamefont {D.}~\bibnamefont {Pines}},\
}\bibfield  {title} {\bibinfo {title} {A collective description of electron
interactions: Iii. coulomb interactions in a degenerate electron gas},\
}\href@noop {} {\bibfield  {journal} {\bibinfo  {journal} {Phys. Rev.}\
}\textbf {\bibinfo {volume} {92}},\ \bibinfo {pages} {609} (\bibinfo {year}
{1953})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Bardeen}\ \emph {et~al.}(1957)\citenamefont
{Bardeen}, \citenamefont {Cooper},\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {J.}~\bibnamefont
{Bardeen}}, \bibinfo {author} {\bibfnamefont {L.~N.}\ \bibnamefont
{Cooper}},\ and\ \bibinfo {author} {\bibfnamefont {J.~R.}\ \bibnamefont
{Schrieffer}},\ }\bibfield  {title} {\bibinfo {title} {Theory of
superconductivity},\ }\href@noop {} {\bibfield  {journal} {\bibinfo
{journal} {Phys. Rev.}\ }\textbf {\bibinfo {volume} {108}},\ \bibinfo {pages}
{1175} (\bibinfo {year} {1957})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Holzwarth}(1973)}]{holzwarth1973four}%
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {G.}~\bibnamefont
{Holzwarth}},\ }\bibfield  {title} {\bibinfo {title} {Four approaches to the
function of inertia in a solvable model},\ }\href@noop {} {\bibfield
{journal} {\bibinfo  {journal} {Nucl. Phys. A}\ }\textbf {\bibinfo {volume}
{207}},\ \bibinfo {pages} {545} (\bibinfo {year} {1973})}\BibitemShut
\bibitem [{\citenamefont {Co’}\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {G.}~\bibnamefont
{Co’}}\ and\ \bibinfo {author} {\bibfnamefont {S.}~\bibnamefont {De~Leo}},\
}\bibfield  {title} {\bibinfo {title} {Hartree--fock and random phase
approximation theories in a many-fermion solvable model},\ }\href@noop {}
{\bibfield  {journal} {\bibinfo  {journal} {Mod. Phys. Lett. A}\ }\textbf
{\bibinfo {volume} {30}},\ \bibinfo {pages} {1550196} (\bibinfo {year}
{2015})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Agassi}(1968)}]{agassi1968validity}%
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {D.}~\bibnamefont
{Agassi}},\ }\bibfield  {title} {\bibinfo {title} {Validity of the {BCS} and
{RPA} approximations in the pairing-plus-monopole solvable model},\
}\href@noop {} {\bibfield  {journal} {\bibinfo  {journal} {Nucl. Phys. A}\
}\textbf {\bibinfo {volume} {116}},\ \bibinfo {pages} {49} (\bibinfo {year}
{1968})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {P{\'e}rez-Fern{\'a}ndez}\ \emph
{et~al.}(2022)\citenamefont {P{\'e}rez-Fern{\'a}ndez}, \citenamefont {Arias},
\citenamefont {Garc{\'\i}a-Ramos},\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {P.}~\bibnamefont
{P{\'e}rez-Fern{\'a}ndez}}, \bibinfo {author} {\bibfnamefont {J.-M.}\
\bibnamefont {Arias}}, \bibinfo {author} {\bibfnamefont {J.-E.}\ \bibnamefont
{Garc{\'\i}a-Ramos}},\ and\ \bibinfo {author} {\bibfnamefont
{L.}~\bibnamefont {Lamata}},\ }\bibfield  {title} {\bibinfo {title} {A
digital quantum simulation of the agassi model},\ }\href@noop {} {\bibfield
{journal} {\bibinfo  {journal} {Physics Letters B}\ ,\ \bibinfo {pages}
{137133}} (\bibinfo {year} {2022})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Carrasco}\ \emph {et~al.}(2016)\citenamefont
{Carrasco}, \citenamefont {Finkel}, \citenamefont {Gonz{\'a}lez-L{\'o}pez},
\citenamefont {Rodr{\'\i}guez},\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {J.~A.}\ \bibnamefont
{Carrasco}}, \bibinfo {author} {\bibfnamefont {F.}~\bibnamefont {Finkel}},
\bibinfo {author} {\bibfnamefont {A.}~\bibnamefont {Gonz{\'a}lez-L{\'o}pez}},
\bibinfo {author} {\bibfnamefont {M.~A.}\ \bibnamefont {Rodr{\'\i}guez}},\
and\ \bibinfo {author} {\bibfnamefont {P.}~\bibnamefont {Tempesta}},\
}\bibfield  {title} {\bibinfo {title} {Generalized isotropic
lipkin-meshkov-glick models: ground state entanglement and quantum
entropies},\ }\href@noop {} {\bibfield  {journal} {\bibinfo  {journal} {J.
Stat. Mech.: Theory Exp}\ }\textbf {\bibinfo {volume} {2016}},\ \bibinfo
{pages} {033114} (\bibinfo {year} {2016})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Hagino}\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {K.}~\bibnamefont
{Hagino}}\ and\ \bibinfo {author} {\bibfnamefont {G.~F.}\ \bibnamefont
{Bertsch}},\ }\bibfield  {title} {\bibinfo {title} {Random-phase
approximation approach to rotational symmetry restoration in a three-level
lipkin model},\ }\href {https://doi.org/10.1103/PhysRevC.61.024307}
{\bibfield  {journal} {\bibinfo  {journal} {Phys. Rev. C}\ }\textbf {\bibinfo
{volume} {61}},\ \bibinfo {pages} {024307} (\bibinfo {year}
{2000})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Jordan}\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {P.}~\bibnamefont
{Jordan}}\ and\ \bibinfo {author} {\bibfnamefont {E.~P.}\ \bibnamefont
{Wigner}},\ }\bibfield  {title} {\bibinfo {title} {{\"u}ber das paulische
{\"a}quivalenzverbot},\ }in\ \href@noop {} {\emph {\bibinfo {booktitle} {The
Collected Works of Eugene Paul Wigner}}}\ (\bibinfo  {publisher} {Springer},\
\bibinfo {year} {1993})\ pp.\ \bibinfo {pages} {109--129}\BibitemShut
\bibitem [{\citenamefont {Nielsen}\ \emph {et~al.}(2005)\citenamefont {Nielsen}
\emph {et~al.}}]{nielsen2005fermionic}%
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {M.~A.}\ \bibnamefont
{Nielsen}} \emph {et~al.},\ }\bibfield  {title} {\bibinfo {title} {The
fermionic canonical commutation relations and the {J}ordan-{W}igner
transform},\ }\href@noop {} {\bibfield  {journal} {\bibinfo  {journal}
{School of Physical Sciences The University of Queensland}\ }\textbf
{\bibinfo {volume} {59}} (\bibinfo {year} {2005})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Sawaya}\ \emph {et~al.}(2020)\citenamefont {Sawaya},
\citenamefont {Menke}, \citenamefont {Kyaw}, \citenamefont {Johri},
\citenamefont {Aspuru-Guzik},\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {N.~P.}\ \bibnamefont
{Sawaya}}, \bibinfo {author} {\bibfnamefont {T.}~\bibnamefont {Menke}},
\bibinfo {author} {\bibfnamefont {T.~H.}\ \bibnamefont {Kyaw}}, \bibinfo
{author} {\bibfnamefont {S.}~\bibnamefont {Johri}}, \bibinfo {author}
{\bibfnamefont {A.}~\bibnamefont {Aspuru-Guzik}},\ and\ \bibinfo {author}
{\bibfnamefont {G.~G.}\ \bibnamefont {Guerreschi}},\ }\bibfield  {title}
{\bibinfo {title} {Resource-efficient digital quantum simulation of d-level
systems for photonic, vibrational, and spin-s hamiltonians},\ }\href@noop {}
{\bibfield  {journal} {\bibinfo  {journal} {Npj Quantum Inf.}\ }\textbf
{\bibinfo {volume} {6}},\ \bibinfo {pages} {1} (\bibinfo {year}
{2020})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Di~Matteo}\ \emph {et~al.}(2021)\citenamefont
{Di~Matteo}, \citenamefont {McCoy}, \citenamefont {Gysbers}, \citenamefont
{Miyagi}, \citenamefont {Woloshyn},\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {O.}~\bibnamefont
{Di~Matteo}}, \bibinfo {author} {\bibfnamefont {A.}~\bibnamefont {McCoy}},
\bibinfo {author} {\bibfnamefont {P.}~\bibnamefont {Gysbers}}, \bibinfo
{author} {\bibfnamefont {T.}~\bibnamefont {Miyagi}}, \bibinfo {author}
{\bibfnamefont {R.}~\bibnamefont {Woloshyn}},\ and\ \bibinfo {author}
{\bibfnamefont {P.}~\bibnamefont {Navr{\'a}til}},\ }\bibfield  {title}
{\bibinfo {title} {Improving {H}amiltonian encodings with the {G}ray code},\
}\href@noop {} {\bibfield  {journal} {\bibinfo  {journal} {Phys. Rev. A}\
}\textbf {\bibinfo {volume} {103}},\ \bibinfo {pages} {042405} (\bibinfo
{year} {2021})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Gray}(1953)}]{gray1953pulse}%
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {F.}~\bibnamefont
{Gray}},\ }\bibfield  {title} {\bibinfo {title} {Pulse code communication},\
}\href@noop {} {\bibfield  {journal} {\bibinfo  {journal} {United States
Patent Number 2632058}\ } (\bibinfo {year} {1953})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Cross}\ \emph {et~al.}(2019)\citenamefont {Cross},
\citenamefont {Bishop}, \citenamefont {Sheldon}, \citenamefont {Nation},\
and\ \citenamefont {Gambetta}}]{cross2019validating}%
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {A.~W.}\ \bibnamefont
{Cross}}, \bibinfo {author} {\bibfnamefont {L.~S.}\ \bibnamefont {Bishop}},
\bibinfo {author} {\bibfnamefont {S.}~\bibnamefont {Sheldon}}, \bibinfo
{author} {\bibfnamefont {P.~D.}\ \bibnamefont {Nation}},\ and\ \bibinfo
{author} {\bibfnamefont {J.~M.}\ \bibnamefont {Gambetta}},\ }\bibfield
{title} {\bibinfo {title} {Validating quantum computers using randomized
model circuits},\ }\href@noop {} {\bibfield  {journal} {\bibinfo  {journal}
{Phys. Rev. A}\ }\textbf {\bibinfo {volume} {100}},\ \bibinfo {pages}
{032328} (\bibinfo {year} {2019})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Zhu}\ \emph {et~al.}(1995)\citenamefont {Zhu},
\citenamefont {Byrd}, \citenamefont {Lu},\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {C.}~\bibnamefont
{Zhu}}, \bibinfo {author} {\bibfnamefont {R.}~\bibnamefont {Byrd}}, \bibinfo
{author} {\bibfnamefont {P.}~\bibnamefont {Lu}},\ and\ \bibinfo {author}
{\bibfnamefont {J.}~\bibnamefont {Nocedal}},\ }\bibfield  {title} {\bibinfo
{title} {A limited memory algorithm for bound constrained optimisation},\
}\href@noop {} {\bibfield  {journal} {\bibinfo  {journal} {SIAM J. Sci. Stat.
Comput.}\ }\textbf {\bibinfo {volume} {16}},\ \bibinfo {pages} {1190}
(\bibinfo {year} {1995})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Nakatsukasa}\ \emph {et~al.}(2007)\citenamefont
{Nakatsukasa}, \citenamefont {Inakura},\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {T.}~\bibnamefont
{Nakatsukasa}}, \bibinfo {author} {\bibfnamefont {T.}~\bibnamefont
{Inakura}},\ and\ \bibinfo {author} {\bibfnamefont {K.}~\bibnamefont
{Yabana}},\ }\bibfield  {title} {\bibinfo {title} {Finite amplitude method
for the solution of the random-phase approximation},\ }\href@noop {}
{\bibfield  {journal} {\bibinfo  {journal} {Phys. Rev. C}\ }\textbf {\bibinfo
{volume} {76}},\ \bibinfo {pages} {024318} (\bibinfo {year}
{2007})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Kortelainen}\ \emph {et~al.}(2015)\citenamefont
{Kortelainen}, \citenamefont {Hinohara},\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {M.}~\bibnamefont
{Kortelainen}}, \bibinfo {author} {\bibfnamefont {N.}~\bibnamefont
{Hinohara}},\ and\ \bibinfo {author} {\bibfnamefont {W.}~\bibnamefont
{Nazarewicz}},\ }\bibfield  {title} {\bibinfo {title} {Multipole modes in
deformed nuclei within the finite amplitude method},\ }\href
{https://doi.org/10.1103/PhysRevC.92.051302} {\bibfield  {journal} {\bibinfo
{journal} {Phys. Rev. C}\ }\textbf {\bibinfo {volume} {92}},\ \bibinfo
{pages} {051302} (\bibinfo {year} {2015})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Nik{\v{s}}i{\'c}}\ \emph {et~al.}(2013)\citenamefont
{Nik{\v{s}}i{\'c}}, \citenamefont {Kralj}, \citenamefont {Tuti{\v{s}}},
\citenamefont {Vretenar},\ and\ \citenamefont
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {T.}~\bibnamefont
{Nik{\v{s}}i{\'c}}}, \bibinfo {author} {\bibfnamefont {N.}~\bibnamefont
{Kralj}}, \bibinfo {author} {\bibfnamefont {T.}~\bibnamefont {Tuti{\v{s}}}},
\bibinfo {author} {\bibfnamefont {D.}~\bibnamefont {Vretenar}},\ and\
\bibinfo {author} {\bibfnamefont {P.}~\bibnamefont {Ring}},\ }\bibfield
{title} {\bibinfo {title} {Implementation of the finite amplitude method for
the relativistic quasiparticle random-phase approximation},\ }\href@noop {}
{\bibfield  {journal} {\bibinfo  {journal} {Phys. Rev. C}\ }\textbf {\bibinfo
{volume} {88}},\ \bibinfo {pages} {044327} (\bibinfo {year}
{2013})}\BibitemShut {NoStop}%
\bibitem [{\citenamefont {Zhang}\ \emph {et~al.}(2022)\citenamefont {Zhang},
\citenamefont {Bjel\ifmmode \check{c}\else \v{c}\fi{}i\ifmmode~\acute{c}\else
\'{c}\fi{}}, \citenamefont {Nik\ifmmode \check{s}\else
\v{s}\fi{}i\ifmmode~\acute{c}\else \'{c}\fi{}}, \citenamefont {Litvinova},
\citenamefont {Ring},\ and\ \citenamefont {Schuck}}]{zhang2021many}%
\BibitemOpen
\bibfield  {author} {\bibinfo {author} {\bibfnamefont {Y.}~\bibnamefont
{Zhang}}, \bibinfo {author} {\bibfnamefont {A.}~\bibnamefont {Bjel\ifmmode
\check{c}\else \v{c}\fi{}i\ifmmode~\acute{c}\else \'{c}\fi{}}}, \bibinfo
{author} {\bibfnamefont {T.}~\bibnamefont {Nik\ifmmode \check{s}\else
\v{s}\fi{}i\ifmmode~\acute{c}\else \'{c}\fi{}}}, \bibinfo {author}
{\bibfnamefont {E.}~\bibnamefont {Litvinova}}, \bibinfo {author}
{\bibfnamefont {P.}~\bibnamefont {Ring}},\ and\ \bibinfo {author}
{\bibfnamefont {P.}~\bibnamefont {Schuck}},\ }\bibfield  {title} {\bibinfo
{title} {Many-body approach to superfluid nuclei in axial geometry},\ }\href
{https://doi.org/10.1103/PhysRevC.105.044326} {\bibfield  {journal} {\bibinfo
{journal} {Phys. Rev. C}\ }\textbf {\bibinfo {volume} {105}},\ \bibinfo
{pages} {044326} (\bibinfo {year} {2022})}\BibitemShut {NoStop}%
\end{thebibliography}%
% Produces the bibliography via BibTeX.

\end{document}

% ****** End of file apssamp.tex ******