#  Neutrino Mass Model in the Context of $\boldsymbol{\Delta(54)\otimes
Z_{2}\otimes Z_{3}\otimes Z_{4}}$ Flavor Symmetries with Inverse Seesaw
Mechanism

Hrishi Bora<EMAIL_ADDRESS>(Corresponding author) Department of
Physics, Tezpur University, Tezpur - 784028, India  Ng. K. Francis
<EMAIL_ADDRESS>Department of Physics, Tezpur University, Tezpur -
784028, India  Animesh Barman<EMAIL_ADDRESS>Department of Physics,
Tezpur University, Tezpur - 784028, India  Bikash Thapa<EMAIL_ADDRESS>Department of Physics, Tezpur University, Tezpur - 784028, India

Abstract
Our analysis involves enhancing the $\Delta(54)$ flavor symmetry model with
Inverse Seesaw mechanism along with two SM Higgs through the incorporation of
distinct flavons. Additionally, we introduce supplementary $Z_{2}\otimes
Z_{3}\otimes Z_{4}$ symmetries to eliminate any undesirable components within
our investigation. The exact tri-bimaximal neutrino mixing pattern undergoes a
deviation as a result of the incorporation of extra flavons, leading to the
emergence of a non-zero reactor angle $\theta_{13}$ that aligns with the
latest experimental findings. It was found that for our model the atmospheric
oscillation parameter occupies the lower octant for normal hierarchy case. We
also examine the parameter space of the model for normal hierarchy to explore
the Dirac CP ($\delta_{CP}$), Jarlskog invariant parameter ($J$) and the
Neutrinoless double-beta decay parameter ($m_{\beta\beta}$) and found it in
agreement with the neutrino latest data. Hence our model may be testable in
the future neutrino experiments.

PACS numbers: 12.60.-i, 14.60.Pq, 14.60.St

##  1 Introduction

Despite the fact that the standard model (SM) has been verified by various
particle physics experiments and observations, there are still several
unresolved queations. These unanswered questions include, but are not limited
to, the source of flavor structure, the matter-antimatter imbalance in the
universe, the existence of additional neutrino flavors, and the enigmatic
properties of dark matter and dark energy. The detection of neutrino
oscillations in 1998 marked the first sign of physics beyond the standard
model. However, many uncertainties still surround neutrino physics, such as
whether the symmetry related to leptonic CP has been broken, what the lightest
absolute neutrino mass value is, and whether the atmospheric mixing angle is
maximal or not. Additionally, it remains unclear whether the neutrino masses
follow the normal hierarchy ($m_{1}<m_{2}<m_{3}$) or the inverted hierarchy
($m_{3}<m_{1}<m_{2}$). It is currently unknown whether neutrinos are Dirac
particles or Majorana particles that violate lepton number. Neutrinoless
double beta decay experiments, if observed, could confirm that neutrinos are
indeed Majorana particles. The combined results of the KamLAND-Zen and GERDA
experiments have established an upper limit for $m_{ee}$ within the range of
0.071–0.161 eV. Several recent reviews on neutrino physics can be found in
references[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13].

The study of neutrino oscillation involves three mixing angles, two of which
are large (the solar angle and the atmospheric angle) and one of which is
relatively small (the reactor angle). In the context of tribimaximal mixing
(TBM), the reactor angle is assumed to be zero and the CP phase cannot be
determined or is not known. However, experiments such as the Daya Bay Reactor
Neutrino Experiment ($sin^{2}2\theta_{13}$ = $0.089\pm 0.010\pm 0.005$)[14]
and RENO Experiment($sin^{2}2\theta_{13}$ = $0.113\pm 0.013\pm 0.019$)
[15]have shown that the reactor mixing angle is not zero. Other experiments
such as MINOS [16], Double Chooz[17], and T2K[18] have also measured
consistent nonzero values for the reactor mixing angle, rendering the TBM
model unrealistic. As a result, alternative models or modifications must be
considered to achieve realistic mixing[15, 17]. The PMNS matrix, which is
generally acknowledged, represents the blending between the various neutrino
flavor states and their corresponding mass eigenstates. Within the context of
a three-flavored model, the PMNS matrix is characterized by three mixing
angles and three CP phases.

$U_{PMNS}=\begin{pmatrix}c_{12}c_{13}&s_{12}c_{13}&s_{13}e^{-\iota\delta}\\\
-s_{12}c_{23}-c_{12}s_{23}s_{13}e^{\iota\delta}&c_{12}c_{23}-s_{12}s_{23}s_{13}e^{\iota\delta}&s_{23}c_{13}\\\
s_{12}s_{23}-c_{12}c_{23}s_{13}e^{\iota\delta}&-c_{12}s_{23}-s_{12}c_{23}s_{13}e^{\iota\delta}&\
c_{23}c_{13}\end{pmatrix}\cdot U_{Maj}$ (1)

where, $c_{ij}=\cos{\theta_{ij}}$, $s_{ij}=\sin{\theta_{ij}}$.The diagonal
matrix $U_{Maj}=diag(1,e^{\iota\alpha},e^{\iota(\beta+\gamma)})$ holds the
Majorana CP phases, $\alpha$ and $\beta$, which become detectable if neutrinos
act as Majorana particles. Discovering neutrinoless double beta decay is
likely the key to proving that neutrinos are Majorana particles, but these
decays are yet to be observed. Symmetry is an essential factor in explaining
this issue, as Wendell Furry proposed the Majorana nature of particles and
studied a kinetic process similar to neutrinoless double beta decay[19, 20].
By producing a pair of electrons, this process breaks the lepton number by two
units, generating Majorana neutrino masses as the electroweak symmetry breaks
$(A,Z)\rightarrow(A,Z+2)+2e^{-}$. The large value of the lepton number
violation scale($\Lambda\sim 10^{14}-10^{15}$GeV) is associated with the
smallness of observed neutrino masses since they are zero in the standard
model[21]. To generate non-zero neutrino mass, it is necessary to develop a
new model beyond the standard model, such as effective theories that use the
Weinberg operator[22].

Several alternative frameworks to the standard model exist that can explain
the origin of neutrino masses. These include the Seesaw Mechanism [23, 24, 25,
26, 27], the Minimal Supersymmetric Standard Model
(MSSM)[28],Supersymmetry[29], the Next-to-Minimal Supersymmetric Standard
Model (NMSSM)[30], string theory[31], Radiative Seesaw Mechanism[32], models
based on extra dimensions[33], and various other models. Numerous neutrino
experiments have demonstrated the presence of tiny but non-zero neutrino
masses and provided evidence of flavor mixing[34, 35].Several researchers have
proposed different patterns of lepton mixing. Phenomenological patterns of
neutrino mixing include Trimaximal (TM1/TM2)[36, 37, 38], Tri-bimaximal
(TBM)[39, 40], Bi-large mixing patterns[41, 42, 43] and Quasi-degenerate
neutrino mass models. Additionally, several models based on non-abelian
discrete symmetries[44] such as $A_{4}$[45],[46, 32], $S_{3}$[32], $S_{4}$,
$\Delta(27)$[47, 48, 49] and $\Delta(54)$[50, 51] have been suggested to
achieve tribimaximal mixing (TBM), and deviations from TBM are obtained by
adding extra flavons.

This study introduces a new approach, the inverse seesaw, using the
$\Delta(54)$ flavour symmetry framework. A previous proposal by the authors of
[[52]] suggested Type I and inverse seesaw for Dirac neutrinos with A4 flavour
symmetry. Lately, endeavors have been made in this particular area,
particularly concerning the type I seesaw, type II seesaw and type I+II for
Dirac neutrinos within the $\Delta(54)$ symmetry [53].The current work
presents a more concise version of the inverse seesaw, utilizing two SM Higgs.
To deviate from the specific TBM neutrino mixing pattern, we incorporated
extra flavons $\chi$, $\eta$, $\zeta$, $\xi$,$\Phi_{S}$ and $\phi$ under
$\Delta(54)$ and expanded the flavon sector of the model. In addition, we
incorporated a $Z_{2}\otimes Z_{3}\otimes Z_{4}$ symmetry into our model to
prevent unwanted terms and facilitate the construction of specific coupling
matrices. We modified the structure of the neutrino mass matrix $M_{\nu}$ that
characterizes the masses of neutrinos and conducted symmetry-based studies.
This approach allows us to thoroughly examine $M_{\nu}$, and investigate the
Jarlskog invariant parameter ($J$) and the NDBD parameter ($m_{ee}$). The
approach we take sets our work apart from that of others.

Our paper’s structure is arranged as follows: Section 2 presents an outline of
the model’s framework, including the fields and their symmetrical
transformation properties. Section 3 contains a numerical analysis and
examination of the neutrino phenomenology results. Lastly, in section 4, we
offer our concluding remarks.

Parameters | NH (3$\sigma$) | IH (3$\sigma$)
---|---|---
$\Delta{m}^{2}_{21}[{10}^{-5}eV^{2}]$ | $6.82\rightarrow 8.03$ | $6.82\rightarrow 8.03$
$\Delta{m}^{2}_{31}[{10}^{-3}eV^{2}]$ | $2.428\rightarrow 2.597$ | $-2.581\rightarrow-2.408$
$\sin^{2}\theta_{12}$ | $0.270\rightarrow 0.341$ | $0.270\rightarrow 0.341$
$\sin^{2}\theta_{13}$ | $0.02029\rightarrow 0.02391$ | $0.02047\rightarrow 0.02396$
$\sin^{2}\theta_{23}$ | $0.406\rightarrow 0.620$ | $0.410\rightarrow 0.623$
$\delta_{CP}$ | $108\rightarrow 404$ | $192\rightarrow 360$
Table 1: The $3\sigma$ range of neutrino oscillation parameters from NuFIT 5.2
(2022) [54]

##  2 Framework of the Model

To achieve the realization of the Inverse seesaw mechanism, it is necessary to
expand the fermion sector of the Standard Model. This can be done by
introducing Right-Handed Neutrinos $N_{R_{1}}$ and $N_{R_{2}}$ and additional
neutral fermions $S_{1}$ and $S_{2}$, which are singlets with respect to the
Standard Model framework, to the existing active neutrinos $\nu$. Superstring
theories can yield certain classes of non-Abelian flavor symmetries, such as
$\Delta(54)$ flavor symmetries, which are present in heterotic orbifold
models. This discussion will focus on the $\Delta(54)$ discrete symmetry,
which contains two $Z_{3}$ subgroups and an $S_{3}$ subgroup. The $S_{3}$
subgroup is considered the minimal non-Abelian discrete symmetry, and the
semi-direct product structure of $\Delta(54)$ between $Z_{3}$ and $S_{3}$
produces triplet irreducible representations. The $\Delta(54)$ symmetry can
therefore result in intriguing models. The $\Delta(54)$ group includes
irreducible representations $1_{+}$, $1_{-}$, $2_{1}$, $2_{2}$, $2_{3}$,
$2_{4}$, $3_{1(1)}$, $3_{1(2)}$, $3_{2(1)}$ and $3_{2(2)}$. While there are
four triplets, the products of $3_{1(1)}\otimes 3_{1(2)}$ , $3_{2(1)}\otimes
3_{2(2)}$ ,$3_{1(1)}\otimes 3_{2(2)}$ and $3_{1(2)}\otimes 3_{2(1)}$ lead to
the trivial singlet. Despite its numerous fascinating aspects, few authors
have considered the $\Delta(54)$ symmetry thus far.

The multiplication rules are as follows:

$3_{1(1)}\otimes 3_{1(1)}=3_{1(2)}\oplus 3_{1(2)}\oplus 3_{2(2)}$
$3_{1(2)}\otimes 3_{1(2)}=3_{1(1)}\oplus 3_{1(1)}\oplus 3_{2(1)}$
$3_{2(1)}\otimes 3_{2(1)}=3_{1(2)}\oplus 3_{1(2)}\oplus 3_{2(2)}$
$3_{2(2)}\otimes 3_{2(2)}=3_{1(1)}\oplus 3_{1(1)}\oplus 3_{2(1)}$
$3_{1(1)}\otimes 3_{1(2)}=1_{+}\oplus 2_{1}\oplus 2_{2}\oplus 2_{3}\oplus
2_{4}$ $3_{1(2)}\otimes 3_{2(1)}=1_{-}\oplus 2_{1}\oplus 2_{2}\oplus
2_{3}\oplus 2_{4}$ $3_{2(1)}\otimes 3_{2(2)}=1_{+}\oplus 2_{1}\oplus
2_{2}\oplus 2_{3}\oplus 2_{4}$ $3_{1(1)}\otimes 3_{2(2)}=1_{-}\oplus
2_{1}\oplus 2_{2}\oplus 2_{3}\oplus 2_{4}$

For two triplets

$a=(a_{1},a_{2},a_{3})$ $b=(b_{1},b_{2},b_{3})$

We can write

$1_{+}=(ab)_{+}=a_{1}b_{1}+a_{2}b_{2}+a_{3}b_{3}$
$1_{-}=(ab)_{-}=a_{1}b_{1}+a_{2}b_{2}+a_{3}b_{3}$

Field | L | $l$ | $H_{1}$ | $H_{2}$ | $\nu_{L}$ | $N_{R_{1}}$ | $N_{R_{2}}$ | $S_{1}$ | $S_{2}$ | $\chi$ | $\eta$ | $\zeta$ | $\xi$ | $\Phi_{S}$ | $\phi$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
SU(2) | 2 | 1 | 1 | 1 | 1 | 2 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1
$\Delta(54)$ | $3_{1(1)}$ | $3_{1(1)}$ | $1_{-}$ | $1_{+}$ | $3_{2(2)}$ | $3_{1(1)}$ | $3_{2(2)}$ | $3_{1(1)}$ | $3_{2(2)}$ | $3_{2(1)}$ | $3_{2(2)}$ | $1_{-}$ | $3_{2(1)}$ | $3_{2(1)}$ | $1_{-}$
Z2 | 1 | 1 | 1 | 1 | -1 | -1 | 1 | -1 | 1 | 1 | 1 | -1 | -1 | -1 | -1
Z3 | $\omega^{2}$ | $\omega$ | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | $\omega$ | $\omega$ | $\omega$
Z4 | 1 | 1 | 1 | 1 | i | 1 | -i | i | 1 | 1 | 1 | 1 | 1 | 1 | i
Table 2: Full particle content of our model

Our model is based on the $\Delta 54$ model where we introduced additional
flavons $\chi$, $\xi$, $\eta$ ,$\zeta$ , $\Phi_{S}$ and $\phi$ to get the
deviation from the precise TBM pattern of neutrino mixing[55]. We put extra
symmetry $Z_{2}\otimes Z_{3}\otimes Z_{4}$ to avoid unwanted terms. Table 2
provides details regarding the composition of the particles and corresponding
charge assignment in accordance with the symmetry group. The triplet
representation of $\Delta 54$ is used to assign the right-handed charged
leptons and the left-handed lepton doublets.

The invariant Yukawa Lagrangian is as follows 111Considering terms upto
dimension-5.:

$\mathcal{L}\equiv\frac{y}{\Lambda}(lL)\chi
H_{1}+\frac{LH_{1}N_{R_{1}}}{\Lambda}y_{\xi}\xi+\frac{LH_{1}N_{R_{1}}}{\Lambda}y_{s}\Phi_{s}+\frac{LH_{2}N_{R_{1}}}{\Lambda}y_{a}\Phi_{s}\\\
+\frac{y_{\scaleto{RN}{3pt}}}{\Lambda}\nu_{L}N_{R_{2}}\eta\zeta+y_{\scaleto{NS}{3pt}}S_{1}N_{R_{2}}\zeta+y_{\scaleto{NS}{3pt}}^{\prime}S_{2}N_{R_{1}}\zeta+\frac{y_{s}}{\Lambda}S_{1}S_{2}\phi^{3}$
(2)

We consider the vacuum expectation values as,

$\displaystyle\langle\Phi_{S}\rangle$ $\displaystyle=(v_{s},v_{s},v_{s})$
$\displaystyle\langle\chi\rangle$ $\displaystyle=(v_{\chi},v_{\chi},v_{\chi})$
$\displaystyle\langle\xi\rangle$ $\displaystyle=(v_{\xi},v_{\xi},v_{\xi})$ (3)
$\displaystyle\langle\eta\rangle$ $\displaystyle=(v_{\eta},v_{\eta},v_{\eta})$
$\displaystyle\langle\zeta\rangle$ $\displaystyle=(v_{\zeta})$
$\displaystyle\langle\phi\rangle$ $\displaystyle=(v_{\phi})$ (4)

The charged lepton mass matrix is given as

$M_{l}=\frac{v_{\scaleto{\chi}{3pt}}H}{\Lambda}\begin{pmatrix}y_{e}&0&0\\\
0&y_{\mu}&0\\\ 0&0&y_{\tau}\end{pmatrix}$ (5)

where, $y_{e}$, $y_{\mu}$ and $y_{\tau}$ are the coupling constants.

###  2.1 Effective neutrino mass matrix

By utilizing the Lagrangian presented, it is possible to derive the mass
matrices that pertain to the neutrino sector once both $\Delta{54}$ and
electroweak symmetry breaking have occurred.

$M_{RN}=\frac{y_{\scaleto{RN}{3pt}}}{\Lambda}\begin{pmatrix}v_{\eta}v_{\zeta}&0&0\\\
0&v_{\eta}v_{\zeta}&0\\\ 0&0&v_{\eta}v_{\zeta}\end{pmatrix}$ (6)

$M_{NS}=\frac{y_{\scaleto{NS}{3pt}}}{\Lambda}\begin{pmatrix}v_{\zeta}&0&0\\\
0&v_{\zeta}&0\\\ 0&0&v_{\zeta}\end{pmatrix}$ (7)

$M_{NS}^{\prime}=\frac{y_{\scaleto{NS}{3pt}}^{\prime}}{\Lambda}\begin{pmatrix}v_{\zeta}&0&0\\\
0&v_{\zeta}&0\\\ 0&0&v_{\zeta}\end{pmatrix}$ (8)

$M_{S}=\frac{y_{\scaleto{S}{3pt}}}{\Lambda^{2}}\begin{pmatrix}v_{\phi}&0&0\\\
0&v_{\phi}&0\\\ 0&0&v_{\phi}\end{pmatrix}$ (9)

$M_{\nu
N}=\frac{v}{\Lambda}\begin{pmatrix}y_{\scaleto{\xi}{3pt}}\xi&y_{\scaleto{s}{3pt}}v_{s}+y_{\scaleto{a}{3pt}}v_{a}&y_{\scaleto{s}{3pt}}v_{s}-y_{\scaleto{a}{3pt}}v_{a}\\\
y_{\scaleto{s}{3pt}}v_{s}-y_{\scaleto{a}{3pt}}v_{a}&y_{\scaleto{\xi}{3pt}}\xi&y_{\scaleto{s}{3pt}}v_{s}+y_{\scaleto{a}{3pt}}v_{a}\\\
y_{\scaleto{s}{3pt}}v_{s}+y_{\scaleto{a}{3pt}}v_{a}&y_{\scaleto{s}{3pt}}v_{s}-y_{\scaleto{a}{3pt}}v_{a}&y_{\scaleto{\xi}{3pt}}\xi\end{pmatrix}$
(10)

Effective neutrino mass matrix is given by

$m_{\nu}=M_{RN}(M_{NS}^{\prime})^{-1}M_{S}M_{NS}^{-1}M_{\nu N}$ (11)
$=\lambda\begin{pmatrix}x&s+a&s-a\\\ s-a&x&s+a\\\ s+a&s-a&x\end{pmatrix}$ (12)

$=\begin{pmatrix}x^{\prime}&s^{\prime}+a^{\prime}&s^{\prime}-a^{\prime}\\\
s^{\prime}-a^{\prime}&x^{\prime}&s^{\prime}+a^{\prime}\\\
s^{\prime}+a^{\prime}&s^{\prime}-a^{\prime}&x^{\prime}\end{pmatrix}$ (13)

where, $x=y_{\scaleto{\xi}{3pt}}\xi$ , $s=y_{\scaleto{s}{3pt}}v_{s}$ and
$a=y_{\scaleto{a}{3pt}}v_{a}$
We now define the Hermitian matrix as

$M_{\nu}=m_{\nu}m_{\nu}^{\dagger}$ (14)

$=\begin{pmatrix}x^{\prime 2}+2(s^{\prime 2}+a^{\prime 2})&s^{\prime
2}-a^{\prime 2}+2s^{\prime}x^{\prime}&s^{\prime 2}-a^{\prime
2}+2s^{\prime}x^{\prime}\\\ s^{\prime 2}-a^{\prime
2}+2s^{\prime}x^{\prime}&x^{\prime 2}+2(s^{\prime 2}+a^{\prime 2})&s^{\prime
2}-a^{\prime 2}+2s^{\prime}x^{\prime}\\\ s^{\prime 2}-a^{\prime
2}+2s^{\prime}x^{\prime}&s^{\prime 2}-a^{\prime
2}+2s^{\prime}x^{\prime}&x^{\prime 2}+2(s^{\prime 2}+a^{\prime
2})\end{pmatrix}$ (15)

where,
$\lambda=\frac{vv_{n}v_{s}v_{\phi}^{3}y_{RN}y_{s}}{{\Lambda}^{4}v_{\zeta}^{2}y_{NS}y_{NS}^{\prime}}$
, $x^{\prime}=x{\lambda}$, $s^{\prime}=s{\lambda}$ and $a^{\prime}=a{\lambda}$

##  3 Numerical Analysis and results

In the earlier section, we illustrated the method of enhancing the $\Delta 54$
model by incorporating supplementary flavons. The subsequent section comprises
a quantitative investigation aimed at examining the effectiveness of the
parameters in generating a departure from TBM in neutrino mixing. The outcomes
of this analysis, restricted to the normal hierarchy scenario, will be
deliberated.

The neutrino mass matrix $m_{\nu}$ can be diagonalized by the PMNS matrix $U$
as

$U^{\dagger}m^{(i)}_{\nu}U^{*}=\textrm{diag(}m_{1},m_{2},m_{3}\textrm{)}$ (16)

We can numerically calculate $U$ using the relation
$U^{\dagger}M_{\nu}U=\textrm{diag(}m_{1}^{2},m_{2}^{2},m_{3}^{2}\textrm{)}$,
where $M_{\nu}=m_{\nu}m^{\dagger}_{\nu}$. The neutrino oscillation parameters
$\theta_{12}$, $\theta_{13}$, $\theta_{23}$ and $\delta$ can be obtained from
$U$ as

$s_{12}^{2}=\frac{\lvert U_{12}\rvert^{2}}{1-\lvert
U_{13}\rvert^{2}},~{}~{}~{}~{}~{}~{}s_{13}^{2}=\lvert
U_{13}\rvert^{2},~{}~{}~{}~{}~{}~{}s_{23}^{2}=\frac{\lvert
U_{23}\rvert^{2}}{1-\lvert U_{13}\rvert^{2}}$ (17)

and $\delta$ may be given by

$\delta=\textrm{sin}^{-1}\left(\frac{8\,\textrm{Im(}h_{12}h_{23}h_{31}\textrm{)}}{P}\right)$
(18)

with

$P=(m_{2}^{2}-m_{1}^{2})(m_{3}^{2}-m_{2}^{2})(m_{3}^{2}-m_{1}^{2})\sin
2\theta_{12}\sin 2\theta_{23}\sin 2\theta_{13}\cos\theta_{13}$ (19)

To assess how the neutrino mixing parameters compare to the most recent
experimental data [54], we adjusted the modified $\Delta 54$ model to fit the
experimental data by minimizing the subsequent $\chi^{2}$ function:

$\chi^{2}=\sum_{i}\left(\frac{\lambda_{i}^{model}-\lambda_{i}^{expt}}{\Delta\lambda_{i}}\right)^{2},$
(20)

where $\lambda_{i}^{model}$ is the $i^{th}$ observable predicted by the model,
$\lambda_{i}^{expt}$ stands for $i^{th}$ experimental best-fit value and
$\Delta\lambda_{i}$ is the 1$\sigma$ range of the observable.

Figure 1: Allowed regions of the model parameters $\lvert x\rvert$,
$\phi_{x}$, $\lvert s\rvert$, $\phi_{s}$ and $\lvert a\rvert$, $\phi_{a}$ in
NH. The best fit values are indicated by red dots.

The parameter of the model space is illustrated in Fig.1, with restrictions
based on the 3$\sigma$ limit of neutrino oscillation data presented in Table1.
The illustration indicates a strong interdependence among various parameters
of the model. The best-fit values for $\lvert x\rvert$, $\lvert s\rvert$,
$\lvert a\rvert$, $\phi_{x}$, $\phi_{s}$ and $\phi_{a}$ obtained are (0.062,
0951, 0.083, 0.505$\pi$, -2.953$\pi$, -0.092$\pi$).

Fig. 2 predicts the expected values of the neutrino oscillation parameters
within the model for NH. The best fit values of $\sin^{2}\theta_{12}$,
$\sin^{2}\theta_{13}$ and $\sin^{2}\theta_{23}$ are (0.320, 0.0214, 0.417)
which are within the 3$\sigma$ range of experimental measurements. Additional
parameters such as $\Delta m_{21}^{2}$, $\Delta m_{31}^{2}$ and $\delta_{CP}$
have their best-fit values, corresponding to $\chi^{2}$-minimum, at
($6.88\times 10^{-5}$ eV, $2.50\times 10^{-3}$ eV, $0.115\pi$).

Fig. 3 gives the correlation between the two mass square difference predicted
by the model for NH. The $\Delta m_{21}^{2}$ and $\Delta m_{31}^{2}$ have
their best-fit values, corresponding to $\chi^{2}$-minimum, at ($6.88\times
10^{-5}$ eV, $2.507\times 10^{-3}$ eV) respectively.

Fig. 4 gives the correlation between the Dirac CP phase and atmospheric angle
and also Dirac CP with the ratio of the two mass squared difference. The best
fit value of $\delta_{CP}$ is predicted to be around 0.115$\pi$.

Thus, the model defined in this work indicates clear deviation from tri-
bimaximal mixing.

Figure 2: Correlation among the mixing angles and among mixing angle and ratio
of mass-squared difference. The best fit value is indicated by the red dot.

Figure 3: Variation of the mass squared differences with mixing angles and
correlation between two mass squared differences.

Figure 4: Correlation between Dirac CP ($\delta_{CP}$) and atmospheric mixing
angle($sin\theta_{23}$) and ratio of mass squared difference respectively. The
best fit value for 3$\sigma$ range is given by red dot.

It is clear from the correlation plot among the neutrino oscillation
parameters that the mixing of neutrinos deviates from the TBM mixing in NH.
The prediction of mixing angle $\theta_{23}$ in NH scenario indicates a
preference towards the lower octant. With the modification of $\Delta 54$
model with additional term, it is possible to deviate from TBM mixing. The
Table 3 shows the best-fit values of the various oscillation parameters.

Parameter | NH
---|---
$\sin^{2}\theta_{12}$ | 0.320
$\sin^{2}\theta_{13}$ | 0.0214
$\sin^{2}\theta_{23}$ | 0.417
$\delta_{CP}/\pi$ | 0.115
$\Delta m_{21}^{2}$ | $6.880\times 10^{-5}$
$\Delta m_{31}^{2}$ | $2.507\times 10^{-3}$
Table 3: Best-fit values for different parameters predicted by the model

Figure 5: Correlation between the Jarlskog invariant parameter($J$) with Dirac
CP($\delta_{CP}$) phase and atmospheric mixing angle($sin^{2}\theta_{23}$)
respectively. The best fit value for 3$\sigma$ range is given by black +.

Prediction for Jarlskog invariant Parameter: In Fig.5 We are sharing further
forecasts generated by the revised model concerning CP-violation’s Jarlskog
parameter($J$) and the effective Majorana mass($m_{ee}$) that defines the
neutrinoless double beta decay. The Jarlskog constant is a quantity that
remains unchanged even after a phase redefinition.

$J=Im\\{{U_{11}U_{22}U_{12}^{*}U_{21}^{*}}\\}=s_{12}c_{13}^{2}s_{12}c_{12}s_{23}c_{23}sin\delta$
(21)

Neutrinoless double beta decay (NDBD): The NDBD phenomenon is crucial in the
field of neutrino physics, as it involves the light Majorana neutrinos. The
process is governed by an effective mass, denoted as $m_{\beta\beta}$, which
can be calculated using the equation:

$m_{\beta\beta}=U^{2}_{Li}m_{i}$ (22)

where $U_{Li}$ are the elements of the first row of the neutrino mixing matrix
$U_{PMNS}$. This equation depends on certain known parameters such as
$\theta_{12}$ and $\theta_{13}$, as well as unknown Majorana phases denoted by
$\alpha$ and $\beta$. The diagonalizing matrix of the light neutrino mass
matrix, denoted by $m_{\nu}$, is represented by $U_{PMNS}$, such that

$m_{\nu}=U_{PMNS}M^{(diag)}_{\nu}U^{T}_{PMNS}$ (23)

where, $M^{(diag)}_{\nu}$ =diag($m_{1}$, $m_{2}$, $m_{3}$). The effective
Majorana mass can be expressed applying the diagonalizing matrix elements and
the mass eigenvalues as follows:

$m_{\beta\beta}=m_{1}c_{12}^{2}c^{2}_{13}+m_{2}s^{2}_{12}c^{2}_{13}e^{2\iota\alpha}+m_{3}s^{2}_{13}e^{2\iota\beta}$
(24)

where $c_{12}$ and $s_{12}$ are the cosine and sine of the mixing angle
$\theta_{12}$, respectively.

After analyzing the restricted parameter space, we have computed the value of
$m_{\beta\beta}$ in the NH scenario. The figure labeled as 6 depicts the
changes in $m_{\beta\beta}$ as per the lightest neutrino mass. Additionally,
the sensitivity range of experiments like GERDA, KamLAND-Zen and LEGEND-1k for
$0\nu\beta\beta$ decay is also illustrated in the same figure. As per the
findings, $m_{\beta\beta}$ falls well within the reach of these NDBD
experiments for normal hierarchy.

Figure 6:  Correlation between Effective Majorana neutrino mass($m_{ee}$) with
lightest neutrino mass($m_{1}$) and Jarlskog invariant($J$).

##  4 Conclusion

We have introduced $\Delta(54)$ flavor model with two SM Higgs along with
$Z_{2}\otimes Z_{3}\otimes Z_{4}$ symmetry that generate a neutrino mass
matrix. We have constructed a flavor-symmetric approach in Inverse Seesaw
mechanism to realize the neutrino masses and mixing to fit present neutrino
oscillation data, including the non-zero reactor angle ($\theta_{13}$) and
Dirac CP ($\delta_{CP}$). We use additional flavons to obtain the desired
mixing pattern. The model under consideration clearly shows deviation from TBM
mixing of the neutrino mixing matrix. The predictions for the neutrino
oscillation parameters derived from the resulting mass matrix are in agreement
with the best-fit values obtained through $\chi^{2}$ analysis. The prediction
of the mixing angles, mass-squared differences, and the Dirac CP phase in the
IH case do not agree with the experimental data. The prediction of mixing
angles in NH case indicates that for the parameter space under consideration,
the model prefers lower octant of $\theta_{23}$.

Furthermore, we investigated the Jarlskog invariant parameter and NDBD within
the context of our $\Delta(54)$ models. The effective Majorana neutrino mass
$|m_{ee}|$ is well within the sensitivity reach of the recent $0\nu\beta\beta$
experiments. In extension of this work, the model can be used to study
leptogenesis and dark matter. The consistency of the model predictions with
the latest neutrino data implies that model may be tested in future neutrino
oscillation experiments.

## Acknowledgements

HB acknowledges Tezpur University, India for Institutional Research
Fellowship. The research of NKF is funded by DST-SERB, India under Grant no.
EMR/2015/001683. AB acknowledges the CSIR, India for Senior Research
Fellowship(file no 09/796(0072)/2017- EMR-1). BT acknowledges the DST, India
for INSPIRE Fellowship vide Grant no. DST/INSPIRE/2018/IF180588. HB is
thankful to Shawan Kumar Jha for fruitful discussions.

## References

  * [1] T. Phong Nguyen, T. Thuy Le, T. T. Hong, and L. T. Hue.  Decay of standard model-like Higgs boson $h\rightarrow\mu\tau$ in a 3-3-1 model with inverse seesaw neutrino masses.  Phys. Rev. D, 97(7):073003, 2018.
  * [2] Stephen F. King, Alexander Merle, Stefano Morisi, Yusuke Shimizu, and Morimitsu Tanimoto.  Neutrino Mass and Mixing: from Theory to Experiment.  New J. Phys., 16:045018, 2014.
  * [3] S. F. King.  Neutrino mass models.  Rept. Prog. Phys., 67:107–158, 2004.
  * [4] S. Cao, A. Nath, T. V. Ngoc, P. T. Quyen, N. T. Hong Van, and Ng. K. Francis.  Physics potential of the combined sensitivity of T2K-II, NO$\nu$A extension, and JUNO.  Phys. Rev. D, 103(11):112010, 2021.
  * [5] Y. H. Ahn and Paolo Gondolo.  Towards a realistic model of quarks and leptons, leptonic $CP$ violation, and neutrinoless $\beta\beta$-decay.  Phys. Rev. D, 91:013007, 2015.
  * [6] Arthur B McDonald.  Nobel lecture: the sudbury neutrino observatory: observation of flavor change for solar neutrinos.  Reviews of Modern Physics, 88(3):030502, 2016.
  * [7] T. Phong Nguyen, T. T. Thuc, D. T. Si, T. T. Hong, and L. T. Hue.  Low-energy phenomena of the lepton sector in an A4 symmetry model with heavy inverse seesaw neutrinos.  PTEP, 2022(2):023B01, 2022.
  * [8] T. T. Hong, N. H. T. Nha, T. Phong Nguyen, L. T. T. Phuong, and L. T. Hue.  Decays h → eaeb, eb → ea$\gamma$, and (g $-$ 2)e,$\mu$ in a 3-3-1 model with inverse seesaw neutrinos.  PTEP, 2022(9):093B05, 2022.
  * [9] Takaaki Kajita.  Nobel lecture: Discovery of atmospheric neutrino oscillations.  Reviews of Modern Physics, 88(3):030501, 2016.
  * [10] Pablo F de Salas, DV Forero, S Gariazzo, P Martínez-Miravé, O Mena, CA Ternes, M Tórtola, and JWF Valle.  2020 global reassessment of the neutrino oscillation picture.  Journal of High Energy Physics, 2021(2):1–36, 2021.
  * [11] Hiroshi Okada and Morimitsu Tanimoto.  Spontaneous cp violation by modulus $\tau$ in a4 model of lepton flavors.  Journal of High Energy Physics, 2021(3):1–27, 2021.
  * [12] Y. H. Ahn, Sin Kyu Kang, and Hyun Min Lee.  Toward a model of quarks and leptons.  Phys. Rev. D, 106(7):075029, 2022.
  * [13] T. Phong Nguyen, L. T. Hue, D. T. Si, and T. T. Thuc.  CP violations in a predictive $A_{4}$ symmetry model.  PTEP, 2020(3):033B04, 2020.
  * [14] F. P. An et al.  Observation of electron-antineutrino disappearance at Daya Bay.  Phys. Rev. Lett., 108:171803, 2012.
  * [15] J. K. Ahn et al.  Observation of Reactor Electron Antineutrino Disappearance in the RENO Experiment.  Phys. Rev. Lett., 108:191802, 2012.
  * [16] P. Adamson et al.  Improved search for muon-neutrino to electron-neutrino oscillations in MINOS.  Phys. Rev. Lett., 107:181802, 2011.
  * [17] Y. Abe et al.  Indication of Reactor $\bar{\nu}_{e}$ Disappearance in the Double Chooz Experiment.  Phys. Rev. Lett., 108:131801, 2012.
  * [18] K. Abe et al.  Indication of Electron Neutrino Appearance from an Accelerator-produced Off-axis Muon Neutrino Beam.  Phys. Rev. Lett., 107:041801, 2011.
  * [19] W. H. Furry.  On transition probabilities in double beta-disintegration.  Phys. Rev., 56:1184–1193, 1939.
  * [20] Stefano Dell’Oro, Simone Marcocci, Matteo Viel, and Francesco Vissani.  Neutrinoless double beta decay: 2015 review.  Adv. High Energy Phys., 2016:2162659, 2016.
  * [21] S. M. Bilenky and Carlo Giunti.  Neutrinoless double-beta decay: A brief review.  Mod. Phys. Lett. A, 27:1230015, 2012.
  * [22] Steven Weinberg.  Baryon and Lepton Nonconserving Processes.  Phys. Rev. Lett., 43:1566–1570, 1979.
  * [23] Tsutomu Yanagida.  Horizontal Symmetry and Masses of Neutrinos.  Prog. Theor. Phys., 64:1103, 1980.
  * [24] Peter Minkowski.  $\mu\to e\gamma$ at a Rate of One Out of $10^{9}$ Muon Decays?  Phys. Lett. B, 67:421–428, 1977.
  * [25] M Gell-Mann.  Ramond and r. slansky,“complex spinors and unified theories”.  Supergravity, P. van Nieuwenhuizen and DZ Freedman eds., North Holland, Amsterdam The Netherlands, 315, 1979.
  * [26] Rabindra N. Mohapatra and Goran Senjanovic.  Neutrino Mass and Spontaneous Parity Nonconservation.  Phys. Rev. Lett., 44:912, 1980.
  * [27] T Yanagida.  Horizontal symmetry and masses of neutrinos, in the proceedings of the.  In Workshop on unified theory and baryon number in the universe. KEK, Tsukuba, Japan, 1979.
  * [28] Csaba Csaki.  The Minimal supersymmetric standard model (MSSM).  Mod. Phys. Lett. A, 11:599, 1996.
  * [29] Ernest Ma.  Supersymmetry and neutrino masses.  PoS, corfu98:047, 1998.
  * [30] Ulrich Ellwanger, Cyril Hugonie, and Ana M. Teixeira.  The Next-to-Minimal Supersymmetric Standard Model.  Phys. Rept., 496:1–77, 2010.
  * [31] Luis E. Ibanez and Angel M. Uranga.  String theory and particle physics: An introduction to string phenomenology.  Cambridge University Press, 2 2012.
  * [32] Ernest Ma.  Verifiable radiative seesaw mechanism of neutrino mass and dark matter.  Phys. Rev. D, 73:077301, 2006.
  * [33] Nima Arkani-Hamed, Savas Dimopoulos, G. R. Dvali, and John March-Russell.  Neutrino masses from large extra dimensions.  Phys. Rev. D, 65:024032, 2001.
  * [34] Max Aker, K Altenmüller, M Arenz, M Babutzka, J Barrett, S Bauer, M Beck, A Beglarian, J Behrens, T Bergmann, et al.  Improved upper limit on the neutrino mass from a direct kinematic method by katrin.  Physical review letters, 123(22):221802, 2019.
  * [35] Ng. K. Francis.  Baryon Asymmetry in Neutrino Mass Models with and without ${\theta}_{13}$, 1 2014.
  * [36] Carl H. Albright and Werner Rodejohann.  Comparing Trimaximal Mixing and Its Variants with Deviations from Tri-bimaximal Mixing.  Eur. Phys. J. C, 62:599–608, 2009.
  * [37] Xiao-Gang He and A. Zee.  Minimal Modification to Tri-bimaximal Mixing.  Phys. Rev. D, 84:053004, 2011.
  * [38] Bikash Thapa and Ng K Francis.  Resonant leptogenesis and TM1 mixing in minimal type-I seesaw model with S4 symmetry.  The European Physical Journal C, 81(12):1–8, 2021.
  * [39] P. F. Harrison, D. H. Perkins, and W. G. Scott.  Tri-bimaximal mixing and the neutrino oscillation data.  Phys. Lett. B, 530:167, 2002.
  * [40] P. F. Harrison and W. G. Scott.  Symmetries and generalizations of tri - bimaximal neutrino mixing.  Phys. Lett. B, 535:163–169, 2002.
  * [41] S. Morisi and E. Peinado.  An A(4) model for lepton masses and mixings.  Phys. Rev. D, 80:113011, 2009.
  * [42] Peng Chen, Gui-Jun Ding, Rahul Srivastava, and José W. F. Valle.  Predicting neutrino oscillations with “bi-large” lepton mixing matrices.  Phys. Lett. B, 792:461–464, 2019.
  * [43] Gui-Jun Ding, Newton Nath, Rahul Srivastava, and José W. F. Valle.  Status and prospects of ‘bi-large’ leptonic mixing.  Phys. Lett. B, 796:162–167, 2019.
  * [44] Stephen F. King and Christoph Luhn.  Neutrino Mass and Mixing with Discrete Symmetry.  Rept. Prog. Phys., 76:056201, 2013.
  * [45] Animesh Barman, Ng. K. Francis, Bikash Thapa, and Ankur Nath.  Nonzero $\theta$13, CP-violation and neutrinoless double beta decay for neutrino mixing in the A4×Z2×Z3 flavor symmetry model.  Int. J. Mod. Phys. A, 38(02):2350012, 2023.
  * [46] Hajime Ishimori and Ernest Ma.  New Simple $A_{4}$ Neutrino Model for Nonzero $\theta_{13}$ and Large $\delta_{CP}$.  Phys. Rev. D, 86:045030, 2012.
  * [47] Ernest Ma.  Near tribimaximal neutrino mixing with $\Delta(27)$ symmetry.  Phys. Lett. B, 660:505–507, 2008.
  * [48] I de Medeiros Varzielas, SF King, and GG Ross.  Neutrino tri-bi-maximal mixing from a non-abelian discrete family symmetry.  Physics Letters B, 648(2-3):201–206, 2007.
  * [49] P. F. Harrison, R. Krishnan, and W. G. Scott.  Deviations from tribimaximal neutrino mixing using a model with $\Delta(27)$ symmetry.  Int. J. Mod. Phys. A, 29(18):1450095, 2014.
  * [50] Hajime Ishimori, Tatsuo Kobayashi, Hiroshi Okada, Yusuke Shimizu, and Morimitsu Tanimoto.  Lepton Flavor Model from $\Delta(54)$ Symmetry.  JHEP, 04:011, 2009.
  * [51] M. A. Loualidi.  Trimaximal Mixing with One Texture Zero from Type II Seesaw and $\Delta$(54) Family Symmetry.  In Beyond Standard Model: From Theory to Experiment, 4 2021.
  * [52] Debasish Borah and Biswajit Karmakar.  $A_{4}$ flavour model for Dirac neutrinos: Type I and inverse seesaw.  Phys. Lett. B, 780:461–470, 2018.
  * [53] Monal Kashav, Labh Singh, and Surender Verma.  Trimaximal mixing and extended magic symmetry in a model of neutrino mass matrix.  arXiv preprint arXiv:2207.13328, 2022.
  * [54] Ivan Esteban, M. C. Gonzalez-Garcia, Michele Maltoni, Thomas Schwetz, and Albert Zhou.  The fate of hints: updated global analysis of three-flavor neutrino oscillations.  JHEP, 09:178, 2020.
  * [55] Guido Altarelli and Ferruccio Feruglio.  Discrete flavor symmetries and models of neutrino mixing.  Reviews of Modern Physics, 82(3):2701–2729, Sep 2010.


envname-P envname#1

# Let’s Sample Step by Step:
for Efficient Reasoning with LLMs

Pranjal Aggarwal1 Aman Madaan2 Yiming Yang2 Mausam1
1Department of Computer Science, Indian Institute of Technology, Delhi
2Language Technologies Institute, Carnegie Mellon University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

A popular approach for improving the correctness of output from large language
models (LLMs) is Self-Consistency – poll the LLM multiple times and output the
most frequent solution. Existing Self-Consistency techniques always draw a
constant number of samples per question, where a better approach will be to
non-uniformly distribute the available budget based on the amount of agreement
in the samples drawn so far. In response, we introduce , a cost-efficient,
model-agnostic technique that dynamically adjusts the number of samples per
question using a lightweight stopping criterion. Our experiments over 13
datasets and two LLMs demonstrate that reduces sample budget by up to 6.0
times with an average accuracy drop of less than 0.1%.111Code and LLM outputs
are available at http://sample-step-by-step.info/.

Figure 1: An overview of : Self-Consistency samples a predetermined number of
answers, whereas iteratively samples until a lightweight Stopping Criteria,
decides to report the majority answer. In the figure, we illustrate an example
where achieves a significant reduction in sampling cost/time from 40 to 10
samples by reporting the majority answer after sampling only ten answers. The
bottom-left graph compares with Self-Consistency on three representative
reasoning datasets, demonstrating that consistently reduces the sample budget
significantly (Average: $2.8\times$) with a negligible drop in accuracy
(Average: $-0.03\%$).

##  1 Introduction

The increasing adoption of large language models (LLMs) across various tasks,
such as text generation and reasoning Wei et al. (2022); Kojima et al. (2022);
Wang et al. (2022a); Mishra et al. (2022), mathematical reasoning (Lewkowycz
et al., 2022; Gao et al., 2022), and code generation (Li et al., 2022; Madaan
et al., 2023), has underscored the importance of improving the correctness of
their outputs. A popular method for achieving this goal is Self-Consistency
Wang et al. (2022b), a majority voting technique where multiple output samples
are generated for a given input, and the final decision is based on the most
frequently occurring output among the samples.

Current Self-Consistency methods typically employ a fixed budget approach,
wherein a predetermined number of samples (e.g., 40) are drawn to make a
decision. However, as LLMs continue to grow in size and complexity, the
sampling time and computational costs associated with majority voting become
increasingly challenging. This challenge is particularly evident in high-
stakes applications like competition-level code generation Li et al. (2022),
where generating a large number of programs, sometimes up to a million, is
essential for maximizing performance.

To address this challenge, we introduce , a cost-efficient, model-agnostic
majority voting technique. employs a lightweight stopping criterion that
dynamically adjusts the number of samples ($n$) for each input, as opposed to
using a fixed budget ($k$). The intuition is that if a clear majority is
established with high confidence after sampling fewer than $k$ answers
($n<k$), there is no need to draw additional samples. Consequently, our method
reduces computational costs without compromising output quality. Our
experiments show that achieves $n<k$ in most cases (on average, $n\sim 0.25k$
for code-davinci-002 and $n\sim 0.55k$ for Vicuna-13B) compared to the fixed-
budget approach of Self-Consistency.

models the probability distribution over unique samples using a Dirichlet
distribution, allowing us to quantify the confidence in the lead of the
majority element over other elements. For instance, if the majority element
has a count of 9 out of the first 10 samples, the likelihood of it remaining
the majority element even after 40 samples is very high ($>99\%$). This allows
to stop sampling at this point, reducing the cost by 30 samples, while Self-
Consistency would continue to sample all 40 answers. As a fast inference-time
technique requiring no additional training, provides a convenient off-the-
shelf option for all pre-trained language models, offering the flexibility to
balance computational cost and performance.

We evaluate on 13 diverse tasks and two LLMs of different scales (Vicuna-13B
and code-davinci-002). Our experimental results show that outperforms Self-
Consistency regarding cost efficiency while maintaining comparable output
quality. On code-davinci-002, reduces the number of samples required by a
factor of $3.7\times$, with an average drop in accuracy: 0.08%. On Vicuna-13B,
requires sampling $2\times$ fewer samples, with almost no drop in accuracy.

In summary, our contributions are:

  * $\bullet$

We propose , a novel and cost-efficient sampling technique for large language
models that dynamically adjusts the number of samples using a lightweight
stopping criterion based on the stability of the majority element.

  * $\bullet$

We develop different stopping criterias for our algorithm, each with its own
advantages and disadvantages, such as providing probabilities, adaptability to
various scenarios, and ease of implementation.

  * $\bullet$

We present experimental results demonstrating that requires fewer samples in
most cases compared to the fixed-budget approach of Self-Consistency, while
maintaining comparable output quality across various natural language
reasoning tasks.

##  2 Methodology

` `

` `

Comparison of (top) and Self-Consistency (bottom). Self-Consistency always
draws a fixed number of samples. In contrast, uses a lightweight stopping
criterion, allowing it to adaptively halt the sampling process, which can lead
to improved efficiency and performance.

###  2.1 Background

#### In-Context Few-Shot Prompting

In-context few-shot prompting is a technique employed by large language models
(LLMs) to learn and generalize from a limited number of examples provided
within the input of a given task. The model can quickly adapt to novel tasks
without fine-tuning or additional training by conditioning the model on a few
examples. Specifically, a prompt $p$ is constructed by concatenating multiple
input-answer example pairs $<x_{i},a_{i}>$. The prompt is then prepended to
the test input $x_{test}$, and the model generates the corresponding answer
$a_{test}$.

#### Self-Consistency

Wang et al. (2022b) proposed Self-Consistency which improved performance over
CoT reasoning by sampling multiple diverse reasoning chains and aggregating
their outputs using a simple majority voting mechanism. However, higher
accuracy is a trade-off for increased computational cost since the model must
be prompted multiple times for the same question.

###  2.2 Adaptive-Consistency

As discussed, the Self-Consistency method consistently samples a predetermined
number of answers ($k$) from the language model before returning the majority
answer. In contrast, the method takes an incremental approach to sampling
outputs from the language model. After generating each sample, employs a
stopping criteria to determine whether it should 1.) produce an additional
sample or 2.) cease sampling and report the current majority answer. This
flexible strategy enables to dynamically adjust the number of samples
generated so far ($n$) for each input. As our experiments demonstrate, $n$ is
typically less than $k$ (up to $4\times$ less in some cases), which suggests
that the method may offer greater cost-efficiency compared to the fixed budget
approach employed by Self-Consistency.

The Algorithm (2) differs from Self-Consistency only in terms of the stopping
criteria. The design of the stopping criteria is crucial to our method, as it
aims to minimize the average number of samples drawn from the LLM while
maximizing accuracy. The simplicity of our algorithm allows for the use of
various stopping criterias interchangeably, each with its own advantages and
disadvantages. We expand on a particular choice of stopping function next.

#### Dirichlet Stopping Criteria

Let $n$ be the number of samples drawn so far, with $m$ unique elements. Let
$v=[v_{1},v_{2},\ldots,v_{m}]$ be the counts of each element, and
$p_{i}=\frac{v_{i}}{n}$ be the normalized count. Considering $n=10$, and $m=3$
(3 unique elements draw), if $v=[8,1,1]$, then we can be more confident that
$v_{1}$ is the answer. On the other hand, if $v=[4,4,2]$, then more samples
need to be drawn. Our goal is to formalize and quantify this intuition.

By convention, let $p_{1}=\max(p_{i})$. We want to assess the stability of
$p_{1}$ as the majority element. Specifically, we want to ask the following
question: what is the probability that $p_{1}$ will be the majority element if
we repeat the process of drawing $n$ samples again? Intuitively, if this
probability is higher than some predetermined threshold $C_{thresh}$, then we
can be more confident in our decision to stop sampling and return $p_{1}$ as
the majority element:

$P(p_{1}>\max_{i=2}^{m}p_{i}\mid v)>C_{thresh}$

To answer this question, we establish a connection with the Dirichlet
distribution. Specifically, we note that the counts $v$ parameterize a
Dirichlet distribution, $\text{Dir}(V)$. 222Dirichlet is a distribution over
multinomials; each draw from Dirichlet is a multinomial distribution. See
Details in Appendix D. Also, $p$ represents a multinomial distribution and is
the expected value of $v$. That is, $p=\mathbb{E}(\text{Dir}(V))$. This
connection allows us to explore the behavior of the sampling process by
drawing more samples from $\text{Dir}(V)$ and observing the stability of
$p_{1}$ as the majority element. To compute the probability of $p_{1}$ being
the majority element, we can integrate the joint probability density function
of the Dirichlet distribution over the appropriate region of the probability
simplex. The integral can be expressed as follows:

$\displaystyle P$ $\displaystyle(p_{1}>\max_{i=2}^{m}p_{i}\mid V)$ (1)
$\displaystyle=\int_{0}^{1}\int_{\mathcal{S}(p_{1}^{\prime})}f(p_{1}^{\prime},p_{2},\ldots,p_{m}\mid
V).$ $\displaystyle\qquad\qquad\qquad\qquad dp_{2}\cdots
dp_{m}dp_{1}^{\prime},$ where $\displaystyle\;\mathcal{S}(p_{1}^{\prime})$
$\displaystyle=\\{(p_{2},\ldots,p_{m})\mid
p_{1}^{\prime}>\max_{i=2}^{m}p_{i},$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\sum_{i=2}^{m}p_{i}=1-p_{1}^{\prime}\\}.$

The function $f(p_{1}^{\prime},p_{2},...,p_{m}|V)$ represents the joint
probability density function of the Dirichlet distribution conditioned on the
counts $V$. In Equation 1, the bounds on the integral for $p_{1}^{\prime}$
range from 0 to 1. The probability simplex $\mathcal{S}(p_{1}^{\prime})$ is
defined for each $p_{1}^{\prime}$ value, such that
$p_{1}^{\prime}>\max_{i=2}^{m}p_{i}$, and the remaining $p_{i}$ values sum to
$1-p_{1}^{\prime}$. This constraint ensures that we are considering all
possible values of $p_{1}^{\prime}$ that would maintain its majority status.
Here we assume, that the number of possible unique answers ($m$) is known,
based on current set of observations ($V$). In Analysis (($\S$ 4.2), we
further evaluate Chinese Restaurant Process (CRP) stopping criteria, which
relaxes this assumption by not requiring the number of possible unique answers
($m$) to be known in advance.

#### Beta Stopping Criteria

Since the number of unique answers in the observation set can be large,
Equation (1) is computationally expensive to solve. As an approximation, we
observe that establishing the majority of $p_{1}$ over the next largest
probability, $p_{2}$, is sufficient for our purpose.

In this setting, the probability in Equation (2.2) simplifies to a Beta
distribution with parameters $(v_{1}+1,v_{2}+1)$ (prior
$\textsc{Beta}{}(1,1)$), and Equation (1) is replaced by Equation (2). This
approximation allows us to efficiently compute the confidence in $p_{1}$ being
the majority, enabling early stopping decisions without incurring substantial
computational overhead.

$\int_{0}^{0.5}p_{2}^{v2}\cdot(1-p_{2})^{v_{1}}dp_{2}$ (2)

Empirically, we show the performance to be similar to Dirichlet stopping
criteria but significantly faster (See Section 4.2). Throughout experiments,
we refer to this Beta Stopping Criteria as .

Intuitively, the stopping criteria in equation (1) ensures we generate only a
few samples for questions, where the majority answer can be inferred from a
small number of samples with high confidence. For example, if nine out of the
first ten answers generated by the model are the same: $a_{1}$, then with
$>99\%$ probability, the majority answer will remain the same even on infinite
generations. Thus we can save more sample generations at the cost of minimal
loss in accuracy.

##  3 Experiments

In this section, we describe the experimental setup for evaluating our
proposed method on various reasoning benchmarks. We use three diverse
categories of reasoning benchmarks, evaluate the performance on two different
language models, and use prompts by PAL Gao et al. (2022) and Self-Consistency
Wang et al. (2022b). We demonstrate marked improvements in terms of reduced
sample generations when compared to Self-Consistency.

#### Benchmarks

We evaluate our method on a diverse set of reasoning benchmarks, encompassing
13 datasets across three distinct categories:

1\. Mathematical Reasoning: To assess mathematical reasoning capabilities, we
utilize the following datasets: GSM-8K Cobbe et al. (2021), SVAMP Patel et al.
(2021), and ASDIV Miao et al. (2020). These datasets consist of grade-school-
level algebra word problems necessitating arithmetic operations and problem-
solving based on contextual information.

2\. Commonsense Reasoning Tasks: We evaluate on four commonsense reasoning
tasks. 1.) StrategyQA Geva et al. (2021) comprises questions that demand the
model to infer a multi-hop strategy with reasoning steps implicitly embedded
in the questions. 2.) Date Understanding entails questions that require the
model to deduce dates from natural language descriptions and perform
arithmetic operations accordingly. 3.) Snarks and 4.) Ruin Names both focus on
emotional understanding tasks.

3\. Logical Reasoning Tasks: We examine the performance of our method on six
diverse logical reasoning tasks. 1.) Tracking Shuffled Objects is a tracking
task that necessitates the model to infer the final state of a system, given
its initial state and a sequence of modifications. 2.) Logical Deduction is a
logical deduction task that demands the model to deduce the order of a
sequence of objects based on a minimal set of conditions. 3.) Boolean
Expressions is a boolean expressions task that evaluates whether a language
model has learned the rules of deductive reasoning, i.e., formal (zeroth-
order) logic associated with the words "and," "or," "not," etc. 4.) Salient
Translation is a salient translation error detection task that requires the
model to identify the type of error in a translation. 5.) Disambiguation QA is
a disambiguation task that necessitates the model to select the person to whom
the pronoun refers. 6.) Penguins describes a table of penguins and requires
the model to answer questions about the penguins’ attributes.

#### Models

We evaluate our method on two different language models: 1\. code-davinci-002:
A GPT-3-based publicly available model Brown et al. (2020) which is a part of
the Codex series Chen et al. (2021) and has 175 billion parameters.333We have
access to Codex models through OpenAI’s researcher access program. However,
since we only need access to the model outputs for our purposes, we will
release them for reproducibility. 2\. Vicuna-13B: Chiang et al. (2023) an
open-source transformer model fine-tuned on instruction-following dataset
Taori et al. (2023) from the base Llama series Touvron et al. (2023).

#### Prompting and Sampling

We use similar prompts as employed in the PAL Gao et al. (2022) and the Chain
of Thought Wei et al. (2022). Specifically, for mathematical reasoning and
Date Understanding tasks, we use the prompts from PAL. For other commonsense
and logical reasoning tasks, we use prompts from CoT Wei et al. (2022).

In terms of sampling, we follow the scheme suggested in the Self-Consistency
Wang et al. (2022b). Specifically, we use a temperature of 0.7 for sampling
and limit the number of generations to a maximum of 40.

#### Hyperparameters

The only hyperparameters in , are those related to parameters in stopping
criterias ($C_{thresh}$). We use a high $C_{thresh}=0.95$ for . By using a
high threshold, we aim to maintain high accuracy and prevent the algorithm
from stopping too early. For other Stopping Criterias, we tune our parameters
on the training set of GSM-8K, and use the same thresholds across all the
datasets. The impact of the chosen threshold on the performance of our method
is further analyzed in the Analysis Section ($\S$ 4.1).

#### Baselines

We compare our method against Self-Consistency, which is the current state-of-
the-art method.

Further, in Analysis 4.2, we evaluate against different stopping criterias
such as Random stopping and Majority (stopping at majority).

#### Evaluation Metrics

We evaluate the performance of our method and the baselines using two metrics:
average generations sampled from the large language models (LLMs) and overall
reasoning accuracy. Our results show that achieves similar performance to
Self-Consistency while often reducing sample budget up to $6\times$ times.

###  3.1 Results

|  | Self-Consistency |  | $\Delta$
---|---|---|---|---
|  | Accuracy | Avg. Gen. | Accuracy | Gen. Reduc. | Acc. Diff. $\uparrow$
GSM-8K | Vicuna-13B | 31.6 | 26.8 | $31.5$ | 1.4$\times$ | $-0.1$
code-davinci-002 | 81.1 | 13.8 | $81.0$ | 2.9$\times$ | $-0.1$
SVAMP | Vicuna-13B | 63.0 | 18.8 | $62.8$ | 2.1$\times$ | $-0.2$
code-davinci-002 | 85.1 | 9.5 | $85.0$ | 4.2$\times$ | $-0.1$
ASDIV | Vicuna-13B | 64.0 | 16.5 | 64.0 | 2.4$\times$ | $0.0$
code-davinci-002 | 83.2 | 10.0 | $83.2$ | 4.0$\times$ | $0.0$
Date Understanding | Vicuna-13B | $59.8$ | 17.3 | 60.2 | 2.3$\times$ | $+0.4$
code-davinci-002 | 80.3 | 10.7 | $79.5$ | 3.7$\times$ | $-0.8$
Tracking Shuffled Objects | Vicuna-13B | $31.8$ | 20.3 | 32.0 | 2.0$\times$ | $+0.2$
code-davinci-002 | 77.2 | 9.7 | $77.1$ | 4.1$\times$ | $-0.1$
Logical Deduction | Vicuna-13B | $51.2$ | 18.1 | 51.4 | 2.2$\times$ | $+0.2$
code-davinci-002 | 89.4 | 8.5 | 89.4 | 4.7$\times$ | $0.0$
StrategyQA | Vicuna-13B | $65.8$ | 16.3 | 65.8 | 2.5$\times$ | $0.0$
code-davinci-002 | 79.0 | 11.9 | $78.8$ | 3.4$\times$ | $-0.2$
Boolean Expressions | Vicuna-13B | 79.2 | 16.2 | $78.4$ | 2.5$\times$ | $-0.8$
code-davinci-002 | 94.5 | 6.6 | 94.5 | 6.0$\times$ | $0.0$
Snarks | Vicuna-13B | $73.2$ | 23.2 | 73.6 | 1.7$\times$ | $+0.4$
code-davinci-002 | 74.0 | 12.7 | 74.0 | 3.1$\times$ | $0.0$
Ruin Names | Vicuna-13B | 43.6 | 33.8 | 43.6 | 1.2$\times$ | $0.0$
code-davinci-002 | 78.0 | 17.2 | 78.0 | 2.3$\times$ | $0.0$
Salient Translation | Vicuna-13B | 28.9 | 28.7 | $28.7$ | 1.2$\times$ | $-0.3$
code-davinci-002 | 64.3 | 11.8 | 64.3 | 3.4$\times$ | $0.0$
Disambiguation QA | Vicuna-13B | 63.7 | 22.8 | $63.5$ | 1.8$\times$ | $-0.3$
code-davinci-002 | $74.9$ | 13.5 | 75.1 | 3.0$\times$ | $+0.1$
Penguins | Vicuna-13B | $46.8$ | 22.9 | 47.3 | 1.7$\times$ | $+0.5$
code-davinci-002 | $83.8$ | 11.0 | 84.0 | 3.6$\times$ | $+0.2$
Average | 67.2 | 16.5 | 67.2 | $2.8\times$ | -0.03

Table 1: Comparison of with Self-Consistency on 13 diverse reasoning datasets.
The table presents the accuracy of Self-Consistency, the average number of
generations (Avg. Gen.) for , and the accuracy of . Self-Consistency always
draws 40 samples. The $\Delta$ columns display the reduction in generations
(Gen. Reduc.) and the difference in accuracy (Acc. Diff.) between Self-
Consistency and . The last row shows the average values across all datasets:
achieves a 2.8 times reduction in sample budget (Gen. Reduc.) compared to
Self-Consistency, while maintaining a minimal average accuracy drop of 0.03%
(Acc. Diff.).

We compare with Self-Consistency in Table 1. Our proposed method consistently
reduces the computational budget while maintaining negligible drops in
accuracy across benchmarks and models. Delta improvements over Self-
Consistency are reported in the two rightmost columns.

On mathematical and logical reasoning tasks, significantly reduces the sample
budget by factors ranging from $1.4\times$ to $6\times$ compared to Self-
Consistency, depending on the task and model, with an average decrease in
accuracy of less than $0.1\%$. This trend holds for both the Vicuna-13B and
code-davinci-002 models, which differ significantly in model size and base
families.

also demonstrates efficiency across a diverse set of tasks, such as emotional
understanding tasks (Snarks and Ruin Names), translation error detection
(Salient Translation), and ambiguity identification (Disambiguation QA). In
these tasks, results in sample budget reductions of $1.2\times$ to
$2.5\times$, with a maximum drop in accuracy of $0.4\%$.

Our experiments demonstrate that effectively enhances the performance of large
language models on diverse reasoning tasks by adaptively adjusting the
stopping criterion. With comparable accuracy to Self-Consistency and
significantly reduced sample budgets, showcases its potential for more
efficient and effective reasoning across various domains, output responses,
and task difficulties.

In summary, our experiments show that enhances large language models’
performance on diverse reasoning tasks by adaptively adjusting the stopping
criterion. achieves accuracy comparable to Self-Consistency while
significantly reducing the sample budget, highlighting ’ potential for more
efficient and effective reasoning in large language models.

##  4 Analysis

###  4.1 Effect of Confidence Threshold in

The confidence threshold, $C_{thresh}$, is a crucial hyperparameter for , as
it determines when to stop sampling based on the desired level of confidence
in the majority element. While we set the threshold to a stringent value of
0.95 for all experiments, in this section, we analyze the impact of varying
$C_{thresh}$ from 0.5 to 1 to understand the trade-offs between model accuracy
and cost-efficiency.

In Figure 2, we present a visualization that examines the relationship between
the confidence threshold, $C_{thresh}$, and the performance of our adaptive
consistency method in terms of both accuracy and cost-efficiency. The x-axis
represents the confidence threshold, varying from 0.5 to 1. The left y-axis
displays the model’s accuracy, while the right y-axis shows the average number
of samples drawn.

The plot shows the expected behavior of two curves: the blue curve (accuracy)
increases gradually and then plateaus, while the red curve (average number of
samples) initially increases linearly and then climbs more steeply. The
plateau in accuracy signifies that the model has reached its maximum
achievable accuracy, and further sampling doesn’t improve it. Meanwhile, the
red curve’s climbing rate indicates that the model requires more samples to
meet increasingly stringent confidence thresholds, highlighting the trade-off
between accuracy and cost efficiency.

(a) GSM-8K (b) Ruin Names

Figure 2: Impact of Confidence Threshold ($C_{thresh}$) on : As $C_{thresh}$
varies, the accuracy of increases gradually, eventually plateauing. Initially,
the average number of generations also increases gradually but then sharply
climbs, reflecting the accuracy-confidence trade-off.

###  4.2 Evaluation of Different Stopping Functions

allows a flexible choice of stopping criteria, based on intended objective and
requirements. Here we evaluate various such stopping functions. Specifically,
we evaluate six different functions: 1.) Random: randomly stopping with a
probability $p$, 2.) Majority: stopping after the most common answer has
majority above a threshold, 3.) Entropy: stopping after entropy of answers is
below a threshold, 4.) Beta: The main stopping criteria used in , based on the
Equation (2), 5.) Dirichlet: The stopping criteria, based on Equation (1). 6.)
Chinese Restaurant Process (CRP): Unlike Dirichlet, CRP makes no assumption on
the number of possible unique answers. Based on the available observations, we
first model the concentration parameter ($\alpha$), then perform monte carlo
simulation to get stability of current majority (See Appendix C.2).

We show results in Table 4. The parameters for all these methods are tuned as
discussed in Section 3. We note, that while Random allows to control the
sample budget consistently (fixed at 10), it is the least effective of all
stopping criterias evaluated. Majority almost consistently underperforms both
Beta and Entropy. While Entropy is competitive to Beta, Entropy lacks human-
interpretable stopping rationale. Moreover, Beta can provide guarantees on the
accuracy. Dirichlet has similar performance to Beta, however is almost
$1200\times$ slower than Beta, because of the expensive multi-variate integral
calculation. This motivates the simplification of Dirichlet into Beta as used
in . CRP is also as effective as Beta, but suffers from same issues of
Dirichlet. We refer readers to Appendix C.1 for more details.

##  5 Related Work

###  5.1 Crowdsourcing and Adaptive Consistency

finds inspiration in techniques from crowdsourcing Lin et al. (2012); Dai et
al. (2013). Traditionally, crowdsourcing involves aggregating diverse human
judgments, which presents challenges in managing resource allocation—knowing
when to query additional contributors or stop based on the consistency of
responses Doan et al. (2011); Quinn and Bederson (2011). Early research
concentrated on probabilistic models estimating the ’true’ answer and worker
reliability Dawid and Skene (1979); Whitehill et al. (2009), later considering
factors like worker expertise, task complexity, and answer quality Raykar et
al. (2010); Welinder et al. (2010). However, rather than addressing these
issues with multiple human contributors, is tailored specifically for LLMs,
optimizing for computational efficiency and output accuracy.

#### Architectures for adaptive computation

A related body of work on adaptive computation aims to preempt computation
based on intermediate representations (Liu et al., 2020; Zhou et al., 2020;
Schuster et al., 2021; Geng et al., 2021; Xin et al., 2020). Schuster et al.
(2022) present clam, a language model that performs language generation
adaptively. Hou et al. (2020) propose Dynamic Bert, which can adapt the depth
and width of the transformer to satisfy various computational constraints.
Xing et al. (2020) propose a dynamic deep neural network with an early-exit
strategy embedded for enhancing the quality of compressed images. Another
direction of work focuses on pruning model weights or training sparse weights
Fan et al. (2019); Jayakumar et al. (2021) to reduce training and inference
time. In contrast to these methods, our approach completely obviates making
any architectural modifications.

#### Inference-time adaptive computation

These methods focus on adaptive computation at inference time without making
architectural modifications to the models. Schwarzschild et al. (2021b, a)
focus on three different generalization tasks. They observe that increasing
the number of test iterations (which corresponds to the network depth in their
setting) helps the models in generalizing better to difficult problems. Madaan
and Yang (2022) leverage two different networks trained for the same task, a
larger variant (slow) and a smaller variant (fast). The switch from fast to
slow happens during inference, based on the complexity of generation at the
current step. Xue et al. (2023) train language models to adaptively read
tokens from a tape bank for each input. Different from these works, our focus
is tasks where the multiple samples are drawn from a model (vs. iteratively
solving a task, which is a focus of these works).

#### Adaptive Sampling in Training and Active Learning

Another line of work focuses on importance-based sampling of input instances
during training (Bengio and Senecal, 2008; Prabhu et al., 2019; Berger et al.,
2017). For instance, Bengio and Senecal (Bengio and Senecal, 2008) introduce
an adaptive importance sampling technique for expediting the training of
neural probabilistic language models. Prabhu et al.(Prabhu et al., 2019)
investigate sampling bias in deep active classification, proposing an
empirical method to mitigate it. Berger et al.(Berger et al., 2017) present an
adaptive sampling scheme for training fully convolutional networks efficiently
in semantic segmentation tasks involving large and imbalanced datasets. These
approaches primarily focus on sampling input instances during the training
phase.

In contrast to the aforementioned methods, our approach centers on adaptively
sampling multiple outputs per input instance during the inference phase,
without soliciting additional labels. Our method is crafted to efficiently
obtain reliable predictions from pretrained language models by adaptively
sampling their outputs, distinguishing it from both adaptive sampling in
training and active learning, which focus on the training phase.

##  6 Conclusion and Future Work

This paper presented , a cost-efficient and model-agnostic technique for
improving the correctness of output from large language models (LLMs) using
dynamic sampling. Our approach builds upon the Self-Consistency method and
introduces a lightweight stopping criterion that allows for adaptive sampling
based on the amount of agreement in the samples drawn so far. is effective
across 13 datasets and two LLMs, reducing the required sample budget by up to
6.0 times while maintaining comparable accuracy, with an average drop of less
than 0.1%.

Our work opens up several avenues for future research. First, investigating
alternative stopping criteria or combining multiple criteria could lead to
even more efficient sampling techniques. Finally, in our current approach, the
majority decision relies on using exact matches to determine the most common
answer. However, this may not always capture the true majority in cases where
the output has minor variations that do not affect the overall correctness or
relevance of the answer. As a future direction, we plan to investigate the use
of approximate matching techniques to identify and count elements that are
close enough as identical.

## Limitations

Despite the promising results of our proposed method, it comes with several
limitations and opportunities for future improvement.

  * $\bullet$

Stopping criterion sensitivity: Our current stopping criterion is based on the
stability of the majority element in the set of samples. While this has shown
to be effective in our experiments, it may not always be the best indicator of
agreement in the samples. There could be cases where the majority element is
not stable enough, but the stopping criterion still triggers, potentially
leading to suboptimal decisions. Exploring alternative or more robust stopping
criteria could help mitigate this issue.

  * $\bullet$

Generalizability: Although we have tested our method across a diverse range of
13 datasets and two different LLMs of contrastive scale, there may still be
tasks or models where the approach might not be as effective. Notably, is
expected to fail where Self-Consistency fails.

  * $\bullet$

Task-specific adaptations: The current implementation of is task-agnostic,
which might limit its performance on specific tasks that could benefit from
task-specific adaptations. Designing specialized versions of for particular
tasks or domains could potentially lead to better performance in those areas.

  * $\bullet$

Reliance on the pretrained LLM: Our method builds upon the Self-Consistency
technique and relies on the pretrained LLM to generate multiple samples.
Consequently, any limitations or biases present in the underlying LLM would
still be carried over to the method. Addressing these issues might require
improvements in the LLM training process itself or the incorporation of
external knowledge sources.

## References

  * Bengio and Senecal (2008) Yoshua Bengio and Jean-Sébastien Senecal. 2008.  Adaptive importance sampling to accelerate training of a neural probabilistic language model.  _IEEE Transactions on Neural Networks_ , 19:713–722.
  * Berger et al. (2017) Lorenz Berger, Eoin R. Hyde, M. Jorge Cardoso, and Sébastien Ourselin. 2017\.  An adaptive sampling scheme to efficiently train fully convolutional networks for semantic segmentation.  In _Annual Conference on Medical Image Understanding and Analysis_.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, S. Arun Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021.  Evaluating large language models trained on code.  _ArXiv_ , abs/2107.03374.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021.  Training verifiers to solve math word problems.  _ArXiv_ , abs/2110.14168.
  * Dai et al. (2013) Peng Dai, C. H. Lin, Mausam, and Daniel S. Weld. 2013.  Pomdp-based control of workflows for crowdsourcing.  _Artif. Intell._ , 202:52–85.
  * Dawid and Skene (1979) A. Philip Dawid and Allan Skene. 1979.  Maximum likelihood estimation of observer error-rates using the em algorithm.  _Journal of The Royal Statistical Society Series C-applied Statistics_ , 28:20–28.
  * Doan et al. (2011) AnHai Doan, Raghu Ramakrishnan, and Alon Y. Halevy. 2011.  Crowdsourcing systems on the world-wide web.  _Communications of the ACM_ , 54:86 – 96.
  * Fan et al. (2019) Angela Fan, Edouard Grave, and Armand Joulin. 2019.  Reducing transformer depth on demand with structured dropout.  _ArXiv_ , abs/1909.11556.
  * Gao et al. (2022) Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2022.  Pal: Program-aided language models.  _ArXiv_ , abs/2211.10435.
  * Geng et al. (2021) Shijie Geng, Peng Gao, Zuohui Fu, and Yongfeng Zhang. 2021.  Romebert: Robust training of multi-exit bert.  _arXiv preprint arXiv:2101.09755_.
  * Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021.  Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.  _Transactions of the Association for Computational Linguistics_ , 9:346–361.
  * Hou et al. (2020) Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2020.  Dynabert: Dynamic bert with adaptive width and depth.  _ArXiv_ , abs/2004.04037.
  * Jayakumar et al. (2021) Siddhant M. Jayakumar, Razvan Pascanu, Jack W. Rae, Simon Osindero, and Erich Elsen. 2021.  Top-kast: Top-k always sparse training.  _ArXiv_ , abs/2106.03517.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large Language Models are Zero-Shot Reasoners.  _arXiv preprint arXiv:2205.11916_.
  * Lewkowycz et al. (2022) Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022.  Solving quantitative reasoning problems with language models.  _arXiv preprint arXiv:2206.14858_.
  * Li et al. (2022) Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom, Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de, Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey, Cherepanov, James Molloy, Daniel Jaymin Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de, Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022.  Competition-level code generation with alphacode.  _Science_ , 378:1092 – 1097.
  * Lin et al. (2012) C. H. Lin, Mausam, and Daniel S. Weld. 2012.  Crowdsourcing control: Moving beyond multiple choice.  _ArXiv_ , abs/1210.4870.
  * Liu et al. (2020) Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. 2020.  Fastbert: a self-distilling bert with adaptive inference time.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6035–6044.
  * Madaan et al. (2023) Aman Madaan, Alexander Shypula, Uri Alon, Milad Hashemi, Parthasarathy Ranganathan, Yiming Yang, Graham Neubig, and Amir Yazdanbakhsh. 2023.  Learning performance-improving code edits.  _arXiv preprint arXiv:2302.07867_.
  * Madaan and Yang (2022) Aman Madaan and Yiming Yang. 2022.  Flowgen: Fast and slow graph generation.  _DyNN workshop at the 39 th International Conference on Machine Learning, Baltimore, Maryland, USA, 2022_.
  * Miao et al. (2020) Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020.  A diverse corpus for evaluating and developing English math word problem solvers.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 975–984, Online. Association for Computational Linguistics.
  * Mishra et al. (2022) Bhavana Dalvi Mishra, Oyvind Tafjord, and Peter Clark. 2022.  Towards teachable reasoning systems: Using a dynamic memory of user feedback for continual system improvement.
  * Patel et al. (2021) Arkil Patel, S. Bhattamishra, and Navin Goyal. 2021.  Are nlp models really able to solve simple math word problems?  In _North American Chapter of the Association for Computational Linguistics_.
  * Prabhu et al. (2019) Ameya Prabhu, Charles Dognin, and Maneesh Kumar Singh. 2019.  Sampling bias in deep active classification: An empirical study.  In _Conference on Empirical Methods in Natural Language Processing_.
  * Quinn and Bederson (2011) Alexander J. Quinn and Benjamin B. Bederson. 2011.  Human computation: a survey and taxonomy of a growing field.  _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_.
  * Raykar et al. (2010) Vikas Chandrakant Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo, Charles Florin, Luca Bogoni, and Linda Moy. 2010.  Learning from crowds.  _J. Mach. Learn. Res._ , 11:1297–1322.
  * Schuster et al. (2022) Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q Tran, Yi Tay, and Donald Metzler. 2022.  Confident adaptive language modeling.  _arXiv preprint arXiv:2207.07061_.
  * Schuster et al. (2021) Tal Schuster, Adam Fisch, Tommi Jaakkola, and Regina Barzilay. 2021.  Consistent accelerated inference via confident adaptive transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4962–4979.
  * Schwarzschild et al. (2021a) Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Arpit Bansal, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. 2021a.  Datasets for Studying Generalization from Easy to Hard Examples.  _arXiv:2108.06011 [cs]_.  ArXiv: 2108.06011.
  * Schwarzschild et al. (2021b) Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. 2021b.  Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 6695–6706. Curran Associates, Inc.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur’elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.  _ArXiv_ , abs/2302.13971.
  * Wang et al. (2022a) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022a.  Rationale-Augmented Ensembles in Language Models.  _arXiv preprints arXiv:2207.00747_.
  * Wang et al. (2022b) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai hsin Chi, and Denny Zhou. 2022b.  Self-consistency improves chain of thought reasoning in language models.  _ArXiv_ , abs/2203.11171.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _ArXiv_ , abs/2201.11903.
  * Welinder et al. (2010) Peter Welinder, Steve Branson, Serge J. Belongie, and Pietro Perona. 2010.  The multidimensional wisdom of crowds.  In _NIPS_.
  * West (1992) Beate West. 1992.  Hyperparameter estimation in dirichlet process mixture models.
  * Whitehill et al. (2009) Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier R. Movellan. 2009.  Whose vote should count more: Optimal integration of labels from labelers of unknown expertise.  In _NIPS_.
  * Xin et al. (2020) Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy J. Lin. 2020.  Deebert: Dynamic early exiting for accelerating bert inference.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Xing et al. (2020) Qunliang Xing, Mai Xu, Tianyi Li, and Zhenyu Guan. 2020.  Early exit or not: Resource-efficient blind quality enhancement for compressed images.  In _European Conference on Computer Vision_.
  * Xue et al. (2023) Fuzhao Xue, Valerii Likhosherstov, Anurag Arnab, Neil Houlsby, Mostafa Dehghani, and Yang You. 2023.  Adaptive computation with elastic input sequence.  _ArXiv_ , abs/2301.13195.
  * Zhou et al. (2020) Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020.  Bert loses patience: Fast and robust inference with early exit.  _Advances in Neural Information Processing Systems_ , 33:18330–18341.

##  Appendix A Experimental Setup

###  A.1 Hyperparameters

The only hyperparameters in , are those related to parameters in stopping
criterias ($C_{thresh}$). We use a high $C_{thresh}=0.95$ for . By using a
high threshold, we aim to maintain high accuracy and prevent the algorithm
from stopping too early. For other Stopping Criterias, we tune our parameters
on the training set of GSM-8K, and use the same thresholds across all the
datasets. The impact of the chosen threshold on the performance of our method
is further analyzed in the Analysis Section ($\S$ 4.1). We further evaluate
all methods on a set of 3 seeds and report the table with standard deviation
in Table 3.

###  A.2 Tools and Framework

For querying code-davinci-002 models Chen et al. (2021), we use the api
library provided by OpenAI444API available at: https://platform.openai.com/.
We use the official code provided for running Vicuna-13B model Chiang et al.
(2023). We run inference on Vicuna-13B models on single A100 gpus. stopping
criteria in are fast to run, and we use a single-core machine. For numerical
integration, we use the Scipy library in Python.

###  A.3 Benchmarks

As described in Section 3, we evaluate our method on 13 diverse reasoning
datasets. Dataset Statistics are presented in Table 2.

Dataset | $|N\\_test|$ | Answer Format
---|---|---
GSM-8K | 1319 | Numerical
ASDIV | 2096 | Numerical
SVAMP | 1000 | Numerical
Date Understanding | 369 | String
Tracking Shuffled Objects | 250 | MCQ
Logical Deduction | 250 | MCQ
StrategyQA | 2279 | Binary
Boolean Expressions | 250 | Binary
Snarks | 250 | Binary
Ruin Names | 178 | MCQ
Salient Translation | 250 | MCQ
Disambiguation QA | 250 | MCQ
Penguins | 146 | MCQ

Table 2: Dataset Statistics. We evaluate on 13 diverse reasoning datasets,
having different difficulty, domains, answer types, sizes ($N\\_test$).

##  Appendix B Results

We present the complete results with standard deviation in Table 3.

|  | Self-Consistency |  | $\Delta$
---|---|---|---|---
|  | Accuracy | Avg. Gen. | Accuracy | Gen. Reduc. | Acc. Diff. $\uparrow$
GSM-8K | Vicuna-13B |  31.6$(\pm 3.0)$ |  26.8$(\pm 2.2)$ |  $31.5$$(\pm 3.0)$ | 1.4$\times$ | $-0.1$
code-davinci-002 |  81.1$(\pm 0.3)$ |  13.8$(\pm 0.0)$ |  $81.0$$(\pm 0.3)$ | 2.9$\times$ | $-0.1$
SVAMP | Vicuna-13B |  63.0$(\pm 0.3)$ |  18.8$(\pm 0.1)$ |  $62.8$$(\pm 0.4)$ | 2.1$\times$ | $-0.2$
code-davinci-002 |  85.1$(\pm 0.3)$ |  9.5$(\pm 0.1)$ |  $85.0$$(\pm 0.3)$ | 4.2$\times$ | $-0.1$
ASDIV | Vicuna-13B |  64.0$(\pm 0.3)$ |  16.5$(\pm 0.2)$ |  64.0$(\pm 0.3)$ | 2.4$\times$ | $0.0$
code-davinci-002 |  83.2$(\pm 0.2)$ |  10.0$(\pm 0.0)$ |  $83.2$$(\pm 0.2)$ | 4.0$\times$ | $0.0$
Date Understanding | Vicuna-13B |  $59.8$$(\pm 0.3)$ |  17.3$(\pm 0.3)$ |  60.2$(\pm 0.4)$ | 2.3$\times$ | $+0.4$
code-davinci-002 |  80.3$(\pm 0.1)$ |  10.7$(\pm 0.3)$ |  $79.5$$(\pm 0.3)$ | 3.7$\times$ | $-0.8$
Tracking Shuffled Objects | Vicuna-13B |  $31.8$$(\pm 1.0)$ |  20.3$(\pm 0.0)$ |  32.0$(\pm 1.2)$ | 2.0$\times$ | $+0.2$
code-davinci-002 |  77.2$(\pm 1.3)$ |  9.7$(\pm 0.1)$ |  $77.1$$(\pm 1.6)$ | 4.1$\times$ | $-0.1$
Logical Deduction | Vicuna-13B |  $51.2$$(\pm 0.8)$ |  18.1$(\pm 0.2)$ |  51.4$(\pm 0.6)$ | 2.2$\times$ | $+0.2$
code-davinci-002 |  89.4$(\pm 0.2)$ |  8.5$(\pm 0.1)$ |  89.4$(\pm 0.2)$ | 4.7$\times$ | $0.0$
StrategyQA | Vicuna-13B |  $65.8$$(\pm 0.5)$ |  16.3$(\pm 0.1)$ |  65.8$(\pm 0.4)$ | 2.5$\times$ | $0.0$
code-davinci-002 |  79.0$(\pm 0.2)$ |  11.9$(\pm 0.2)$ |  $78.8$$(\pm 0.1)$ | 3.4$\times$ | $-0.2$
Boolean Expressions | Vicuna-13B |  79.2$(\pm 0.6)$ |  16.2$(\pm 0.3)$ |  $78.4$$(\pm 0.3)$ | 2.5$\times$ | $-0.8$
code-davinci-002 |  94.5$(\pm 0.4)$ |  6.6$(\pm 0.1)$ |  94.5$(\pm 0.4)$ | 6.0$\times$ | $0.0$
Snarks | Vicuna-13B |  $73.2$$(\pm 1.0)$ |  23.2$(\pm 0.7)$ |  73.6$(\pm 0.8)$ | 1.7$\times$ | $+0.4$
code-davinci-002 |  74.0$(\pm 1.0)$ |  12.7$(\pm 0.4)$ |  74.0$(\pm 1.5)$ | 3.1$\times$ | $0.0$
Ruin Names | Vicuna-13B |  43.6$(\pm 2.1)$ |  33.8$(\pm 0.6)$ |  43.6$(\pm 2.1)$ | 1.2$\times$ | $0.0$
code-davinci-002 |  78.0$(\pm 0.9)$ |  17.2$(\pm 0.1)$ |  78.0$(\pm 0.6)$ | 2.3$\times$ | $0.0$
Salient Translation | Vicuna-13B |  28.9$(\pm 2.4)$ |  28.7$(\pm 2.5)$ |  $28.7$$(\pm 2.5)$ | 1.2$\times$ | $-0.3$
code-davinci-002 |  64.3$(\pm 0.2)$ |  11.8$(\pm 0.5)$ |  64.3$(\pm 0.2)$ | 3.4$\times$ | $0.0$
Disambiguation QA | Vicuna-13B |  63.7$(\pm 0.7)$ |  22.8$(\pm 1.0)$ |  $63.5$$(\pm 1.1)$ | 1.8$\times$ | $-0.3$
code-davinci-002 |  $74.9$$(\pm 0.8)$ |  13.5$(\pm 0.6)$ |  75.1$(\pm 0.7)$ | 3.0$\times$ | $+0.1$
Penguins | Vicuna-13B |  $46.8$$(\pm 1.8)$ |  22.9$(\pm 0.7)$ |  47.3$(\pm 1.9)$ | 1.7$\times$ | $+0.5$
code-davinci-002 |  $83.8$$(\pm 0.9)$ |  11.0$(\pm 0.4)$ |  84.0$(\pm 0.6)$ | 3.6$\times$ | $+0.2$
Average | 67.2$(\pm 0.8)$ | 16.5$(\pm 0.5)$ | 67.2$(\pm 0.8)$ | $2.8\times$ | -0.03

Table 3: Comparison of with Self-Consistency on 13 diverse reasoning datasets.
The table presents the accuracy of Self-Consistency, the average number of
generations (Avg. Gen.) for , and the accuracy of . Self-Consistency always
draws 40 samples. The $\Delta$ columns display the reduction in generations
(Gen. Reduc.) and the difference in accuracy (Acc. Diff.) between Self-
Consistency and . The last row shows the average values across all datasets:
achieves a 2.8 times reduction in sample budget (Gen. Reduc.) compared to
Self-Consistency, while maintaining a minimal average accuracy drop of 0.03%
(Acc. Diff.).

##  Appendix C Analysis

###  C.1 Stopping Criterias

This section follows from the main discussion in Section 4.2. We evaluate
different stopping criterias for . We evaluate 6 different functions:

  1. 1.

Random: randomly stopping with a probability $p$, 2.)

  2. 2.

Majority: stopping after the most common answer has a majority above a
threshold,

  3. 3.

Entropy: stopping after the entropy of answers is below a threshold,

  4. 4.

Beta: The main stopping criteria used in , based on the Equation (2),

  5. 5.

Dirichlet: The stopping criteria, based on Equation (1).

  6. 6.

Chinese Restaurant Process (CRP): The stopping criteria, which models
probability as chinese restaurant process making no assumption on possible
number of unique answers.

For comparison, we tune the $C_{thresh}$ in each case on the training set of
GSM-8K dataset. Results are presented in Table 4.

###  C.2 Chinese Restaurant Process

In the Dirichlet stopping criteria, we assume that the number of unique
answers that can be generated by the LLM is known in advance (and equal to the
number of unique answers in the current observation set). However, this
assumption may not hold for datasets such as GSM-8K, where numerical answers
are expected. The Chinese Restaurant Process (CRP) is a generalization of the
Dirichlet process that addresses this limitation by not making any assumption
on the number of unique answers.

In CRP, we consider a list of same answers as a cluster, denoted by $c_{i}$,
where $i$ is the index of the cluster. Let $n_{i}$ be the number of elements
in cluster $c_{i}$, and $n$ be the total number of elements across all
clusters. The probability of a new answer belonging to an existing cluster
$c_{i}$ is directly proportional to the size of the cluster, and is given by:

$P(c_{i})=\frac{n_{i}}{n+\alpha},$ (3)

whereas the probability that a new unseen answer will form a new cluster is
given by:

$P(c_{new})=\frac{\alpha}{n+\alpha},$ (4)

where $\alpha$ is the concentration parameter, which parameterizes the
probability of generating a new answer.

Our goal is to calculate the probability that the current majority cluster in
observations will remain the same even with more generations. The first task
is to estimate the concentration parameter $\alpha$. We use the approximation
proposed by West (1992) to model the $\alpha$ as

$p(\alpha|k,n)\approx G(a+k-1,b+\gamma+\log(n)),$ (5)

where $k$ is the number of unique answers (clusters) in the current
observation, $n$ is the total number of answers, $a$ and $b$ are priors and
both set equal to 1, and $\gamma$ is Euler’s constant and
$G(\alpha;a+k-1,b+\gamma+\log(n))$ denotes the probability density function of
the Gamma distribution with shape parameter $a+k-1$ and rate parameter
$b+\gamma+\log(n)$.

We sample $\alpha$ multiple times (100), and for each sample, we run Monte-
Carlo Simulation (1000 simulations) based on the CRP probability modeling.
Each simulation starts from from current set of observations, and performed
till 40 generations are sampled. The probability that the current majority
cluster remains the majority is then given by:

$P(\text{majority})=\frac{1}{N_{\alpha}N_{MCS}}\sum_{i=1}^{N_{\alpha}}\sum_{j=1}^{N_{MCS}}I(\text{majority}_{n}^{40}),$
(6)

where $N_{\alpha}$ is the number of times we sample $\alpha$, $N_{MCS}$ is the
number of Monte-Carlo Simulations, and $I(\text{majority}_{n}^{40})$ is an
indicator function that equals 1 if the current majority remains the majority
after $40$ generations, and 0 otherwise.

|  | Random | Majority |  Entropy () | Beta | Dirichlet | CRP
---|---|---|---|---|---|---|---
|  | Average $\downarrow$ | Accuracy $\uparrow$ | Average $\downarrow$ | Accuracy $\uparrow$ | Average $\uparrow$ | Accuracy $\uparrow$ | Average $\uparrow$ | Accuracy $\uparrow$ | Average $\uparrow$ | Accuracy $\uparrow$ | Average $\uparrow$ | Accuracy $\uparrow$
|  | Generations | Generations | Generations | Generations | Generations | Generations
GSM-8K | Vicuna-13B | 26.0 | $30.1$ | $26.1$ | $31.6$ | $26.3$ | $31.5$ | $26.8$ | $31.5$ | $28.2$ | 31.7 | 25.6 | $31.5$
code-davinci-002 | $13.8$ | $76.9$ | $13.3$ | $80.5$ | $15.3$ | $81.0$ | $13.8$ | $81.0$ | $15.2$ | 81.1 | 13.2 | 81.1
ASDIV | Vicuna-13B | $28.0$ | $63.2$ | 14.8 | $63.7$ | $15.8$ | $63.9$ | $16.5$ | 64.0 | $17.7$ | 64.0 | $16.9$ | 64.0
code-davinci-002 | $13.8$ | $81.9$ | 9.2 | $83.1$ | $11.5$ | 83.3 | $10.0$ | $83.2$ | $10.7$ | $83.1$ | $10.7$ | $83.1$
SVAMP | Vicuna-13B | $28.0$ | $61.3$ | 17.1 | $62.5$ | $17.7$ | $62.6$ | $18.8$ | $62.8$ | $19.7$ | 62.9 | $18.2$ | $62.8$
code-davinci-002 | $13.4$ | $83.3$ | 8.4 | $84.8$ | $10.7$ | $85.1$ | $9.5$ | $85.0$ | $10.3$ | 85.1 | $9.8$ | $85.0$
Date Understanding | Vicuna-13B | $28.0$ | $58.3$ | 15.3 | $59.5$ | $16.0$ | $59.9$ | $17.3$ | 60.2 | $18.5$ | $59.9$ | $16.9$ | $59.9$
code-davinci-002 | $13.2$ | $76.4$ | 9.7 | $78.7$ | $11.6$ | $79.9$ | $10.7$ | $79.5$ | $11.9$ | 80.5 | $10.7$ | $79.8$
Tracking Shuffled Objects | Vicuna-13B | $27.9$ | $31.8$ | 15.0 | 33.0 | $18.4$ | $32.0$ | $20.3$ | $32.0$ | $23.3$ | $32.0$ | $19.6$ | $31.8$
code-davinci-002 | $13.5$ | $76.3$ | 7.0 | $76.8$ | $11.5$ | $76.9$ | $9.7$ | $77.1$ | $11.5$ | 77.2 | $10.2$ | $77.1$
Logical Deduction | Vicuna-13B | $27.9$ | $50.5$ | 12.9 | $51.2$ | $15.8$ | 51.4 | $18.1$ | 51.4 | $20.9$ | $51.2$ | $18.3$ | 51.4
code-davinci-002 | $13.7$ | $88.3$ | 5.9 | 89.6 | $10.1$ | 89.6 | $8.5$ | $89.4$ | $10.2$ | $89.2$ | $9.3$ | $89.4$
StrategyQA | Vicuna-13B | $28.1$ | $65.1$ | 11.7 | $65.5$ | $14.5$ | $65.8$ | $16.3$ | $65.8$ | $18.7$ | 65.8 | $17.0$ | $65.7$
code-davinci-002 | $13.4$ | $76.6$ | 7.2 | $77.8$ | $14.9$ | $78.5$ | $11.9$ | $78.8$ | $14.5$ | 78.9 | $11.4$ | 78.9
Boolean Expressions | Vicuna-13B | $27.6$ | $78.0$ | 10.4 | $76.8$ | $14.8$ | $78.3$ | $16.2$ | $78.4$ | $19.1$ | 78.8 | $17.0$ | $78.5$
code-davinci-002 | $13.1$ | $93.4$ | 4.3 | $94.3$ | $8.2$ | 94.5 | $6.6$ | 94.5 | $8.2$ | 94.5 | $7.9$ | $94.4$
Snarks | Vicuna-13B | $28.4$ | $70.3$ | 18.1 | $72.1$ | $20.3$ | $73.0$ | $23.2$ | 73.6 | $25.8$ | 73.6 | $22.9$ | 73.6
code-davinci-002 | $13.6$ | $71.6$ | 10.5 | 74.0 | $12.1$ | 74.0 | $12.7$ | 74.0 | $14.2$ | $73.4$ | $12.3$ | $73.2$
Ruin Names | Vicuna-13B | 28.3 | $40.6$ | $30.4$ | $43.9$ | $31.9$ | $43.7$ | $33.8$ | $43.6$ | $34.0$ | $43.6$ | $32.0$ | 44.0
code-davinci-002 | 13.8 | $71.7$ | $17.5$ | $77.7$ | $18.6$ | 78.1 | $17.2$ | $78.0$ | $17.6$ | $76.8$ | $16.4$ | 78.1
Salient Translation | Vicuna-13B | $24.9$ | $27.7$ | 24.6 | $28.5$ | $26.3$ | $28.0$ | $28.7$ | $28.7$ | $29.4$ | $28.8$ | $26.9$ | 28.9
code-davinci-002 | $14.0$ | $62.5$ | 9.9 | 64.7 | $13.1$ | $64.3$ | $11.8$ | $64.3$ | $13.7$ | $64.1$ | $11.7$ | $64.3$
Disambiguation QA | Vicuna-13B | $27.9$ | $62.9$ | 18.3 | $63.5$ | $20.1$ | $63.1$ | $22.8$ | $63.5$ | $25.4$ | 63.9 | $22.1$ | $63.3$
code-davinci-002 | $13.7$ | $72.1$ | 10.4 | $73.9$ | $15.9$ | $74.9$ | $13.5$ | $75.1$ | $16.3$ | 75.2 | $13.2$ | 75.2
Penguins | Vicuna-13B | $27.9$ | $45.6$ | 19.7 | $46.3$ | $20.7$ | 47.3 | $22.9$ | 47.3 | $25.1$ | 47.3 | $22.1$ | 47.3
code-davinci-002 | $13.3$ | $81.4$ | 9.0 | $83.3$ | $13.1$ | $83.8$ | $11.0$ | $84.0$ | $12.9$ | $84.0$ | $11.0$ | 84.5

Table 4: Comparison of various Stopping Criterias in . In general, Beta
outperforms Random and Majority by decent margins across all datasets. Beta
has comparable performance to Dirichlet, but the latter is much slower.
Entropy performs similarly to Beta but lacks human-interpretable stopping
rationale.

###  C.3 Effect of Confidence Threshold on

We follow the discussion in Section 4.1, and present complete results on all
datasets for code-davinci-002.

(a) GSM-8K (b) ASDIV (c) SVAMP (d) Date Understanding (e) Tracking Shuffled
Objects (f) Logical Deduction (g) StrategyQA (h) Boolean Expressions (i)
Snarks (j) Ruin Names (k) Salient Translation (l) Disambiguation QA (m)
Penguins

Figure 3: Impact of Confidence Threshold ($C_{thresh}$) on : As $C_{thresh}$
varies, the accuracy of increases gradually, eventually plateauing. Initially,
the average number of generations also increases gradually but then sharply
climbs, reflecting the accuracy-confidence trade-off. The trend is observed
almost consistently across all datasets.

##  Appendix D Derivation of Dirichlet stopping criteria

Consider for a given input ($I$), the model can generate one of $m$ distinct
answers $A:=\\{a_{1},a_{2},\ldots a_{m}\\}$. Define the probability of
generating an answer given input as $p_{i}:=P(a_{i}\mid I)$. Now, consider an
observation set ($O$) with counts of each of $a_{i}$ as $v_{i}$, such that
$\sum_{i=1}^{m}v_{i}=n$. Now, without loss of generality, consider
$p_{1}>\max_{i=2}^{m}p_{i}$. Now, based on Equation (2.2), we need to find the
probability:

$P(p_{1}>\max_{i=2}^{m}p_{i}\mid O)$

.

However, here the $p_{i}$s are latent variables, and only $O$ is available to
us. We next make the following
Assumption 1: The vector $\vec{p}=\\{p_{1},p_{2}\ldots p_{m}\\}$ is sampled
from uniform distribution over $(m-1)$-simplex.
Thus, $p_{1}=1-\sum_{i=1}^{m-1}p_{i}$. Since the observation set follows a
multinomial distribution with parameters $\vec{p}$, conditional joint
probability distribution of $O$ given $\vec{p}$ can be written as:

$P(O\mid\vec{p})=\frac{n!}{\prod_{i=1}^{m}(v_{i}!)}\prod_{i=1}^{m}p_{i}^{v_{i}}=Dir(v_{1}+1,v_{2}+1\ldots
v_{m}+1)$

, where $Dir$ represents the dirichlet distribution with $v_{i}+1$, as its
parameters. Applying Baye’s Rule,

$P(\vec{p}\mid O)=\frac{P(O\mid\vec{p})\cdot P(\vec{p})}{P(O)}$

. Here $P(O)$ is a normalizing constant and can be omitted for computation.
From Assumption 1, since $\vec{p}$ is sampled from uniform distribution,

$P(\vec{p})=\prod_{i=2}^{m}dp_{i}$

. Thus conditional joint probability distribution of $\vec{p}$ given $O$ can
be written as:

$P(\vec{p}\mid O)=Dir(v_{1}+1,v_{2}+1\ldots v_{m}+1)\;dp_{m}dp_{m-1}\ldots
dp_{2}$

.

Now we can integrate the above equation over a subset of $(m-1)$-simplex, such
that $p_{1}>\max_{i=2}^{m}p_{i}$. This gives us the equation:

$\displaystyle P$ $\displaystyle(p_{1}>\max_{i=2}^{m}p_{i}\mid O)$ (7)
$\displaystyle=\int_{0}^{1}\int_{\mathcal{S}(p_{1}^{\prime})}P(\vec{p}\mid O)$
$\displaystyle\qquad\qquad\qquad\qquad dp_{2}\cdots dp_{m}dp_{1}^{\prime},$
where $\displaystyle\;\mathcal{S}(p_{1}^{\prime})$
$\displaystyle=\\{(p_{2},\ldots,p_{m})\mid
p_{1}^{\prime}>\max_{i=2}^{m}p_{i},$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\sum_{i=2}^{m}p_{i}=1-p_{1}^{\prime}\\}.$

We note that the integration has no closed-form solution, and we use numerical
approximation to compute the above integral.

#### Defining region of integration: $\mathcal{S}(p_{1}^{\prime})$

Next, for computation of Equation (7), we need to precisely calculate the
limits of each integration such that they represent the region
$\mathcal{S}(p_{1}^{\prime})$. We do so by noting the following constraints on
$p_{i}$: 1.) The $p_{i}=0$ is valid $\forall 2\leq i\leq m$, 2.) Given
$\\{p_{m},p_{m-1}\ldots p_{i+1}$ are fixed and in region
$\mathcal{S}(p_{1}^{\prime})$, $p_{i}<\frac{1-\sum_{j=i+1}^{m}p_{j}}{2}$ as
else $p_{i}\geq p_{1}$ which is not allowed, 3.) Since
$p_{1}>\max_{j=i+1}^{m}p_{j}$, so
$p_{i}<1-\sum_{j=i+1}^{m}p_{j}-\max_{j=i+1}^{m}p_{j}$, as else the $\vec{p}$,
will lie outside the $(m-1)$-simplex, which is invalid. The first condition
makes the lower limit for each integration 0, and the minimum of condition 2
and condition 3 gives the upper bound (limit) on each of the integrations.

####  Beta stopping criteria

Due to $m-1$ dimensions integrations involved, with $m$ often getting larger
than 10, computing Equation (7) is not efficient. Instead, we observe that
establishing the majority of $p_{1}$ over the next largest probability,
$p_{2}$, is sufficient for our purpose. Then, pdf simplifies to Beta
distribution with parameters $v_{1}+1,v_{2}+1$, and Equation (7) simplifies
to:

$\int_{0}^{0.5}p_{2}^{v2}\cdot(1-p_{2})^{v_{1}}dp_{2}$ (8)

We use Scipy library in Python numerically compute the above equation.