∎

e1e-mail<EMAIL_ADDRESS>
11institutetext: College of Marine Science and Technology, Hainan Tropical
Ocean University, Sanya, 572022, China
2Department of Astronomy, Nanjing University, Nanjing, Jiangshu 210000, China

# Strongly screening on electron capture for Nuclides 52,53,59,60Fe by the
Shell-Model Monte Carlo method in pre-supernova

Jing-Jing  Liue1,addr1  Qiu-He  Peng 2 Dong-Mei  Liuaddr1

(Received: date / Accepted: date)

###### Abstract

The death of the massive stars due to supernova explosion is a key ingredient
in stellar evolution, stellar population synthesis. The electron capture (EC)
plays a vital role in supernovae explosions. According to the Shell-Model
Monte Carlo (SMMC) method , basing on the Random Phase Approximation (RPA) and
Linear Response Theory Model (LRTM), we study the strongly screening EC rates
of nuclides 52,53,59,60Fe in presupernova. The results show that the screening
rates can decrease about 18.66%. We compare our results with those of Fuller
et al. (FFN), Aufderheide et al. (AUFD), and Nabi et al. (NKK)in the case with
and without strong electron screening(SES). For the case without SES, our
calculations are in very good agreement with those of AUFD in relatively high
density surroundings and the maximum error is within 0.35% at $\rho_{7}=100$
(e. g., even-even nuclei 60Fe). However, for odd-A nuclei 59Fe, our rates are
close to one, one, two order magnitude smaller than those of FFN, AUFD, and
NKK. For the case with SES, our screening results are about three, two orders
magnitude, and 7.27% lower than those of FFN, AUFD, NKK for 60Fe,
respectively, and it is lower about two, two orders magnitude, and 12.42% than
those of FFN, AUFD, NKK for odd-A nuclide 59Fe.

###### Keywords:

stars: supernovae, stars: evolution, Physical Date and Processes: nuclear
reactions.

##  1 Introduction

It is well known that supernovae not only plays a critical role in the
universe, but also supernovae are major sources of nucleosynthesis of stellar
evolution and galactic chemical evolution. However, the driving mechanisms are
still not well understood for two typical supernova (i.e. the core-collapse
(type II) and thermonuclear (type Ia) supernovae). Some researches show that
electron captures(hereafter EC) on medium-heavy nuclei play an important role
in both types of supernovae. The weak interaction (e.g., EC and Beta decay)
leads up the unstable nuclear burning and iron nucleus collapse in the process
of the supernova explosions r1 ; r2 ; r3 . Therefore, some pioneer works on EC
are investigated by Fuller, et al.r4 ; r5 (FFN); Aufderheide, et al.r6 ; r7
(AUFD); According to the Shell-Model Monte Carlo methodr71 ; r72 ; r73 ; r74 ,
Langanke, et al.r8 ; r9 ; and Juodagalvis, et al. r10  in supernova. Liu, et
al.r11 ; r12 ; r13 ; r14 ; r15 ; r16 ; r17 ; r18 ; r19 ; r2 ; r3 ; r20 ; r1 ;
r21  Nabi, et al. r22 (NKK) also discussed the weak interaction reactions in
explosive stellar environments due to the importance of EC.

Nonetheless, the problem on supernova explosion has always been the
interesting and challenging issue in the fields of astrophysics. On the other
hand, the strong electron screening(SES) has been raised a strong interesting
and challenging problem among nuclear astrophysicist in pre-supernova stellar
evolution and nucleosynthesis.

In the process of supernova explosion, what role on earth should the EC play
in stars? How do the temperature and the density affect on the EC rates? what
role will the SES play in stars? How does SES affect on the EC rates? These
problems show that it is extremely important and useful to calculate
accurately the EC rates for the research of supernova explosion and numerical-
simulation. It is also extremely necessary for us to understand, solve and
calculate accurately the SES and screening corrections in stellar interior for
the relativistic degenerate electron liquid.

Some works r6 ; r7 ; r23 ; r24 ; r14 ; r16 ; r4 ; r5  show that nuclides
52,53,59,60Fe are very important and dominated during the process of supernova
explosions. Thus the rates of them are widely investigated by some eminent
scholars (e.g. r6 ; r7 ; r23 ; r24 ) in supernova. In the same environment,
Liu, et al. and Gutierrez, et al.r18 ; r25  also discussed the weak
interaction reactions on 52,53,59,60Fe. However, their works seem not to
consider the influence of SES on EC. The problems about SES has already been
discussed by Bravo et al.r26 , and Liu et al.r27 . The works mentioned above
show that it is extremely important and necessary to calculate accurately the
screening corrections to the EC rates in dense stars.

The effects of the charge screening on the nuclear physics(e.g. electron
capture and beta decay) Come at least from three factors. Firstly, the
screening potential changes the electron Coulomb wave function in the process
nuclear reaction. Secondly, the electron screening potential decreases the
energy of incident electron joining the capture reaction, which generally also
decreases the electron capture rates of the nuclei. Thirdly, the electron
screening increases the energy of atomic nucleus(i.e. the single particle
energy will increases) in the process of nuclear reaction, thus increases the
nuclear reaction rates. Finally, the electron screening evidently and
effectively decreases the number of the higher-energy electrons, of which the
energy is more than the threshold of the capture reaction. The screening
relatively increases the threshold of the reaction and decreases the capture
rates, but increases for the beta decay rates.

Due to the importance of SES about the EC of 52,53,59,60Fe in astrophysical
environments, in this paper, we focus on these nuclei and reinvestigate their
EC rates according to the Shell-Model Monte Carlo (hereafter SMMC) method, and
the theory of Random Phase Approximation (hereafter RPA) r23 . For the case
without SES, we analyse the error factor $C$ and compared our
results($\lambda_{ec}^{0}$(LJ)) with those of AUFD ($\lambda_{ec}^{0}$(AUFD)),
which is based on the theory of Brink Hypothesis r6 ; r7 . Furthermore, we
discuss the electron capture cross section (hereafter ECCS) and the rates of
the change of electron fraction (hereafter RCEF) in process of EC by using the
theory of RPA. The comparisons of our rates with those of FFN r8 , AUFD r6
,and NKK r22  are presented in table format in the case without SES. On the
other hand, basing on the linear response theory r28 , we investigate the
strongly screening rates and the screening factors $C_{1}$. In order to
understand the influence of SES on EC, we also compare our screening rates
with those of FFN, AUFD, and NKK. We find the influence of SES on the rates is
very significant.

Our work differs from previous works r6 ; r7 ; r4 ; r5 ; r22  about the
discussion of EC. Firstly, the works of FFN and AUFD are based on the theory
of Brink Hypothesis(BH). Basing on quasi-particle random phase approximation,
NKK also discussed the EC reaction in the case without SES. We analyze the EC
process for iron group nuclei and derive new screening results by using the
Shell-Model Monte Carlo (SMMC) method r23  and basing on LRTM r28  and RPA
theoryr8 ; r9 . We also make detailed comparison of the results for the strong
screening rates and no-screening rates among the calculations by FFN, AUFD,
and NKK. Secondly, our discussions differs from Ref.r29 , which analyzed the
EC by using the method of BH and basing on the plasma ion ball strong
screening model(PIBSSM). PIBSSM is a very rough model and BH also is a very
poor approximation, which assumes that the Gamow-Teller strength distribution
on excited states is the same as for the ground state, only shifted by the
excitation energy of the state. Finally, we analyze the effect of SES by the
linear response theory model(LRTM) These screening rates of iron group nuclide
may be universal, very important and helpful for the researches of supernova
explosion and numerical simulation.

The present paper is organized as follows: in the next section, we analyses
the EC rates in stellar interiors in the case with and without strong electron
screening. Some numerical results and discussion are given in Section 3. And
some conclusions are summarized in Section 4.

##  2 The EC in stellar interiors

###  2.1 The EC process in the case without SES

The stellar electron capture rates for the $k$ th nucleus (Z, A) in thermal
equilibrium at temperature $T$ is given by a sum over the initial parent
states $i$ and the final daughter states $f$ r4 ; r5 ; r6 ; r7

$\lambda_{k}^{0}=\lambda_{ec}^{0}=\sum_{i}\frac{2(J_{i}+1)\exp(\frac{-E_{i}}{kT})}{G(Z,A,T)}\sum_{f}\lambda_{if}$
(1)

The EC rate from one of the initial states to all possible final states is
$\lambda_{if}$. The $J_{i}$ and $E_{i}$ are the spin and excitation energies
of the parent states, $G(Z,A,T)$ is the nuclear partition function and given
by

$G(Z,A,T)=\sum_{i}(2J_{i}+1)\exp(-\frac{E_{i}}{kT})$ (2)

Using the level density formula, $\vartheta(E,J,\pi)$ , the contribution from
the excite states is discussed. Thus the nuclear partition function
approximately becomes r7

$\displaystyle G(Z,A,T)$ $\displaystyle\approx$
$\displaystyle(2J_{0}+1)+\int_{0}^{\infty}dE\int_{J,\pi}dJd\pi(2J_{i}+1)$ (3)
$\displaystyle\times\vartheta(E,J,\pi)\exp(-\frac{E_{i}}{kT})$

where the level density is given by r12 ; r34

$\vartheta(E,J,\pi)=\frac{1}{\sqrt{2\pi}\psi}\frac{\sqrt{\pi}}{12a^{\frac{1}{4}}}\times\frac{\exp[2\sqrt{a(E-\delta)}]}{(E-\delta)^{\frac{5}{4}}}f(E,J,\pi)$
(4)

where

$f(E,J,\pi)=\frac{1}{2}\frac{(2J+1)}{2\psi^{2}}\exp[-\frac{J(J+1)}{2\psi^{2}}]$
(5)

where $a$ is the level density parameter, $\delta$ is the backshift (pairing
correction). $\psi$ is defined as

$\psi=(\frac{2m_{u}AR^{2}}{2\hbar^{2}})^{\frac{1}{2}}[\frac{(E-\delta)}{a}]^{\frac{1}{4}}$
(6)

where $R$ is the radius and $m_{u}=\frac{1}{N_{A}}$ is the atomic mass unit.

Based on the RPA theory with a global parameterization of the single particle
numbers, the EC rates in the case without SES is related to the electron
capture cross-section by r10

$\lambda_{ec}^{0}(\rm{LJ})=\frac{1}{\pi^{2}\hbar^{3}}\sum_{if}\int^{\infty}_{\varepsilon_{0}}p^{2}_{e}\sigma_{ec}(\varepsilon_{e},\varepsilon_{i},\varepsilon_{f})f(\varepsilon_{e},U_{F},T)d\varepsilon_{e}$
(7)

where $\varepsilon_{0}=\max(Q_{if},1)$. $p_{e}=\sqrt{\varepsilon_{e}-1}$ is
the momenta of the incoming electron, and $\varepsilon_{e}$ is the total rest
mass and kinetic energies of the incoming electron, $U_{F}$ is the electron
chemical potential, $T$ is the electron temperature. Note that in this paper
all of the energies and the momenta are respectively in units of $m_{e}c^{2}$
and $m_{e}c$, where $m_{e}$ is the electron mass and $c$ is the light speed in
vacuum. The electron Fermi-Dirac distribution is defined as

$f=f(\varepsilon_{e},U_{F},T)=[1+\exp(\frac{\varepsilon_{e}-U_{F}}{kT})]^{-1}$
(8)

Due to the energy conservation, the electron, proton and neutron energies are
related to the neutrino energy, and $\rm{Q}$-value for the capture reactionr32

$Q_{i,f}=\varepsilon_{e}-\varepsilon_{\nu}=\varepsilon_{n}-\varepsilon_{\nu}=\varepsilon^{n}_{f}-\varepsilon^{p}_{i}$
(9)

and we have

$\varepsilon^{n}_{f}-\varepsilon^{p}_{i}=\varepsilon^{\ast}_{if}+\hat{\mu}+\Delta_{np}$
(10)

where $\hat{\mu}=\mu_{n}-\mu_{p}$, the difference between neutron and proton
chemical potentials in the nucleus and
$\Delta_{np}=M_{n}c^{2}-M_{p}c^{2}=1.293Mev$, the neutron and the proton mass
difference. $Q_{00}=M_{f}c^{2}-M_{i}c^{2}=\hat{\mu}+\Delta_{np}$, with $M_{i}$
and $M_{f}$ being the masses of the parent nucleus and the daughter nucleus
respectively; $\varepsilon^{\ast}_{if}$ corresponds to the excitation energies
in the daughter nucleus at the states of the zero temperature.

The electron chemical potential is found by inverting the expression for the
lepton number density

$n_{e}=\frac{8\pi}{(2\pi)^{3}}\int^{\infty}_{0}p^{2}_{e}(f_{-e}-f_{+e})dp_{e}$
(11)

where $f_{-e}=[1+\exp(\frac{\varepsilon_{e}-U_{F}}{kT})]^{-1}$ and
$f_{+e}=[1+\exp(\frac{\varepsilon_{e}+U_{F}}{kT})]^{-1}$ are the electron and
positron distribution functions respectively, $k$ is the Boltzmann constant.

According to the Shell-Model Monte Carlo method, which discussed the Gamow-
Teller strength distributions, the total cross section by EC is given by r23

$\displaystyle\sigma_{ec}$ $\displaystyle=$
$\displaystyle\sigma_{ec}(E_{e})=\sum_{if}\frac{(2J_{i}+1)\exp(-\beta
E_{i})}{Z_{A}}\sigma_{fi}(E_{e})$ $\displaystyle=$ $\displaystyle
6g^{2}_{wk}\int
d\xi(E_{e}-\xi)^{2}\frac{G^{2}_{A}}{12\pi}S_{GT^{+}}(\xi)F(Z,E_{e})$

where $\beta=1/T_{N}$ is the inverse temperature, $T_{N}$ is the nuclear
temperature and in unit of Mev, and $E_{e}=\varepsilon_{e}$ is the electron
energy. $S_{GT^{+}}$ is the Gamow-teller(GT) strength distribution, which is
as a function of the transition energy $\xi$. The $g_{wk}=1.1661\times
10^{-5}\rm{Gev^{-2}}$ is the weak coupling constant and $G_{A}$ is the axial
vector form-factor which at zero momentum is $G_{A}=1.25$.
$F(Z,\varepsilon_{e})$ is the Coulomb wave correction which is the ratio of
the square of the electron wave function distorted by the coulomb scattering
potential to the square of wave function of the free electron.

The SMMC method is also used to calculate the response function $R_{A}(\tau)$
of an operator $\hat{A}$ at an imaginary-time $\tau$. By using a spectral
distribution of initial and final states $|i\rangle$ and $|f\rangle$ with
energies $E_{i}$ and $E_{f}$. $R_{A}(\tau)$ is given by r8

$R_{A}(\tau)=\frac{\sum_{if}(2J_{i}+1)\exp(-\beta
E_{i})\exp(-\tau(E_{f}-E_{i}))|\langle
f|\hat{A}|i\rangle|^{2}}{\sum_{i}(2J_{i}+1)\exp(-\beta E_{i})}$ (13)

Note that the total strength for the operator is given by $R(\tau=0)$. The
strength distribution is given by

$S_{GT^{+}}(E)=\frac{\sum_{if}\delta(E-E_{f}+E_{i})(2J_{i}+1)\exp(-\beta
E_{i})|\langle f|\hat{A}|i\rangle|^{2}}{\sum_{i}(2J_{i}+1)\exp(-\beta E_{i})}$
(14)

which is related to $R_{A}(\tau)$ by a Laplace Transform,
$R_{A}(\tau)=\int_{-\infty}^{\infty}S_{GT^{+}}(E)\exp(-\tau E)dE$. Note that
here $E$ is the energy transfer within the parent nucleus, and that the
strength distribution $S_{GT^{+}}(E)$ has units of $\rm{Mev^{-1}}$.

The presupernova EC rates in the case without SES is given by folding the
total cross section with the flux of a degenerate relativistic electron gas r8

$\displaystyle\lambda_{ec}^{0}(\rm{LJ})=\frac{\ln
2}{6163}\int^{\infty}_{0}d\xi S_{GT}\frac{c^{3}}{(m_{e}c^{2})^{5}}$
$\displaystyle\int^{\infty}_{p_{0}}dp_{e}p^{2}_{e}(-\xi+\varepsilon_{e})^{2}F(Z,\varepsilon_{e})f(\varepsilon_{e},U_{F},T)~{}~{}~{}(\rm{s}^{-1})$
(15)

where the $\xi$ is the transition energy of the nucleus, and
$f(\varepsilon_{n},U_{F},T)$ is the electron distribution function. The
$p_{0}$ is defined as

$p_{0}=\left\\{\begin{array}[]{ll}~{}\sqrt{\bigtriangleup
Q_{if}^{2}-m_{e}^{2}c^{4}}~{}~{}~{}~{}~{}~{}(Q_{if}<-m_{e}c^{2})\\\
~{}0~{}~{}~{}~{}~{}~{}(\rm{otherwise}).\end{array}\right.$ (16)

In the case without SES, we define the error factors $C$, which compare our
results of $\lambda^{0}_{ec}$(LJ), which discussed by method of SMMC with
those of $\lambda^{0}_{ec}$(AFUD), which calculated basing on the method of
Brink Hypothesis by AUFD.

$C=\frac{(\lambda^{0}_{ec}\rm{(LJ)}-\lambda^{0}_{ec}\rm{(AUFD)})}{\lambda^{0}_{ec}\rm{(LJ)}}$
(17)

On the other hand, the RCEF plays a key role in stellar evolution and
presupernova outburst. In order to understand how would the SES effect on
RCEF, the RCEF due to EC reaction on the $k$ th nucleus in SES is defined as

$\dot{Y^{ec}_{e}}(k)=-\frac{X_{k}}{A_{k}}\lambda_{k}$ (18)

where $\lambda_{k}$ is the EC rates; $X_{k}$ is the mass fraction of the $k$
th nucleus and $A_{k}$ is the mass number of the $k$ th nucleus.

###  2.2 The EC process in the case with SES

Using the linear response theory, Itoh et al.r28  calculated the screening
potential for relativistic degenerate electrons. We name this the linear
response theory model (hereafter LRTM) with SES. The electron is strongly
degenerate in our considerable regime of the density-temperature. The
condition is expressed as

$T\ll T_{F}=5.930\times
10^{9}\\{[1+1.018(\frac{Z}{A})^{2/3}(10\rho_{7})^{2/3}]^{1/2}-1\\},$ (19)

where $\rho_{7}$ is the density in units of $10^{7}\rm{g/cm^{3}}$,
$T_{\rm{F}}$ is the electron Fermi temperature, Z and A are the atomic number
and mass number of nucleus considered, respectively.

Based on the relativistic random-phase approximation, the static longitudinal
dielectric function due to the relativistically degenerate electron liquid
calculated by Jancovici et al. r33 . The electron potential energy, which
takes into account the strong screening by the relativistically degenerate
electron liquid, is written as

$V(r)=-\frac{Ze^{2}(2k_{\rm{F}})}{2k_{\rm{F}}r}\frac{2}{\pi}\int_{0}^{\infty}\frac{\rm{sin}[(2k_{\rm{F}}r)]q}{q\epsilon(q,0)}dq,$
(20)

where $\epsilon(q,0)$ is Jancovici s static longitudinal dielectric function
and $k_{\rm{F}}$ is the electron Fermi wavenumber.

Using the linear response theory, r28  calculated the screening potential for
relativistic degenerate electrons. We name this as linear response theory
model (hereafter LRTM) with SES. A more precise screening potential in LRTM is
given by

$D=7.525\times
10^{-3}Z(\frac{10z\rho_{7}}{A})^{\frac{1}{3}}J(r_{s},R)(\rm{Mev})$ (21)

where $J(r_{s},R)$, $r_{s}$ and $R$ can be found in Ref. r13 . The formula
(21) is valid for $10^{-5}\leq r_{s}\leq 10^{-1},0\leq R\leq 50$ conditions,
which are usually fulfilled in the pre-supernova environment.

If the electron is strongly screened and the screening energy is high enough
in order not to be neglected in high density plasma. Its energy will decrease
from $\varepsilon$ to $\varepsilon^{{}^{\prime}}=\varepsilon-D$ in the decay
reaction due to electron screening. At the same time, the screening relatively
decreases the number of high energy electrons with energies higher than the
threshold energy for electron capture. The threshold energy increases from
$\varepsilon_{0}$ to $\varepsilon_{s}=\varepsilon_{0}+D$. Thus the EC rates
with SES becomes

$\displaystyle\lambda^{s}_{ec}(\rm{LJ})=\frac{\ln
2}{6163}\int^{\infty}_{0}d\xi S_{GT^{+}}\frac{c^{3}}{(m_{e}c^{2})^{5}}$
$\displaystyle\int^{\infty}_{p_{0}}dp_{e}p^{2}_{e}(-\xi+\varepsilon_{e})F(Z,\varepsilon_{e})f(\varepsilon_{e},U_{F},T)$
$\displaystyle=\frac{\ln 2}{6163}\int^{\infty}_{0}d\xi
S_{GT^{+}}\frac{c^{3}}{(m_{e}c^{2})^{5}}$
$\displaystyle\int^{\infty}_{\varepsilon_{s}}d\varepsilon^{{}^{\prime}}\varepsilon{{}^{\prime}}(\varepsilon^{{}^{\prime}2}-1)^{\frac{1}{2}}(-\xi+\varepsilon^{{}^{\prime}})^{2}F(Z,\varepsilon^{{}^{\prime}})f(\varepsilon_{e},U_{F},T)$
(22)

We define the screening factors $C_{1}$ in the care with and without SES in
order to understand the effect of SES on the EC process as follows:

$C_{1}=\frac{\lambda^{s}_{ec}(\rm{LJ})}{\lambda^{0}_{ec}(\rm{LJ})}$ (23)

##  3 The results and disscusion

Figure 1 shows the ECCS of nuclides 52,53,59,60Fe as a function of electron
energy at temperature $T_{9}=9,11$. We find with increasing of electron
energy, the ECCS increases greatly. The higher the temperature, the faster the
changes of ECCS becomes. It is because that the higher the temperature, the
larger the electron energy and electron chemical potential are. So even more
electrons will join in the EC process due to their energy is greater than the
Q-values. Furthermore, the Gamow-Teller transition would be dominated in this
process at high temperature surroundings.

As we all know, the trigger of the electron capture requires a minimum
electron energy given by the mass splitting between parent and daughter (i.e.
$Q_{\rm{if}}$). This threshold is lowered by the internal excitation energy at
finite temperature. For even-even parent nuclei the Gamow-Teller strength
centered at daughter excitation energies of order of 2Mev at low temperatures.
Therefore, the ECCS for these parent nuclei increase drastically within the
first couple of MeV of electron energies above threshold. But for odd-A nuclei
the Gamow-Teller distribution will peak at noticeably higher daughter
excitation energies at low temperatures. So the ECCS are shifted to higher
electron energies in comparison to even-even parent nuclei by about 3 MeV.

Figure 1: The ECCS for nuclides 52,53,59,60Fe as a function of the electron
energy at the temperature of $T_{9}=9,Y_{e}=0.44$ and $T_{9}=11,Y_{e}=0.43$
and density of $\rho_{7}=5.86$

Figure 2: The EC rates for nuclides 52,53,59,60Fe as a function of the density
$\rho_{7}$ at the temperature of $T_{9}=3.34,Y_{e}=0.45$ and
$T_{9}=12.6,Y_{e}=0.41$

Figure 3: The EC rates for nuclides 52,53,59,60Fe as a function of the density
$\rho_{7}$ at the temperature of $T_{9}=3.40,Y_{e}=0.47$ and
$T_{9}=7.33,Y_{e}=0.41$

Figure 4: The theoretical $S_{\rm{GT}^{+}}$ for nuclei 60Fe as a function of
the excitation energy $E$ at the ground state($0^{+}$) and 1st excited
state($2^{+}$).

Figure 5: The RCEF due to EC process for nuclides 52,53,59,60Fe as a function
of the density $\rho_{7}$ at the temperature of $T_{9}=3.40,Y_{e}=0.47$ and
$T_{9}=7.33,Y_{e}=0.41$

Figure 6: The factor $C$ for nuclides 52,53,59,60Fe as a function of the
density $\rho_{7}$ at the temperature of $T_{9}=3.34,Y_{e}=0.45$ and
$T_{9}=12.6,Y_{e}=0.41$

Figure 7: The screening factor $C_{1}$ for for nuclides 52,53,59,60Fe as a
function of the density $\rho_{7}$ at the temperature of
$T_{9}=0.133,Y_{e}=0.48$ and $T_{9}=0.74,Y_{e}=0.48$

Figure 8: The screening factor $C_{1}$ for nuclides 52,53,59,60Fe as a
function of the density $\rho_{7}$ at the temperature of
$T_{9}=3.80,Y_{e}=0.45$ and $T_{9}=7.99,Y_{e}=0.43$

Figure 9: The screening factor $C_{1}$ for for nuclides 52,53,59,60Fe as a
function of the density $\rho_{7}$ at the temperature of
$T_{9}=11.33,Y_{e}=0.41$ and $T_{9}=12.6,Y_{e}=0.40$

The EC rates as function of $\rho_{7}$ at some typical astrophysics condition
are shown in Figure 2 and 3. We detailed discuss the EC process according to
SMMC method, especially for the contribution for EC due to the GT transition
base on RPA theory. We find the EC rates are increased greatly and may be in
excess of six orders of magnitude with increasing of density (e. g. for 60Fe
at $T_{9}=3.40,Y_{e}=0.47$). On the other hand, the density make the different
effect on the EC rates for different nuclides due to different Q-values and
transition orbits.

According to our calculations, we find that the GT transition in EC reaction
may not be dominant at a relative low temperature. This process can be
dominated by low energy transition. On the contrary, the distribution of the
electron gas should satisfy the Fermi-Dirac distribution under the condition
of high temperature and density. The GT transition strength of nuclei is
distributed in the form of the centrosymmetric Gaussian function about the GT
resonance point. So the energies of the electrons, which can participate in
the GT resonance transitions are not symmetric in relative high energy range.

The Gamow-Teller strength distributions from shell model Monte Carlo studies
of fp-shell nuclei play an important role in the pre-collapse evolution of
supernovae. The GT+ transitions, which change protons into neutrons, have so
far been addressed only qualitatively in presupernova simulations because of
insufficient experimental information, assuming the GT+ strength to reside in
a single resonance whose energy relative to the daughter ground state has been
parametrized phenomenologically r4 ; r5 . (n, p) experiments show that the GT+
strength is fragmented over many states, while the total strength is
significantly quenched compared to the single particle model.

As an example, we plot the strength distributions $S_{\rm{GT}^{+}}$ as a
function of daughter state excitation energy for nuclei 60Fe. we show the
calculated strength functions for GT+ for the two parent states, the ground
state ($0^{+}$) and first excited state ($2^{+}$) of 60Fe in Figure 4. We
consider and reproduce the first few low-lying levels in 60Fe, which are 0,
1.1, 2.2, 2.4MeV correspondingly to the spin parity of $0^{+}$, $2^{+}$,
$0^{+}$, $2^{+}$. We find the peak of $S_{\rm{GT}^{+}}$ will get to 1.562,
0.223MeV-1 at 0.5MeV, 3.40MeV of daughter nuclei 60Mn for ground state and 1st
excited state, respectively. The total GT strength distribution
$B(\rm{GT})_{tot}$ for the ground state ($0^{+}$) and first excited state
($2^{+}$) is 9.47, 8.19 MeV, respectively. From above discussion, by simply
displacing the ground state strength distribution by the excitation energy,
one can see that the GT distribution for the excited state may be not
qualitatively inferred from the information of the ground state. In fact we
think an average value of the excited state distributions may be the most
standard distribution, which would appear to be the one pertaining to the
excited states.

The RCEF is very sensitivity parameter in the electron capture process. We
find the RCEF decreases and even more than by four orders of magnitude for
60Fe at $T_{9}=7.33$ in Figures 5. With increasing of the density and
temperature, the electron chemical potential becomes so high that large
numbers of electrons join to the EC process. Thus the RCEF reduce greatly.

It is well known that the electron capture (EC) plays a vital role in
supernovae explosions. AUFD expended FFN’s work based on shell model. Their
works are based on the theory of Brink Hypothesis , which detailed discussed
by FFN in the case without SES. We analyze the EC process and derive new
results according to the Shell-Model Monte Carlo (SMMC) method and the Random
Phase Approximation (RPA) theory. In this paper, we define the error factors C
in order to compare our results ($\lambda_{ec}^{0}$(LJ)) with those of AUFD (
$\lambda_{ec}^{0}$(AUFD)). We hope to find some difference between these two
methods at different typical stellar conditions.

Figure 6 shows the results of the error factors C as a function of density
$\rho_{7}$. One can see that with increasing of density, the factor C reduces
greatly. According to our calculations, we find that our results is agreed
reasonably well with AUFD in a high density environment (e. g. $\rho_{7}=100$)
and the maximum error is within 0.35%. On the other hand, in a relative low
density surroundings the maximum error is within 3.982% (e. g.
$\rho_{7}=10,Y_{e}=0.41,T_{9}=12.6$).

The EC rates on these iron group nuclei are important from the oxygen shell
burning phase up to the end of convective core silicon burning phase of
massive stars. Some pioneer works on EC rates have been done by FFN, AUFD, and
NKK. As example, the comparisons of several EC rates(i.e. FFN’s, AUFD’s,
NKK’s, and ours) for nuclides 59Fe and 60Fe are presented in a tabular form at
$\rho_{7}=4010,Y_{e}=0.41,T_{9}=7.33$ in the case without SES. We find it is
well agreement between ours and AUFD’s for even-even nuclide 60Fe. The factor
C is about 0.832, 3.848, and 1.7267 corresponding to those of AUFD, FFN and
NKK. The comparisons for odd-A nuclide 59Fe show that the rates of FFN, AUFD,
and NKK are close to by one, one, and two order magnitude bigger than ours.

Table 2 presents the comparison of our strongly screening results with those
of FFN, AUFD, NKK. From the results of $s_{i}$ (i=1, 2, 3, 4), one can
conclude that the strongly screening rates are about three and two orders
magnitude lower than those of FFN and AUFD for even-even nuclide 60Fe,
respectively, but is about two orders magnitude for odd-A nuclide 59Fe. On the
other hand, due to SES, our screening rates decreases about 12.42%, 7.27%
comparing to those of NKK for odd-A nuclide 59Fe and even-even nuclide 60Fe,
respectively.

The screening factors $C_{1}$ is plotted as a function of $\rho_{7}$ from
figure 7 to 9. Due to SES, we find the rates decrease greatly and even more
than by $\sim 18.66$% and $\sim 17.80$% in Figure 6. The lower the
temperature, the larger the effect on EC rates becomes. This is due to the
fact that the SES mainly decreased the number of higher energy electrons,
which can actively join in the EC reaction. On the other hand, the lower the
temperature(e. g. in Figure 7), the larger the effect on $C_{1}$ is. However,
the higher temperature(e. g., in Figure 8), the higher the average electron
energy becomes, thus the smaller the effect on $C_{1}$ is, due to the
relatively low screening potential. One can also find from Figure 7 to 9 that
the screening factor is nearly the same at higher density and independent on
the temperature and density. The reason is that at higher density surroundings
the electron energy is mainly determined by its Fermi energy, which is
strongly decided by density.

Because of relative low electron screening potential at the low density, we
find that the lower the density, the smaller the effect on EC becomes. As the
density increases, the $C_{1}$ increases gradually due to the increases of the
screening potential. As the density further increases, the factor $C_{1}$ will
close to identical at relative high density. This is because the electron
energy is mainly determined by Fermi energy at higher density and the effect
is relatively weakened by temperature. As the density increases, the
electronic Fermi and shielding potential increases. The ratio between
shielding potential and Fermi energy has nothing to do with density
approximatively.

Of course, we know the screening of nuclear electric charges with a high
electron density means a short screening length, which means a lower
enhancement factor from Coulomb wave correction. However, even a relatively
short electric charge screening length will not have much effect on the
overall rate due to the weak interaction is effectively a contact potential. A
bigger effect is that electrons are bound in the plasma. Table 3 in detail
shows the numerical calculations about the relationship by the minimums value
of screening factor $C_{1min}$ between the rates in the case with and without
SES. For example, the EC rates of nuclei 52,53,59,60Fe are decreased about
$\sim 1.40$%, $\sim 2.12$%, $\sim 17.80$%, $\sim 18.66$% at
$T_{9}=0.133,Y_{e}=0.485$, respectively.

The Q-value of electron capture for some neutron rich nuclei (e.g. 60Fe) has
not been measured, so that the EC Q-value has to be estimated with a mass
formal by FFN. FFN used the Semiempirical atomic mass formula (see Ref.r34 ),
thus the Q-value used in the effective rates are quite different. On the other
hand, For odd-A nuclei (e.g. 59Fe), FFN places the centroid of the GT strength
at too low excitation energies (see the discussions in Ref.r5 ). Their rates
are thus somewhat overestimated. Using the nuclear shell model, AUFD expanded
the FFN’s works. AUFD analyzed the nuclear excited level by a simple
calculation on the nuclear excitation level transitions. The capture rates are
made up of the lower energy transition rates between the ground states and the
higher energy transition rates between GT resonance states. The works of FFN
and AUFD may be an oversimplification and therefore the accuracy is limited.
they adopt the so-called Brink s hypothesis in their calculations. This
hypothesis assumes that the Gamow-Teller strength distribution on excited
states is the same as for the ground state, only shifted by the excitation
energy of the state. This hypothesis is used because no experimental data is
available for the Gamow-Teller strength distributions from excited states and
they did not employ any microscopic theory to calculate the Gamow- Teller
strength functions from excited states.

Using the pn-QRPA theory, NKK expanded the FFN’s works and analyzed nuclear
excitation energy distribution. They have taken into consideration the
particle emission processes, which constrain the parent excitation energies.
The pn-QRPA theory calculates stronger Gamow-Teller strength distribution from
these excited states compared to those assumed using Brink s hypothesis.
However in the GT transitions considered process in their works, only low
angular momentum states are considered.

The method of SMMC is actually adopts an average of GT intensity distribution
of electron capture and the calculated results are in good agreement with
experiments, but the results for most nuclei are generally smaller than other
methods, especially for some odd-A nuclides(e.g. 59Fe). The charge exchange
reactions (p, n) and (n, p) make it possible to observe in the process of weak
interaction, especially for the information of the total GT strength
distribution in nuclei. The experimental information is particularly rich for
some iron nuclei and it is the availability of both $\rm{GT}^{+}$ and
$\rm{GT}^{-}$, which makes it possible to study the problem of renormalization
of $\sigma\tau$ operators in detail. We have calculated the total GT strength
in a full p-f shell calculation, resulting in
$\rm{B}(\rm{GT})=g_{A}^{2}|\langle\vec{\sigma}\tau_{+}\rangle|^{2}$, where
$g_{\rm{A}}^{2}$ is axial-vector coupling constant.

For example, in presupernova the electron capture reaction on 59Fe is
dominated by the wave functions of the parent and daughter states. The total
GT strength for 59Fe in a full p-f shell calculation, is resulting in
$\rm{B}(\rm{GT})=10.1g_{A}^{2}$ r4 . An average of the GT strength
distribution is in fact obtained by SMMC method. A reliable replication of the
GT distribution in the nucleus is carried out and detailed analysis by using
an amplification of the electronic shell model. Thus the method is relative
accuracy.

Summing up the above discussion, basing on RPA and linear response theory, by
using the method of SMMC, we have discussed the EC rates in SES. One can see
that the SES has an evident effect on EC rates for different nuclei,
particularly for heavier nuclides whose threshold is negative (e.g. 59,60Fe)
at relative lower temperature and higher density environment. According to
above calculations and discussion, one can conclude that the strongly
screening rates are decreased greatly and may be in excess of $\sim 18.66$%
based on the RPA theory and SMMC method.

Table 1: The comparisons of our calculations of EC rates in the case without SES for nuclides 59Fe and 60Fe with those of FFN r5 , AUFD r7  and NKK r22  at $\rho_{7}=4010,Y_{e}=0.41,T_{9}=7.33$. The ratio computes as $k_{i}=\frac{\lambda_{ec}^{0}(i)}{\lambda_{ec}^{0}(\rm{LJ})}$, $\lambda_{ec}^{0}(i)$ ($i=1,2,3$) is the rates for FFN, AUFD, and NKK respectively in the care without SES. Nuclide |  $\lambda_{ec}^{0}$(FFN) |  $\lambda_{ec}^{0}$(AUFD) |  $\lambda_{ec}^{0}$(NKK) | $\lambda_{ec}^{0}(\rm{LJ})$ | $k_{1}$ | $k_{2}$ | $k_{3}$
---|---|---|---|---|---|---|---
59Fe | 7.20e$+$02 | 7.43e$+$02 | 2.7e$+$02 | 7.789e$+$01 | 9.244 | 9.539 | 34.704
60Fe | 6.73e$+$01 | 1.44e$+$01 | 3.02$+$01 | 1.749e$+$01 | 3.848 | 0.823 | 1.7267
Table 2: The comparisons of our calculations of EC rates for nuclides 59Fe and 60Fe with those of FFNr5 , AUFD r7  and NKK r22  at $\rho_{7}=33,Y_{e}=0.45,T_{9}=4.24$. The ratio computes as $s_{j}=\frac{\lambda_{ec}^{s}(\rm{LJ})}{\lambda_{ec}^{0}(j)}$, $\lambda_{ec}^{0}(j)$ ($j=1,2,3,4$) is the rates for FFN, AUFD, NKK, and ours respectively in the care without SES. Nuclide |  $\lambda_{ec}^{0}$(FFN) |  $\lambda_{ec}^{0}$(AUFD) |  $\lambda_{ec}^{0}$(NKK) | $\lambda_{ec}^{0}(\rm{LJ})$ | $\lambda_{ec}^{s}(\rm{LJ})$ | $s_{1}$ | $s_{2}$ | $s_{3}$ | $s_{4}$
---|---|---|---|---|---|---|---|---|---
59Fe | 6.30e$-$03 | 5.30e$-$03 | 6.20e$-$05 | 5.63e$-$05 | 5.43e$-$05 | 8.6190e$-$03 | 1.0245e$-$02 | 0.8758 | 0.9644
60Fe | 4.60e$-$03 | 1.00e$-$03 | 1.10e$-$05 | 1.08e$-$05 | 1.02e$-$05 | 2.2174e$-$03 | 1.0200e$-$02 | 0.9273 | 0.9444
Table 3: The minimums value of strong screening factor $C_{1}$, which is comparisons of the screening rates with those of no-screening rate for some typical astronomical condition when $1\leq\rho_{7}\leq 200$. | $T_{9}=0.133,Y_{e}=0.485$ |  | $T_{9}=0.74,Y_{e}=0.481$ |  | $T_{9}=3.80,Y_{e}=0.45$ |  | $T_{9}=7.99,Y_{e}=0.43$
---|---|---|---|---|---|---|---
Nuclide | $\rho_{7}$ | $C_{\rm{min}}$ |  | $\rho_{7}$ | $C_{\rm{min}}$ |  | $\rho_{7}$ | $C_{\rm{min}}$ |  | $\rho_{7}$ | $C_{\rm{min}}$
52Fe | 25 | 0.9986 |  | 18 | 0.9997 |  | 19 | 0.9998 |  | 41 | 0.9999
53Fe | 10 | 0.9788 |  | 10 | 0.9854 |  | 9 | 0.9960 |  | 8 | 0.9984
59Fe | 15 | 0.8220 |  | 15 | 0.9670 |  | 13 | 0.9944 |  | 12 | 0.9978
60Fe | 26 | 0.8134 |  | 26 | 0.9641 |  | 14 | 0.9937 |  | 21 | 0.9971

##  4 Conclusions

In this paper, basing on RPA theory and LRTM, by using SMMC method, we have
carried out an estimation for the EC rates of 52,53,59,60Fe in the case with
and without SES. Meanwhile, the ECCS and the RCEF are discussed in SES. We
also detailed compare our results with those of FFN, AUFD, and NKK, which are
in the case without SES.

Firstly, We find the influence on ECCS is very obvious and significant by
temperature under the condition of SES. With increasing of electron energy,
the ECCS increases greatly. The RCEF is very sensitivity parameter in the EC
process and the RCEF decreases and even more than by four orders of magnitude
(e.g. for 60Fe at $T_{9}=7.33$).

Secondly, for the case without SES, the EC rates increase greatly and ever
exceed by six orders of magnitude(e. g. for 60Fe at $T_{9}=3.40,Y_{e}=0.47$).
We compare our results with those of AFUD due to different methods for
calculating the EC rates. One can find our calculations are in very good
agreement with those of AUFD in relative high density surroundings (e. g.
$\rho_{7}=100$) and the maximum error is within 0.35%. However, it is within
3.982% in a relative low density surroundings(e. g.
$\rho_{7}=10,Y_{e}=0.41,T_{9}=15.6$). On the other hand, as examples, we also
discuss the comparisons of our calculated rates with those of FFN, AUFD and
NKK of 59Fe and 60Fe. We find it is well agreement between our results and
AUFD’s for even-even nuclide 60Fe (i. e. the factor C is about 0.832, but is
3.848, 1.7267 corresponding to FFN and NKK, respectively). The comparisons for
odd-A nuclide 59Fe show that the rates of FFN, AUFD, and NKK are close to by
one, one, and two order magnitude bigger than ours.

Finally, for the case with SES, by using SMMC method, we discuss the strongly
screening rates in supernovae explosive stellar environments basing on RPA and
linear response theory. We compare our strongly screening results with those
of FFN, AUFD, and NKK in the case without SES. One can find that the strongly
screening rates are about three and two orders magnitude lower than those of
FFN and AUFD for even-even nuclei 60Fe, respectively, but it is lower about
two orders magnitude for odd-A nuclei 59Fe. Our screening rates are decreased
about 12.42%, 7.27% comparing to those of NKK for odd-A nuclide 59Fe and even-
even nuclide 60Fe, respectively. However, according to our calculations, our
strongly screening rates ($\lambda_{ec}^{s}$(LJ)) are decreased greatly and
even exceed $\sim 18.66$% corresponding to those of $\lambda_{ec}^{0}$(LJ) in
the case without SES.

As we all know, the EC by SES play an important role in the dynamics process
of the collapsing core of a massive star. It is main parameter which leads to
a supernova explosion and stellar collapse. It also is quite relevant for
simulations in the process of collapse and explosion for massive star. The SES
also strongly influence on the cooling rate and evolutionary timescale in EC
and beta decay process. Our calculations may be helpful for study of the
stellar and galactic evolution and nucleosynthesis calculations. The results
we derived, may become a good foundation for the future investigation of the
evolution of late-type stars, the nature of mechanism of supernova explosions
and the numerical simulation of supernovas.

###### Acknowledgements.

This work is supported by the National Natural Science Foundation of China
under grants 11565020, and the Counterpart Foundation of Sanya under grant
2016PT43, the Special Foundation of Science and Technology Cooperation for
Advanced Academy and Regional of Sanya under grant 2016YD28, the Scientific
Research Staring Foundation for 515 Talented Project of Hainan Tropical Ocean
University under grant RHDRC201701, and the Natural Science Foundation of
Hainan province under grant 114012.

## References

  * (1) J. J. Liu, & W. M. Gu, ApJS., 224: 29(2016)
  * (2) J. J. Liu, MNRAS., 438: 930(2014)
  * (3) J. J. Liu, & D. M. Liu., Ap&SS., 361: 246(2016)
  * (4) G. M. Fuller, W. A. Fowler, and M. J. Newman, ApJ., 42: 447(1980)
  * (5) G. M. Fuller, W. A. Fowler, and M. J. Newman, ApJS., 48: 279(1982)
  * (6) M. B. Aufderheide, G. E. Brown, T. T. S. kuo, D. B. Stout, and P. Vogel., ApJ., 362: 241(1990)
  * (7) M. B. Aufderheide, I. Fushikii, S. E. Woosely, and D. H. Hartmanm, ApJS., 91: 389(1994)
  * (8) M.H., Johnson, & B. A., Lippmann, PhRv., 76: 828(1949)
  * (9) W. E., Ormand, D. J., Dean, C. W. Johnson, et al., PhRvC.,49: 1422(1994)
  * (10) S. E., Koonin, D. J., Dean, and K., Langanke, PhRep.,278: 1(1997)
  * (11) Y., Alhassid, D. J., Dean, S. E., Koonin, et al., PhRvL.,72: 613(1994)
  * (12) K. Langanke & G. Martinez-Pinedo, Phys. Lett. B., 436: 19(1998)
  * (13) K. Langanke & G. Martinez-Pinedo, , Nuclear Phys. A., 673: 481(2000)
  * (14) A. Juodagalvis, K. Langanke, W. R. Hix, et al., Nuclear Phys. A., 848: 454(2010)
  * (15) Z. F. Gao, N. Wang, J. P. Yuan, L. Jiang, D. L. Song, ApS&S., 332: 129(2011)
  * (16) J. J. Liu, and Z. Q. Luo., Chin. Phys. Lett., 16: 1861(2007)
  * (17) J. J. Liu, and Z. Q. Luo., Chin. Phys., 16: 2671(2007)
  * (18) J. J. Liu, and Z. Q. Luo., Chin. Phys., 16: 3624(2007)
  * (19) J. J. Liu, and Z. Q. Luo., Chin.Phys. C., 32: 108(2008)
  * (20) J. J. Liu, and Z. Q. Luo., Comm.Theo. Phys., 49: 239(2008)
  * (21) J. J. Liu, X. P. Kang, et al., Chin. Phys. C., 35: 243(2011)
  * (22) J. J. Liu, Chin. Phys. C., 34: 171(2010)
  * (23) J. J. Liu, Chin. Phys.C., 34: 190(2010)
  * (24) J. J. Liu, RAA., 16: 30(2016)
  * (25) J. J. Liu, Chin. Phys. C., 37: 51018(2013)
  * (26) J. Nabi, and H. V. Klapdor-Kleingrothaus, EPJA, 337: 339(1999)
  * (27) D. J. Dean, K. Langanke, L. Chatterjee, P. B. Radha, and M. R. Strayer, Phys. Rev. C., 58: 536(1998)
  * (28) A. Heger, S. E. Woosley, G. Martinez-Pinedo and K. Langanke, ApJ., 560: 307(2001)
  * (29) J. Gutierrez, E. Garcia-Berro, I. Iben, et al., ApJ., 459: 701(1996)
  * (30) E. Bravo and D. Garcia-Senz., MNRAS., 307: 984(1999)
  * (31) J. J. Liu, Chin. Phys. B., 19: 099601(2010)
  * (32) N. Itoh, N. Tomizawa , M. Tamamura, et al., ApJ., 579: 380(2002)
  * (33) Z. Q. Luo, and Q. H. Peng, ChA&A, 25: 1( 2001)
  * (34) J. A. Holmes, S. E. Woosley, W. A. Fowler, B. A. Zimmerman, Atomic Data and Nuclear Data Tables., 18: 305(1986)
  * (35) F-K. Thielemann, J. W. Truran, and M. Arnould, ana..work., 525: 540(1986)
  * (36) J. Cooperstein, and J. Wambach., Nuclear Phys. A, 420: 591(1984)
  * (37) B. Jancovici, Nuovo Cimento, 25: 428(1962)
  * (38) P. A. Seeger, and W. M. Howard, Nucl. Phys. A, 238: 491(1975)


# Second Language Acquisition of Neural Language Models

Miyu Oba1 Tatsuki Kuribayashi2,3 Hiroki Ouchi1,4 Taro Watanabe1
1Nara Institute of Science and Technology
2MBZUAI 3Tohoku University 4RIKEN
{oba.miyu.ol2, hiroki.ouchi<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

With the success of neural language models (LMs), their language acquisition
has gained much attention. This work sheds light on the second language (L2)
acquisition of LMs, while previous work has typically explored their first
language (L1) acquisition. Specifically, we trained bilingual LMs with a
scenario similar to human L2 acquisition and analyzed their cross-lingual
transfer from linguistic perspectives. Our exploratory experiments
demonstrated that the L1 pretraining accelerated their linguistic
generalization in L2, and language transfer configurations (e.g., the L1
choice, and presence of parallel texts) substantially affected their
generalizations. These clarify their (non-)human-like L2 acquisition in
particular aspects.111Our codes are available at
https://github.com/mlieynua/sla-of-nlm

##  1 Introduction

Cross-lingual transferability of language models (LMs) has attracted much
attention. For example, large English LMs show some translation performance
even when using a small amount of non-English languages as training data
(Brown et al., 2020; Shi et al., 2023), which indicates the efficient language
transfer from English to others. Such cross-lingual transferability has been
evaluated by holistic measures, such as perplexity and accuracy on downstream
tasks (Papadimitriou and Jurafsky, 2020; Deshpande et al., 2022; Blevins et
al., 2022). On the other hand, there is much room for investigating them from
_linguistic perspectives_ ; e.g., grammatical knowledge acquisition and
language transfer tendencies among languages.

Figure 1: Experimental Procedure. First, we pretrain the monolingual masked
language model on the first language (first language acquisition; L1
acquisition). Then, the model is additionally trained under the bilingual
setting (second language acquisition; L2 acquisition). Finally, we analyze the
effect of L1 on L2 via a grammatical judgment test in L2.

In this study, we investigate the cross-lingual transferability of LMs from a
perspective of second language (L2) acquisition. Our main research question is
_how first language (L1) acquisition of LMs affects the efficiency of grammar
acquisition in L2_. To answer this question, we design an experimental
procedure (Section 2): (i) pretraining LMs in a certain language (assumed to
be the L1 speakers), (ii) further training them in English as an L2, and (iii)
evaluating and analyzing their linguistic generalization in L2. As L1s, we
chose four languages with different levels of difficulty in transferring to
English, i.e., French, German, Russian, and Japanese. The size of training
data is restricted to match the human-like L2 acquisition scenario, which
enables better comparison with human L2 acquisition tendencies and, hopefully,
provides insights into L2 acquisition from a computational linguistic
perspective.

We begin with exploring the inductive biases of several L2 training methods
(Section 3). Specifically, we compared some variations of L2 data settings,
such as training on only the L2 texts or on L1–L2 translation pairs. We
observed that, for example, feeding L1–L2 translation pairs into LMs slowed
down their L2 grammar acquisition, compared to only feeding L2 monolingual
texts every two epochs.

In our main experiments, we conducted exploratory analyses of the effects of
L1 training on L2 grammar acquisition (Section 4). We gained three main
insights. First, L1 knowledge promotes better linguistic generalization in L2
(Section 4.1). Second, different L1s incur different generalizations in L2
(Section 4.2). More specifically, Japanese and Russian are far behind French
and German, which is consistent with the human-defined difficulty levels of
language transfer (Chiswick and Miller, 2004). Third, L1 pretraining gives
different effects on different types of grammar items (Section 4.3). In
particular, morphological and syntactic items get larger gains than semantic
and syntax&semantic items.

In more detail, we analyzed the process of L2 acquisition (Section 5). We
investigated how L2 knowledge acquisition progresses (Section 5.1) and found
that L2 knowledge acquisition does not progress so much until seeing the whole
dataset many times (e.g., 50-100 times), implying their data inefficiency.
Furthermore, we also observed the L1 knowledge degrade during L2 acquisition;
this motivates us to balance the source–target linguistic knowledge during
language transfer.

##  2 Second language acquisition of LMs

#### Overview:

We are interested in how L1 knowledge affects the linguistic generalization of
LMs in L2. Figure 1 shows an overview of the experimental procedure. First, in
our L1 acquisition simulation, we train LMs on a monolingual corpus of a
specific language. Second, in our L2 acquisition simulation, we additionally
train the pretrained LMs with a corpus including L2 texts (English). Finally,
we evaluate the grammatical judgment ability of the LMs in the L2 (English)
using BLiMP (Warstadt et al., 2020).

###  2.1 Language exposure

Lang. |  Family |  Order |  Script |  Rank
---|---|---|---|---
French |  IE |  SVO |  Alphabet |  $1$
German |  IE |  SOV |  Alphabet |  $2$
Russian |  IE |  SVO |  Cyrillic |  $3$
Japanese |  N-IE |  SOV |  Kana/Kanji |  $4$
English |  IE |  SVO |  Alphabet |  -
Table 1: Characteristics of the four languages used in our experiment. English
is employed as L2, and the others are L1s. “Rank” indicates the transfer
difficulty from the corresponding language to English, based on the linguistic
distance and FSI rank; a higher value indicates a greater gap to English from
the language acquisition perspective. “Family” indicates if Indo-European (IE)
or not (N-IE). “Order” indicates canonical word order in the corresponding
language.

#### First and second languages:

We used French, German, Russian, and Japanese as L1 and employed English as L2
(Table 1). We expect that the transfer to English becomes more difficult in
order of French, German, Russian, and Japanese from multiple perspectives:
linguistic distance (Grimes and Grimes, 2002; Chiswick and Miller, 2004) and
learning difficulty level222https://www.state.gov/foreign-language-training/
Note that these difficulty levels only indicate the difficulty of transferring
from English to a specific language. In our study, we tentatively assume the
symmetry of the source and target language in terms of learning difficulty..

#### L1 acquisition:

We first train LMs in particular L1 language using a monolingual corpus of
approximately 100M words sampled from CC-100 (Conneau et al., 2020; Wenzek,
2020). The corpus size is roughly similar to the number of words exposed to
humans during language acquisition. We trained the models with 100
epochs.333This number of epochs might be cognitively-implausible since humans
would not face the same example 100 times, but it is also argued that the
memory of language experience continues to affect the learning multiple times
(Bybee, 2013).

#### L2 acquisition:

We then further train the L1 LMs under bilingual input (Section 3). We trained
the models with 100 epochs, but the intermediate checkpoints will also be
analyzed to track the process of L2 acquisition in Section 5. We used Tatoeba
(Tiedemann, 2012)444https://opus.nlpl.eu/Tatoeba.php as parallel corpora in L2
acquisition. Tatoeba is a multilingual parallel corpus consisting of example
sentences originally collected for human foreign language learners. From the
L2 acquisition perspective, this amount would be large enough for human
learners to learn the top 95% English words in frequency (Nation, 2014).

Note that there would be several scenarios of human L2 learning/acquisition,
such as through language classes or natural communications. Following Krashen
et al. (1979), we refer to L2 acquisition as the latter scenario of acquiring
L2 through natural language exposure, e.g., raw texts.

###  2.2 Learners

We largely followed the settings of the cross-lingual language model (XLM)
(Conneau and Lample, 2019), which is typically used in cross-lingual language
modeling in the field of natural language processing (NLP). In short, this is
a Transformer-based bidirectional LM, but the input consists of bilingual text
pairs. The tokens in the bilingual text were randomly masked, and the model
predicts the masked tokens on both L1 and L2 sides. During the L1 training,
the L1 side is the only input.

The bilingual XLM is trained from scratch (L1 training and L1–L2 training),
rather than using the off-the-shelf pre-trained XLM that is trained across
dozens of languages (Conneau and Lample, 2019; Conneau et al., 2020). From a
cognitive perspective, such a super-multilingual setting is unrealistic since
humans hardly face dozens of languages in a multilingual acquisition scenario.
Rather, we hope that such a bilingual transfer will gain much attention from
the adjacent areas, such as pedagogy and cognitive science of exploring human
second language acquisition/learning.

Technically, we randomly initialized the parameters of the XLM (18M),
constructed a bilingual tokenizer using byte pair encoding on the bilingual
texts, and trained the model separately for each L1–L2 combination. For each
L1–L2 setting, we trained four models with different seeds; the results
reported in Sections 4 and 5 are the average of scores from four models. See
Table 5 in Appendix for the hyperparameters and detailed settings.

Figure 2: Training settings investigated in Section 3: (i) L1–L2 text pairs
without translation relationship (TLM-nopara), (ii) translation pairs (TLM-
para), and (iii) a mixed setting where parallel L1 text is removed every other
epoch (TLM-drop).

###  2.3 Evaluation

#### Dataset:

We used BLiMP (Warstadt et al., 2020), a benchmark of English grammatical
judgment test to evaluate the models’ L2 linguistic generalization. The
dataset consists of 12 test suites; each corresponds to a specific linguistic
phenomenon and falls into one of four coarse linguistic categories:
morphology, syntax, semantics, and syntax&semantics. Each test suite has 1,000
minimal sentence pairs. Each pair consists of grammatically acceptable and
unacceptable ones as follows:

  * a.

Many teenagers were helping themselves.

  * b.

* Many teenagers were helping herself.

#### Grammatical judgement:

To select one sentence in each pair, we adopted pseudo-perplexity, commonly
used in exploring the linguistic behaviors of LMs (Lau et al., 2020).
Specifically, if the model can assign a lower pseudo-perplexity to the
grammatical sentence than to the paired ungrammatical one, we regard it as
correct. Following Salazar et al. (2020), psuedo-perplexity ($\mathrm{PPPL}$)
of sentence $\boldsymbol{s}=[w_{1},w_{2},\cdots,w_{n}]$ is computed using the
bidirectional LM $\theta$:

$\mathrm{PPPL}(\boldsymbol{s})=\prod_{t=1}^{n}p_{\theta}(w_{t}|\boldsymbol{s}_{\setminus
w_{t}})^{\frac{1}{|\boldsymbol{s}|}}\;\;,$ (1)

where $w_{t}$ denotes the $t$-th token in sentence, and
$\boldsymbol{s}_{\setminus w_{t}}$ denotes all the tokens in the sentence
except for $w_{t}$; $[w_{1},\cdots,w_{t-1},w_{t+1},\cdots,w_{n}]$. The
probability of $w_{t}$ given its bidirectional context
$\boldsymbol{s}_{\setminus w_{t}}$ is calculated by the model $\theta$. Based
on the selected sentences, we calculated an accuracy score on each test suit
of BliMP. We also report the macro-average of accuracies among all the test
suits. Note that all the accuracy scores reported in the tables/figures in
this paper are multiplied by 100 for readability.

##  3 Preliminary experiment: L2 exposure configurations

First, we investigate the inductive bias of L2 training settings. While
existing studies use parallel data as an input for cross-lingual training
(Conneau and Lample, 2019), we investigate the bias in this setting from L2
grammar acquisition perspectives.

#### Settings:

We set up the three training settings with different input data: (i) L1–L2
text pairs without the translation relationship (TLM-nopara), (ii) L1-L2
translation pairs (TLM-para), and (iii) a mixed setting where L2 text
concatenated with L1 parallel text is used as input or only L2 text is used as
input (TLM-drop)555In other words, the bilingual and monolingual settings are
switched alternately for each epoch.. An overview of the experimental settings
is shown in Figure 2 (see details in Appendix A). Note that the original XLM
(Conneau and Lample, 2019) adopts a setting similar to the TLM-drop. In this
experiment, we report the macro-average BLiMP accuracies across the test
suites. Table 2 shows the results (see Table 7 for the results in fine-grained
test suits).

#### Translation pairs does not facilitate L2 acquisition:

One notable point in Table 2 is that the results in the TLM-nopara setting
were better than those in the TLM-para setting.666Statistical differences
between the settings are tested across seeds$\times$languages with Mann-
Whitney U tests ($p$=4.6e-2 for TLM-para and TLM-nopara, $p$=1.0e-2 for TLM-
nopara and TLM-drop)  This suggests that parallel data input does not
facilitate L2 acquisition. Perhaps, the TLM-para task was too easy for LMs to
learn syntactic knowledge; the TLM-para task could partially be solved solely
by relying on lexical knowledge, i.e., capturing the lexical correspondences
between the tokens in L1 and L2 sentences and predicting the word found only
in one of them. In this sense, the TLM-nopara setting, by contrast, might
impose a more difficult problem on LMs and promote their effective learning of
linguistic knowledge.

#### Switching between using L2 text with and without its parallel L1 text
during training facilitates L2 acquisition:

Another notable point is that the TLM-drop was the most effective for
acquiring linguistic knowledge in L2 for LMs.6 Since we switch between using
L1 text as input or not every epoch, there is a possibility that monolingual
and bilingual training have a complementary positive effect. In addition, this
might mitigate the training-inference mismatch in evaluating LMs’ linguistic
knowledge using BLiMP. Here, a single sentence is used as input, which is
compatible with the phase of using only L2 text during the TLM-drop training.
In subsequent experiments, we will use the TLM-drop setting as it was the most
effective training setting for L2 grammar acquisition.

Model (TLM) | Settings | First language
---|---|---
para. | drop | Fr | De | Ru | Ja
nopara |  |  | $52.0$ | $57.6$ | $51.2$ | $52.5$
para | ✓ |  | $51.1$ | $53.6$ | $48.9$ | $51.3$
drop | ✓ | ✓ | $\mathbf{58.0}$ | $\mathbf{61.1}$ | $\mathbf{52.8}$ | $\mathbf{56.2}$
Table 2: Performance of bilingual LMs on BLiMP in different training settings.
The _para._ column indicates whether parallel corpus was used. The drop column
indicates whether the L1-side input is removed every other epoch.

##  4 Experiments: L1$\rightarrow$L2 effects on linguistic generalization

| Morphology | Syntax | Semantics | Syntax & Sem.
---|---|---|---|---
Lang. | L1 |  Overall |  Ana. agr |  D-n agr |  Irregular |  S-v agr |  Arg. str |  Ellipsis |  Filler-gap |  Island |  NPI |  Quantifiers |  Binding |  Ctrl. Rais.
Fr | ✓ | $58.0$ | $55.8$ | $69.5$ | $73.0$ | $60.4$ | $55.4$ | $67.7$ | $54.6$ | $52.2$ | $40.5$ | $56.5$ | $51.8$ | $58.6$
$\Delta$ | $5.3$ | $2.3$ | $14.5$ | $3.8$ | $8.8$ | $7.2$ | $17.0$ | $4.5$ | $-0.8$ | $3.5$ | $-1.2$ | $1.9$ | $1.5$
De | ✓ | $61.1$ | $43.1$ | $68.7$ | $69.3$ | $67.0$ | $53.1$ | $63.5$ | $68.2$ | $47.7$ | $54.6$ | $80.5$ | $65.2$ | $52.2$
$\Delta$ | $5.2$ | $5.9$ | $11.1$ | $-11.5$ | $14.3$ | $4.6$ | $14.7$ | $4.8$ | $4.5$ | $9.8$ | $4.6$ | $1.4$ | $-2.2$
Ru | ✓ | $52.8$ | $52.9$ | $58.6$ | $72.7$ | $54.9$ | $47.0$ | $54.2$ | $52.4$ | $49.3$ | $32.8$ | $56.2$ | $40.7$ | $61.4$
$\Delta$ | $0.7$ | $-3.1$ | $3.2$ | $0.3$ | $2.9$ | $0.5$ | $5.2$ | $4.1$ | $-1.9$ | $1.1$ | $-4.5$ | $-0.1$ | $0.3$
Ja | ✓ | $56.2$ | $61.5$ | $65.8$ | $70.5$ | $53.0$ | $52.1$ | $55.3$ | $51.3$ | $54.0$ | $41.0$ | $50.6$ | $61.0$ | $57.8$
$\Delta$ | $1.5$ | $0.7$ | $5.0$ | $-3.3$ | $1.1$ | $2.0$ | $4.3$ | $1.1$ | $4.6$ | $2.2$ | $-2.9$ | $0.8$ | $2.3$
Table 3: English (L2) grammatical knowledge of bilingual LMs with different L1s. Overall indicates the macro-average accuracy over all linguistic phenomena. The rows with ✓in the L1 column exhibit the accuracy of bilingual LMs on BLiMP. The rows with $\Delta$ in the L1 column exhibit the performance difference between the LMs with and without L1 pretraining. The coarse categories (e.g., Morphology) are from the metadata of BLiMP.  L1 | Morph. | Syntax | Semantics | Syn.&Sem.
---|---|---|---|---
Fr | $7.3$ | $7.0$ | $1.2$ | $1.7$
De | $5.0$ | $7.2$ | $7.2$ | $-0.4$
Ru | $0.8$ | $1.9$ | $-1.7$ | $0.1$
Ja | $0.9$ | $3.0$ | $-0.3$ | $1.5$
Avg. | $\mathbf{3.5}$ | $\mathbf{4.8}$ | $\mathbf{1.6}$ | $\mathbf{0.7}$
Table 4: Performance difference between the LMs with and without L1
pretraining for each coarse category of grammatical items (morphology, syntax,
semantics, and syntax&semantics).  Figure 3: Grammar learning trajectories in
each test suite on BLiMP (the L2 side). The x-axis denotes the epoch during L2
acquisition, and the y-axis denotes the accuracy in the corresponding test
suite.

We investigate how _pretraining with L1_ affects L2 grammar acquisition in
LMs. We exploratorily compare the linguistic generalization of LM trained in
the settings with or without L1 pretraining. Table 3 shows the grammaticality
judgment ability after additional training. The Overall column indicates the
macro-average accuracy score across the grammar items. The $\Delta$ rows show
the difference in BLiMP accuracy between models with and without pretraining.
Here, the models without pretraining were trained only with bilingual corpus
without L1 monolingual corpus pretraining.

###  4.1 L1s promote L2 generalization

Table 3 shows the effect of pretraining with L1 on L2 grammar acquisition.
Most of the $\Delta$ values are positive; i.e., the models pretrained with L1s
achieved better results than those without pretraining. This demonstrates that
pretraining in a particular language generally improves English grammatical
ability.777The standard deviation over the different seeds is an average of
1.2 (0–100 scale) for the scores in Table 3, which means that the seed
randomness does not overturn the conclusion. This positive effect is in light
of the assumptions that different languages share some grammatical universals,
and learners could use such a common property in language transfer Cook
(1985). Ri and Tsuruoka (2022) also reported that pretraining in a natural
language other than English improves the overall syntactic parsing performance
in English. Our results are consistent with such positive effects. Besides,
our experiment provides the results of each of more fine-grained grammatical
items related to morphological, syntactic, and semantic phenomena.

###  4.2 Differences in L1s

The $\Delta$ values in the Overall column in Table 3 differ across the L1s.
French is the highest, followed closely by German, and Japanese and Russian
are far behind these two languages; pretraining in French and German is much
more effective than in Japanese and Russian. This ordering shows parallels
with the presumed language learning difficulty order: French, German, Russian,
and Japanese. This suggests that the difficulty of acquiring an L2 grammatical
ability is somewhat close between LMs and humans.

###  4.3 Differences in grammar items

The $\Delta$ scores in Table 3 exhibit that different grammatical items obtain
different degrees of gains. Table 4 shows the average $\Delta$ scores for each
course grammar category. There was a general tendency for morphological and
syntactic items to get larger gains from the L1 pretraining than semantic and
syntax&semantic items except for particular settings, e.g., Irregular. It has
been shown that linguistic phenomena related to semantics such as NPI
(negative polarity item) and Quantifiers were relatively difficult for LMs to
learn Warstadt et al. (2020). Based on this, there is a concern that LMs
failed to learn enough such linguistic knowledge in L1 to transfer it to
another language.

###  4.4 Differences in L1$\times$grammar-item

Notably, in specific combinations of L1s and grammar items, the L1 pretraining
hurt the L2 generalization, i.e., negative transfer problem. For example, the
performance in the Irregular item was not so much enhanced or even degraded by
L1 pretraining. The Irregular (Irregular forms) item targets English-specific
irregular verb conjugations; its less effect by L1 pretraining is due to the
concern that the irregular patterns generally could not be predictable by
other language’s knowledge.

We also found that the same grammatical item was affected differently
depending on the L1. For example, the $\Delta$ values on the Filler-gap item
in Table 3 differ across the L1s, e.g., 4.8 in German and 1.1 in Japanese. At
least in this Filler-gap aspect, there is an interesting parallel between our
results and linguistic notions; Japanese is the only language where gap
precedes filler in wh-construction among the L1s we used, and the transfer
from Japanese to English was indeed limited ($\Delta=1.1$) compared to other
L1s. There might be a possibility that such linguistic (dis)similarities are
reflected in the results. Nevertheless, concluding the exact consistency
between our L1$\times$grammar-item results (Table 3) and the L1–L2 grammatical
similarity requires further interdisciplinary research.

##  5 Analysis: acquisition process

This section sheds light on the process of L2 acquisition. We investigate how
L2 knowledge acquisition progresses (Section 5.1) and how original L1
knowledge changes during L2 acquisition (Section 5.2). As for the L1 knowledge
during L2 acquisition, there is a concern, for example, that LMs exhibit
catastrophic forgetting about their L1.

Figure 4: Grammar learning trajectories in each test suite on CLAM (the L1
side). The x-axis denotes the epoch during L2 acquisition, and the y-axis
denotes the accuracy in the corresponding test suite. The upper parts show the
scores of the bilingual LMs (L1 pretraining and bilingual training). The lower
parts show the scores of the L1-only LMs (L1 pretraining and further training
on L1 texts collected from the parallel corpus).

###  5.1 L1$\rightarrow$L2 effects

#### Settings:

We evaluate the L2 linguistic knowledge of intermediate checkpoints of LMs
(Figure 3). Specifically, we evaluate LMs after {1, 2, 3, 4, 5, 10, 20, 30,
40, 50, 100} training epochs. Note that we analyzed the same models used in
Section 4; the results after 100 epochs are the same ones reported in Section
4.

#### General improvement after dozens of epochs:

The trajectory of the Overall scores in Figure 3 suggest that linguistic
ability generally improves along with the number of epochs. There was a
tendency for large improvements to emerge after dozens of epochs; in other
words, the models began to acquire L2 knowledge after seeing the same examples
many times, e.g., 50-100 times. Note that humans are argued to acquire a
vocabulary after encountering the same word about 12 times Nation (2014), and
of course, the lexical and syntactic acquisition is not comparable, but the
observation that the L2 knowledge improves after 50-100 rounds of the corpus
may be in the direction that LMs are inefficient at acquiring a new language.

#### Differences in grammar items:

Focusing on the general trajectory shapes for each grammatical item, we
observed at least four patterns: (i) spike-at-the-end (D-N agr., Irregular,
S-V agr.), (ii) flat (Arg.str., Ctrl.Rais., Island), (iii) bumpy (Ana.agr.,
Ellipsis, NPI, Quantifiers), and (iv) mixed (Filler-gap, Binding). In
addition, these groups roughly mirror the linguistic categories of the grammar
items (morphology, syntax, semantics, and syntax&semantics); for example, all
the items in the spike-at-the-end group are morphological phenomena, while all
the semantic categories (NPI, Quantifiers) yielded the bumpy patterns. Note
that existing studies reported that low-level (e.g., morphological) linguistic
skills could be acquired earlier and vice versa (Liu et al., 2021; Blevins et
al., 2022); but at least in our cognitively-inspired bilingual training
scenario, we did not observe such an explicit tendency.

#### Inter-L1s differences:

In terms of the change of inter-L1s differences of accuracies in each grammar
item, there are several different patterns: (i) converging (Irregular,
Island), (ii) diverging (Arg.str., Binding, D-N agr., S-V agr.), and (iii)
none of them. Considering Irregular as an example of the converging group, the
accuracies were substantially different across the L1s in the initial stage of
training; these differences, however, gradually reduced along with epochs. On
the other hand, considering S-V agr. as an example of the diverging group, the
accuracies gradually differed in the latter stage of training among the LMs.
In the third group, the inter-L1s accuracy differences remain the same or
unstable during L2 tranining (Ana.agr., Ctrl.Rais., Ellipsis, Filler-gap, NPI,
Quantifiers). At least the former two groups imply that pretraining with
different L1s differently affects the efficiency of L2 acquisition (e.g.,
different slopes for different L1s).

To sum up, we clarified that different L1s and grammar items exhibit different
learning dynamics of LMs. The cognitive plausibility of these patterns could
be the next important investigation.

###  5.2 L2$\rightarrow$L1 effects

In contrast to the previous analysis, which analyzed the impact in the
L1$\rightarrow$L2 direction, we further analyze the L2$\rightarrow$L1 impact.
In applied linguistics, L1–L2 impact in both directions is of interest; for
example, it is reported that the L1 ability is sometimes hurt by the increase
of L2 exposure (Kecskes, 2008; Haman et al., 2017).

#### Settings:

In the same way as Section 5.1, we evaluate the grammatical knowledge of LMs
during L2 training, but the focus is on the L1 knowledge. We used a
multilingual benchmark of grammatical judgment test CLAMS (Mueller et al.,
2020). This dataset consists of seven syntactic test suites for several
languages. Similarly to BLiMP, the dataset consists of pairs of sentences,
where one is grammatical, and the other is ungrammatical. We report the
accuracy scores in terms of whether the LMs could assign lower pseudo-
perplexity to the grammatically correct sentence. We analyze French-L1,
German-L1, and Russian-L1 LMs since this dataset covers these three L1s.

As a baseline, we also evaluate the L1-only LMs that are first pretrained in
L1, then additionally trained with the L1-side texts collected from the
corresponding parallel corpus; that is, the only difference between bilingual
LMs and L1-only baselines was the presence of L2 texts during the L2
acquisition phase.

#### L2 effects once occur, but diminish:

Figure 4 shows the results (see Table 8 for the exact scores). We found that
the L1 knowledge is greatly influenced by the L2, especially at the initial
stage of L2 training. For example, the French-L1 and German-L1 LMs gained
positive effects, and the Russian-L1 LM got a negative influence (the top row
of Figure 4). In addition, in the latter stage, the L2 effects on the L1
knowledge gradually fade, in either a good or bad sense, and the L1 linguistic
knowledge is converging to the original level. For example, French-L1 and
German-L1 LMs exhibited better performance after bilingual language modeling
once, e.g., at the 10 epochs, but the gain decreased after further bilingual
training.

#### L2 negatively affects L1 knowledge:

In Figure 4), the L1 knowledge in bilingual LMs was competitive or even
inferior to that in L1-only LMs in the end, although there were also
exceptional cases in specific grammar items in German. For example, in the
French-L1 LMs, the L1 syntactic generalization performance after L2 training
converged below 0.7 points, while the L1-only baseline model generally
achieved above 0.7 points. Contrasting the L1 results with L2 ones (Section
5.1), there is an asymmetry effect between L1 and L2. From a perspective of
developing linguistically better multi-lingual LMs, these suggest that
balancing the linguistic knowledge of the languages, especially enhancing the
L1 knowledge, during language transfer is challenging even if the model is
exposed to L1 during bilingual training. Addressing these challenges with, for
example, some regularization will be a promising direction from an engineering
perspective.

##  6 Related work

#### Cognitively-motivated analysis of neural LMs:

Investigations into the ability of neural models in language acquisition began
in the 1980s with the interest of whether language can be acquired without
innate knowledge (Rumelhart and McClelland, 1986; Pinker and Prince, 1988).
The initial investigation was made with simple neural networks; after the
development of Neural NLP (Manning, 2015), the classical questions posed by
cognitive science are currently revisited (Kirov and Cotterell, 2018). A
typical movement is a growing interest in the probing of neural LMs’
linguistic knowledge (Linzen et al., 2016; Warstadt and Bowman, 2020). In this
context, our study analyzes L2 acquisition in neural LMs, while existing
studies have typically focused on L1 acquisition.

#### Language transfer in computational models:

Language transfer of NLP models is actively researched from both engineering
and scientific perspectives. In the engineering context, to mitigate the
English-centric focus of NLP techniques, models that can handle more languages
have been developed (Dong et al., 2015; Conneau and Lample, 2019; Conneau et
al., 2020). From the scientific perspective, the mechanism and linguistic
properties of LMs’ language transfer have been explored (Pérez-Mayos et al.,
2021; Tyler et al., 2022; Blevins et al., 2022), sometimes beyond the transfer
between natural languages (Ri and Tsuruoka, 2022; Papadimitriou and Jurafsky,
2020). One of the motivations of such analyses is to quantify the transferable
universals behind (non-)languages. Notably, simulation of L2 acquisition is
also explored from pedagogical motivations (Settles et al., 2018).

#### Language transfer in humans:

L2 acquisition/learning has long been studied in applied linguistics,
psycholinguistics, and pedagogy fields (Krashen, 1981; Hatch, 1983; Ellis,
2010). These fields articulated several hypotheses/theories on human language
learning, e.g., input hypothesis (Krashen, 1977). Analyzing the LMs’ L2
acquisition in a more direct light with these hypotheses would be interesting
for future work.

##  7 Conclusions

We have investigated the L2 acquisition of LMs, especially focusing on their
grammatical knowledge in L1 and L2. Specifically, we have trained bilingual
LMs under a similar scenario to the human L2 acquisition and then analyzed
their cross-lingual transfer. Our experiments have demonstrated that L1
pretraining promotes their linguistic generalization in L2, and there are
interesting variations in L1 pretraining effects with respect to the L1
choice, training settings, and grammar items. The results have also implied
that their L2 acquisition is not human-like in particular aspects.

## Limitations

### Coverage of experiments

There is room for further exploration in terms of model architectures, data,
and evaluation settings in our study. Experiments in more diverse settings
would enhance the generality of the conclusions.

#### Models:

The architecture was fixed to XLM (14M), although we tested four models with
different seeds in each setting. Specifically, testing unidirectional LMs will
be in a reasonable direction, considering the humans’ incremental language
processing. Related to this, the measure of pseudo-perplexity might also
induce unintended biases in our results, this is a common metric in NLP
though. In addition, there are different methods to fine-tune the model to
multiple languages, e.g., using adapters. Comparing these methods with our
scheme will be an interesting direction.

#### Data:

There are possible variations of L1–L2 combinations. Although we selected L1
from one of the four languages (German, French, Russian, Japanese) and fixed
L2 as English, increasing the coverage of languages will lead to more
generalized conclusions. Furthermore, the performance of LMs was generally not
so good on the BLiMP dataset. We suspect that this is due to the limited L2
training data size; it is also worth exploring scaling up the experiments into
typical NLP experiments.

#### Evaluation:

While our focus is on morphology, syntax, and semantic generalization, L2
acquisition studies are conducted from broader perspectives, such as the
growth of vocabulary size. In addition, our observations are from the
perspective of the LMs; the contrast between LMs’ and humans’ L2 learning is
more important from an interdisciplinary perspective.

### Performance was overall poor

One reviewer is concerned that our results on the BLiMP are generally near the
chance level, and it may be difficult to derive findings from such poor
results. We thank the reviewer and would like to share our thoughts and
limitations here.

First, comparisons to the chance rate are not always meaningful. In BLiMP, the
task is typically to select a correct generalization over an incorrect one.
Occasionally, neural models overly prefer incorrect generalizations more than
chance level. For instance, in the linear vs. hierarchical generalization
contrast, neural models often favor linear, causing accuracy to drop near 0,
far below chance McCoy et al. (2018). In such cases, achieving accuracies
around 50 indicates more than random guessing, as models avoid an excessive
preference for incorrect generalizations, moving toward a more neutral stance.
Thus, we believe that it is also worthwhile to observe how much performance
improves from below the chance level.

Furthermore, in BLiMP’s finer-grained test suits, our models sometimes exhibit
an accuracy of 0 or 100 (resulting in an overall score of around 50.0),
highlighting that our models do not always act as random guessing baselines.
The full results across more fine-grained test suits are shown in Appendix 7.

## Ethics Statement

There might be a possibility that the texts we used (CC-100 and Tatoeba) have
socially biased, despite their popular use in the NLP community. We adopted
cognitively-plausible restricted settings with respect to data size, which can
potentially be aligned with environmentally friendly, green NLP.

## Acknowledgements

We would like to express our gratitude for the anonymous reviewers who
provided many insightful comments that have improved our paper. Special thanks
also go to the members of NAIST NLP Labolayory and Tohoku NLP Laboratory for
the interesting comments and energetic discussions. This work was supported by
JSPS KAKENHI Grant Number JP19K20351.

## References

  * Blevins et al. (2022) Terra Blevins, Hila Gonen, and Luke Zettlemoyer. 2022.  Analyzing the mono- and cross-lingual pretraining dynamics of multilingual language models.  In _Proceedings of EMNLP_ , pages 3575–3590, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Bybee (2013) Joan L. Bybee. 2013.  49 Usage-based Theory and Exemplar Representations of Constructions.  In _The Oxford Handbook of Construction Grammar_. Oxford University Press.
  * Chiswick and Miller (2004) Barry Chiswick and Paul Miller. 2004.  Linguistic distance: A quantitative measure of the distance between english and other languages.  Technical Report 1246, Institute of Labor Economics (IZA).
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of ACL_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  Cross-lingual language model pretraining.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Cook (1985) Vivian J Cook. 1985.  Chomsky’s universal grammar and second language learning.  _Applied linguistics_ , 6(1):2–18.
  * Deshpande et al. (2022) Ameet Deshpande, Partha Talukdar, and Karthik Narasimhan. 2022.  When is BERT multilingual? isolating crucial ingredients for cross-lingual transfer.  In _Proceedings of NAACL_ , pages 3610–3623, Seattle, United States. Association for Computational Linguistics.
  * Dong et al. (2015) Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015.  Multi-task learning for multiple language translation.  In _Proceedings of ACL_ , pages 1723–1732, Beijing, China. Association for Computational Linguistics.
  * Ellis (2010) Rod Ellis. 2010.  Second language acquisition, teacher education and language pedagogy.  _Language teaching_ , 43(2):182–201.
  * Grimes and Grimes (2002) Barbara F Grimes and JE Grimes. 2002.  Ethnologue: Languages of the world. 14, h edition.  _Dallas, TX: SIL International_.
  * Haman et al. (2017) Ewa Haman, Zofia Wodniecka, Marta Marecka, Jakub Szewczyk, Marta Białecka-Pikul, Agnieszka Otwinowska, Karolina Mieszkowska, Magdalena Łuniewska, Joanna Kołak, Aneta Miękisz, Agnieszka Kacprzak, Natalia Banasik, and Małgorzata Foryś-Nogala. 2017.  How does L1 and L2 exposure impact L1 performance in bilingual children? evidence from Polish-English migrants to the united kingdom.  _Front. Psychol._ , 8:1444.
  * Hatch (1983) Evelyn Marcussen Hatch. 1983.  _Psycholinguistics: A second language perspective._ ERIC.
  * Kecskes (2008) Istvan Kecskes. 2008.  The effect of the second language on the first language.  _Babylonia_ , 2(2):31–34.
  * Kirov and Cotterell (2018) Christo Kirov and Ryan Cotterell. 2018.  Recurrent neural networks in linguistic theory: Revisiting pinker and prince (1988) and the past tense debate.  _Transactions of the Association for Computational Linguistics_ , 6:651–665.
  * Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007\.  Moses: Open source toolkit for statistical machine translation.  In _Proceedings of ACL_ , pages 177–180, Prague, Czech Republic. Association for Computational Linguistics.
  * Krashen (1977) Stephen Krashen. 1977.  Some issues relating to the monitor model.  _On Tesol_ , 77(144-158).
  * Krashen (1981) Stephen Krashen. 1981.  Second language acquisition.  _Second Language Learning_ , 3(7):19–39.
  * Krashen et al. (1979) Stephen D Krashen, Michael A Long, and Robin C Scarcella. 1979.  Age, rate and eventual attainment in second language acquisition.  _TESOL Quarterly_ , 13(4):573–582.
  * Lau et al. (2020) Jey Han Lau, Carlos Armendariz, Shalom Lappin, Matthew Purver, and Chang Shu. 2020\.  How furiously can colorless green ideas sleep? sentence acceptability in context.  _Transactions of the Association for Computational Linguistics_ , 8:296–310.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016.  Assessing the ability of LSTMs to learn syntax-sensitive dependencies.  _Transactions of the Association for Computational Linguistics_ , 4:521–535.
  * Liu et al. (2021) Zeyu Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A. Smith. 2021\.  Probing across time: What does RoBERTa know and when? In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 820–842, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Manning (2015) Christopher D. Manning. 2015.  Last words: Computational linguistics and deep learning.  _Computational Linguistics_ , 41(4):701–707.
  * McCoy et al. (2018) R Thomas McCoy, Robert Frank, and Tal Linzen. 2018.  Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks.  In _In Proceedings of CogSci_ , pages 2093–2098, Madison, WI.
  * Mueller et al. (2020) Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, Natalia Talmina, and Tal Linzen. 2020.  Cross-linguistic syntactic evaluation of word prediction models.  In _Proceedings of ACL_ , pages 5523–5539, Online. Association for Computational Linguistics.
  * Nation (2014) Paul Nation. 2014.  How much input do you need to learn the most frequent 9,000 words?  _Reading in a Foreign Language_ , 26(2):2.
  * Neubig and Mori (2010) Graham Neubig and Shinsuke Mori. 2010.  Word-based partial annotation for efficient corpus construction.  In _Proceedings of LREC_ , Valletta, Malta. European Language Resources Association (ELRA).
  * Neubig et al. (2011) Graham Neubig, Yosuke Nakata, and Shinsuke Mori. 2011.  Pointwise prediction for robust, adaptable Japanese morphological analysis.  In _Proceedings of ACL_ , pages 529–533, Portland, Oregon, USA. Association for Computational Linguistics.
  * Papadimitriou and Jurafsky (2020) Isabel Papadimitriou and Dan Jurafsky. 2020.  Learning Music Helps You Read: Using transfer to study linguistic structure in language models.  In _Proceedings of EMNLP_ , pages 6829–6839, Online. Association for Computational Linguistics.
  * Pérez-Mayos et al. (2021) Laura Pérez-Mayos, Alba Táboas García, Simon Mille, and Leo Wanner. 2021.  Assessing the syntactic capabilities of transformer-based multilingual language models.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3799–3812, Online. Association for Computational Linguistics.
  * Pinker and Prince (1988) Steven Pinker and Alan Prince. 1988.  On language and connectionism: Analysis of a parallel distributed processing model of language acquisition.  _Cognition_ , 28(1):73–193.
  * Ri and Tsuruoka (2022) Ryokan Ri and Yoshimasa Tsuruoka. 2022.  Pretraining with artificial language: Studying transferable knowledge in language models.  In _Proceedings of ACL_ , pages 7302–7315, Dublin, Ireland. Association for Computational Linguistics.
  * Rumelhart and McClelland (1986) David E Rumelhart and James L McClelland. 1986.  On learning the past tenses of english verbs.  In _Parallel distributed processing: Explorations in the microstructure of cognition_ , pages 216–271. MIT Press, Cambridge, MA.
  * Salazar et al. (2020) Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020.  Masked language model scoring.  In _Proceedings of ACL_ , pages 2699–2712, Online. Association for Computational Linguistics.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of ACL_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Settles et al. (2018) Burr Settles, Chris Brust, Erin Gustafson, Masato Hagiwara, and Nitin Madnani. 2018\.  Second language acquisition modeling.  In _Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 56–65, New Orleans, Louisiana. Association for Computational Linguistics.
  * Shi et al. (2023) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023.  Language models are multilingual chain-of-thought reasoners.  In _The Eleventh International Conference on Learning Representations_.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in opus.  In _Proceedings of LREC_ , Istanbul, Turkey. European Language Resources Association (ELRA).
  * Tyler et al. (2022) Chang Tyler, Tu Zhuowen, and Benjamin Berge. 2022.  The geometry of multilingual language model representations.  In _Proceedings of EMNLP_ , pages 119–136, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Warstadt and Bowman (2020) Alex Warstadt and Samuel R. Bowman. 2020.  Can neural networks acquire a structural bias from raw linguistic data?  In _In Proceedings of CogSci_.
  * Warstadt et al. (2020) Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020.  BLiMP: The benchmark of linguistic minimal pairs for English.  _Transactions of the Association for Computational Linguistics_ , 8:377–392.
  * Wenzek (2020) Guillaume Wenzek. 2020.  Ccnet: Extracting high quality monolingual datasets from web crawl data.  In _Proceedings of LREC_ , pages 4003–4012, Marseille, France. European Language Resources Association.

dropout | $0.1$
---|---
attention_dropout | $0.1$
accumulate_gradients | $4$
emb_dim | $256$
ffn_embed_dim | $1024$
gelu_activation | True
Optimizer | adam_inverse_sqrt
| lr=$0.00020$
| eps=$0.000001$
| warmup_updates=$30000$
| beta1=$0.9$
| beta2=$0.999$
| weight_decay=$0.01$
epoch | 100
n_heads | 8
n_layers | 12
clip_grad_norm | $1.0$
amp | $2$
fp16 | True
Table 5: Hyperparameters of the LMs Name |  Version |  License
---|---|---
fastBPE |  0.1.0 |  MIT License
kytea |  0.4.7 |  Apache 2.0
mosesdecoder |  0.4.0 |  LGPL 2.1
XLM |  0.1.0 |  CC BY-NC 4.0
BLiMP |  0.1.0 |  CC BY-NC 4.0
CC100 (CC-Net) |  1.0.0 |  MIT License
CLAMS |  0.1.0 |  Apache 2.0
Tatoeba |  v2022-03-03 |  CC–BY 2.0 FR
Table 6: The versions and licenses of used tools and datasets. These tools and
datasets used in this study were designed for the purposes of research and
language learning.

##  Appendix A Experimental Procedure

We list the hyperparameters in Table 5. The versions and licenses of used
tools and datasets are listed in Table 6.

#### L1 Acquisition:

We used mosesdecoder (Koehn et al., 2007) as French, German and Russian
tokenizer, kytea888http://www.phontron.com/kytea/(Neubig et al., 2011; Neubig
and Mori, 2010) as Japanese tokenizer and segmented words into subwords with
fastBPE999https://github.com/glample/fastBPE(Sennrich et al., 2016) The
dataset was split into train/dev/test in the ratio of 8:1:1. We set 14,000
vocabulary size for any language. We trained our models with 4 parallel GPUs
(VRAM 48G), which took 6 days per model.

#### L2 Acquisition:

We added English tokens from the parallel corpus into BPE codes and vocabulary
used in L1 Acquisition and removed duplicated tokens and vocabulary. As for
models not using a monolingual corpus, we created the codes and vocabulary
using a parallel corpus of both L1 and L2. We used mosesdecoder (Koehn et al.,
2007) as English tokenizer. As for other languages, we use tokenizers the same
as L1 Acquisition. The dataset was split into train/dev/test in the ratio of
8:1:1. As we increased the number of vocabularies in the embedding layer, the
weights/biases in the final layer were also increased. Our four LMs were
trained with different 3 seeds and reported their averages as results.
Compared models in our preliminary experiment (Sec. 3) are shown in Figure 2.
We trained our models with 2–4 GPUs (VRAM 48G), which took around 5 hours per
model.

Coarse | Specific | Challenge | First Language
---|---|---|---
Fr |  De |  Ru |  Ja
Morphology | ANA.AGR | anaphor_gender_agreement | $61.5$ | $23.7$ | $52.7$ | $57.7$
anaphor_number_agreement | $50.1$ | $62.5$ | $53.1$ | $65.4$
D-N AGR | determiner_noun_agreement_1 | $80.8$ | $74.5$ | $69.1$ | $67.9$
determiner_noun_agreement_2 | $70.1$ | $76.5$ | $62.6$ | $79.8$
determiner_noun_agreement_irregular_1 | $65.8$ | $63.0$ | $58.3$ | $54.5$
determiner_noun_agreement_irregular_2 | $64.9$ | $72.3$ | $56.5$ | $78.3$
determiner_noun_agreement_with_adj_1 | $77.9$ | $69.2$ | $59.9$ | $61.1$
determiner_noun_agreement_with_adj_2 | $62.8$ | $66.9$ | $56.1$ | $65.7$
determiner_noun_agreement_with_adj_irregular_1 | $72.7$ | $64.1$ | $54.0$ | $50.5$
determiner_noun_agreement_with_adj_irregular_2 | $61.1$ | $63.0$ | $51.9$ | $68.4$
IRREGULAR | irregular_past_participle_adjectives | $75.1$ | $61.6$ | $95.3$ | $79.1$
irregular_past_participle_verbs | $70.9$ | $77.0$ | $50.0$ | $62.0$
S-V AGR | distractor_agreement_relational_noun | $50.9$ | $65.5$ | $39.8$ | $42.6$
distractor_agreement_relative_clause | $48.6$ | $51.7$ | $46.1$ | $45.4$
irregular_plural_subject_verb_agreement_1 | $64.2$ | $68.9$ | $56.7$ | $54.4$
irregular_plural_subject_verb_agreement_2 | $65.5$ | $74.5$ | $63.6$ | $58.9$
regular_plural_subject_verb_agreement_1 | $65.9$ | $72.2$ | $60.7$ | $59.5$
regular_plural_subject_verb_agreement_2 | $67.3$ | $69.2$ | $62.5$ | $57.2$
Syntax | ARG.STR | animate_subject_passive | $65.1$ | $55.9$ | $51.1$ | $53.9$
animate_subject_trans | $44.7$ | $44.1$ | $31.7$ | $37.3$
causative | $39.6$ | $53.4$ | $35.8$ | $38.8$
drop_argument | $57.5$ | $44.6$ | $44.0$ | $54.1$
inchoative | $51.6$ | $43.5$ | $37.7$ | $45.2$
intransitive | $56.0$ | $47.1$ | $40.5$ | $52.4$
passive_1 | $61.4$ | $62.6$ | $60.9$ | $67.7$
passive_2 | $62.7$ | $70.0$ | $65.1$ | $66.2$
transitive | $59.8$ | $56.8$ | $56.1$ | $53.8$
ELLIPSIS | ellipsis_n_bar_1 | $52.1$ | $50.2$ | $46.4$ | $46.0$
ellipsis_n_bar_2 | $83.2$ | $76.8$ | $62.1$ | $64.5$
FILLER-GAP | wh_questions_object_gap | $34.7$ | $87.2$ | $36.6$ | $31.4$
wh_questions_subject_gap | $66.8$ | $89.2$ | $64.9$ | $61.8$
wh_questions_subject_gap_long_distance | $68.8$ | $95.5$ | $63.8$ | $62.4$
wh_vs_that_no_gap | $68.4$ | $97.1$ | $60.5$ | $63.2$
wh_vs_that_no_gap_long_distance | $61.3$ | $97.8$ | $56.1$ | $53.5$
wh_vs_that_with_gap | $39.8$ | $6.6$ | $40.1$ | $38.8$
wh_vs_that_with_gap_long_distance | $42.6$ | $4.1$ | $45.2$ | $47.7$
ISLAND | adjunct_island | $49.0$ | $52.5$ | $50.0$ | $56.8$
complex_NP_island | $47.7$ | $43,0$ | $43.9$ | $53.9$
coordinate_structure_constraint_complex_left_branch | $33.7$ | $40.2$ | $41.1$ | $39.0$
coordinate_structure_constraint_object_extraction | $68.8$ | $78.9$ | $57.7$ | $53.9$
left_branch_island_echo_question | $36.6$ | $37.2$ | $27.1$ | $37.2$
left_branch_island_simple_question | $63.1$ | $71.7$ | $68.0$ | $69.1$
sentential_subject_island | $50.5$ | $50.1$ | $52.3$ | $55.4$
wh_island | $68.0$ | $8.1$ | $54.5$ | $66.6$
Semantics | NPI | matrix_question_npi_licensor_present | $46.2$ | $46.0$ | $14.3$ | $30.9$
npi_present_1 | $52.1$ | $94.5$ | $41.3$ | $72.3$
npi_present_2 | $56.4$ | $94.8$ | $39.5$ | $72.5$
only_npi_licensor_present | $1.0$ | $0.2$ | $0.1$ | $0.2$
only_npi_scope | $35.9$ | $55.2$ | $50.4$ | $52.1$
sentential_negation_npi_licensor_present | $32.1$ | $34.2$ | $39.6$ | $11.8$
sentential_negation_npi_scope | $59.6$ | $57.6$ | $44.6$ | $47.1$
QUANTIFIERS | existential_there_quantifiers_1 | $85.2$ | $84.0$ | $69.6$ | $62.3$
existential_there_quantifiers_2 | $7.5$ | $61.2$ | $4.2$ | $2.8$
superlative_quantifiers_1 | $45.2$ | $92.0$ | $60.6$ | $55.4$
superlative_quantifiers_2 | $87.9$ | $84.9$ | $90.3$ | $82.0$
Syntax & Semantics | BINDING | principle_A_c_command | $61.4$ | $49.6$ | $69.8$ | $53.5$
principle_A_case_1 | $52.0$ | $99.8$ | $11.4$ | $99.9$
principle_A_case_2 | $62.0$ | $58.7$ | $54.8$ | $49.6$
principle_A_domain_1 | $41.3$ | $93.7$ | $0.7$ | $83.9$
principle_A_domain_2 | $51.3$ | $62.9$ | $49.9$ | $45.2$
principle_A_domain_3 | $52.1$ | $49.4$ | $48.5$ | $48.7$
principle_A_reconstruction | $42.4$ | $42.2$ | $49.6$ | $46.3$
CTRL. RAIS | existential_there_object_raising | $68.2$ | $53.8$ | $68.5$ | $69.9$
existential_there_subject_raising | $55.1$ | $53.3$ | $69.7$ | $51.1$
expletive_it_object_raising | $70.2$ | $54.7$ | $67.9$ | $69.1$
tough_vs_raising_1 | $56.4$ | $44.6$ | $27.2$ | $62.3$
tough_vs_raising_2 | $43.2$ | $54.8$ | $73.8$ | $36.8$
Table 7: Results for each fine-grained test suit in BLiMP. Model | L1 | Epoch |  long_vp_coord |  obj_rel_within_anim |  simple_agrmt |  vp_coord |  obj_rel_across_anim |  prep_anim |  subj_rel
---|---|---|---|---|---|---|---|---|---
TLM-drop | Fr | 5 | $90.9$ | $75.7$ | $78.2$ | $81.4$ | $85.1$ | $77.3$ | $78.5$
50 | $75.7$ | $66.4$ | $71.2$ | $72.3$ | $69.2$ | $69.1$ | $70.7$
100 | $67.3$ | $63.4$ | $64.6$ | $66.5$ | $65.2$ | $64.9$ | $64.9$
De | 5 | $98.7$ | $83.6$ | $88.5$ | $87.3$ | $89.8$ | $86.4$ | $87.6$
50 | $85.9$ | $77.4$ | $82.9$ | $81.3$ | $81.3$ | $81.0$ | $81.4$
100 | $67.9$ | $65.6$ | $72.8$ | $71.0$ | $73.4$ | $70.6$ | $71.4$
Ru | 5 | $82.1$ | $77.6$ | $75.0$ | $69.4$ | $59.0$ | $73.5$ | $69.6$
50 | $74.1$ | $77.0$ | $74.7$ | $74.6$ | $73.5$ | $74.1$ | $75.3$
100 | $64.2$ | $75.1$ | $74.7$ | $77.9$ | $75.9$ | $74.3$ | $76.6$
L1-only | Fr | 5 | $92.3$ | $79.3$ | $72.0$ | $75.6$ | $76.3$ | $77.3$ | $80.1$
50 | $97.2$ | $82.3$ | $74.9$ | $77.6$ | $78.3$ | $77.3$ | $80.0$
100 | $94.5$ | $80.1$ | $72.4$ | $77.1$ | $77.3$ | $76.7$ | $80.9$
De | 5 | $54.7$ | $60.1$ | $61.1$ | $71.6$ | $71.3$ | $74.0$ | $72.1$
50 | $56.8$ | $58.2$ | $55.7$ | $67.2$ | $67.9$ | $71.4$ | $70.5$
100 | $53.8$ | $57.4$ | $56.4$ | $68.0$ | $66.3$ | $68.9$ | $66.0$
Ru | 5 | $85.7$ | $77.9$ | $87.5$ | $85.6$ | $86.9$ | $84.9$ | $85.5$
50 | $90.8$ | $83.2$ | $88.6$ | $87.7$ | $88.1$ | $89.3$ | $91.8$
100 | $92.1$ | $72.3$ | $81.8$ | $82.9$ | $82.2$ | $82.7$ | $86.4$
Table 8: Results for each fine-grained test suit in CLAMS.