Total: | $15.2^{+0.5}_{-0.3}$ | $12.4^{+0.4}_{-0.8}$ | $12.9^{+0.1}_{-0.1}$ | $14.0^{+0.5}_{-1.4}$ | $14.3^{+0.3}_{-1.2}$ | $<13.9$ | $12.9^{+0.3}_{-1.0}$ | $<13.5$ | $13.2^{+0.3}_{-0.8}$ | N/C | $14.1^{+0.5}_{-1.5}$ | $13.7^{+0.3}_{-0.9}$ | $<11.0$ | $<13.2$
J1338$+$0311, $z_{\rm em,corr}=0.02604$
0.02543 | -178 | $15^{+13}_{-5}$ | N/A | $12.9^{+0.6}_{-1.6}$ | $13.0^{+1.1}_{-1.7}$ | N/C | $10.1^{+2.4}_{-0.1}$ | N/C | $14.1^{+0.4}_{-0.9}$ | $13.3^{+1.6}_{-2.4}$ | $13.2^{+0.5}_{-1.4}$ | $<15.0$ | N/C | $13.2^{+0.7}_{-0.7}$ | $14.6^{+0.6}_{-1.6}$ | N/A
0.02566 | -110 | $80^{+30}_{-20}$ | N/A | $13.4^{+0.2}_{-0.2}$ | $13.6^{+0.3}_{-0.5}$ | N/C | $10.1^{+2.5}_{-0.1}$ | N/C | $13.7^{+0.6}_{-1.6}$ | $10.1^{+4.6}_{-0.1}$ | $10.3^{+1.1}_{-0.3}$ | $15.0^{+0.2}_{-0.3}$ | N/C | $10.5^{+2.0}_{-0.5}$ | N/A | N/A
0.02603 | 2 | $77^{+18}_{-13}$ | $20.2^{+0.1}_{-0.1}$ | $13.6^{+0.2}_{-0.3}$ | $13.5^{+0.3}_{-0.3}$ | N/C | $15.2^{+0.1}_{-0.1}$ | N/C | $14.2^{+0.2}_{-0.6}$ | $14.9^{+0.2}_{-0.2}$ | $11.5^{+0.8}_{-1.5}$ | $15.0^{+0.2}_{-0.2}$ | N/C | $14.6^{+0.4}_{-1.0}$ | $<14.6$ | N/A
0.02639 | 146 | $50^{+10}_{-13}$ | N/A | $12.7^{+0.4}_{-0.3}$ | $12.6^{+0.3}_{-2.6}$ | N/C | $14.3^{+0.3}_{-2.5}$ | N/C | $14.2^{+0.2}_{-0.3}$ | $13.9^{+1.0}_{-2.8}$ | $<12$ | $14.2^{+0.4}_{-0.8}$ | N/C | $<14.0$ | $<14$ | N/A
Total: | $20.2^{+0.1}_{-0.1}$ | $13.9^{+0.2}_{-0.2}$ | $13.9^{+0.4}_{-0.2}$ | N/C | $15.2^{+0.1}_{-0.2}$ | N/C | $<14.3$ | $<16$ | $<13.8$ | $15.5^{+0.1}_{-0.1}$ | N/C | $<14.6$ | $<14.3$ | N/A
J2106$+$0909, $z_{\rm em,corr}=0.0437$
0.04423 | 135 | $30^{+10}_{-10}$ | $13.7^{+0.2}_{-0.2}$ | $<13.0$ | $<13.2$ | N/C | $<14.8$ | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A
J2130$-$0025, $z_{\rm em,corr}=0.0200$
0.01967 | -146 | $26^{+6}_{-4}$ | $18.8^{+0.1}_{-0.1}$ | $13.8^{+0.1}_{-0.2}$ | $14.6^{+0.9}_{-0.5}$ | $15.3^{+0.9}_{-0.5}$ | $<14.6$ | N/C | $13.1^{+0.3}_{-1.1}$ | $<14.7$ | $<13.7$ | $14.6^{+0.9}_{-0.5}$ | N/C | $13.6^{+0.4}_{-1.9}$ | $<11$ | N/A

Table 6: Fit results to cloudy photo-ionization model of absorption systems Quasar | $z_{\rm abs}$ | v | $\log N_{\rm HI}$ | $\log N_{\rm H_{tot}}$ | [X/H] | $\log q$ | $\log I_{\rm UV}/n_{\rm H}$ | $F_{*}$
---|---|---|---|---|---|---|---|---
|  | km s-1 | cm-2 | cm-2 |  |  | Drain/cm-3 |
J0755+3911 | Average: | $13.5^{+0.3}_{-0.3}$ | $16.5^{+0.3}_{-0.3}$ | $1.2^{+0.2}_{-0.2}$ | $-1.5^{+0.3}_{-0.3}$ | $2.0^{+0.3}_{-0.2}$ | $0.0^{+0.4}_{-0.0}$
J0758$+$4219 | 0.03138 | -115 | $14.2^{+0.3}_{-0.2}$ | $16.9^{+1.0}_{-0.7}$ | $0.7^{+0.6}_{-0.8}$ | $-2.4^{+1.0}_{-0.4}$ | $1.0^{+1.0}_{-0.5}$ | $<1$
| 0.03202 | 52 | $14.2^{+0.3}_{-0.1}$ | $17.8^{+0.2}_{-0.2}$ | $0.6^{+0.3}_{-0.3}$ | $-2.0^{+0.3}_{-0.3}$ | $1.4^{+0.3}_{-0.3}$ | $0.0^{+0.4}_{-0.0}$
| 0.03226 | 130 | $15.0^{+0.4}_{-0.3}$ | $17.6^{+0.3}_{-0.3}$ | $0.9^{+0.4}_{-0.4}$ | $-2.1^{+0.6}_{-0.6}$ | $1.4^{+0.4}_{-0.6}$ | $0.2^{+0.4}_{-0.2}$
| 0.03261 | 230 | N/A | N/A | N/A | N/A | N/A | N/A
| Average: | $15.3^{+0.3}_{-0.2}$ | $18.1^{+0.3}_{-0.4}$ | $0.8^{+0.3}_{-0.3}$ | $-2.0^{+0.3}_{-0.4}$ | $1.4^{+0.3}_{-0.4}$ | $0.1^{+0.5}_{-0.1}$
J0838$+$2453 | Average(A): | $13.2^{+0.2}_{-0.3}$ | $16.7^{+0.3}_{-0.3}$ | $1.9^{+0.3}_{-0.3}$ | $-0.8^{+0.2}_{-0.5}$ | $2.7^{+0.3}_{-0.5}$ | $0.0^{+0.1}_{-0.0}$
| Average(B): | $13.8^{+0.2}_{-0.3}$ | $17.1^{+1.5}_{-0.3}$ | $0.5^{+1.5}_{-0.5}$ | $-1.1^{+0.5}_{-0.7}$ | $2.4^{+0.5}_{-0.7}$ | $<1$
J0950+4309 | 0.01713 | -2 | $17.6^{+0.3}_{-0.7}$ | $18.9^{+0.3}_{-0.5}$ | $-1.1^{+0.5}_{-0.3}$ | $-3.8^{+0.3}_{-0.3}$ | $-0.5^{+0.3}_{-0.2}$ | $0.0^{+0.2}_{-0.0}$
| 0.01696 | -52 | $14.9^{+0.3}_{-0.3}$ | $19.4^{+0.4}_{-0.5}$ | $-0.5^{+0.4}_{-0.5}$ | $-1.2^{+0.4}_{-0.3}$ | $2.3^{+0.3}_{-0.4}$ | $0.0^{+0.3}_{-0.0}$
| 0.01630 | -247 | N/A | N/A | N/A | N/A | N/A | N/A
| Average: | $16.8^{+0.8}_{-0.3}$ | $19.0^{+0.6}_{-0.2}$ | $-0.6^{+0.2}_{-0.7}$ | $-3.2^{+0.2}_{-0.2}$ | $-0.1^{+0.4}_{-0.2}$ | $0.0^{+0.2}_{-0.0}$
J1237$+$4447 | 0.05960 | -151 | $14.9^{+0.6}_{-0.3}$ | $18.2^{+1.3}_{-0.6}$ | $-0.2^{+0.7}_{-1.1}$ | $-2.0^{+0.9}_{-0.5}$ | $1.5^{+0.6}_{-0.8}$ | $<1$
| 0.05993(A) | -57 | $17.0^{+0.4}_{-0.3}$ | $19.3^{+0.8}_{-0.9}$ | $-1.6^{+1.0}_{-1.0}$ | $-3.1^{+0.6}_{-0.5}$ | $0.0^{+0.9}_{-0.3}$ | $0.0^{+0.2}_{-0.0}$
| 0.05993(B) | -57 | $14.8^{+0.7}_{-0.3}$ | $18.1^{+1.4}_{-0.7}$ | $-0.7^{+0.6}_{-1.4}$ | $-2.6^{+1.1}_{-0.5}$ | $0.9^{+1.2}_{-0.5}$ | $0.0^{+0.2}_{-0.0}$
| 0.06119 | 302 | N/A | N/A | N/A | N/A | N/A | N/A
| Average(A): | $17.2^{+0.2}_{-0.7}$ | $19.4^{+0.8}_{-0.5}$ | $-1.8^{+0.8}_{-0.8}$ | $-2.9^{+0.9}_{-0.6}$ | $0.3^{+0.9}_{-0.6}$ | $0.2^{+0.0}_{-0.2}$
| Average(B): | $15.2^{+0.5}_{-0.3}$ | $18.1^{+0.9}_{-0.5}$ | $-0.2^{+0.5}_{-1.0}$ | $-2.5^{+0.6}_{-0.4}$ | $0.9^{+0.6}_{-0.5}$ | $0.0^{+0.2}_{-0.0}$
J1338$+$0311 | Average: |  | $20.2^{+0.1}_{-0.1}$ | $20.5^{+0.2}_{-0.2}$ | $-0.4^{+0.4}_{-0.3}$ | $-2.8^{+0.3}_{-0.2}$ |  | $0.5^{+0.3}_{-0.3}$
J2106$+$0909 | 0.04423 | 135 | N/A | N/A | N/A | N/A | N/A | N/A
J2130$-$0025 | 0.01967 | -110 | $18.8^{+0.1}_{-0.1}$ | $21.1^{+0.4}_{-0.6}$ | $-1.1^{+0.2}_{-0.2}$ | $-2.1^{+0.4}_{-0.5}$ | $1.4^{+0.4}_{-0.6}$ | $0.1^{+0.3}_{-0.1}$

Figure 17: The posterior probability density function (PDF) of fitting
parameters of H i and Si iii lines in four HST COS spectra shown in Fig. 3.
The first three columns from left to right show the PDF of $b$-parameter,
$\log N({\rm HI})$, $\log N({\rm SiIII})$. The color of curves correspond to
the color of velocity components in Fig. 3. Dashed area represents the 68%
confidence interval around the value with the maximum probability. For the
case of J1237$+$4447 the light and heavy dashed areas show two different
solutions (see text). The rightmost column shows the PDF of the total H i
column density and the total Si iii column density (summed over the
components) by red and black curves, respectively.

Figure 18: The comparison of radial velocity, metallicity and ionization
parameter derived from the fitting to MaNGA maps of the galaxy 1-166736 and
ones measured in the absorption system towards the quasar J0950$+$4309\. A
detailed description of panels is presented in the Appendix B.

Figure 19: The comparison of radial velocity, metallicity and ionization
parameter derived from the fitting to MaNGA maps of the galaxy 1-180522 and
ones measured in the absorption system towards the quasar J2130$-$0025\. The
detailed description is presented in the Appendix B.

Figure 20: The comparison of radial velocity, metallicity and ionization
parameter derived from the fitting to MaNGA maps of the galaxy 1-635629 and
ones measured in the absorption system towards the quasar J2130$-$0025\. The
detailed description is presented in the Appendix B.

Figure 21: The comparison of radial velocity, metallicity and ionization
parameter derived from the fitting to MaNGA maps of the galaxy 1-44487 and
ones measured in the absorption system towards the quasar J0758$+$4219\. The
detailed description of panels is presented in the Appendix B.

Figure 22: The comparison of radial velocity derived from the fitting to MaNGA
maps of the galaxy 1-575668 and ones measured in the absorption system towards
the quasar J1237$+$4447 The detailed description of panels is presented in the
Appendix B.

Figure 23: The comparison of radial velocity derived from the fitting to MaNGA
maps of the galaxy 1-113242 and ones measured in the absorption system towards
the quasar J2106$+$0909 The detailed description of panels is presented in the
Appendix B.

Figure 24: The comparison of radial velocity, metallicity and ionization
parameter derived from the fitting to MaNGA maps of the galaxy 1-71974 and
ones measured in the absorption system towards the AGN J0755$+$3911\. The
detailed description of panels is presented in the Appendix B.

Figure 25: The comparison of radial velocity derived from the fitting to MaNGA
maps of the galaxy 1-385099 and ones measured in the absorption system towards
the quasar J0838$+$2453 The detailed description of panels is presented in the
Appendix B.

Figure 26: The comparison of radial velocity, metallicity and ionization
parameter derived from the fitting to MaNGA maps of the galaxy 12-192116 and
ones measured in the absorption system towards the AGN J1338$+$2620\. The
detailed description of panels is presented in the Appendix B.  Figure 27: Fit
to absorption lines at $z_{\rm abs}=0.01714$ in the spectrum of J0950$+$4309\.
The synthetic profile is shown in red and the contribution from each
component, associated with the studied galaxy, is shown in green, blue and
purple (and orange). Dashed vertical lines represent the position of each
component. Vertical sticks indicate the position of other absorption lines,
associated with the MW (magenta sticks) and remote galaxies. Figure 28: Fit to
absorption lines at $z_{\rm abs}=0.06013$ in the spectrum of J1237$+$4447\.
Lines are as in Fig. 27.  Figure 29: Fit to absorption lines at $z_{\rm
abs}=0.01966$ in the spectrum of J2130$-$0025\. Lines are as in Fig. 27.
Figure 30: Fit to absorption lines at $z_{\rm abs}=0.03181$ in the spectrum of
J0758$+$4219\. Lines are as in Fig. 27. Figure 31: Fit to absorption lines at
$z_{\rm abs}=0.02604$ in the spectrum of J1338$+$2620\. Lines are as in Fig.
27.  Figure 32: Fit to absorption lines at $z_{\rm abs}=0.03362$ in the
spectrum of J0755$+$3911\. Lines are as in Fig. 27.  Figure 33: Fit to
absorption lines at $z_{\rm abs}=0.02843$ in the spectrum of J0838$+$2453\.
Lines are the same as in Fig. 27.  Figure 34: Fit to absorption lines at
$z_{\rm abs}=0.04375$ in the spectrum of J2106$+$0909\. Lines are as in Fig.
27.

Figure 35: The cases of non-detections. The absorption lines of H i, Si iii,
Si ii, N v in the HST COS spectra of J1653$+$3945, J1709$+$3421 and
J1629$+$4007 at the redshifts of the corresponding galaxies (1-594755,
1-561034, 1-564490, respectively). The synthetic profile is shown in red. The
dashed vertical line represents the likely the position of absorption system.
Vertical sticks indicate the position of absorption lines, associated with the
Milky Way (MW, magenta sticks) and remote galaxies.

Figure 36: The fit results to measured species column densities in our sample
with cloudy simulation. Dashed vertical lines and light shaded areas represent
the most probable value and its uncertainty. The light shaded area without the
line represent upper limit. The histograms show distributions of species
column densities in cloudy simulation.

## References

  * Asplund et al. (2009) Asplund, M., Grevesse, N., Sauval, A. J., et al. 2009, ARA&A, 47, 481, doi: 10.1146/annurev.astro.46.060407.145222
  * Augustin et al. (2018) Augustin, R., Péroux, C., Møller, P., et al. 2018, MNRAS, 478, 3120, doi: 10.1093/mnras/sty1287
  * Balashev et al. (2017) Balashev, S. A., Noterdaeme, P., Rahmani, H., et al. 2017, MNRAS, 470, 2890, doi: 10.1093/mnras/stx1339
  * Balashev et al. (2019) Balashev, S. A., Klimenko, V. V., Noterdaeme, P., et al. 2019, MNRAS, 490, 2668, doi: 10.1093/mnras/stz2707
  * Bekiaris et al. (2016) Bekiaris, G., Glazebrook, K., Fluke, C. J., & Abraham, R. 2016, MNRAS, 455, 754, doi: 10.1093/mnras/stv2292
  * Bizyaev et al. (2017) Bizyaev, D., Walterbos, R. A. M., Yoachim, P., et al. 2017, ApJ, 839, 87, doi: 10.3847/1538-4357/aa6979
  * Blanc et al. (2015) Blanc, G. A., Kewley, L., Vogt, F. P. A., & Dopita, M. A. 2015, ApJ, 798, 99, doi: 10.1088/0004-637X/798/2/99
  * Blanton et al. (2017) Blanton, M. R., Bershady, M. A., Abolfathi, B., et al. 2017, AJ, 154, 28, doi: 10.3847/1538-3881/aa7567
  * Bouché et al. (2007) Bouché, N., Cresci, G., Davies, R., et al. 2007, ApJ, 671, 303, doi: 10.1086/522221
  * Bouché et al. (2012) Bouché, N., Murphy, M. T., Péroux, C., et al. 2012, MNRAS, 419, 2, doi: 10.1111/j.1365-2966.2011.19500.x
  * Bundy et al. (2015) Bundy, K., Bershady, M. A., Law, D. R., et al. 2015, ApJ, 798, 7, doi: 10.1088/0004-637X/798/1/7
  * Cherinka et al. (2019) Cherinka, B., Andrews, B. H., Sánchez-Gallego, J., et al. 2019, AJ, 158, 74, doi: 10.3847/1538-3881/ab2634
  * Dopita et al. (2013) Dopita, M. A., Sutherland, R. S., Nicholls, D. C., Kewley, L. J., & Vogt, F. P. A. 2013, ApJS, 208, 10, doi: 10.1088/0067-0049/208/1/10
  * Draine (1978) Draine, B. T. 1978, ApJS, 36, 595, doi: 10.1086/190513
  * Drory et al. (2015) Drory, N., MacDonald, N., Bershady, M. A., et al. 2015, AJ, 149, 77, doi: 10.1088/0004-6256/149/2/77
  * Foreman-Mackey et al. (2013) Foreman-Mackey, D., Hogg, D. W., Lang, D., & Goodman, J. 2013, PASP, 125, 306, doi: 10.1086/670067
  * Fumagalli et al. (2016) Fumagalli, M., Cantalupo, S., Dekel, A., et al. 2016, MNRAS, 462, 1978, doi: 10.1093/mnras/stw1782
  * Gehrels (1986) Gehrels, N. 1986, ApJ, 303, 336, doi: 10.1086/164079
  * Girelli et al. (2020) Girelli, G., Pozzetti, L., Bolzonella, M., et al. 2020, A&A, 634, A135, doi: 10.1051/0004-6361/201936329
  * Grand et al. (2017) Grand, R. J. J., Gómez, F. A., Marinacci, F., et al. 2017, MNRAS, 467, 179, doi: 10.1093/mnras/stx071
  * Gunn et al. (2006) Gunn, J. E., Siegmund, W. A., Mannery, E. J., et al. 2006, AJ, 131, 2332, doi: 10.1086/500975
  * Hsu et al. (2011) Hsu, W. H., Putman, M. E., Heitsch, F., et al. 2011, AJ, 141, 57, doi: 10.1088/0004-6256/141/2/57
  * Jenkins (2009) Jenkins, E. B. 2009, ApJ, 700, 166, doi: 10.1088/0004-637X/700/2/1299
  * Johnson et al. (2021) Johnson, C., Plesha, R., Jedrzejewski, R., Frazer, E., & Dashtamirova, D. 2021, 21.  https://www.stsci.edu/files/live/sites/www/files/home/hst/instrumentation/cos/documentation/instrument-science-reports-isrs/_documents/COS_ISR_2021_03.pdf
  * Jones & Nair (2019) Jones, A. M., & Nair, P. 2019, in American Astronomical Society Meeting Abstracts, Vol. 233, American Astronomical Society Meeting Abstracts #233, 407.01
  * Karki et al. (2023) Karki, A., Kulkarni, V. P., Weng, S., et al. 2023, MNRAS, 524, 5524, doi: 10.1093/mnras/stad2134
  * Kauffmann et al. (2003) Kauffmann, G., Heckman, T. M., Tremonti, C., et al. 2003, MNRAS, 346, 1055, doi: 10.1111/j.1365-2966.2003.07154.x
  * Kewley et al. (2001) Kewley, L. J., Dopita, M. A., Sutherland, R. S., Heisler, C. A., & Trevena, J. 2001, ApJ, 556, 121, doi: 10.1086/321545
  * Khaire & Srianand (2019) Khaire, V., & Srianand, R. 2019, MNRAS, 484, 4174, doi: 10.1093/mnras/stz174
  * Kulkarni et al. (2022) Kulkarni, V. P., Bowen, D. V., Straka, L. A., et al. 2022, ApJ, 929, 150, doi: 10.3847/1538-4357/ac5fab
  * Kulkarni et al. (2010) Kulkarni, V. P., Khare, P., Som, D., et al. 2010, New A, 15, 735, doi: 10.1016/j.newast.2010.05.006
  * Law et al. (2016) Law, D. R., Cherinka, B., Yan, R., et al. 2016, AJ, 152, 83, doi: 10.3847/0004-6256/152/4/83
  * Lehner et al. (2018) Lehner, N., Wotta, C. B., Howk, J. C., et al. 2018, ApJ, 866, 33, doi: 10.3847/1538-4357/aadd03
  * Lofthouse et al. (2020) Lofthouse, E. K., Fumagalli, M., Fossati, M., et al. 2020, MNRAS, 491, 2057, doi: 10.1093/mnras/stz3066
  * Lofthouse et al. (2023) —. 2023, MNRAS, 518, 305, doi: 10.1093/mnras/stac3089
  * Martin et al. (2012) Martin, C. L., Shapley, A. E., Coil, A. L., et al. 2012, ApJ, 760, 127, doi: 10.1088/0004-637X/760/2/127
  * Masters et al. (2019) Masters, K. L., Stark, D. V., Pace, Z. J., et al. 2019, MNRAS, 488, 3396, doi: 10.1093/mnras/stz1889
  * Mingozzi et al. (2020) Mingozzi, M., Belfiore, F., Cresci, G., et al. 2020, A&A, 636, A42, doi: 10.1051/0004-6361/201937203
  * Morton (2004) Morton, D. C. 2004, ApJS, 151, 403, doi: 10.1086/382157
  * Mowla et al. (2019) Mowla, L. A., van Dokkum, P., Brammer, G. B., et al. 2019, ApJ, 880, 57, doi: 10.3847/1538-4357/ab290a
  * Muzahid et al. (2018) Muzahid, S., Fonseca, G., Roberts, A., et al. 2018, MNRAS, 476, 4965, doi: 10.1093/mnras/sty529
  * Navarro et al. (1997) Navarro, J. F., Frenk, C. S., & White, S. D. M. 1997, ApJ, 490, 493, doi: 10.1086/304888
  * Nelson et al. (2020) Nelson, D., Sharma, P., Pillepich, A., et al. 2020, MNRAS, 498, 2391, doi: 10.1093/mnras/staa2419
  * Noterdaeme et al. (2021) Noterdaeme, P., Balashev, S., Krogager, J. K., et al. 2021, A&A, 646, A108, doi: 10.1051/0004-6361/202038877
  * Pérez-Ràfols et al. (2015) Pérez-Ràfols, I., Miralda-Escudé, J., Lundgren, B., et al. 2015, MNRAS, 447, 2784, doi: 10.1093/mnras/stu2645
  * Péroux et al. (2011) Péroux, C., Bouché, N., Kulkarni, V. P., York, D. G., & Vladilo, G. 2011, MNRAS, 410, 2237, doi: 10.1111/j.1365-2966.2010.17598.x
  * Péroux et al. (2012) —. 2012, MNRAS, 419, 3060, doi: 10.1111/j.1365-2966.2011.19947.x
  * Péroux et al. (2014) Péroux, C., Kulkarni, V. P., & York, D. G. 2014, MNRAS, 437, 3144, doi: 10.1093/mnras/stt2084
  * Péroux et al. (2020) Péroux, C., Nelson, D., van de Voort, F., et al. 2020, MNRAS, 499, 2462, doi: 10.1093/mnras/staa2888
  * Péroux et al. (2016) Péroux, C., Quiret, S., Rahmani, H., et al. 2016, MNRAS, 457, 903, doi: 10.1093/mnras/stw016
  * Péroux et al. (2019) Péroux, C., Zwaan, M. A., Klitsch, A., et al. 2019, MNRAS, 485, 1595, doi: 10.1093/mnras/stz202
  * Péroux et al. (2022) Péroux, C., Weng, S., Karki, A., et al. 2022, MNRAS, 516, 5618, doi: 10.1093/mnras/stac2546
  * Putman et al. (2003) Putman, M. E., Staveley-Smith, L., Freeman, K. C., Gibson, B. K., & Barnes, D. G. 2003, ApJ, 586, 170, doi: 10.1086/344477
  * Putman et al. (2002) Putman, M. E., de Heij, V., Staveley-Smith, L., et al. 2002, AJ, 123, 873, doi: 10.1086/338088
  * Salem et al. (2016) Salem, M., Bryan, G. L., & Corlies, L. 2016, MNRAS, 456, 582, doi: 10.1093/mnras/stv2641
  * Schroetter et al. (2016) Schroetter, I., Bouché, N., Wendt, M., et al. 2016, ApJ, 833, 39, doi: 10.3847/1538-4357/833/1/39
  * Schroetter et al. (2019) Schroetter, I., Bouché, N. F., Zabl, J., et al. 2019, MNRAS, 490, 4368, doi: 10.1093/mnras/stz2822
  * Smee et al. (2013) Smee, S. A., Gunn, J. E., Uomoto, A., et al. 2013, AJ, 146, 32, doi: 10.1088/0004-6256/146/2/32
  * Stark et al. (2021) Stark, D. V., Masters, K. L., Avila-Reese, V., et al. 2021, MNRAS, 503, 1345, doi: 10.1093/mnras/stab566
  * Tumlinson et al. (2013) Tumlinson, J., Thom, C., Werk, J. K., et al. 2013, ApJ, 777, 59, doi: 10.1088/0004-637X/777/1/59
  * van de Voort et al. (2021) van de Voort, F., Bieri, R., Pakmor, R., et al. 2021, MNRAS, 501, 4888, doi: 10.1093/mnras/staa3938
  * van de Voort et al. (2019) van de Voort, F., Springel, V., Mandelker, N., van den Bosch, F. C., & Pakmor, R. 2019, MNRAS, 482, L85, doi: 10.1093/mnrasl/sly190
  * Wake et al. (2017) Wake, D. A., Bundy, K., Diamond-Stanic, A. M., et al. 2017, AJ, 154, 86, doi: 10.3847/1538-3881/aa7ecc
  * Welty et al. (1999) Welty, D. E., Frisch, P. C., Sonneborn, G., & York, D. G. 1999, ApJ, 512, 636, doi: 10.1086/306795
  * Wendt et al. (2021) Wendt, M., Bouché, N. F., Zabl, J., Schroetter, I., & Muzahid, S. 2021, MNRAS, 502, 3733, doi: 10.1093/mnras/stab049
  * Weng et al. (2023) Weng, S., Péroux, C., Karki, A., et al. 2023, MNRAS, 519, 931, doi: 10.1093/mnras/stac3497
  * Werk et al. (2013) Werk, J. K., Prochaska, J. X., Thom, C., et al. 2013, ApJS, 204, 17, doi: 10.1088/0067-0049/204/2/17
  * Werk et al. (2014) Werk, J. K., Prochaska, J. X., Tumlinson, J., et al. 2014, ApJ, 792, 8, doi: 10.1088/0004-637X/792/1/8
  * Westfall et al. (2019) Westfall, K. B., Cappellari, M., Bershady, M. A., et al. 2019, AJ, 158, 231, doi: 10.3847/1538-3881/ab44a2
  * Willick et al. (1997) Willick, J. A., Courteau, S., Faber, S. M., et al. 1997, ApJS, 109, 333, doi: 10.1086/312983


# Language acquisition: do children and language models follow similar
learning stages?

Linnea Evanson
Meta AI Paris;
Laboratoire des systèmes perceptifs
École normale supérieure
PSL University
<EMAIL_ADDRESS>
&Yair Lakretz
Cognitive Neuroimaging Unit
CEA, INSERM
Université Paris-Saclay
NeuroSpin Center
<EMAIL_ADDRESS>
&Jean-Rémi King11footnotemark: 1
Meta AI Paris;
Laboratoire des systèmes perceptifs
École normale supérieure
PSL University
<EMAIL_ADDRESS>
Equal Contribution

###### Abstract

During language acquisition, children follow a typical sequence of learning
stages, whereby they first learn to categorize phonemes before they develop
their lexicon and eventually master increasingly complex syntactic structures.
However, the computational principles that lead to this learning trajectory
remain largely unknown. To investigate this, we here compare the learning
trajectories of deep language models to those of children. Specifically, we
test whether, during its training, GPT-2 exhibits stages of language
acquisition comparable to those observed in children aged between 18 months
and 6 years. For this, we train 48 GPT-2 models from scratch and evaluate
their syntactic and semantic abilities at each training step, using 96 probes
curated from the BLiMP, Zorro and BIG-Bench benchmarks. We then compare these
evaluations with the behavior of 54 children during language production. Our
analyses reveal three main findings. First, similarly to children, the
language models tend to learn linguistic skills in a systematic order. Second,
this learning scheme is parallel: the language tasks that are learned last
improve from the very first training steps. Third, some – but not all –
learning stages are shared between children and these language models.
Overall, these results shed new light on the principles of language
acquisition, and highlight important divergences in how humans and modern
algorithms learn to process natural language.

##  1 Introduction

Language acquisition is marked by a series of successive stages (Dupoux, 2018;
Kuhl, 2004; Werker, 2018). Within their first year of existence, humans
infants successively acquire prosody contours (Mehler et al., 1988), phonetic
categories (Werker and Tees, 1984; Kuhl et al., 1992; Mazuka et al., 2011) and
frequent words (Tincoff and Jusczyk, 1999; Bergelson and Swingley, 2012). They
then learn to produce basic syntactic structures (_e.g._ “The boy sang” or
“The boy fell”), questions (_e.g._ “What sound does a cow make?”) and nested
syntactic structures (_e.g._ “The boy that I saw sang”), at approximately 12,
30, and 42 months, respectively (Friedmann et al., 2021). Even though some
children may take slightly longer to learn than others, there is a set order
in which children acquire various syntactic structures Friedmann and Reznick
(2021).

Our understanding of the entire learning trajectory of children remains very
coarse, however. This partly stems from the difficulty of measuring linguistic
skills in young children. In babies, experimenters typically measure eye gaze
and sucking rate while children process linguistic stimuli, as these reflexive
behaviors are known to increase during surprising events. Such “implicit”
approaches have successfully been used to assess whether non-speaking infants
detect linguistic violations (Zamuner, 2006), distinguish lexical from
grammatical words (Shi et al., 1999) or discriminate their native language
from a foreign language Mehler et al. (1988); Kuhl et al. (2006); Nazzi et al.
(2000). In older children, linguistic skills can also be more explicitly
measured from spontaneous speech and sentence repetition. For example, a
recent study by Friedmann et al. (2021), to which we compare our work in this
paper, quantified the extent to which 18 month to 6 year-old children produce
variably complex syntactic structures. For both of these approaches, however,
the measures from children at such early ages can be noisy and fragmented.

Interestingly, these issues do not apply to modern language models. Deep
learning architectures trained to predict words from their proximal contexts
have proved immensely effective at learning to process natural language
(Radford et al., 2019; Devlin et al., 2019). Unlike humans, these algorithms
can be easily probed during training, at any time point and rate, and with
unlimited number of test stimuli, without interfering with their language
acquisition (Jawahar et al., 2019; Manning et al., 2020; Bowman and Dahl,
2021). Furthermore, high-performing deep nets have been shown to implicitly
Lakretz et al. (2019); Gulordava et al. (2018) or explicitly learn to
represent and use syntactic structures (Manning et al., 2020), as well as to
use features such as concreteness and lexical class to learn language Chang
and Bergen (2022). Finally, and importantly, these deep neural networks have
recently been shown to represent lexical, syntactic and compositional
representations similarly to the adult brain Jain and Huth (2018); Toneva and
Wehbe (2019); Caucheteux and King (2022); Pasquiou et al. (2022, 2023);
Caucheteux et al. (2023). Evidencing similar learning trajectories in children
and language models could thus provide an invaluable framework to better
understand the computational principles underlying language acquisition.

Here, we compare the trajectory of language acquisition between human children
and modern language models. We focus on three main questions. First, do these
models learn linguistic skills in a systematic order? Second, is this
trajectory sequential or parallel? Third, is this trajectory similar to that
of children? These hypotheses are illustrated in Figure 1.

Specifically, we train 48 GPT-2 architectures (Radford et al., 2019) from
scratch, using a standard next-word prediction objective. We then evaluate, at
each training step, their linguistic abilities with 96 semantic and syntactic
probes curated from the BLiMP, Zorro and BIG-Bench benchmarks (Warstadt et
al., 2020; Huebner et al., 2021; Srivastava et al., 2022). Finally, we compare
a subset of these probes to the behavior of 54 children aged, between 18
months and 6 years (Friedmann et al., 2021).

##  2 Approach

###  2.1 Language models

We consider two main language models. First, we use a pretrained language
model – GPT-2 – as provided by HuggingFace 111https://huggingface.co/gpt2 and
pretrained on 40 GB of data (Radford et al., 2019). Second, we separately
train 48 versions of a 12-layer GPT-2 model from scratch. We train each model
on WikiText103 Merity et al. (2016) with a distinct random seed to set its
initial parameters and data-loader. Each model is evaluated on all linguistic
probes every 100 training steps. Further training details are provided in
Appendix B.

###  2.2 Zero-shot linguistic probes

Zero-shot linguistic probes are sentences or phrases crafted to evaluate
whether a model has learned a particular linguistic skill, without training or
fine-tuning the model on that particular skill. In practice, a zero-shot probe
consists of comparing the estimated probability of a grammatical sentence with
that of a matched ungrammatical sentence. This two-alternative forced-choice
approach can be compared to "acceptability judgements", classically used in
linguistics (Warstadt et al., 2019).

We evaluate our models on 96 different linguistic probes, curated from three
open source benchmarks, the details of which are presented in Appendix C.

Specifically, we compare the probability of each sentence in a
grammatical/ungrammatical pair by evaluating the sum of the logarithm of the
loss output by the softmax layer:

$\sum_{i=0}^{n_{g}}log(f(X_{g})_{i})<\sum_{j=0}^{n_{u}}log(f(X_{u})_{j})$ (1)

with $f$ the softmax layer of the language model, $X_{g}$ and $X_{u}$ the
grammatical and ungrammatical sentences, respectively, and $n_{g}$ and
$n_{u}$, the number of tokens in the grammatical and ungrammatical sentences,
respectively.

The accuracy of a given probe is the percentage of pairs where the estimated
probability of the grammatical sentence is higher than that of the
ungrammatical sentence.

###  2.3 Assessing learning trajectory

To evaluate whether the trajectory of language acquisition is shared across
models, we rank the probes by their "acquisition time", _i.e._ the number of
steps taken by a model to reach 90% of its final accuracy on a particular
probe, for each model independently. We then assess the correlation of ranks
between all pairs of the 48 models and take the average of these correlations.
To estimate the statistical significance of this average correlation we redo
this calculation for all possible model pairs after shuffling the ranks of one
of the models in each pair. We repeat this permutation 1,000 times, getting
1,000 values for this shuffled correlation. If in all cases this shuffled
correlation is lower than the true average correlation, then the order of
acquisition time is shared across models with p $<$ 0.001.

###  2.4 Parallel versus Sequential learning

Language acquisition may be characterized by a "sequential" or a "parallel"
learning scheme (Figure 1). "Sequential" learning designates the case where a
complex skill does not start to be learned before simpler skills are mastered.
By contrast, "Parallel" learning designates the case where all skills are
acquired simultaneously, but at different speeds. The null hypothesis is that
the order in which an agent learns linguistic skills varies across agents. To
determine the learning scheme of language models, we consider whether the
probes have a positive derivative in the first three checkpoints (parallel
learning) or not (sequential learning), and whether they have statistically
different learning rates (by performing a one-way ANOVA test) across the three
groups.

Figure 1:  Hypotheses. Skill performance (y-axis) as a function of training
(x-axis) illustrated on three tasks (colors) and two agents (model or
children). Sequential learning implies that the learning of a complex skill
(e.g. C, shown in red) does not start before simplest skills (e.g. A in green
and B in blue) are fully learned. Parallel learning implies that all skills
are acquired simultaneously, but at different speeds. Sequential and parallel
learning may cross an arbitrary performance threshold at the same training
step. The frequency at which we can probe artificial networks during learning
is much greater than what is realistically possible in children, giving us the
timescale granularity to distinguish sequential and parallel learning
trajectories. We also present a null hypothesis, that artificial networks with
different random seeds may learn skills in a different order.

###  2.5 Assessing linguistic skill from children’s behavior

Friedmann et al. (2021) studied 54 Hebrew-speaking children between the ages
of 18 - 71 months and investigated the emergence of 11 linguistic phenomena,
which the authors propose to organize into three stages (details in Appendix
A). For our analysis we select the following tests, one from each stage:

  * •

Stage 1: Simple sentences in subject-verb (SV) order

  * •

Stage 2: Wh Questions

  * •

Stage 3: Relative Clauses

Data collection consisted of spontaneous speech samples produced by each child
at home. Each sample was then manually annotated to detect the presence of
each of the linguistic phenomena. A linguistic phenomenon was considered
learned if and only if it was present in the speech sample. Speech samples had
a mean length of 151 utterances per sample and standard deviation of 37. The
aggregated data was made available directly in the original paper (under
Creative Commons Attribution 4.0 International License), and here used for
comparison with our language models. In Table 1 we show which probes in the
models matched with these tests.

Stage | Children | Language Model
---|---|---
1 | Simple sentences in Subject-Verb (SV) order | SV agreement across simple sentences
2 | Wh-questions | SV agreement in questions
3 | Relative Clauses (RCs) | SV agreement across object RCs
Table 1:  Linguistic phenomena in children selected for comparison with probes
in the language models.

##  3 Results

Figure 2: The performance of the models on each linguistic probe over time is
smoothed using a moving average filter with window size of 6 checkpoints then
the number of steps required to reach 90% of final accuracy (acquisition time)
is calculated. Probes are ordered by increasing average acquisition time.
Results shown for 48 models. Only probes which have final accuracy greater
than chance (50%) are shown. This demonstrates that probes tend to be learned
in the same order by all agents with R = 0.743, p < 0.001, disproving the null
hypothesis.

We aim to compare the learning trajectories of deep language models to those
observed in 54 children aged between 18 months and 6 years. For this, we
trained variants of GPT-2 models Radford et al. (2019) from 48 different
random seeds with the WikiText103 dataset Merity et al. (2016) and evaluated
each model on 96 linguistic probes every 100 steps.

At the end of this training, 64 probes (66%) were achieved above chance level
(50% accuracy) by all models. In comparison, a pretrained version of GPT-2
large Radford et al. (2019) provided by Hugging
Face222https://huggingface.co/tftransformers/gpt2-large, and trained on a much
larger dataset33340 GB compared to the 181 MB of WikiText103, achieves above-
chance performance on 93 of the 96 probes.

###  3.1 A systematic learning trajectory

For clarity, we focus on the learning dynamics of the probes that ultimately
achieve above-chance performance in our training. Figure 2 lists all probes
learned above chance level, ordered by their average acquisition time. We
perform the permutation analysis outlined in 2.3, to evaluate whether the
order of acquisition is shared between models, and find that their order of
acquisition is correlated with $R=0.743$ and p $<$ 0.001. These results
suggest that there is a systematic learning trajectory among models.

###  3.2 Learning is parallel across linguistic tasks

Figure 3: Linguistic probes grouped into 3 groups to avoid plotting one line
per probe. The probes in each group correspond to the colours in Figure 2.
Probes which have final accuracy less than chance (50%) are placed in their
own group, and tend to zero likely due to biases towards plural verbs in
English (c.f. 3.6). Shading is standard error of the mean across probes in the
group. This figure demonstrates that linguistic skills are learned in parallel
not in sequence.

Are these linguistic skills learned sequentially or in parallel (Figure 1)? To
address this question, we evaluate whether each linguistic probe starts to
improve from the very first training steps but with different rates (i.e. a
“parallel” learning scheme) or, on the contrary, whether some probes only
start to improve once others have reached a particular performance (i.e. a
“sequential” learning scheme). As the individual learning trajectories of each
probe were noisy, we group the 64 linguistic probes into three categories:
early, middle and late acquisition time (Figure 3).

Overall, we observe parallel learning between the three groups: their
performances all increase from the beginning of training: 95% of tests in all
three groups have a positive derivative within the first three hundred steps.
However, they have different learning rates, as evaluated with a one-way ANOVA
test on the learning rate (i.e. change of accuracy over time) obtained in each
group and in each model ($p<10^{-23}$).

###  3.3 Comparison with children

Figure 4: Comparing language model performance on linguistic probes to
children’s performance. Example sentences observed in children were originally
in Hebrew Friedmann et al. (2021). Non-white indicates the phenomena is
learned by the agent. The threshold for considering a probe learned by the
model is performance above 55%.

Do these learning trajectories match the behavior of human children? For the
three probes that correspond to the three stages identified in children’s
language acquisition (Table 1), we observe that the order in which these three
probes are learned by the language models is the same as those of children
(Figure 4). This effect is robust across random seeds: 46 of our 48 GPT-2
models follows this order, where chance level is $(1/3!)^{46}=1.60e^{-36}$.
For this subset of language abilities, models and children thus seem to
acquire syntactic skills in a similar order.

###  3.4 Learning of theoretically-defined stages is parallel

A

B

C

Figure 5: Subject verb agreement. Shading is standard error of the mean across
seeds. S: Singular. P: Plural. A. Parallel learning is observed in the three
stages defined by Friedmann et al. (2021), when results are averaged across
congruent and incongruent cases. B. Subject verb agreement on incongruent
cases which indicate whether the model understands syntax. The networks do not
learn some syntax structures as the incongruent case of Short Nested Outer
does not reach above chance level. C. Development trajectories of the bias
towards plural number in English.

In part 3.1, we showed that GPT-2 learns its language abilities in parallel.
Does this learning scheme also characterize the three syntactic skills
investigated in children? To address this question, we now look at the
learning curves of the skills defined in Table 1, as well as an additional
probe: Nounpp, as it can be separated into congruent and incongruent cases
which is important for the analysis in section 3.5. Overall, we observe that
these probes are all learned in parallel in the model (Figure 5A).

###  3.5 Models use both syntax and heuristics

Both an understanding of syntactic rules and a superficial heuristics can lead
to above-chance performances on these probes. Indeed, in many sentences
(_e.g._ The cat [subject] of the lady [attractor] is [verb] hungry.), the
number of the verb is congruent with the number of the adjacent attractor,
even if the two are not related syntactically. To verify that the GPT-2 models
effectively learn the syntactic rules, we thus separately examine congruent
and incongruent cases. Incongruent cases require knowledge of the syntax of
the sentence as the correct verb number is different from the number of the
attractor. Empirically, we observe that the models do not learn the
incongruent case in stage three above chance level, and just barely reach
chance level on the incongruent case in stage two (Figure 5B), indicating that
our models are using heuristics rather than syntactic rules to achieve high
accuracy on the congruent cases (leading to above chance performance on the
probe overall in Figure 5A). On the contrary, the pretrained GPT-2 large
achieves above 75% accuracy also on the incongruent cases of these probes.
Thus for the models trained on the WikiText103, syntax is only learned for
stages one and two, and heuristics seem to explain the above chance accuracy
in stage three. A larger training dataset is required to learn syntax and not
only heuristics for the most difficult examples.

###  3.6 Impact of number biases in congruent and incongruent sentences

In previous work, it was found that a variety of language models have a bias
towards plural English verbs, and several studies Jumelet et al. (2019);
Lakretz et al. (2021a, b) determined that LSTM-based models have a default
number and gender prediction preference. To examine whether number bias has a
significant effect on our analysis, we compare congruent sentences with only
singular or only plural verbs and incongruent sentences with a plural or a
singular verb. Accuracy on predicting plural verbs increases sharply from the
start of the training and then drops. By contrast, the accuracy of singular
cases first drops and then rises (Figure 5C), indicating that the models are
biased towards plural verbs at the beginning of training. This bias is
overcome for the stage one probe but for stage two and three it remains
throughout training. This explains the initial downward trend in Group 3 and
why the unlearned probes tend toward zero in Figure 3.

##  4 Discussion

The stages followed by children to acquire language has been the topic of
intense research (Dupoux, 2018; Kuhl, 2004; Werker, 2018). While this learning
trajectory is becoming clearer for sub-lexical representations (Dupoux, 2018),
the acquisition of higher-level syntactic and semantic processing remains
largely unclear. Here, we approach this long-lasting question through the lens
of a deep language architecture, GPT-2 (Radford et al., 2019), to test whether
this model follows a learning trajectory similar to children.

###  4.1 Language acquisition: similarities and differences between humans and
GPT-2

First, we show that GPT-2 models tend to learn a battery of linguistic
phenomena (Warstadt et al., 2020; Lakretz et al., 2019; Huebner et al., 2021)
in a consistent order. It is the reliability of the acquisition trajectory
that allows a direct comparison with the learning trajectory of children
Friedmann et al. (2021). However, this consistency in GPT-2 models may result
from two non-mutually exclusive factors, that remain to be disentangled:
either the acquisition time of each linguistic phenomenon relates to its
frequency in natural language (e.g. Simple subject-verb-complement are more
frequent in natural language than nested syntactic structures; Karlsson 2007),
and/or it relates to their intrinsic complexity (e.g. sentences with nested
structure require more operations to be composed than simple sentences).
Future work systematically controlling for these relative frequencies is thus
necessary to distinguish these two possibilities, and would build upon work by
Weber et al. (2021) who found that less frequent linguistic phenomena can be
learned from fewer examples, though later in training.

Second, we show that the order in which linguistic skills are acquired is
similar between children and GPT-2 – at least on the syntactic phenomena that
were evaluated in these two cohorts, and with the limitation of using number
agreement as a proxy to whether the models acquire the corresponding syntactic
structure. Similarly to children, GPT-2 models master subject-verb agreement
in SV sentences before they master it in questions, or across nested center-
embedded clauses (object-relative clauses). This result thus complements a
series of studies comparing modern language models and humans. For example, a
recent study showed that transformers trained on child-directed data can
achieve comparable accuracy on linguistic probes to large pre-trained models
Huebner et al. (2021). Similarly, several studies have recently shown that the
representations of GPT-2 become increasingly similar to those of the adult
human brain during its training Caucheteux and King (2022). Finally, Lavechin
et al. (2022) showed that models trained on audio in a self-supervised fashion
learn phoneme and lexical abilities in a similar trajectory to children.

###  4.2 A work in progress

It is important to stress that significant work remains to be done before
drawing any definitive conclusions about the similarities between language
acquisition in humans and algorithms.

First, we only consider a single architecture (GPT-2, Radford et al. (2019))
with a unique textual corpus (WikiText103). Testing whether our results hold
true independently of the model and training corpus remains an important
milestone for future research.

Second, linguistic abilities are not tested with the same protocols in
children and in the models: the models are explicitly tested on next word
prediction, with a two-alternative forced-choice metric, whereas children were
implicitly evaluated on their ability to spontaneously use specific syntactic
structures during natural speech.

Third, there were only three linguistic features that were directly comparable
between the model probes and the data in children, and all were syntactic.
This leaves a significant margin of progress to modulate our conclusion, and
investigate whether the lexicon, narrative structures, pragmatics and world-
knowledge are acquired in the same order in humans and algorithms.

Fourth, the order in which some linguistic skills were learned by GPT-2 does
not trivially fit with linguistic theory. For example, the probe “Simple”,
which examines subject-verb agreement in a simple sentence, was one of the
last probes to be learned by GPT-2 (it is part of group three in Figure 2). By
contrast, "Wh Questions Subject Gap Long Distance" was among the first probes
to be to be learned, even though it would be expected to be much harder than
“Simple”. This unexpected result may be due to the way we approximate
"Acquisition Time", namely, the moment when the probes reaches 90% of the
final accuracy. Consequently, probes with very low final accuracy could end up
with a shorter Acquisition Time, because noise may lead to crossing the 90%
threshold relatively quickly.

Finally, we show that our models appear to use heuristics rather than a deep
understanding of syntax for the most difficult linguistic probes (incongruent
numbers between verbs and their attractors) and were biased towards plural
English verbs. While our models learn only 66% of tasks to above chance level,
a larger GPT-2 pretrained on considerably more texts successfully perform on
97% of the tasks, and has an accuracy above 75% on the incongruent examples,
meaning this bias and reliance on heuristics could potentially be solved by
training on a larger dataset.

In sum, additional work remains necessary to identify the exact elements of
convergence and divergence between the acquisition of language in models and
in children.

###  4.3 Fueling the debate between nativism versus empiricism

The present study fuels a long-lasting debate on the acquisition of language.
While “empiricists“ argue that language can be acquired with a statistical
approach (Clark, 2002; Kolodny et al., 2015; Chater and Christiansen, 2018;
McCauley and Christiansen, 2019), “nativists“ maintain that this ability
depends on a core and innate operation, specific to humans (Chomsky, 1959,
1971).

The present study shows how modern language models may contribute to resolving
this debate, by systematically studying which components of a model (e.g.
architecture) or properties of the training data (e.g., frequency of sentence
structures) contribute to shape the trajectory of language acquisition. Claims
about an innate Universal Grammar could be understood as an inductive bias of
a language model, implemented in its architecture and dynamics, which tightly
constrains learning trajectories across models. If this bias is hierarchical
(rather than linear) then this could lead to learning trajectories that follow
the structure of the syntactic tree, consistently with the hypothesis of three
linguistic stages presented by Friedmann et al. (2021) in humans and what we
find in this study in language models. Indeed, neural language models have
been previously shown to have a weak inductive bias towards hierarchical
processing McCoy et al. (2020); Kharitonov and Chaabouni (2020), which can
partially explain our results.

This result echos the recent observation that syntactic trees spontaneously
emerge in the middle layers of neural language models (Hewitt and Manning,
2019). Together, these elements thus suggest that modern neural networks
provide fruitful models of language acquisition and could reconcile or settle
the confronting theories of language acquisition (Warstadt and Bowman, 2022).

###  4.4 Conclusion

Overall, the similarities identified between children and GPT-2 suggest that
there may be a small set of means by which to efficiently acquire language.
This result is anything but trivial: humans and deep neural networks have
extraordinarily different architectures, training, and language exposure. If
generalized, this systematic learning trajectory would support the existence
of an intrinsic hierarchy of linguistic structures that both machines and
humans must climb, be that through inductive biases or properties of the
training data, to master the faculty of language. And while these hypotheses
remain open, the path to resolve them has never been clearer.

## Acknowledgements

We would like to thank Dieuwke Hupkes, Naama Friedmann, Marco Baroni and the
attendees of the EviL meetings for their comments and suggestions.

This project has received funding from the European Union’s Horizon 2020
research and innovation program under the Marie Skłodowska-Curie grant
agreement No 945304, for L.E for her work at PSL. This work was funded in part
by FrontCog grant ANR-17-EURE-0017 for the work of L.E. and J.R.K. for their
work at PSL.

## References

  * Bergelson and Swingley (2012) Elika Bergelson and Daniel Swingley. 2012.  At 6–9 months, human infants know the meanings of many common nouns.  _Proceedings of the National Academy of Sciences_ , 109:3253–3258.
  * Bowman and Dahl (2021) Samuel R. Bowman and George Dahl. 2021.  What will it take to fix benchmarking in natural language understanding? _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4843–4855.
  * Caucheteux et al. (2023) Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. 2023.  Evidence of a predictive coding hierarchy in the human brain listening to speech.  _Nature Human Behaviour_ , pages 1–12.
  * Caucheteux and King (2022) Charlotte Caucheteux and Jean Rémi King. 2022.  Brains and algorithms partially converge in natural language processing.  _Communications Biology_ , 5(1).
  * Chang and Bergen (2022) Tyler A. Chang and Benjamin K. Bergen. 2022.  Word acquisition in neural language models.  _Transactions of the Association for Computational Linguistics_ , 10:1–16.
  * Chater and Christiansen (2018) Nick Chater and Morten H Christiansen. 2018.  Language acquisition as skill learning.  _Current opinion in behavioral sciences_ , 21:205–208.
  * Chomsky (1959) Noam Chomsky. 1959.  Review of verbal behavior.  35(1):26–58.  Publisher: Linguistic Society of America.
  * Chomsky (1971) Noam Chomsky. 1971.  _Problems of Knowledge and Freedom_.  New York,: W.W. Norton.
  * Clark (2002) Alexander Clark. 2002.  Unsupervised language acquisition: Theory and practice.  _arXiv preprint cs/0212024_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova Google, and AI Language. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Dupoux (2018) Emmanuel Dupoux. 2018.  Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner.  _Cognition_ , 173:43–59.
  * Friedmann et al. (2021) Naama Friedmann, Adriana Belletti, and Luigi Rizzi. 2021.  Growing trees: The acquisition of the left periphery.  _Glossa: a journal of general linguistics_ , 39(1).
  * Friedmann and Reznick (2021) Naama Friedmann and Julia Reznick. 2021.  Stages rather than ages in the acquisition of movement structures: Data from sentence repetition and 27696 spontaneous clauses.  _Glossa: a journal of general linguistics_ , 39(1).
  * Gulordava et al. (2018) Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018.  Colorless green recurrent networks dream hierarchically.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1195–1205, New Orleans, Louisiana. Association for Computational Linguistics.
  * Hewitt and Manning (2019) John Hewitt and Christopher D Manning. 2019.  A structural probe for finding syntax in word representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4129–4138.
  * Huebner (2022) Philip Huebner. 2022.  Unmasked.  _https://github.com/phueb/UnMasked_.  (Accessed 2023/05/24).
  * Huebner et al. (2021) Philip A Huebner, Elior Sulem, Cynthia Fisher, and Dan Roth. 2021.  BabyBERTa : Learning More Grammar With Small-Scale Child-Directed Language.  _Proceedings of the 25th Conference on Computational Natural Language Learning_ , pages 624–646.
  * Jain and Huth (2018) Shailee Jain and Alexander Huth. 2018.  Incorporating context into language encoding models for fmri.  In _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc.
  * Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019.  What does bert learn about the structure of language? _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3651–3657.
  * Jumelet et al. (2019) Jaap Jumelet, Willem Zuidema, and Dieuwke Hupkes. 2019.  Analysing neural language models: Contextual decomposition reveals default reasoning in number and gender assignment.  _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 1–11.
  * Karlsson (2007) Fred Karlsson. 2007.  Constraints on multiple center-embedding of clauses.  _Journal of Linguistics_ , 43(2):365–392.
  * Kharitonov and Chaabouni (2020) Eugene Kharitonov and Rahma Chaabouni. 2020.  What they do when in doubt: a study of inductive biases in seq2seq learners.  _arXiv:2006.14953_.
  * Kolodny et al. (2015) Oren Kolodny, Arnon Lotem, and Shimon Edelman. 2015.  Learning a generative probabilistic grammar of experience: A process-level model of language acquisition.  _Cognitive Science_ , 39(2):227–267.
  * Kuhl (2004) Patricia K. Kuhl. 2004.  Early language acquisition: cracking the speech code.  _Nature Reviews Neuroscience_ , 5:831–843.
  * Kuhl et al. (2006) Patricia K Kuhl, Erica Stevens, Akiko Hayashi, Toshisada Deguchi, Shigeru Kiritani, and Paul Iverson. 2006.  Fast-track report infants show a facilitation effect for native language phonetic perception between 6 and 12 months.  _Developmental Science_ , 9:13–21.
  * Kuhl et al. (1992) Patricia K. Kuhl, Karen A. Williams, Francisco Lacerda, Kenneth N. Stevens, and Bjorn Lindblom. 1992.  Linguistic experience alters phonetic perception in infants by 6 months of age.  _Science_ , 255:606–608.
  * Lakretz et al. (2021a) Yair Lakretz, Théo Desbordes, Dieuwke Hupkes, and Stanislas Dehaene. 2021a.  Causal transformers perform below chance on recursive nested constructions, unlike humans.  _arXiv:2110.07240_.
  * Lakretz et al. (2021b) Yair Lakretz, Dieuwke Hupkes, Alessandra Vergallito, Marco Marelli, Marco Baroni, and Stanislas Dehaene. 2021b.  Mechanisms for handling nested dependencies in neural-network language models and humans.  _Cognition_ , 213:104699.  Special Issue in Honour of Jacques Mehler, Cognition’s founding editor.
  * Lakretz et al. (2019) Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. 2019.  The emergence of number and syntax units in LSTM language models.  _Association for Computational Linguistics_ , pages 11–20.
  * Lavechin et al. (2022) Marvin Lavechin, Maureen De Seyssel, Hadrien Titeux, Hervé Bredin, Guillaume Wisniewski, Alejandrina Cristia, and Emmanuel Dupoux. 2022.  Statistical learning bootstraps early language acquisition.  _PsyArXiv_.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016.  Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies.  _Transactions of the Association for Computational Linguistics_ , 4:521–535.
  * Manning et al. (2020) Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. 2020.  Emergent linguistic structure in artificial neural networks trained by self-supervision.  _Proceedings of the National Academy of Sciences of the United States of America_ , 117:30046–30054.
  * Mazuka et al. (2011) Reiko Mazuka, Yvonne Cao, Emmanuel Dupoux, and Anne Christophe. 2011.  The development of a phonological illusion: A cross-linguistic study with japanese and french infants.  _Developmental Science_ , 14:693–699.
  * McCauley and Christiansen (2019) Stewart M McCauley and Morten H Christiansen. 2019.  Language learning as language use: A cross-linguistic model of child language development. _Psychological review_ , 126(1):1.
  * McCoy et al. (2020) R Thomas McCoy, Robert Frank, and Tal Linzen. 2020.  Does syntax need to grow on trees? sources of hierarchical inductive bias in sequence-to-sequence networks.  _Transactions of the Association for Computational Linguistics_ , 8:125–140.
  * Mehler et al. (1988) Jacques Mehler, Peter Jusczyk, Ghislaine Lambertz, Nilofar Halsted, Josiane Bertoncini, and Claudine Amiel-Tison. 1988.  A precursor of language acquisition in young infants.  _Cognition_ , 29:143–178.
  * Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016.  Pointer sentinel mixture models.  _arXiv:1609.07843_.
  * Nazzi et al. (2000) Thierry Nazzi, Peter W. Jusczyk, and Elizabeth K. Johnson. 2000.  Language discrimination by english-learning 5-month-olds: Effects of rhythm and familiarity.  _Journal of Memory and Language_ , 43:1–19.
  * Pasquiou et al. (2022) Alexandre Pasquiou, Yair Lakretz, John Hale, Bertrand Thirion, and Christophe Pallier. 2022.  Neural language models are not born equal to fit brain data, but training helps.  In _ICML 2022-39th International Conference on Machine Learning_ , page 18.
  * Pasquiou et al. (2023) Alexandre Pasquiou, Yair Lakretz, Bertrand Thirion, and Christophe Pallier. 2023\.  Information-restricted neural language models reveal different brain regions’ sensitivity to semantics, syntax and context.  _arXiv:2302.14389_.
  * Pruksachatkun et al. (2020) Yada Pruksachatkun, Phil Yeres, Haokun Liu, Jason Phang, Phu Mon Htut, Alex Wang, Ian Tenney, and Samuel R. Bowman. 2020.  jiant: A software toolkit for research on general-purpose text understanding models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 109–117, Online. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language Models are Unsupervised Multitask Learners.  _Semantic Scholar_.  (Accessed 2023-05-04).
  * Shi et al. (1999) Rushen Shi, Janet F Werker, and James L Morgan. 1999.  Newborn infants’ sensitivity to perceptual cues to lexical and grammatical words.  _Cognition_ , 72:B11–B21.
  * Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _arXiv 2206.04615_.
  * Tincoff and Jusczyk (1999) Ruth Tincoff and Peter W. Jusczyk. 1999.  Some beginnings of word comprehension in 6-month-olds.  _Psychological Science_ , 10:172–175.
  * Toneva and Wehbe (2019) Mariya Toneva and Leila Wehbe. 2019.  Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain).  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Warstadt and Bowman (2022) Alex Warstadt and Samuel R Bowman. 2022.  What artificial neural networks can tell us about human language acquisition.  _arXiv:2208.07998_.
  * Warstadt et al. (2020) Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng Fu Wang, and Samuel R. Bowman. 2020.  Erratum: “blimp: The benchmark of linguistic minimal pairs for english”.  _Transactions of the Association for Computational Linguistics_ , 8:867–868.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641.
  * Weber et al. (2021) Lucas Weber, Jaap Jumelet, Elia Bruni, and Dieuwke Hupkes. 2021.  Language modelling as a multi-task problem.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2049–2060, Online. Association for Computational Linguistics.
  * Werker (2018) Janet F. Werker. 2018.  Perceptual beginnings to language acquisition.  _Applied Psycholinguistics_ , 39:703–728.
  * Werker and Tees (1984) Janet F. Werker and Richard C. Tees. 1984.  Cross-language speech perception: Evidence for perceptual reorganization during the first year of life.  _Infant Behavior and Development_ , 7:49–63.
  * Zamuner (2006) Tania S. Zamuner. 2006.  Sensitivity to word-final phonotactics in 9- to 16-month-old infants.  _Infancy_ , 10:77–95.

## Appendix

##  Appendix A Tests in Children

Detailed description of tests available in children, in the three linguistic
stages defined by Friedmann et al. (2021):

  * •

Stage 1: Subject-Verb Simple, Subject-Verb Unaccusative, Verb-Subject
Unaccusative

  * •

Stage 2: Root WH-Argument, WH-Adjunct Excluding Why, Preposed Adverb, Root y/n

  * •

Stage 3: Why, Relative Clause, Topicalisation, Embedding

The probes chosen for comparison (stated in Table 1), were the only probes
that matched well with one of the test available in children. In addition
Nounpp was examined in the models, as it fits into the linguistic stage 2,
and, as it is part of the BIG-Bench probes, could be separated into congruent
and incongruent sentences.

##  Appendix B Model Training

To evaluate linguistic abilities of a high-performance language model, we
first use the HuggingFace pretrained GPT-2 large which has 774M parameters and
is trained on 40GB of data. This model has one-shot perplexity of 22 on
WikiText103 Radford et al. (2019).

Then, to evaluate how linguistic abilities vary with language acquisition, we
separately trained 48 models (each with a distinct random seed which set the
model’s initial parameters and the seed of the dataloader) using the 12-layer
GPT-2 architecture Radford et al. (2019) provided by
HuggingFace444https://huggingface.co/gpt2 on WikiText103 Merity et al. (2016)
with a learning rate of $10^{-5}$ and a batch size of 16 distributed over 8
GPUs, making a total batch size of 64 and a context size of 128. Training was
stopped when then validation loss plateaued, reaching final perplexity of 28
after 10 epochs. This is lower perplexity than the one-shot performance of the
HuggingFace pretrained 12-layer GPT-2 which was 37.5, which is logical as our
model was trained specifically on this dataset.

In all cases we used the pretrained tokenizer which has vocabulary size of
50,257. All other parameters were the default training arguments for the
transformer provided by HuggingFace. The HuggingFace architectures are
publicly available under an MIT license, and WikiText103 is available under
Creative Commons Attribution-ShareAlike License.

##  Appendix C Linguistic Probe Benchmarks

We use three different zero-shot benchmarks. The first benchmark, ‘BLiMP’ (The
Benchmark of Linguistic Minimal Pairs for English) Warstadt et al. (2020)
contains 67 different probes, each in the form of 1,000 pairs of grammatical
and ungrammatical sentences designed to isolate a specific linguistic
phenomenon. Adult human performance on BLiMP is 96.4% Warstadt et al. (2020).
The second benchmark, ‘Zorro’ 555https://github.com/phueb/Zorro, was developed
with a vocabulary frequent in child-directed corpora. Zorro contains 13
probes, each consisting of 2,000 pairs of sentences. Finally, the third
benchmark is the Subject-Verb Agreement Task of BIG-Bench Srivastava et al.
(2022); Lakretz et al. (2019, 2021b); Linzen et al. (2016); Gulordava et al.
(2018). We focus on the syntactic probes, namely: "Simple English" which
contains 600 pairs, "NounPP" which contains 2,400 pairs, and "Short Nested
Inner", "Short Nested Outer", "Long Nested Inner" and "Long Nested Outer"
which each contain 4,096 pairs of grammatical and ungrammatical sentences.

Accuracy on a linguistic probe is evaluated with the Jiant Pruksachatkun et
al. (2020) and UnMasked method Huebner (2022). In practice, sentences are
input to the model in batches of 300, with padding on the left to make all
sentences the length of the longest sentence in the batch. The logit values of
punctuation are discarded when estimating the probability of a sentence.

Zorro, Jiant and UnMasked are publicly available under the MIT License, BLiMP
under a CC BY License, and BIG-Bench under the Apache License 2.0.