* [131] Dvali, G.  $S$-Matrix and Anomaly of de Sitter.  _Symmetry_ 13, 3, DOI: 10.3390/sym13010003 (2020).  2012.02133.
  * [132] Obied, G., Ooguri, H., Spodyneiko, L. & Vafa, C.  De Sitter Space and the Swampland.  _arXiv preprint_ (2018).  1806.08362.
  * [133] Garg, S. K. & Krishnan, C.  Bounds on Slow Roll and the de Sitter Swampland.  _J. High Energy Phys._ 11, 075, DOI: 10.1007/JHEP11(2019)075 (2019).  1807.05193.
  * [134] Ooguri, H., Palti, E., Shiu, G. & Vafa, C.  Distance and de Sitter Conjectures on the Swampland.  _Phys. Lett. B_ 788, 180–184, DOI: 10.1016/j.physletb.2018.11.018 (2019).  1810.05506.
  * [135] Agrawal, P., Obied, G., Steinhardt, P. J. & Vafa, C.  On the Cosmological Implications of the String Swampland.  _Phys. Lett. B_ 784, 271–276, DOI: 10.1016/j.physletb.2018.07.040 (2018).  1806.09718.
  * [136] Dvali, G., Gomez, C. & Zell, S.  Quantum Breaking Bound on de Sitter and Swampland.  _Fortsch. Phys._ 67, 1800094, DOI: 10.1002/prop.201800094 (2019).  1810.11002.
  * [137] Ade, P. A. R. _et al._ BICEP/Keck XII: Constraints on axionlike polarization oscillations in the cosmic microwave background.  _Phys. Rev. D_ 103, 042002, DOI: 10.1103/PhysRevD.103.042002 (2021).  2011.03483.
  * [138] Freese, K., Frieman, J. A. & Olinto, A. V.  Natural inflation with pseudo - Nambu-Goldstone bosons.  _Phys. Rev. Lett._ 65, 3233–3236, DOI: 10.1103/PhysRevLett.65.3233 (1990).
  * [139] Fujita, T., Minami, Y., Murai, K. & Nakatsuka, H.  Probing axionlike particles via cosmic microwave background polarization.  _Phys. Rev. D_ 103, 063508, DOI: 10.1103/PhysRevD.103.063508 (2021).  2008.02473.
  * [140] Fujita, T., Murai, K., Nakatsuka, H. & Tsujikawa, S.  Detection of isotropic cosmic birefringence and its implications for axionlike particles including dark energy.  _Phys. Rev. D_ 103, 043509, DOI: 10.1103/PhysRevD.103.043509 (2021).  2011.11894.
  * [141] Takahashi, F. & Yin, W.  Kilobyte Cosmic Birefringence from ALP Domain Walls.  _J. Cosmol. Astropart. Phys._ 04, 007, DOI: 10.1088/1475-7516/2021/04/007 (2021).  2012.11576.
  * [142] Mehta, V. M. _et al._ Superradiance in string theory.  _J. Cosmol. Astropart. Phys._ 07, 033, DOI: 10.1088/1475-7516/2021/07/033 (2021).  2103.06812.
  * [143] Nakagawa, S., Takahashi, F. & Yamada, M.  Cosmic Birefringence Triggered by Dark Matter Domination.  _Phys. Rev. Lett._ 127, 181103, DOI: 10.1103/PhysRevLett.127.181103 (2021).  2103.08153.
  * [144] Alvey, J. & Escudero Abenza, M.  Constraints on global symmetry breaking in quantum gravity from cosmic birefringence measurements.  _Phys. Lett. B_ 823, 136752, DOI: 10.1016/j.physletb.2021.136752 (2021).  2106.04226.
  * [145] Choi, G., Lin, W., Visinelli, L. & Yanagida, T. T.  Cosmic birefringence and electroweak axion dark energy.  _Phys. Rev. D_ 104, L101302, DOI: 10.1103/PhysRevD.104.L101302 (2021).  2106.12602.
  * [146] Obata, I.  Implications of the Cosmic Birefringence Measurement for the Axion Dark Matter Search.  _arXiv preprint_ (2021).  2108.02150.
  * [147] Abitbol, M. H., Hill, J. C. & Johnson, B. R.  Foreground-Induced Biases in CMB Polarimeter Self-Calibration.  _Mon. Not. Roy. Astron. Soc._ 457, 1796–1803, DOI: 10.1093/mnras/stw030 (2016).  1512.06834.
  * [148] Planck Collaboration XI.  Planck 2018 results. XI. Polarized dust foregrounds.  _Astron. Astrophys._ 641, A11, DOI: 10.1051/0004-6361/201832618 (2020).  1801.04945.
  * [149] Weiland, J. L., Addison, G. E., Bennett, C. L., Halpern, M. & Hinshaw, G.  An Examination of Galactic Polarization with Application to the Planck TB Correlation.  _Astrophys. J._ 893, 119, DOI: 10.3847/1538-4357/ab7ea6 (2020).  1907.02486.
  * [150] Clark, S. E., Kim, C.-G., Hill, J. C. & Hensley, B. S.  The Origin of Parity Violation in Polarized Dust Emission and Implications for Cosmic Birefringence.  _Astrophys. J._ 919, 53, DOI: 10.3847/1538-4357/ac0e35 (2021).  2105.00120.
  * [151] Huffenberger, K. M., Rotti, A. & Collins, D. C.  The Power Spectra of Polarized, Dusty Filaments.  _Astrophys. J._ 899, 31, DOI: 10.3847/1538-4357/ab9df9 (2020).  1906.10052.
  * [152] Guth, A. H.  The Inflationary Universe: A Possible Solution to the Horizon and Flatness Problems.  _Phys. Rev._ D23, 347–356, DOI: 10.1103/PhysRevD.23.347 (1981).
  * [153] Sato, K.  First Order Phase Transition of a Vacuum and Expansion of the Universe.  _Mon. Not. Roy. Astron. Soc._ 195, 467–479 (1981).
  * [154] Linde, A. D.  A New Inflationary Universe Scenario: A Possible Solution of the Horizon, Flatness, Homogeneity, Isotropy and Primordial Monopole Problems.  _Phys. Lett._ 108B, 389–393, DOI: 10.1016/0370-2693(82)91219-9 (1982).
  * [155] Albrecht, A. & Steinhardt, P. J.  Cosmology for Grand Unified Theories with Radiatively Induced Symmetry Breaking.  _Phys. Rev. Lett._ 48, 1220–1223, DOI: 10.1103/PhysRevLett.48.1220 (1982).
  * [156] Steinhardt, P. J. & Turok, N.  A Cyclic model of the universe.  _Science_ 296, 1436–1439, DOI: 10.1126/science.1070462 (2002).  hep-th/0111030.
  * [157] Nayeri, A., Brandenberger, R. H. & Vafa, C.  Producing a scale-invariant spectrum of perturbations in a Hagedorn phase of string cosmology.  _Phys. Rev. Lett._ 97, 021302, DOI: 10.1103/PhysRevLett.97.021302 (2006).  hep-th/0511140.
  * [158] Brandenberger, R. & Peter, P.  Bouncing Cosmologies: Progress and Problems.  _Found. Phys._ 47, 797–850, DOI: 10.1007/s10701-016-0057-0 (2017).  1603.05834.
  * [159] Mukhanov, V. F.  Quantum Universe.  _Phys. Usp._ 59, 1021–1027, DOI: 10.3367/UFNe.2016.07.037857 (2016).
  * [160] Tristram, M. _et al._ Improved limits on the tensor-to-scalar ratio using BICEP and Planck.  _arXiv preprint_ (2021).  2112.07961.
  * [161] Caprini, C. & Figueroa, D. G.  Cosmological Backgrounds of Gravitational Waves.  _Class. Quant. Grav._ 35, 163001, DOI: 10.1088/1361-6382/aac608 (2018).  1801.04268.
  * [162] Lyth, D. H.  What would we learn by detecting a gravitational wave signal in the cosmic microwave background anisotropy?  _Phys. Rev. Lett._ 78, 1861–1863, DOI: 10.1103/PhysRevLett.78.1861 (1997).  hep-ph/9606387.
  * [163] Lyth, D. H. & Riotto, A.  Particle physics models of inflation and the cosmological density perturbation.  _Phys. Rept._ 314, 1–146, DOI: 10.1016/S0370-1573(98)00128-8 (1999).  hep-ph/9807278.
  * [164] Lasky, P. D. _et al._ Gravitational-wave cosmology across 29 decades in frequency.  _Phys. Rev. X_ 6, 011035, DOI: 10.1103/PhysRevX.6.011035 (2016).  1511.05994.
  * [165] Adshead, P., Giblin, J. T., Pieroni, M. & Weiner, Z. J.  Constraining Axion Inflation with Gravitational Waves across 29 Decades in Frequency.  _Phys. Rev. Lett._ 124, 171301, DOI: 10.1103/PhysRevLett.124.171301 (2020).  1909.12843.
  * [166] Campeti, P., Komatsu, E., Poletti, D. & Baccigalupi, C.  Measuring the spectrum of primordial gravitational waves with CMB, PTA and Laser Interferometers.  _J. Cosmol. Astropart. Phys._ 01, 012, DOI: 10.1088/1475-7516/2021/01/012 (2021).  2007.04241.
  * [167] Kite, T., Ravenni, A., Patil, S. P. & Chluba, J.  Bridging the gap: spectral distortions meet gravitational waves.  _Mon. Not. Roy. Astron. Soc._ 505, 4396, DOI: 10.1093/mnras/stab1558 (2021).  2010.00040.
  * [168] Bailes, M. _et al._ Gravitational-wave physics and astronomy in the 2020s and 2030s.  _Nature Rev. Phys._ 3, 344–366, DOI: 10.1038/s42254-021-00303-8 (2021).
  * [169] Chluba, J. _et al._ New horizons in cosmology with spectral distortions of the cosmic microwave background.  _Exper. Astron._ 51, 1515–1554, DOI: 10.1007/s10686-021-09729-5 (2021).  1909.01593.
  * [170] Starobinsky, A. A.  A New Type of Isotropic Cosmological Models Without Singularity.  _Phys. Lett._ 91B, 99–102, DOI: 10.1016/0370-2693(80)90670-X (1980).
  * [171] Watanabe, Y. & Komatsu, E.  Improved Calculation of the Primordial Gravitational Wave Spectrum in the Standard Model.  _Phys. Rev. D_ 73, 123515, DOI: 10.1103/PhysRevD.73.123515 (2006).  astro-ph/0604176.
  * [172] Saikawa, K. & Shirai, S.  Primordial gravitational waves, precisely: The role of thermodynamics in the Standard Model.  _J. Cosmol. Astropart. Phys._ 05, 035, DOI: 10.1088/1475-7516/2018/05/035 (2018).  1803.01038.
  * [173] Kite, T., Chluba, J., Ravenni, A. & Patil, S. P.  Clarifying transfer function approximations for the large-scale gravitational wave background in $\Lambda$CDM.  _Mon. Not. Roy. Astron. Soc._ 509, 1366–1376, DOI: 10.1093/mnras/stab3125 (2021).  2107.13351.
  * [174] Gertsenshtein, M. E. & Pustovoit, V. I.  On the Detection of Low Frequency Gravitational Waves.  _Sov. Phys. JETP_ 16, 433 (1962).
  * [175] Moss, G. E., Miller, L. R. & Forward, R. L.  Photon-noise-limited laser transducer for gravitational antenna.  _Appl. Opt._ 10, 2495–2498, DOI: 10.1364/AO.10.002495 (1971).
  * [176] Abbott, B. P. _et al._ Observation of Gravitational Waves from a Binary Black Hole Merger.  _Phys. Rev. Lett._ 116, 061102, DOI: 10.1103/PhysRevLett.116.061102 (2016).  1602.03837.
  * [177] Amaro-Seoane, P. _et al._ Laser Interferometer Space Antenna.  _arXiv preprint_ (2017).  1702.00786.
  * [178] Sesana, A. _et al._ Unveiling the gravitational universe at $\mu$-Hz frequencies.  _Exper. Astron._ 51, 1333–1383, DOI: 10.1007/s10686-021-09709-9 (2021).  1908.11391.
  * [179] Baibhav, V. _et al._ Probing the nature of black holes: Deep in the mHz gravitational-wave sky.  _Exper. Astron._ 51, 1385–1416, DOI: 10.1007/s10686-021-09741-9 (2021).  1908.11390.
  * [180] Sedda, M. A. _et al._ The missing link in gravitational-wave astronomy: A summary of discoveries waiting in the decihertz range.  _Exper. Astron._ 51, 1427–1440, DOI: 10.1007/s10686-021-09713-z (2021).  2104.14583.
  * [181] Bertoldi, A. _et al._ AEDGE: Atomic experiment for dark matter and gravity exploration in space.  _Exper. Astron._ 51, 1417–1426, DOI: 10.1007/s10686-021-09701-3 (2021).
  * [182] Arzoumanian, Z. _et al._ The NANOGrav 12.5 yr Data Set: Search for an Isotropic Stochastic Gravitational-wave Background.  _Astrophys. J. Lett._ 905, L34, DOI: 10.3847/2041-8213/abd401 (2020).  2009.04496.
  * [183] Antoniadis, J. _et al._ The International Pulsar Timing Array second data release: Search for an isotropic gravitational wave background.  _Mon. Not. Roy. Astron. Soc._ 510, 4873–4887, DOI: 10.1093/mnras/stab3418 (2022).  2201.03980.
  * [184] Moore, C. J., Mihaylov, D. P., Lasenby, A. & Gilmore, G.  Astrometric Search Method for Individually Resolvable Gravitational Wave Sources with Gaia.  _Phys. Rev. Lett._ 119, 261102, DOI: 10.1103/PhysRevLett.119.261102 (2017).  1707.06239.
  * [185] Seljak, U.  Measuring polarization in cosmic microwave background.  _Astrophys. J._ 482, 6, DOI: 10.1086/304123 (1997).  astro-ph/9608131.
  * [186] Weinberg, S.  _Gravitation and Cosmology: Principles and Applications of the General Theory of Relativity_ (John Wiley and Sons, New York, 1972).
  * [187] Nariai, H. & Tomita, K.  On the removal of initial singularity in a big-bang universe in terms of a renormalized theory of gravitation. 2. criteria for obtaining a physically reasonable model.  _Prog. Theor. Phys._ 46, 776–786, DOI: 10.1143/PTP.46.776 (1971).
  * [188] Komatsu, E. & Futamase, T.  Complete constraints on a nonminimally coupled chaotic inflationary scenario from the cosmic microwave background.  _Phys. Rev. D_ 59, 064029, DOI: 10.1103/PhysRevD.59.064029 (1999).  astro-ph/9901127.
  * [189] Futamase, T. & Maeda, K.-i.  Chaotic Inflationary Scenario in Models Having Nonminimal Coupling With Curvature.  _Phys. Rev. D_ 39, 399–404, DOI: 10.1103/PhysRevD.39.399 (1989).
  * [190] Fakir, R. & Unruh, W. G.  Improvement on cosmological chaotic inflation through nonminimal coupling.  _Phys. Rev. D_ 41, 1783–1791, DOI: 10.1103/PhysRevD.41.1783 (1990).
  * [191] Bezrukov, F. L. & Shaposhnikov, M.  The Standard Model Higgs boson as the inflaton.  _Phys. Lett. B_ 659, 703–706, DOI: 10.1016/j.physletb.2007.11.072 (2008).  0710.3755.
  * [192] Spokoiny, B. L.  Inflation and Generation of Perturbations in Broken Symmetric Theory of Gravity.  _Phys. Lett. B_ 147, 39–43, DOI: 10.1016/0370-2693(84)90587-2 (1984).
  * [193] Accetta, F. S., Zoller, D. J. & Turner, M. S.  Induced Gravity Inflation.  _Phys. Rev. D_ 31, 3046, DOI: 10.1103/PhysRevD.31.3046 (1985).
  * [194] Fakir, R. & Unruh, W. G.  Induced gravity inflation.  _Phys. Rev. D_ 41, 1792–1795, DOI: 10.1103/PhysRevD.41.1792 (1990).
  * [195] Weinberg, S.  Damping of tensor modes in cosmology.  _Phys. Rev. D_ 69, 023503, DOI: 10.1103/PhysRevD.69.023503 (2004).  astro-ph/0306304.
  * [196] Maldacena, J. M.  Non-Gaussian features of primordial fluctuations in single field inflationary models.  _J. High Energy Phys._ 05, 013, DOI: 10.1088/1126-6708/2003/05/013 (2003).  astro-ph/0210603.
  * [197] Komatsu, E. _et al._ First year Wilkinson Microwave Anisotropy Probe (WMAP) observations: tests of gaussianity.  _Astrophys. J. Suppl._ 148, 119–134, DOI: 10.1086/377220 (2003).  astro-ph/0302223.
  * [198] Komatsu, E.  Hunting for Primordial Non-Gaussianity in the Cosmic Microwave Background.  _Class. Quant. Grav._ 27, 124010, DOI: 10.1088/0264-9381/27/12/124010 (2010).  1003.6097.
  * [199] Planck Collaboration IX.  Planck 2018 results. IX. Constraints on primordial non-Gaussianity.  _Astron. Astrophys._ 641, A9, DOI: 10.1051/0004-6361/201935891 (2020).  1905.05697.
  * [200] Hinshaw, G. _et al._ Nine-Year Wilkinson Microwave Anisotropy Probe (WMAP) Observations: Cosmological Parameter Results.  _Astrophys. J. Suppl._ 208, 19, DOI: 10.1088/0067-0049/208/2/19 (2013).  1212.5226.
  * [201] Lozanov, K. D.  Lectures on Reheating after Inflation.  _arXiv preprint_ (2019).  1907.04402.
  * [202] Kodama, H. & Sasaki, M.  Cosmological Perturbation Theory.  _Prog. Theor. Phys. Suppl._ 78, 1–166, DOI: 10.1143/PTPS.78.1 (1984).
  * [203] Tomita, K.  Non-Linear Theory of Gravitational Instability in the Expanding Universe.  _Progress of Theoretical Physics_ 37, 831–846, DOI: 10.1143/PTP.37.831 (1967).
  * [204] Matarrese, S., Mollerach, S. & Bruni, M.  Second order perturbations of the Einstein-de Sitter universe.  _Phys. Rev. D_ 58, 043504, DOI: 10.1103/PhysRevD.58.043504 (1998).  astro-ph/9707278.
  * [205] Ananda, K. N., Clarkson, C. & Wands, D.  The Cosmological gravitational wave background from primordial density perturbations.  _Phys. Rev. D_ 75, 123518, DOI: 10.1103/PhysRevD.75.123518 (2007).  gr-qc/0612013.
  * [206] Baumann, D., Steinhardt, P. J., Takahashi, K. & Ichiki, K.  Gravitational Wave Spectrum Induced by Primordial Scalar Perturbations.  _Phys. Rev. D_ 76, 084019, DOI: 10.1103/PhysRevD.76.084019 (2007).  hep-th/0703290.
  * [207] Carney, D., Fischler, W., Kovetz, E. D., Lorshbough, D. & Paban, S.  Rapid field excursions and the inflationary tensor spectrum.  _J. High Energy Phys._ 11, 042, DOI: 10.1007/JHEP11(2012)042 (2012).  1209.3848.
  * [208] Cook, J. L. & Sorbo, L.  Particle production during inflation and gravitational waves detectable by ground-based interferometers.  _Phys. Rev. D_ 85, 023534, DOI: 10.1103/PhysRevD.85.023534 (2012).  [Erratum: Phys.Rev.D 86, 069901 (2012)], 1109.0022.
  * [209] Biagetti, M., Fasiello, M. & Riotto, A.  Enhancing Inflationary Tensor Modes through Spectator Fields.  _Phys. Rev._ D88, 103518, DOI: 10.1103/PhysRevD.88.103518 (2013).  1305.7241.
  * [210] Senatore, L., Silverstein, E. & Zaldarriaga, M.  New Sources of Gravitational Waves during Inflation.  _J. Cosmol. Astropart. Phys._ 1408, 016, DOI: 10.1088/1475-7516/2014/08/016 (2014).  1109.0542.
  * [211] Cai, Y.-F., Jiang, J., Sasaki, M., Vardanyan, V. & Zhou, Z.  Beating the Lyth Bound by Parametric Resonance during Inflation.  _Phys. Rev. Lett._ 127, 251301, DOI: 10.1103/PhysRevLett.127.251301 (2021).  2105.12554.
  * [212] Martineau, P. & Brandenberger, R.  A Back-reaction Induced Lower Bound on the Tensor-to-Scalar Ratio.  _Mod. Phys. Lett. A_ 23, 727–735, DOI: 10.1142/S0217732308026777 (2008).  0709.2671.
  * [213] Brandenberger, R. H., Nayeri, A., Patil, S. P. & Vafa, C.  Tensor Modes from a Primordial Hagedorn Phase of String Cosmology.  _Phys. Rev. Lett._ 98, 231302, DOI: 10.1103/PhysRevLett.98.231302 (2007).  hep-th/0604126.
  * [214] Brandenberger, R. H., Nayeri, A., Patil, S. P. & Vafa, C.  String gas cosmology and structure formation.  _Int. J. Mod. Phys. A_ 22, 3621–3642, DOI: 10.1142/S0217751X07037159 (2007).  hep-th/0608121.
  * [215] Domènech, G.  Scalar Induced Gravitational Waves Review.  _Universe_ 7, 398, DOI: 10.3390/universe7110398 (2021).  2109.01398.
  * [216] Barnaby, N. _et al._ Gravity waves and non-Gaussian features from particle production in a sector gravitationally coupled to the inflaton.  _Phys. Rev. D_ 86, 103508, DOI: 10.1103/PhysRevD.86.103508 (2012).  1206.6117.
  * [217] Ratra, B.  Cosmological ‘seed’ magnetic field from inflation.  _Astrophys. J. Lett._ 391, L1–L4, DOI: 10.1086/186384 (1992).
  * [218] Demozzi, V., Mukhanov, V. & Rubinstein, H.  Magnetic fields from inflation?  _J. Cosmol. Astropart. Phys._ 08, 025, DOI: 10.1088/1475-7516/2009/08/025 (2009).  0907.1030.
  * [219] Barnaby, N., Namba, R. & Peloso, M.  Observable non-gaussianity from gauge field production in slow roll inflation, and a challenging connection with magnetogenesis.  _Phys. Rev. D_ 85, 123523, DOI: 10.1103/PhysRevD.85.123523 (2012).  1202.1469.
  * [220] Maleknejad, A., Sheikh-Jabbari, M. & Soda, J.  Gauge Fields and Inflation.  _Phys. Rept._ 528, 161–261, DOI: 10.1016/j.physrep.2013.03.003 (2013).  1212.2921.
  * [221] Kobayashi, T. & Afshordi, N.  Schwinger Effect in 4D de Sitter Space and Constraints on Magnetogenesis in the Early Universe.  _J. High Energy Phys._ 10, 166, DOI: 10.1007/JHEP10(2014)166 (2014).  1408.4141.
  * [222] Namba, R., Peloso, M., Shiraishi, M., Sorbo, L. & Unal, C.  Scale-dependent gravitational waves from a rolling axion.  _J. Cosmol. Astropart. Phys._ 01, 041, DOI: 10.1088/1475-7516/2016/01/041 (2016).  1509.07521.
  * [223] Shiraishi, M., Hikage, C., Namba, R., Namikawa, T. & Hazumi, M.  Testing statistics of the CMB B -mode polarization toward unambiguously establishing quantum fluctuation of the vacuum.  _Phys. Rev. D_ 94, 043506, DOI: 10.1103/PhysRevD.94.043506 (2016).  1606.06082.
  * [224] Barnaby, N., Namba, R. & Peloso, M.  Phenomenology of a Pseudo-Scalar Inflaton: Naturally Large Nongaussianity.  _J. Cosmol. Astropart. Phys._ 04, 009, DOI: 10.1088/1475-7516/2011/04/009 (2011).  1102.4333.
  * [225] Seto, N.  Prospects for direct detection of circular polarization of gravitational-wave background.  _Phys. Rev. Lett._ 97, 151101, DOI: 10.1103/PhysRevLett.97.151101 (2006).  astro-ph/0609504.
  * [226] Seto, N. & Taruya, A.  Measuring a Parity Violation Signature in the Early Universe via Ground-based Laser Interferometers.  _Phys. Rev. Lett._ 99, 121101, DOI: 10.1103/PhysRevLett.99.121101 (2007).  0707.0535.
  * [227] Crowder, S. G., Namba, R., Mandic, V., Mukohyama, S. & Peloso, M.  Measurement of Parity Violation in the Early Universe using Gravitational-wave Detectors.  _Phys. Lett. B_ 726, 66–71, DOI: 10.1016/j.physletb.2013.08.077 (2013).  1212.4165.
  * [228] Barnaby, N. & Peloso, M.  Large Nongaussianity in Axion Inflation.  _Phys. Rev. Lett._ 106, 181301, DOI: 10.1103/PhysRevLett.106.181301 (2011).  1011.1500.
  * [229] Cook, J. L. & Sorbo, L.  An inflationary model with small scalar and large tensor nongaussianities.  _J. Cosmol. Astropart. Phys._ 11, 047, DOI: 10.1088/1475-7516/2013/11/047 (2013).  1307.7077.
  * [230] Bartolo, N. _et al._ Science with the space-based interferometer LISA. IV: Probing inflation with gravitational waves.  _J. Cosmol. Astropart. Phys._ 12, 026, DOI: 10.1088/1475-7516/2016/12/026 (2016).  1610.06481.
  * [231] Kim, J. & Komatsu, E.  Limits on anisotropic inflation from the Planck data.  _Phys. Rev. D_ 88, 101301, DOI: 10.1103/PhysRevD.88.101301 (2013).  1310.1605.
  * [232] Maleknejad, A. & Sheikh-Jabbari, M. M.  Non-Abelian Gauge Field Inflation.  _Phys. Rev. D_ 84, 043515, DOI: 10.1103/PhysRevD.84.043515 (2011).  1102.1932.
  * [233] Maleknejad, A. & Sheikh-Jabbari, M. M.  Gauge-flation: Inflation From Non-Abelian Gauge Fields.  _Phys. Lett._ B723, 224–228, DOI: 10.1016/j.physletb.2013.05.001 (2013).  1102.1513.
  * [234] Adshead, P. & Wyman, M.  Chromo-Natural Inflation: Natural inflation on a steep potential with classical non-Abelian gauge fields.  _Phys. Rev. Lett._ 108, 261302, DOI: 10.1103/PhysRevLett.108.261302 (2012).  1202.2366.
  * [235] Adshead, P. & Wyman, M.  Gauge-flation trajectories in Chromo-Natural Inflation.  _Phys. Rev. D_ 86, 043530, DOI: 10.1103/PhysRevD.86.043530 (2012).  1203.2264.
  * [236] Sheikh-Jabbari, M. M.  Gauge-flation Vs Chromo-Natural Inflation.  _Phys. Lett. B_ 717, 6–9, DOI: 10.1016/j.physletb.2012.09.014 (2012).  1203.2265.
  * [237] Dall’Agata, G.  Chromo-Natural inflation in Supergravity.  _Phys. Lett. B_ 782, 139–142, DOI: 10.1016/j.physletb.2018.05.020 (2018).  1804.03104.
  * [238] McDonough, E. & Alexander, S.  Observable Chiral Gravitational Waves from Inflation in String Theory.  _J. Cosmol. Astropart. Phys._ 11, 030, DOI: 10.1088/1475-7516/2018/11/030 (2018).  1806.05684.
  * [239] Agrawal, P., Fan, J. & Reece, M.  Clockwork Axions in Cosmology: Is Chromonatural Inflation Chrononatural?  _J. High Energy Phys._ 10, 193, DOI: 10.1007/JHEP10(2018)193 (2018).  1806.09621.
  * [240] Holland, J., Zavala, I. & Tasinato, G.  On chromonatural inflation in string theory.  _J. Cosmol. Astropart. Phys._ 12, 026, DOI: 10.1088/1475-7516/2020/12/026 (2020).  2009.00653.
  * [241] Maleknejad, A.  Chiral Anomaly in SU(2)R-Axion Inflation and the New Prediction for Particle Cosmology.  _J. High Energy Phys._ 21, 113, DOI: 10.1007/JHEP06(2021)113 (2020).  2103.14611.
  * [242] Maleknejad, A.  SU(2)R and its axion in cosmology: A common origin for inflation, cold sterile neutrinos, and baryogenesis.  _Phys. Rev. D_ 104, 083518, DOI: 10.1103/PhysRevD.104.083518 (2021).  2012.11516.
  * [243] Maleknejad, A. & Erfani, E.  Chromo-Natural Model in Anisotropic Background.  _J. Cosmol. Astropart. Phys._ 03, 016, DOI: 10.1088/1475-7516/2014/03/016 (2014).  1311.3361.
  * [244] Wolfson, I., Maleknejad, A. & Komatsu, E.  How attractive is the isotropic attractor solution of axion-SU(2) inflation?  _J. Cosmol. Astropart. Phys._ 09, 047, DOI: 10.1088/1475-7516/2020/09/047 (2020).  2003.01617.
  * [245] Wolfson, I., Maleknejad, A., Murata, T., Komatsu, E. & Kobayashi, T.  The isotropic attractor solution of axion-SU(2) inflation: universal isotropization in Bianchi type-I geometry.  _J. Cosmol. Astropart. Phys._ 09, 031, DOI: 10.1088/1475-7516/2021/09/031 (2021).  2105.06259.
  * [246] Adshead, P. & Liu, A.  Anisotropic Massive Gauge-flation.  _J. Cosmol. Astropart. Phys._ 07, 052, DOI: 10.1088/1475-7516/2018/07/052 (2018).  1803.07168.
  * [247] Adshead, P., Martinec, E. & Wyman, M.  Gauge fields and inflation: Chiral gravitational waves, fluctuations, and the Lyth bound.  _Phys. Rev._ D88, 021302, DOI: 10.1103/PhysRevD.88.021302 (2013).  1301.2598.
  * [248] Dimastrogiovanni, E. & Peloso, M.  Stability analysis of chromo-natural inflation and possible evasion of Lyth’s bound.  _Phys. Rev._ D87, 103501, DOI: 10.1103/PhysRevD.87.103501 (2013).  1212.5184.
  * [249] Maleknejad, A. & Komatsu, E.  Production and Backreaction of Spin-2 Particles of $SU(2)$ Gauge Field during Inflation.  _J. High Energy Phys._ 05, 174, DOI: 10.1007/JHEP05(2019)174 (2019).  1808.09076.
  * [250] Adshead, P., Martinec, E. & Wyman, M.  Perturbations in Chromo-Natural Inflation.  _J. High Energy Phys._ 09, 087, DOI: 10.1007/JHEP09(2013)087 (2013).  1305.2930.
  * [251] Agrawal, A., Fujita, T. & Komatsu, E.  Large tensor non-Gaussianity from axion-gauge field dynamics.  _Phys. Rev._ D97, 103526, DOI: 10.1103/PhysRevD.97.103526 (2018).  1707.03023.
  * [252] Agrawal, A., Fujita, T. & Komatsu, E.  Tensor Non-Gaussianity from Axion-Gauge-Fields Dynamics : Parameter Search.  _J. Cosmol. Astropart. Phys._ 1806, 027, DOI: 10.1088/1475-7516/2018/06/027 (2018).  1802.09284.
  * [253] Dimastrogiovanni, E. _et al._ Non-Gaussianity from Axion-Gauge Fields Interactions during Inflation.  _J. Cosmol. Astropart. Phys._ 11, 029, DOI: 10.1088/1475-7516/2018/11/029 (2018).  1806.05474.
  * [254] Fujita, T., Namba, R. & Obata, I.  Mixed Non-Gaussianity from Axion-Gauge Field Dynamics.  _J. Cosmol. Astropart. Phys._ 04, 044, DOI: 10.1088/1475-7516/2019/04/044 (2019).  1811.12371.
  * [255] Fujita, T., Murai, K., Obata, I. & Shiraishi, M.  Gravitational wave trispectrum in the axion-SU(2) model.  _J. Cosmol. Astropart. Phys._ 01, 007, DOI: 10.1088/1475-7516/2022/01/007 (2022).  2109.06457.
  * [256] Alexander, S. & Yunes, N.  Chern-Simons Modified General Relativity.  _Phys. Rept._ 480, 1–55, DOI: 10.1016/j.physrep.2009.07.002 (2009).  0907.2562.
  * [257] Saito, S., Ichiki, K. & Taruya, A.  Probing polarization states of primordial gravitational waves with CMB anisotropies.  _J. Cosmol. Astropart. Phys._ 09, 002, DOI: 10.1088/1475-7516/2007/09/002 (2007).  0705.3701.
  * [258] Contaldi, C. R., Magueijo, J. & Smolin, L.  Anomalous CMB polarization and gravitational chirality.  _Phys. Rev. Lett._ 101, 141101, DOI: 10.1103/PhysRevLett.101.141101 (2008).  0806.3082.
  * [259] Mirzagholi, L., Komatsu, E., Lozanov, K. D. & Watanabe, Y.  Effects of Gravitational Chern-Simons during Axion-SU(2) Inflation.  _J. Cosmol. Astropart. Phys._ 06, 024, DOI: 10.1088/1475-7516/2020/06/024 (2020).  2003.05931.
  * [260] Namba, R., Dimastrogiovanni, E. & Peloso, M.  Gauge-flation confronted with Planck.  _J. Cosmol. Astropart. Phys._ 11, 045, DOI: 10.1088/1475-7516/2013/11/045 (2013).  1308.1366.
  * [261] Maleknejad, A.  Axion Inflation with an SU(2) Gauge Field: Detectable Chiral Gravity Waves.  _J. High Energy Phys._ 07, 104, DOI: 10.1007/JHEP07(2016)104 (2016).  1604.03327.
  * [262] Caldwell, R. R. & Devulder, C.  Axion Gauge Field Inflation and Gravitational Leptogenesis: A Lower Bound on B Modes from the Matter-Antimatter Asymmetry of the Universe.  _Phys. Rev. D_ 97, 023532, DOI: 10.1103/PhysRevD.97.023532 (2018).  1706.03765.
  * [263] Watanabe, Y. & Komatsu, E.  Gravitational Wave from Axion-SU(2) Gauge Fields: Effective Field Theory for Kinetically Driven Inflation.  _arXiv preprint_ (2020).  2004.04350.
  * [264] Dimastrogiovanni, E., Fasiello, M. & Fujita, T.  Primordial Gravitational Waves from Axion-Gauge Fields Dynamics.  _J. Cosmol. Astropart. Phys._ 1701, 019, DOI: 10.1088/1475-7516/2017/01/019 (2017).  1608.04216.
  * [265] Iarygina, O. & Sfakianakis, E. I.  Gravitational waves from spectator Gauge-flation.  _J. Cosmol. Astropart. Phys._ 11, 023, DOI: 10.1088/1475-7516/2021/11/023 (2021).  2105.06972.
  * [266] Nieto, C. M. & Rodriguez, Y.  Massive Gauge-flation.  _Mod. Phys. Lett. A_ 31, 1640005, DOI: 10.1142/S0217732316400058 (2016).  1602.07197.
  * [267] Adshead, P., Martinec, E., Sfakianakis, E. I. & Wyman, M.  Higgsed Chromo-Natural Inflation.  _J. High Energy Phys._ 12, 137, DOI: 10.1007/JHEP12(2016)137 (2016).  1609.04025.
  * [268] Adshead, P. & Sfakianakis, E. I.  Higgsed Gauge-flation.  _J. High Energy Phys._ 08, 130, DOI: 10.1007/JHEP08(2017)130 (2017).  1705.03024.
  * [269] Obata, I., Miura, T. & Soda, J.  Chromo-Natural Inflation in the Axiverse.  _Phys. Rev. D_ 92, 063516, DOI: 10.1103/PhysRevD.92.063516 (2015).  [Addendum: Phys.Rev.D 95, 109902 (2017)], 1412.7620.
  * [270] Obata, I. & Soda, J.  Chiral primordial gravitational waves from dilaton induced delayed chromonatural inflation.  _Phys. Rev._ D93, 123502, DOI: 10.1103/PhysRevD.93.123502 (2016).  1602.06024.
  * [271] Domcke, V., Mares, B., Muia, F. & Pieroni, M.  Emerging chromo-natural inflation.  _J. Cosmol. Astropart. Phys._ 04, 034, DOI: 10.1088/1475-7516/2019/04/034 (2019).  1807.03358.
  * [272] Fujita, T., Sfakianakis, E. & Shiraishi, M.  Tensor Spectra Templates for Axion-Gauge Fields Dynamics during Inflation.  _J. Cosmol. Astropart. Phys._ 05, 057, DOI: 10.1088/1475-7516/2019/05/057 (2019).  1812.03667.
  * [273] Schwinger, J. S.  On gauge invariance and vacuum polarization.  _Phys. Rev._ 82, 664–679, DOI: 10.1103/PhysRev.82.664 (1951).
  * [274] Lozanov, K. D., Maleknejad, A. & Komatsu, E.  Schwinger Effect by an $SU(2)$ Gauge Field during Inflation.  _J. High Energy Phys._ 02, 041, DOI: 10.1007/JHEP02(2019)041 (2019).  1805.09318.
  * [275] Domcke, V., Ema, Y., Mukaida, K. & Sato, R.  Chiral Anomaly and Schwinger Effect in Non-Abelian Gauge Theories.  _J. High Energy Phys._ 03, 111, DOI: 10.1007/JHEP03(2019)111 (2019).  1812.08021.
  * [276] Mirzagholi, L., Maleknejad, A. & Lozanov, K. D.  Production and backreaction of fermions from axion-$SU(2)$ gauge fields during inflation.  _Phys. Rev. D_ 101, 083528, DOI: 10.1103/PhysRevD.101.083528 (2020).  1905.09258.
  * [277] Maleknejad, A.  Dark Fermions and Spontaneous $CP$ violation in $SU(2)$-axion Inflation.  _J. High Energy Phys._ 07, 154, DOI: 10.1007/JHEP07(2020)154 (2020).  1909.11545.
  * [278] Fujita, T., Namba, R. & Tada, Y.  Does the detection of primordial gravitational waves exclude low energy inflation?  _Phys. Lett._ B778, 17–21, DOI: 10.1016/j.physletb.2017.12.014 (2018).  1705.01533.
  * [279] Ishiwata, K., Komatsu, E. & Obata, I.  Axion-gauge field dynamics with backreaction.  _J. Cosmol. Astropart. Phys._ 03, 010, DOI: 10.1088/1475-7516/2022/03/010 (2022).  2111.14429.
  * [280] Papageorgiou, A., Peloso, M. & Unal, C.  Nonlinear perturbations from the coupling of the inflaton to a non-Abelian gauge field, with a focus on Chromo-Natural Inflation.  _J. Cosmol. Astropart. Phys._ 09, 030, DOI: 10.1088/1475-7516/2018/09/030 (2018).  1806.08313.
  * [281] Papageorgiou, A., Peloso, M. & Unal, C.  Nonlinear perturbations from axion-gauge fields dynamics during inflation.  _J. Cosmol. Astropart. Phys._ 07, 004, DOI: 10.1088/1475-7516/2019/07/004 (2019).  1904.01488.
  * [282] Fujita, T., Nakatsuka, H., Mukaida, K. & Murai, K.  SU($N$)-natural inflation.  _arXiv preprint_ (2021).  2110.03228.
  * [283] Dahal, S. _et al._ Four-year Cosmology Large Angular Scale Surveyor (CLASS) Observations: On-sky Receiver Performance at 40, 90, 150, and 220 GHz Frequency Bands.  _Astrophys. J._ 926, 33, DOI: 10.3847/1538-4357/ac397c (2022).  2107.08022.
  * [284] Hu, W., Hedman, M. M. & Zaldarriaga, M.  Benchmark parameters for CMB polarization experiments.  _Phys. Rev. D_ 67, 043004, DOI: 10.1103/PhysRevD.67.043004 (2003).  astro-ph/0210096.
  * [285] Rosset, C. _et al._ Beam mismatch effects in cosmic microwave background polarization measurements.  _Astron. Astrophys._ 464, 405–415, DOI: 10.1051/0004-6361:20042230 (2007).  astro-ph/0410544.
  * [286] O’Dea, D., Challinor, A. & Johnson, B. R.  Systematic errors in cosmic microwave background polarization measurements.  _Mon. Not. Roy. Astron. Soc._ 376, 1767–1783, DOI: 10.1111/j.1365-2966.2007.11558.x (2007).  astro-ph/0610361.
  * [287] Shimon, M., Keating, B., Ponthieu, N. & Hivon, E.  CMB Polarization Systematics Due to Beam Asymmetry: Impact on Inflationary Science.  _Phys. Rev. D_ 77, 083003, DOI: 10.1103/PhysRevD.77.083003 (2008).  0709.1513.
  * [288] Bao, C. _et al._ The Impact of the Spectral Response of an Achromatic Half-Wave Plate on the Measurement of the Cosmic Microwave Background Polarization.  _Astrophys. J._ 747, 97, DOI: 10.1088/0004-637X/747/2/97 (2012).  1112.3057.
  * [289] Duivenvoorden, A. J., Adler, A. E., Billi, M., Dachlythra, N. & Gudmundsson, J. E.  Probing frequency-dependent half-wave plate systematics for CMB experiments with full-sky beam convolution simulations.  _Mon. Not. Roy. Astron. Soc._ 502, 4526–4539, DOI: 10.1093/mnras/stab317 (2021).  2012.10437.
  * [290] Giardiello, S. _et al._ Detailed study of HWP non-idealities and their impact on future measurements of CMB polarization anisotropies from space.  _Astron. Astrophys._ 658, A15, DOI: 10.1051/0004-6361/202141619 (2022).  2106.08031.
  * [291] Abitbol, M. H. _et al._ The Simons Observatory: gain, bandpass and polarization-angle calibration requirements for B-mode searches.  _J. Cosmol. Astropart. Phys._ 05, 032, DOI: 10.1088/1475-7516/2021/05/032 (2021).  2011.02449.
  * [292] Vielva, P. _et al._ Polarization angle requirements for CMB B-mode experiments. Application to the LiteBIRD satellite.  _arXiv preprint_ (2022).  2202.01324.
  * [293] Keating, B., Shimon, M. & Yadav, A.  Self-Calibration of CMB Polarization Experiments.  _Astrophys. J._ 762, L23, DOI: 10.1088/2041-8205/762/2/L23 (2013).  1211.5734.
  * [294] Krachmalnicoff, N. _et al._ In-flight polarization angle calibration for LiteBIRD: blind challenge and cosmological implications.  _J. Cosmol. Astropart. Phys._ 01, 039, DOI: 10.1088/1475-7516/2022/01/039 (2022).  2111.09140.
  * [295] Johnson, B. R. _et al._ A CubeSat for Calibrating Ground-Based and Sub-Orbital Millimeter-Wave Polarimeters (CalSat).  _J. Astron. Inst._ 04, 1550007, DOI: 10.1142/S2251171715500075 (2015).  1505.07033.
  * [296] Kaufman, J. P., Keating, B. G. & Johnson, B. R.  Precision Tests of Parity Violation over Cosmological Distances.  _Mon. Not. Roy. Astron. Soc._ 455, 1981–1988, DOI: 10.1093/mnras/stv2348 (2016).  1409.8242.
  * [297] Nati, F. _et al._ POLOCALC: a Novel Method to Measure the Absolute Polarization Orientation of the Cosmic Microwave Background.  _J. Astron. Inst._ 06, 1740008, DOI: 10.1142/S2251171717400086 (2017).  1704.02704.
  * [298] Casas, F. J. _et al._ L2-CalSat: A Calibration Satellite for Ultra-Sensitive CMB Polarization Space Missions.  _Sensors_ 21, 3361, DOI: 10.3390/s21103361 (2021).
  * [299] Pogosian, L., Shimon, M., Mewes, M. & Keating, B.  Future CMB constraints on cosmic birefringence and implications for fundamental physics.  _Phys. Rev. D_ 100, 023507, DOI: 10.1103/PhysRevD.100.023507 (2019).  1904.07855.
  * [300] Shiraishi, M.  Tensor Non-Gaussianity Search: Current Status and Future Prospects.  _Front. Astron. Space Sci._ 6, 49, DOI: 10.3389/fspas.2019.00049 (2019).  1905.12485.
  * [301] Duivenvoorden, A. J., Meerburg, P. D. & Freese, K.  CMB B-mode non-Gaussianity: optimal bispectrum estimator and Fisher forecasts.  _Phys. Rev. D_ 102, 023521, DOI: 10.1103/PhysRevD.102.023521 (2020).  1911.11349.
  * [302] Hiramatsu, T., Komatsu, E., Hazumi, M. & Sasaki, M.  Reconstruction of primordial tensor power spectra from B-mode polarization of the cosmic microwave background.  _Phys. Rev. D_ 97, 123511, DOI: 10.1103/PhysRevD.97.123511 (2018).  1803.00176.
  * [303] Campeti, P., Poletti, D. & Baccigalupi, C.  Principal component analysis of the primordial tensor power spectrum.  _J. Cosmol. Astropart. Phys._ 09, 055, DOI: 10.1088/1475-7516/2019/09/055 (2019).  1905.08200.
  * [304] Ford, L. H. & Parker, L.  Quantized Gravitational Wave Perturbations in Robertson-Walker Universes.  _Phys. Rev. D_ 16, 1601–1608, DOI: 10.1103/PhysRevD.16.1601 (1977).
  * [305] Birrell, N. D. & Davies, P. C. W.  _Quantum Fields in Curved Space_.  Cambridge Monographs on Mathematical Physics (Cambridge Univ. Press, Cambridge, UK, 1984).
  * [306] Mukhanov, V. F., Feldman, H. A. & Brandenberger, R. H.  Theory of cosmological perturbations. Part 1. Classical perturbations. Part 2. Quantum theory of perturbations. Part 3. Extensions.  _Phys. Rept._ 215, 203–333, DOI: 10.1016/0370-1573(92)90044-Z (1992).
  * [307] Lifshitz, E.  On the gravitational stability of the expanding universe.  _J. Phys. (USSR)_ 10, 116, DOI: 10.1007/s10714-016-2165-8 (1946).
  * [308] Kiefer, C., Polarski, D. & Starobinsky, A. A.  Quantum to classical transition for fluctuations in the early universe.  _Int. J. Mod. Phys. D_ 7, 455–462, DOI: 10.1142/S0218271898000292 (1998).  gr-qc/9802003.
  * [309] Komatsu, E.  _The pursuit of non-gaussian fluctuations in the cosmic microwave background_.  Ph.D. thesis, Tohoku U. (2001).  astro-ph/0206039.
  * [310] Grishchuk, L. P. & Sidorov, Y. V.  On the Quantum State of Relic Gravitons.  _Class. Quant. Grav._ 6, L161–L165, DOI: 10.1088/0264-9381/6/9/002 (1989).
  * [311] Grishchuk, L. P. & Sidorov, Y. V.  Squeezed quantum states of relic gravitons and primordial density fluctuations.  _Phys. Rev. D_ 42, 3413–3421, DOI: 10.1103/PhysRevD.42.3413 (1990).
  * [312] Polarski, D. & Starobinsky, A. A.  Semiclassicality and decoherence of cosmological perturbations.  _Class. Quant. Grav._ 13, 377–392, DOI: 10.1088/0264-9381/13/3/006 (1996).  gr-qc/9504030.
  * [313] Martin, J., Vennin, V. & Peter, P.  Cosmological Inflation and the Quantum Measurement Problem.  _Phys. Rev. D_ 86, 103524, DOI: 10.1103/PhysRevD.86.103524 (2012).  1207.2086.
  * [314] Allen, B., Flanagan, E. E. & Papa, M. A.  Is the squeezing of relic gravitational waves produced by inflation detectable?  _Phys. Rev. D_ 61, 024024, DOI: 10.1103/PhysRevD.61.024024 (2000).  gr-qc/9906054.
  * [315] Kanno, S. & Soda, J.  Detecting nonclassical primordial gravitational waves with Hanbury-Brown–Twiss interferometry.  _Phys. Rev. D_ 99, 084010, DOI: 10.1103/PhysRevD.99.084010 (2019).  1810.07604.
  * [316] Sachs, R. K. & Wolfe, A. M.  Perturbations of a cosmological model and angular variations of the microwave background.  _Astrophys. J._ 147, 73–90, DOI: 10.1007/s10714-007-0448-9 (1967).


11institutetext:  1Department of Computer Science, Drexel University, PA
2Department of Computer Science, Smith College, Massachusetts, MA
ek826@drexel.edu,imi25@drexel.edu,nhs39<EMAIL_ADDRESS>
# Generative Artificial Intelligence Reproducibility and Consensus

Edward Kim1 Isamu Isozaki1 Naomi Sirkin1 Michael Robson2

###### Abstract

We performed a billion locality sensitive hash comparisons between
artificially generated data samples to answer the critical question - can we
reproduce the results of generative AI models? Reproducibility is one of the
pillars of scientific research for verifiability, benchmarking, trust, and
transparency. Futhermore, we take this research to the next level by verifying
the “correctness” of generative AI output in a non-deterministic, trustless,
decentralized network. We generate millions of data samples from a variety of
open source diffusion and large language models and describe the procedures
and trade-offs between generating more verses less deterministic output.
Additionally, we analyze the outputs to provide empirical evidence of
different parameterizations of tolerance and error bounds for verification.
For our results, we show that with a majority vote between three independent
verifiers, we can detect image generated perceptual collisions in generated AI
with over 99.89% probability and less than 0.0267% chance of intra-class
collision. For large language models (LLMs), we are able to gain 100%
consensus using greedy methods or n-way beam searches to generate consensus
demonstrated on different LLMs. In the context of generative AI training, we
pinpoint and minimize the major sources of stochasticity and present gossip
and synchronization training techniques for verifiability. Thus, this work
provides a practical, solid foundation for AI verification, reproducibility,
and consensus for generative AI applications.

##  1 Introduction

Generative Artificial Intelligence (GenAI) represents one of the most
impactful and consumer pervasive advancements in artificial intelligence
technology in recent years. This form of AI is designed to learn the
distribution of the training data, and sample from the learned manifold to
create content, e.g. text, images, music, or other complex signals. It a
significant shift from traditional AI models that are primarily used to
analyze and interpret data typically seen in discriminative supervised tasks
such as classification or regression. This raises issues of scientific
reproducibility due to the statistical nature of generative AI.
Reproducibility is essential as it enables the independent verification of
results, ensuring that findings from a machine learning model or algorithm are
valid and reliable. This is also important for understanding of a model’s
behavior, its strengths, and weaknesses. Reproducibility allows for the
independent and accurate benchmarking and comparison of ML models through the
open sharing of datasets and code; however, given the current trajectory of
large ML models, understanding the issues and challenges around
reproducibility is increasingly important.

As a case in point, AI is growing exponentially in every aspect - in usage and
adoption, as well as in cost to train, model parameterization, data, and
compute. For example, ChatGPT reached 1 million users in just 5 days -a feat
that took 3.5 years for Netflix, and 2 years for Twitter. OpenAI’s GPT-3
required over over $12M USD in training costs, and the carbon footprint of
training this model was equivalent to the output of 126 danish homes for an
entire year [1]. This does not even cover the cost of obtaining and labeling
massive amounts of data. The cost, size, and compute necessary for GPT-4 is an
order of magnitude larger, and intractable to train or deploy by anyone except
for a handful of the largest industry players. However, advancements in model
quantization, hardware, and cloud computing have made it possible to train and
deploy increasingly complex models on consumer grade hardware. Thus, a
significant effort is underway in the field to democratize artificial
intelligence.

One of the main goals of the democratization of AI is to expand the benefits
of AI beyond a small group of elite researchers and companies, and to ensure
that everyone has access to the tools and resources they need to take
advantage of AI. While open source software is a core component, there is also
the need for “open source” hardware. This concept is not new and has been
explored in decentralized cloud networks. For example, BOINC [2], which stands
for Berkeley Open Infrastructure for Network Computing, is an open-source
software platform for distributed computing. It allows volunteers to donate
unused processing power from their personal computers to scientific research
projects. The system is designed to manage and utilize the computing resources
of thousands of volunteers across the globe, effectively creating a massive,
distributed supercomputer. This allows researchers to conduct large-scale
computations without the need for a dedicated, or centralized infrastructure.
Some well-known projects include SETI@home [3], which searches for
extraterrestrial intelligence, and Folding@home [4], which simulates protein
folding for disease research.

However, a decentralized cloud infrastructure must utilize trustless computing
principles that do not normally apply when dealing with a known entity. In a
decentralized ecosystem, you do not know who you are interacting with, and the
dangers of malicious or adversarial actors increases by orders of magnitude.
To address these issues, the contributions of this work are the following. We
investigate and present techniques to scale up the computational capabilities
of a decentralized network. We present algorithms that can execute ML tasks
and monitor other nodes for fraudulent output and, in our results, provide
empirical evidence of different parameterizations of tolerance and error
bounds for verification. In the context of generative AI training, we pinpoint
and minimize the major sources of stochasticity and present gossip and
synchronization training techniques for verifiability. In essence, this work
demonstrates that we can reproduce and verify generative AI work in both image
and language generation with minimal overhead and extremely high precision in
a decentralized, trustless machine learning network.

##  2 Background

###  2.1 Background in Generative AI

Algorithmically, deep learning currently dominates nearly all applications of
artificial intelligence and machine learning, and has shown incredible success
in the past several years. These improvements can be attributed to multiple
factors, where two major contributors were access to large amounts of data,
and large amounts of compute power. Today, training and running these models
requires an enormous amount of compute, usually accelerated in cloud
infrastructure using high-powered GPU hardware.

Since the introduction of Generative Adversarial Networks (GANs) [5],
generative AI has made remarkable strides. Today, it’s used in a wide range of
applications, from creating realistic imagery and generating art to
synthesizing high-quality speech and writing. Stable Diffusion [6], is a deep
learning, text-to-image model primarily used to generate detailed images
conditioned on text descriptions. It uses a latent diffusion model, which
involves training the model to remove successive applications of Gaussian
noise on training images, functioning as a sequence of denoising autoencoders.
The model consists of three parts: a variational autoencoder (VAE) [7], a
U-Net [8], and text encoder [9]. The VAE compresses an image from pixel space
to a smaller dimensional latent space, capturing a more semantic meaning of
the image. Gaussian noise is then iteratively applied to this compressed
latent representation during forward diffusion. The U-Net block, composed of a
ResNet [10] backbone, denoises the output from forward diffusion to obtain a
latent representation. The VAE decoder then generates the final image by
converting this representation back into pixel space. This process can be
conditioned on a string of text, an image, or another modality. Large language
models (LLMs) are another class of generative AI that have been trained on
large textual datasets. These models have shifted the focus of natural
language processing research away from training specialized supervised models
for specific tasks. Despite being trained on simple tasks such as predicting
the next word in a sentence, LLMs with sufficient training and parameter
counts capture much of the syntax and semantics of human language and
demonstrate considerable general knowledge within their training corpus.

###  2.2 Background in Decentralized Verification and Consensus

The field of machine learning verification is in its infancy. In the
traditional definition, verification involves making a compelling argument
that the system will not misbehave under a broad range of circumstances [11].
This involves the need to consider unusual inputs crafted by an adversary, not
just naturally occurring inputs as testing alone is insufficient to provide
security guarantees [12].

However, we are dealing with a nuanced type of verification of correctness. In
the context of a decentralized network, a consensus model is a mechanism that
ensures all participants in a distributed network agree on the content of a
shared database. It is the protocol by which the nodes in the network agree on
a single version of the truth, despite the presence of faulty nodes or those
with malicious intent. The consensus model is the core idea to minimize the
need for trust in a blockchain system, where consensus ensures that every
transaction is validated according to a set of agreed-upon rules. Unlike
traditional centralized systems where a single authority validates
transactions, in a blockchain, multiple nodes participate in the validation
process [13]. This decentralization enhances security and transparency but
requires the following properties to be effective [14]. (1) Agreement - all
honest nodes must agree on the same value. (2) Validity - if all honest nodes
propose the same value, they must decide on that value. (3) Termination -
every honest node must eventually reach a decision. (4) Integrity - a node
decides on a value at most once. In other words, once a node has made a
decision, it cannot change it. And (5) Fault Tolerance - the consensus model
should be able to function correctly even if some nodes fail or act
maliciously.

Consensus is not the only way to verify in a trustless system; you can also
use a cryptographic proof. A SNARK (Succinct Non-Interactive ARguments of
Knowledge) is a cryptographic primitive that allows one party, called the
prover, to convince another party, called the verifier, that a given statement
is true, without the verifier needing to perform the actual computation.
Cryptographic techniques like Zero-Knowledge Proofs (ZKPs) and verifiable
computing enable a party to prove that a computation was performed correctly
without revealing the details of the computation itself. These methods provide
strong guarantees of correctness while preserving privacy. Despite these
benefits, cryptographic methods are extremely computationally expensive, often
times adding 10,000x more work on the prover. While existing SNARKs exist and
have been demonstrated to work on smaller neural network models [15], their
proving times are unusable for the large diffusion and language models being
deployed today. Thus, reproducibility of machine learning goes beyond
scientific inquiry; rather it also has applications in consensus mechanisms to
verify generative machine learning tasks.

###  2.3 Why Reproducability in Generative AI is Hard

As stated previously, in order to have an effective consensus model, and
ultimately verify the correctness of machine learning tasks in a distributed,
decentralized system, honest nodes must come to agreement and must agree on
the same value. However, in the realm of machine learning and deep learning,
the parallel nature of the computations can introduce a slight non-
determinism. This is because floating-point operations are not exactly
associative, and when executed in parallel, the order in which they are
executed can vary. This can cause tiny differences in the results, which may
accumulate over time. Additionally, some GPUs and hardware accelerators have
built-in randomness. For example, the NVIDIA Tensor Cores use probabilistic
rounding [16], which can make results slightly different even if everything
else is kept constant.

Software-wise, machine learning frameworks often introduce non-determinism.
For example, PyTorch, like many other deep learning frameworks, involves
operations that can yield different results across multiple executions, even
when using identical seeds [17], see Figure 1. This non-deterministic behavior
can be attributed to factors such as the use of multi-threading, which can
lead to race conditions, or specific hardware and software configurations that
introduce variability. While you can limit the sources of non-deterministic
behavior or use deterministic algorithms instead of non-deterministic ones
where available, this often comes at the cost of performance. As a concrete
example, CUDA convolution operations, which use the cuDNN library, can be a
source of non-determinism. This is due to the benchmarking feature of cuDNN,
which can select different algorithms for convolution operations based on the
size parameters [18]. Disabling this feature can lead to more deterministic
but potentially slower performance.

Figure 1: Identical runs of the same seed and prompt to generative AI does not
yield the same bit level result, even on the same machine. Crytographic hashes
that are designed for exact matches exhibit the avalanche effect, while
perceptual hashes (aHash, pHash, dHash, cHash) are more stable and exhibit
locality sensitive hashing.

##  3 Methodology

Assuming a set of machine learning nodes are performing inference or training
work on generative AI, how do we know the machine learning node did what it
was supposed to do? Can we reproduce its results? Recall in the typical
deterministic consensus setting, multiple nodes can re-run the instructions
and check the results. This checking mechanism is typically performed by
computing a cryptographic hash of the output or state. If the instructions
were all run correctly, then any validator should have identical state hashes.
Any minor change in the state (like altering a single bit) leads to a
substantial change in the output hash, i.e. each of the output bits changes
with a 50% probability. This effect, avalanche effect, can be seen in common
hash functions such as the SHA-1 or MD5 hash function. If a single bit is
modified, the resulting hash sum becomes entirely different. This makes it
extremely difficult to predict the output of the hash function based on a
given input, which is a key aspect of its security.

###  3.1 Locality-Sensitive Hashing

In our case, we employ different algorithms that exhibit the Locality-
sensitive hashing (LSH) property that probabilistically groups similar input
items into the same “buckets”. Unlike traditional hashing techniques that aim
to minimize hash collisions, LSH intentionally maximizes them. This technique
can also be viewed as a method for reducing the dimensionality of high-
dimensional data, where high-dimensional input items are transformed into
lower-dimensional versions while maintaining the relative distances between
items. We utilize the following set of perceptual hashes (a hash string that
approximates the visual characteristics of an image), that are commonly used
in image hashing,

Average Hash (aHash): This is a type of perceptual hash that works by resizing
the image to a small, fixed resolution, converting it to grayscale,
calculating the mean pixel value, and then generating a hash based on whether
each pixel is above or below the mean.

Perceptual Hash (pHash): This is a more complex type of perceptual hash that
involves the Discrete Cosine Transform (DCT). The image is resized and
converted to grayscale, the DCT is applied, the top-left portion of the DCT
matrix (which represents the lowest frequencies) is retained, the mean value
is calculated excluding the first element, and a hash is generated based on
whether each value is above or below the mean.

Difference Hash (dHash): This is another type of perceptual hash that works by
comparing the relative gradients of the pixel values. The image is resized and
converted to grayscale, each pixel is compared to its neighbor, and a hash is
generated based on whether each pixel is greater than or less than its
neighbor.

Color Hash (cHash): This involves generating a hash based on the colors in an
image. It involves resizing the image to a small, fixed resolution and
generating a hash based on the quantized color values of each pixel.

These types of perceptual hashes are used in search by image, e.g. Google
images searching [19], or things like identifying songs with the same
fingerprint (Shazam) [20]. For our purposes, the perceptual hashes work well,
but there are some slight variations in the hash as shown by Figure 2.

[469] [361] [695] [168] [555] [695] (a) (b) (c) (d) (e) (f) (g)

Figure 2: Generated images from SD v1.5 on seven different GPUs (mix of
3060ti, 3070ti, 3080ti, and 3090) using the prompt, “A photo of {class}”,
where class is from the ImageNet dataset. The first three columns, class id
(469) [caldron, cauldron], (361) [skunk, polecat], and (695) [padlock] have
identical perceptual hashes. The last three rows, (168) [redbone], (555) [fire
engine, fire truck], (686) [oil filter] have perceptual hashes with the most
extreme hamming distances we observed in our generated data (up to 5). Visual
differences can be seen in the writing, the color of the truck, and the circle
in the oil filter.

###  3.2 Generative Image and Language Models

Our image generation experiments are centered around stable diffusion models
and fine-tuned variants. Stable diffusion supports the ability to generate new
images from scratch through the use of a text prompt describing elements to be
included or omitted from the output. The architecture of the model includes a
variational autoencoder (VAE), a U-Net, and text encoder. Gaussian noise is
iteratively applied to the compressed latent representation during forward
diffusion. The U-Net denoises the output from forward diffusion backwards to
obtain a latent representation. Finally, the VAE decoder generates the final
image by converting the representation back into pixel space.

Popular fine-tuning methods have been employed to personalize the diffusion
models, including fine-tuning all of the weights, or low rank and textual
fine-tuning methods.

####  3.2.1 Low-Rank Adapation

The LoRA, or Low-Rank Adaptation [21], proposes a new method for adapting
large-scale pre-trained language models to specific tasks or domains. Full
fine-tuning (retraining all model parameters) becomes less feasible as the
size of models continue to expand, i.e. fine-tuning the 175 billion parameters
in GPT-3 is prohibitively expensive.

Low-Rank Adaptation (LoRA) freezes the pre-trained model weights and
introduces trainable rank decomposition matrices into each layer of the
Transformer architecture. This approach significantly reduces the number of
trainable parameters for downstream tasks. LoRA can reduce the number of
trainable parameters by orders of magnitude. In the context of image
generation, LoRAs have been used to guide the diffusion process towards
particular concepts or visual representations.

####  3.2.2 Textual Inversion

Textual inversions [22] are an alternative approach to personalizing text-to-
image generation. Using only a small number of images (typically 3-5 images)
of a user-provided concept, such as an object or a style, inversions learn to
represent visual concepts through new “words” in the embedding space of a
frozen text-to-image model. These words can be composed into natural language
sentences, guiding personalized creation. Interestingly, a single word
embedding is sufficient for capturing unique and varied concepts.

###  3.3 Large Language Model Generation

Text generation is crucial for many NLP tasks, including open-ended text
generation, summarization, translation, and more. The text generation process,
known as decoding, can be customized to improve the quality of the generated
output and reduce repetition. However, in our case, our goal is to produce
quality output that can be verified by other nodes.

The generation configuration that yields deterministic results uses a simple
decoding strategy called greedy search, which picks the token with the highest
probability as the next token. For a more globally “aware” generation
strategy, we can also utilize beam-search decoding, which keeps several
hypotheses at each time step and eventually chooses the hypothesis that has
the overall highest probability for the entire sequence.

We minimize the stocasticity of the output by turning off stragegies such as
multinomial sampling which randomly selects the next token based on the
probability distribution over the entire vocabulary, and Beam-Search
Multinomial Sampling.

##  4 Experiments and Results

For our experiments, we set up a heterogeneous, decentralized GPU network
consisting of (8x) 3080ti GPUs running on 1x PCI-e risers, (4x) 3070ti GPUs
running on GPU risers, (4x) 3060ti GPUs, and two other machines each running a
single 3090 connected directly to the motherboard via PCI-e 16x slots. There
are an additional (4x) A40 Nvidia GPUs connected in an NLink configuration. In
total, the tasks described below were performed on a mix of 22 total GPUs.
Unless specified in the experiment, the GPUs were selected to perform a task
at random. Each machine was running in an Ubuntu 22 environment with CUDA 12
and torch 2.0.

We structure our results as follows: (1) We first compute the independent
likelihoods of determinism in several image and language models. (2) We then
use these likelihoods to compute an algorithm to verify machine learning
inference tasks on a decentralized network. (3) Next, we describe how to
verify in the training setting, and lastly (4) share the derived generated
dataset for reproducibility and to benefit the community at large.

###  4.1 Consensus Likelihoods of Image and Language Generation

In our first experiment, generate images from different variants of stable
diffusion [6], including the base v1.5 model, a fine-tuned version of stable
diffusion, LoRA or low rank adapatations [21], and textual inversions [22]. We
use a template “A photo of {}”, where the class variable inserted is a class
from the ILSVRC challenge [23]. We generate 7 images with identical prompts
and seeds, for a total of approximately 7000 images per image model. The
results of our generation can be seen in Table 1.

We compute four different types of perceptual hashes and compare the results.
Each class (group of 7 images) is hashed and compared with each other. The
mode hash is selected as the “correct” one, and the number of outliers are
computed across all thousand categories and reported as the outliers. We also
record the number of outliers when allowing for a tolerance of 1 (Ot$<$=1) or
2 (Ot$<=$2) hash discrepancies (hamming distance of 1 or 2). Additionally, we
report the average hamming distance (aDist) of the outliers and the time to
compute a single hash on a 512x512 image. While the color hash technique
yields the best accuracy results, the average hash demonstrates nearly the
same performance, yet with a speed up of approximately 3.5x. The consensus
percentages are computed by the number of outliers over the total number of
images hashed. This can be used as the likelihood that an image generated with
the same prompt and seed will generate the same perceptual hash.

Method | # Im. | Consensus | Outli. | Ot¡=1 | Ot¡=2 | aDist. | Time
---|---|---|---|---|---|---|---
Perceptual Hash |  |  |  |  |  |  |
Stable Diffusion (v1.5) [6] | 6985 | 92.9/ 92.9/ 99.7% | 496 | 496 | 21 | 2.104 | 0.0022s
Fine Tuned Diffusion [6] | 6997 | 82.5/ 82.5/ 96.7% | 1224 | 1224 | 233 | 2.615 | 0.0021s
Low Rank Adapatation [21] | 6928 | 76.9/ 76.9/ 94.1% | 1598 | 1598 | 410 | 2.871 | 0.0022s
Textual Inversion [22] | 7035 | 76.4/ 76.4/ 94.8% | 1657 | 1647 | 365 | 2.931 | 0.0022s
Difference Hash |  |  |  |  |  |  |
Stable Diffusion (v1.5) [6] | 6985 | 89.1/ 97.6/ 99.3% | 764 | 169 | 46 | 1.327 | 0.0020s
Fine Tuned Diffusion [6] | 6997 | 74.1/ 89.9/ 95.5% | 1815 | 708 | 312 | 1.772 | 0.0019s
Low Rank Adapatation [21] | 6928 | 65.5/ 84.9/ 92.4% | 2386 | 1042 | 525 | 1.940 | 0.0019s
Textual Inversion [22] | 7035 | 71.5/ 88.1/ 94.2% | 2008 | 840 | 407 | 2.052 | 0.0019s
Average Hash |  |  |  |  |  |  |
Stable Diffusion (v1.5) [6] | 6985 |  95.0/ 99.2/ 99.8% | 346 | 55 | 11 | 1.210 | 0.0019s
Fine Tuned Diffusion [6] | 6997 | 86.6/ 95.3/ 98.0% | 968 | 327 | 140 | 1.638 | 0.0019s
Low Rank Adapatation [21] | 6928 | 80.2/ 94.0/ 97.7% | 1372 | 412 | 157 | 1.545 | 0.0018s
Textual Inversion [22] | 7035 | 82.5/ 94.7/ 98.1% | 1229 | 370 | 137 | 1.666 | 0.0018s
Color Hash |  |  |  |  |  |  |
Stable Diffusion (v1.5) [6] | 6985 |  96.8/ 98.5/ 99.5% | 227 | 105 | 37 | 1.700 | 0.0074s
Fine Tuned Diffusion [6] | 6997 | 92.1/ 96.2/ 98.8% | 555 | 265 | 86 | 1.754 | 0.0073s
Low Rank Adapatation [21] | 6928 | 90.6/ 95.6/ 98.7% | 652 | 303 | 93 | 1.719 | 0.0082s
Textual Inversion [22] | 7035 | 89.2/ 94.6/ 98.2% | 759 | 379 | 124 | 1.760 | 0.0070s
Table 1: Generated images from different variants of stable diffusion [6],
including the base v1.5 model, a fine-tuned version of stable diffusion, LoRA
or low rank adapatations [21], and textual inversions [22]. Approximately 7
images with identical prompts and seeds were generated per 1000 classes in the
ImageNet class categories. The number of outliers to the majority hash in the
class is shown with tolerances of 0, 1 (Ot$<$=1), and 2 (Ot$<$=2). The average
distance of all outliers is shown as aDist, and the time per image hash is
presented in the last column.

For language generation, we sample several open source LLMs and run the same
prompt over four different machines and GPUs. The five LLMs tested were the
Wizard Vicuna 13B [24] in 4-bit quantizaiton mode, the Vicuna 7B [25] in 8-bit
quantization, Red Pajama 7B [26] in 8-bit quantization, the Red Pajama 3B
[26], and GPT-J 6B [27] in 8-bit quantization. We utilized the “instruct”
version of the LLM and provided a prompt, “### Human: Please write a
description about {name} ### Assistant:”, where the name comes from the 1000
ImageNet classes. A total of 4000 sentence generations were created across
different decoding techniques and beam searches as shown in Table 2. Even with
the GPU non-determinism, we saw no stocasticity in the LLM outputs when greedy
or n-beam methods were used to decode. When explicitly specifying multinomial
sampling where the model selects the next token based on the probability
distribution over the entire vocabulary, we do observe the expected non-
deterministic behavior. As a final note, we believe that although we did not
observe any mismatched tokens between GPUs, we believe that there is a non-
zero chance of a token mismatch in the greedy and n-beam case. The nature of
the floating point operations and error drift, with a special edge case
probability that is near the border of two words, could possibly flip the
generated next word; however, this would be an extremely rare edge case.

Method | # Gen. | Greedy(30) | Greedy(60) | Beam N=5 | Beam N=10 | Multin.
---|---|---|---|---|---|---
Wizard Vicuna 13B [24] | 4000 | Yes | Yes | Yes | Yes | No
Vicuna 7B [25] | 4000 | Yes | Yes | Yes | Yes | No
Red Pajama 7B [26] | 4000 | Yes | Yes | Yes | Yes | No
Red Pajama 3B [26] | 4000 | Yes | Yes | Yes | Yes | No
GPT-J 6B [27] | 4000 | Yes | Yes | Yes | Yes | No
Table 2: Generated 4,000 descriptions from different LLMs of different
parameter sizes. Different decoing strategies include greedy methods of 30 max
tokens and 60 max tokens, beam search of size 5 and 10, as well as multinomial
decoding. All greedy and beam search strategies were empirically deterministic
across different machines and GPUs. The multinomial case was non-deterministic
as expected. A “Yes” indicates observed deterministic behavior.

###  4.2 Tolerance in Reproducability

Given the independent likelihoods of determinism computed from the previous
experiment, we can now derive an machine learning algorithm for a
decentralized network. We provide posterior probabilities of detecting
incorrect or fraudulent behavior in the case that we assume the majority of
nodes in the network are honest (Figure 3(a)), or the case where we require a
super-majority (Figure 3(b)). We use, $P(X=k)=\binom{n}{k}p^{k}(1-p)^{n-k}$ ,
where the binomial distribution gives the probability of getting exactly $k$
successes (defined as generating the correct perceptual hash) over $n$
independent verifiers. The $P(X=k)$ is the probability of getting exactly k
verifications, where $p$ is the probability of generating a correct output
(derived from above).

Verification of Correctness - Type I Error \- For majority vote and tolerance
of 2, we can achieve 99.843%, 99.988%, and 99.999% verification with 3, 5, and
7 independent verifiers. For super majority (greater than 2/3s) and tolerance
of 2, we can achieve 99.692%, 99.960%, and 99.999% verification accuracy with
4, 7, and 10 independent verifiers; these probabilities demonstrate strong
verification with a minimal redudant work.

(a) Simple Majority

(b) 2/3 Super Majority

Figure 3: Graphs of probabilities that independent verifiers can spot
incorrect or fraudlent behavior given different likelihoods of deterministic
generation. In the simple majority case, we assume a majority of nodes are
honest, and in the super majority case, we have a stricter assumption that
over 2/3s are honest.

Another type of error is the accidental or malicious generation of a
perceptual hash without performing the task, or guessing a hash based upon the
given prompt. To assess the risk of this error, we generate over 1.3M images
using stable diffusion. Our simulation mimics a scenario where a malicious
actor sees the prompt, and tries to guess at a perceptual hash. Thus, these
guess images are generated using identical prompts as the one provided (a
photo of {}), but with a different seed.

Verification of Correctness - Type II Error \- For each category, we generated
1300 images (a total of 1.3M), and perform an all-to-all average hash
comparison and count collisions. The total number of hash comparisons here is
approximately 0.85 billion. The probability of an intra-class perceptual hash
collision is less than 0.0267%, when allowing for a hamming distance tolerance
of $<$=2. This indicates that there is nearly zero percent chance that an
adversary would be able to guess the perceptual hash - even with information
about the prompt given.

###  4.3 Verification of Generative Training

We now turn our attention to the case of generative AI training. In
particular, we focus on the textual inversion fine-tuning case; this is a
likely scenario in a distributed, heterogeneous GPU network. Textual Inversion
is a technique for capturing novel concepts from a small number of example
images. It learns new “words” in the text encoder’s embedding space, which are
used within text prompts for personalized image generation. Importantly, the
weights of the U-Net and VAE are frozen, thus, the only parameters that are
able to change and reduce the loss are the 768-dimensional word embedding.

Similar to the inference case, we would need the ability to perform a fraud
proof over training epochs. Thus, we need to identify the major sources of
controllable stocasticity and minimize them to allow for replication. Figure
4, identifies and minimizes the six controllable sources of randomness
including, horizontal flips of the data, random choice of the template,
training data shuffling, gaussian noise for the latents, random choice of
timestep, and the noise scheduler.

Figure 4: Six major sources of stocasticity in the fine-tuning process of
textual inversion. The randomness is minimized at each of these six points in
order to gain verifiability in the training process. Even with control over
these parameters via seeds or parameter settings, the process still remains
non-deterministic.

We remove these sources of stocasticity and train several dozens of textual
inversions randomly sampled from the huggingface concepts library
111https://huggingface.co/sd-concepts-library. For each concept, we present
5-10 exemplar images, and train the model with the following hyperparameters:
learning rate is 5e-04, maximum train steps is 2000, batch size is 4, gradient
accumulation is 1, and checkpoints are generated every 50 steps. The model is
trained in mixed 16-bit precision. Each textual inversion train is performed
six times. The first three times are deterministic runs - we minimize all
controllable sources of stochasticity. The next three runs are ablations over
the possible sources of randomness. For run 4, we allow horizontal flips in
the training process, for run 5 we allow the input dataset to be shuffled, and
for run 6, we do not set the seed for training and noise. We take the six runs
and perform PCA over the 768-dimensional space and project the checkpoint
embeddings onto principal components 1 and 2. The plots of the training
checkpoints of all six runs can be seen in Figure 5.

(a) $<$popcorn-toy$>$

(b) $<$kitchenrobot$>$

(c) $<$ugly-sonic$>$

(d) $<$clothes$>$

(e) $<$kirby$>$

(f) $<$car-toy$>$

Figure 5: Textual inversion training results from randomly selected objects in
the sd concepts library (https://huggingface.co/sd-concepts-library). The
graphs show six runs of 2000 iterations fine-tuning a new token based upon a
small set of images (3-10). Three runs minimize the amount of stocasticity
within the training process, “DeterR1,2,3”. “StocHF0.5” includes a random
horizontal flip of the image, “Shuffle” allows the dataset to be shuffled, and
“Noseed” does not provide a random seed for the noise. The deterministic runs
mostly overlap with some drift over the plot of the 768-dimensional vector
projected on the 2 principal components of the training data.  Figure 6:
Training in a textual inversion with resyncing of checkpoints every 300
iterations on the $<$kitchenrobot$>$ inversion. (1) Three training instances
(Sync Deter1,2,3) are introduced and synchronized to DeterR1 during training.
(2) At the start, there is some amount of variation within the training but
within an error bounds of all three Deterministic Runs. (3) By the end of
training 2000 iterations, the Synchronized trainings and Deterministic R1, are
nearly identical. The graph shows 4 overlapping vector projections from the
DeterR1 and Sync Deter1,2,and 3.

We observe that by minimizing the stocasticity leads to nearly deterministic
training. There are cases in Figure 5(b) and Figure 5(e) where the random
error begins to propagate and cause drift in the final textual embeddings. To
control for this, we demonstrate a gossip training procedure in Figure 6 where
the checkpoints between deterministic nodes can be synchronized every several
checkpoints (here we do every 6 checkpoints, or 300 epochs). Given the gossip
training mechanism, the end embedding weights of the verifiers follow the
trajectory of the trainer and provide much tighter error bounds.

##  5 Conclusion

In conclusion, we demonstrate that we are able to reproduce and verify the
correctness of machine learning tasks in a distributed, decentralized network.
We tackle a particularly challenging task in ML of verifying both the
inference and training of generative AI. We present the likelihoods of an
honest node producing an output that can be verified by other independent
nodes in the network and in the process generated millions of data samples and
billions of hash comparisons.

We provided empirical evidence of different parameterizations of tolerance and
error bounds for verification as the groundwork for a fraud proof in a
blockchain network. Our results show that with minimal overhead and extremely
high precision, we can verify generative AI work in both image and language
generation, and a malicious actor has close to zero percent chance of
exploiting the algorithm. The study also identified and minimized the major
sources of stochasticity in generative AI training and presented gossip and
synchronization training techniques for verifiability.

In summary, we provide a robust and practical foundation for AI verification,
reproducibility, and consensus, significantly reducing the need for trust in a
decentralized network.

## References

  * [1] Lasse F Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan.  Carbontracker: Tracking and predicting the carbon footprint of training deep learning models.  arXiv preprint arXiv:2007.03051, 2020.
  * [2] David P Anderson.  Boinc: A system for public-resource computing and storage.  In Fifth IEEE/ACM international workshop on grid computing, pages 4–10. IEEE, 2004.
  * [3] David P Anderson, Jeff Cobb, Eric Korpela, Matt Lebofsky, and Dan Werthimer.  Seti@ home: an experiment in public-resource computing.  Communications of the ACM, 45(11):56–61, 2002.
  * [4] Adam L Beberg, Daniel L Ensign, Guha Jayachandran, Siraj Khaliq, and Vijay S Pande.  Folding@ home: Lessons from eight years of volunteer distributed computing.  In 2009 IEEE International Symposium on Parallel & Distributed Processing, pages 1–8. IEEE, 2009.
  * [5] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * [6] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, June 2022.
  * [7] Diederik P Kingma, Max Welling, et al.  An introduction to variational autoencoders.  Foundations and Trends® in Machine Learning, 12(4):307–392, 2019.
  * [8] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  * [9] Ang Li, Allan Jabri, Armand Joulin, and Laurens Van Der Maaten.  Learning visual n-grams from web data.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4183–4192, 2017.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [11] Weiming Xiang, Patrick Musau, Ayana A Wild, Diego Manzanas Lopez, Nathaniel Hamilton, Xiaodong Yang, Joel Rosenfeld, and Taylor T Johnson.  Verification for machine learning, autonomy, and neural networks survey.  arXiv preprint arXiv:1810.01989, 2018.
  * [12] Ian Goodfellow and Nicolas Papernot.  The challenge of verification and testing of machine learning.  Cleverhans-blog, 2017.
  * [13] Arati Baliga.  Understanding blockchain consensus models.  Persistent, 4(1):14, 2017.
  * [14] Md Sadek Ferdous, Mohammad Jabed Morshed Chowdhury, Mohammad A Hoque, and Alan Colman.  Blockchain consensus algorithms: A survey.  arXiv preprint arXiv:2001.07091, 2020.
  * [15] Daniel Kang, Tatsunori Hashimoto, Ion Stoica, and Yi Sun.  Scaling up trustless dnn inference with zero-knowledge proofs.  arXiv preprint arXiv:2210.08674, 2022.
  * [16] Hiroyuki Ootomo and Rio Yokota.  Recovering single precision accuracy from tensor cores while surpassing the fp32 theoretical peak performance.  The International Journal of High Performance Computing Applications, 36(4):475–491, 2022.
  * [17] Reproducibility.  https://pytorch.org/docs/stable/notes/randomness.html.  Accessed: 2023-07-01.
  * [18] Marc Jorda, Pedro Valero-Lara, and Antonio J Pena.  Performance evaluation of cudnn convolution algorithms on nvidia volta gpus.  IEEE Access, 7:70461–70473, 2019.
  * [19] Testing different image hash functions.  https://content-blockchain.org/research/testing-different-image-hash-functions/.  Accessed: 2023-07-01.
  * [20] Locality sensitive hashing for similar item search.  https://towardsdatascience.com/locality-sensitive-hashing-for-music-search-f2f1940ace23.  Accessed: 2023-07-01.
  * [21] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [22] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  arXiv preprint arXiv:2208.01618, 2022.
  * [23] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [24] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.  Wizardlm: Empowering large language models to follow complex instructions, 2023.
  * [25] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
  * [26] Together Computer.  Redpajama: An open source recipe to reproduce llama training dataset, 2023.
  * [27] Ben Wang.  Mesh-Transformer-JAX: Model-Parallel Implementation of Transformer Language Model with JAX.  https://github.com/kingoflolz/mesh-transformer-jax, May 2021.