Kyung Hee University, Department of Physics, Seoul, Korea J.
Goh0000-0002-1129-2083

Sejong University, Seoul, Korea H. S. Kim0000-0002-6543-9191, Y. Kim, S. Lee

Seoul National University, Seoul, Korea J. Almond, J.H. Bhyun, J.
Choi0000-0002-2483-5104, S. Jeon0000-0003-1208-6940, J.
Kim0000-0001-9876-6642, J.S. Kim, S. Ko0000-0003-4377-9969, H.
Kwon0009-0002-5165-5018, H. Lee0000-0002-1138-3700, S. Lee, B.H.
Oh0000-0002-9539-7789, S.B. Oh0000-0003-0710-4956, H. Seo0000-0002-3932-0605,
U.K. Yang, I. Yoon0000-0002-3491-8026

University of Seoul, Seoul, Korea W. Jang0000-0002-1571-9072, D.Y. Kang, Y.
Kang0000-0001-6079-3434, D. Kim0000-0002-8336-9182, S. Kim0000-0002-8015-7379,
B. Ko, J.S.H. Lee0000-0002-2153-1519, Y. Lee0000-0001-5572-5947, J.A. Merlin,
I.C. Park0000-0003-4510-6776, Y. Roh, I.J. Watson0000-0003-2141-3413, S.
Yang0000-0001-6905-6553

Yonsei University, Department of Physics, Seoul, Korea S.
Ha0000-0003-2538-1551, H.D. Yoo0000-0002-3892-3500

Sungkyunkwan University, Suwon, Korea M. Choi0000-0002-4811-626X, M.R.
Kim0000-0002-2289-2527, H. Lee, Y. Lee0000-0001-6954-9964, I.
Yu0000-0003-1567-5548

College of Engineering and Technology, American University of the Middle East
(AUM), Dasman, Kuwait T. Beyrouthy, Y. Maghrbi0000-0002-4960-7458

Riga Technical University, Riga, Latvia K. Dreimanis0000-0003-0972-5641, A.
Gaile0000-0003-1350-3523, G. Pikurs, A. Potrebko0000-0002-3776-8270, M.
Seidel0000-0003-3550-6151, V. Veckalns550000-0003-3676-9711

University of Latvia (LU), Riga, Latvia N.R. Strautnieks0000-0003-4540-9048

Vilnius University, Vilnius, Lithuania M. Ambrozas0000-0003-2449-0158, A.
Juodagalvis0000-0002-1501-3328, A. Rinkevicius0000-0002-7510-255X, G.
Tamulaitis0000-0002-2913-9634

National Centre for Particle Physics, Universiti Malaya, Kuala Lumpur,
Malaysia N. Bin Norjoharuddeen0000-0002-8818-7476, I.
Yusuff560000-0003-2786-0732, Z. Zolkapli

Universidad de Sonora (UNISON), Hermosillo, Mexico J.F.
Benitez0000-0002-2633-6712, A. Castaneda Hernandez0000-0003-4766-1546, H.A.
Encinas Acosta, L.G. Gallegos Maríñez, M. León Coello0000-0002-3761-911X, J.A.
Murillo Quijada0000-0003-4933-2092, A. Sehrawat0000-0002-6816-7814, L.
Valencia Palomo0000-0002-8736-440X

Centro de Investigacion y de Estudios Avanzados del IPN, Mexico City, Mexico
G. Ayala0000-0002-8294-8692, H. Castilla-Valdez0009-0005-9590-9958, E. De La
Cruz-Burelo0000-0002-7469-6974, I. Heredia-De La Cruz570000-0002-8133-6467, R.
Lopez-Fernandez0000-0002-2389-4831, C.A. Mondragon Herrera, D.A. Perez
Navarro0000-0001-9280-4150, A. Sánchez Hernández0000-0001-9548-0358

Universidad Iberoamericana, Mexico City, Mexico C. Oropeza
Barrera0000-0001-9724-0016, M. Ramírez García0000-0002-4564-3822

Benemerita Universidad Autonoma de Puebla, Puebla, Mexico I.
Pedraza0000-0002-2669-4659, H.A. Salazar Ibarguen0000-0003-4556-7302, C. Uribe
Estrada0000-0002-2425-7340

University of Montenegro, Podgorica, Montenegro I. Bubanja, N.
Raicevic0000-0002-2386-2290

University of Canterbury, Christchurch, New Zealand P.H.
Butler0000-0001-9878-2140

National Centre for Physics, Quaid-I-Azam University, Islamabad, Pakistan A.
Ahmad0000-0002-4770-1897, M.I. Asghar, A. Awais0000-0003-3563-257X, M.I.M.
Awan, H.R. Hoorani0000-0002-0088-5043, W.A. Khan0000-0003-0488-0941

AGH University of Krakow, Faculty of Computer Science, Electronics and
Telecommunications, Krakow, Poland V. Avati, L. Grzanka0000-0002-3599-854X, M.
Malawski0000-0001-6005-0243

National Centre for Nuclear Research, Swierk, Poland H.
Bialkowska0000-0002-5956-6258, M. Bluj0000-0003-1229-1442, B.
Boimska0000-0002-4200-1541, M. Górski0000-0003-2146-187X, M.
Kazana0000-0002-7821-3036, M. Szleper0000-0002-1697-004X, P.
Zalewski0000-0003-4429-2888

Institute of Experimental Physics, Faculty of Physics, University of Warsaw,
Warsaw, Poland K. Bunkowski0000-0001-6371-9336, K. Doroba0000-0002-7818-2364,
A. Kalinowski0000-0002-1280-5493, M. Konecki0000-0001-9482-4841, J.
Krolikowski0000-0002-3055-0236

Warsaw University of Technology, Warsaw, Poland K. Pozniak0000-0001-5426-1423,
W. Zabolotny0000-0002-6833-4846

Laboratório de Instrumentação e Física Experimental de Partículas, Lisboa,
Portugal M. Araujo0000-0002-8152-3756, D. Bastos0000-0002-7032-2481, C. Beirão
Da Cruz E Silva0000-0002-1231-3819, A. Boletti0000-0003-3288-7737, M.
Bozzo0000-0002-1715-0457, P. Faccioli0000-0003-1849-6692, M.
Gallinaro0000-0003-1261-2277, J. Hollar0000-0002-8664-0134, N.
Leonardo0000-0002-9746-4594, T. Niknejad0000-0003-3276-9482, M.
Pisano0000-0002-0264-7217, J. Seixas0000-0002-7531-0842, J.
Varela0000-0003-2613-3146

Faculty of Physics, University of Belgrade, Belgrade, Serbia P.
Adzic0000-0002-5862-7397, P. Milenovic0000-0001-7132-3550

VINCA Institute of Nuclear Sciences, University of Belgrade, Belgrade, Serbia
M. Dordevic0000-0002-8407-3236, J. Milosevic0000-0001-8486-4604, V. Rekovic

Centro de Investigaciones Energéticas Medioambientales y Tecnológicas
(CIEMAT), Madrid, Spain M. Aguilar-Benitez, J. Alcaraz
Maestre0000-0003-0914-7474, M. Barrio Luna, Cristina F.
Bedoya0000-0001-8057-9152, M. Cepeda0000-0002-6076-4083, M.
Cerrada0000-0003-0112-1691, N. Colino0000-0002-3656-0259, B. De La
Cruz0000-0001-9057-5614, A. Delgado Peris0000-0002-8511-7958, D. Fernández Del
Val0000-0003-2346-1590, J.P. Fernández Ramos0000-0002-0122-313X, J.
Flix0000-0003-2688-8047, M.C. Fouz0000-0003-2950-976X, O. Gonzalez
Lopez0000-0002-4532-6464, S. Goy Lopez0000-0001-6508-5090, J.M.
Hernandez0000-0001-6436-7547, M.I. Josa0000-0002-4985-6964, J. León
Holgado0000-0002-4156-6460, D. Moran0000-0002-1941-9333, Á. Navarro
Tobar0000-0003-3606-1780, C. Perez Dengra0000-0003-2821-4249, A. Pérez-Calero
Yzquierdo0000-0003-3036-7965, J. Puerta Pelayo0000-0001-7390-1457, I.
Redondo0000-0003-3737-4121, D.D. Redondo Ferrero0000-0002-3463-0559, L.
Romero, S. Sánchez Navas0000-0001-6129-9059, L. Urda Gómez0000-0002-7865-5010,
J. Vazquez Escobar0000-0002-7533-2283, C. Willmott

Universidad Autónoma de Madrid, Madrid, Spain J.F. de
Trocóniz0000-0002-0798-9806

Universidad de Oviedo, Instituto Universitario de Ciencias y Tecnologías
Espaciales de Asturias (ICTEA), Oviedo, Spain B. Alvarez
Gonzalez0000-0001-7767-4810, J. Cuevas0000-0001-5080-0821, J. Fernandez
Menendez0000-0002-5213-3708, S. Folgueras0000-0001-7191-1125, I. Gonzalez
Caballero0000-0002-8087-3199, J.R. González Fernández0000-0002-4825-8188, E.
Palencia Cortezon0000-0001-8264-0287, C. Ramón Álvarez0000-0003-1175-0002, V.
Rodríguez Bouza0000-0002-7225-7310, A. Soto Rodríguez0000-0002-2993-8663, A.
Trapote0000-0002-4030-2551, C. Vico Villalba0000-0002-1905-1874, P.
Vischia0000-0002-7088-8557

Instituto de Física de Cantabria (IFCA), CSIC-Universidad de Cantabria,
Santander, Spain S. Bhowmik0000-0003-1260-973X, S. Blanco
Fernández0000-0001-7301-0670, J.A. Brochero Cifuentes0000-0003-2093-7856, I.J.
Cabrillo0000-0002-0367-4022, A. Calderon0000-0002-7205-2040, J. Duarte
Campderros0000-0003-0687-5214, M. Fernandez0000-0002-4824-1087, C. Fernandez
Madrazo0000-0001-9748-4336, G. Gomez0000-0002-1077-6553, C. Lasaosa
García0000-0003-2726-7111, C. Martinez Rivero0000-0002-3224-956X, P. Martinez
Ruiz del Arbol0000-0002-7737-5121, F. Matorras0000-0003-4295-5668, P. Matorras
Cuevas0000-0001-7481-7273, E. Navarrete Ramos0000-0002-5180-4020, J. Piedra
Gomez0000-0002-9157-1700, C. Prieels, L. Scodellaro0000-0002-4974-8330, I.
Vila0000-0002-6797-7209, J.M. Vizan Garcia0000-0002-6823-8854

University of Colombo, Colombo, Sri Lanka M.K. Jayananda0000-0002-7577-310X,
B. Kailasapathy580000-0003-2424-1303, D.U.J. Sonnadara0000-0001-7862-2537,
D.D.C. Wickramarathna0000-0002-6941-8478

University of Ruhuna, Department of Physics, Matara, Sri Lanka W.G.D.
Dharmaratna0000-0002-6366-837X, K. Liyanage0000-0002-3792-7665, N.
Perera0000-0002-4747-9106, N. Wickramage0000-0001-7760-3537

CERN, European Organization for Nuclear Research, Geneva, Switzerland D.
Abbaneo0000-0001-9416-1742, E. Auffray0000-0001-8540-1097, G.
Auzinger0000-0001-7077-8262, J. Baechler, D. Barney0000-0002-4927-4921, A.
Bermúdez Martínez0000-0001-8822-4727, M. Bianco0000-0002-8336-3282, B.
Bilin0000-0003-1439-7128, A.A. Bin Anuar0000-0002-2988-9830, A.
Bocci0000-0002-6515-5666, E. Brondolin0000-0001-5420-586X, C.
Caillol0000-0002-5642-3040, T. Camporesi0000-0001-5066-1876, G.
Cerminara0000-0002-2897-5753, N. Chernyavskaya0000-0002-2264-2229, M.
Cipriani0000-0002-0151-4439, D. d’Enterria0000-0002-5754-4303, A.
Dabrowski0000-0003-2570-9676, A. David0000-0001-5854-7699, A. De
Roeck0000-0002-9228-5271, M.M. Defranchis0000-0001-9573-3714, M.
Deile0000-0001-5085-7270, M. Dobson0009-0007-5021-3230, F. Fallavollita59, L.
Forthomme0000-0002-3302-336X, G. Franzoni0000-0001-9179-4253, W.
Funk0000-0003-0422-6739, S. Giani, D. Gigi, K. Gill0009-0001-9331-5145, F.
Glege0000-0002-4526-2149, L. Gouskos0000-0002-9547-7471, M.
Haranko0000-0002-9376-9235, J. Hegeman0000-0002-2938-2263, T.
James0000-0002-3727-0202, J. Kieseler0000-0003-1644-7678, N.
Kratochwil0000-0001-5297-1878, S. Laurila0000-0001-7507-8636, P.
Lecoq0000-0002-3198-0115, E. Leutgeb0000-0003-4838-3306, C.
Lourenço0000-0003-0885-6711, B. Maier0000-0001-5270-7540, L.
Malgeri0000-0002-0113-7389, M. Mannelli0000-0003-3748-8946, A.C.
Marini0000-0003-2351-0487, F. Meijers0000-0002-6530-3657, S.
Mersi0000-0003-2155-6692, E. Meschi0000-0003-4502-6151, F.
Moortgat0000-0001-7199-0046, M. Mulders0000-0001-7432-6634, S. Orfanelli, F.
Pantaleo0000-0003-3266-4357, M. Peruzzi0000-0002-0416-696X, A.
Petrilli0000-0003-0887-1882, G. Petrucciani0000-0003-0889-4726, A.
Pfeiffer0000-0001-5328-448X, M. Pierini0000-0003-1939-4268, D.
Piparo0009-0006-6958-3111, H. Qu0000-0002-0250-8655, D.
Rabady0000-0001-9239-0605, G. Reales Gutiérrez, M. Rovere0000-0001-8048-1622,
H. Sakulin0000-0003-2181-7258, S. Scarfi0009-0006-8689-3576, M.
Selvaggi0000-0002-5144-9655, A. Sharma0000-0002-9860-1650, P.
Silva0000-0002-5725-041X, P. Sphicas600000-0002-5456-5977, A.G. Stahl
Leiton0000-0002-5397-252X, A. Steen0009-0006-4366-3463, S.
Summers0000-0003-4244-2061, D. Treille0009-0005-5952-9843, P.
Tropea0000-0003-1899-2266, A. Tsirou, D. Walter0000-0001-8584-9705, J.
Wanczyk610000-0002-8562-1863, K.A. Wozniak0000-0002-4395-1581, P.
Zejdl0000-0001-9554-7815, W.D. Zeuner

Paul Scherrer Institut, Villigen, Switzerland T.
Bevilacqua620000-0001-9791-2353, L. Caminada620000-0001-5677-6033, A.
Ebrahimi0000-0003-4472-867X, W. Erdmann0000-0001-9964-249X, R.
Horisberger0000-0002-5594-1321, Q. Ingram0000-0002-9576-055X, H.C.
Kaestli0000-0003-1979-7331, D. Kotlinski0000-0001-5333-4918, C.
Lange0000-0002-3632-3157, M. Missiroli620000-0002-1780-1344, L.
Noehte620000-0001-6125-7203, T. Rohe0009-0005-6188-7754

ETH Zurich - Institute for Particle Physics and Astrophysics (IPA), Zurich,
Switzerland T.K. Aarrestad0000-0002-7671-243X, K.
Androsov610000-0003-2694-6542, M. Backhaus0000-0002-5888-2304, A.
Calandri0000-0001-7774-0099, K. Datta0000-0002-6674-0015, A. De
Cosa0000-0003-2533-2856, G. Dissertori0000-0002-4549-2569, M. Dittmar, M.
Donegà0000-0001-9830-0412, F. Eble0009-0002-0638-3447, M.
Galli0000-0002-9408-4756, K. Gedia0009-0006-0914-7684, F.
Glessgen0000-0001-5309-1960, C. Grab0000-0002-6182-3380, D.
Hits0000-0002-3135-6427, W. Lustermann0000-0003-4970-2217, A.-M.
Lyon0009-0004-1393-6577, R.A. Manzoni0000-0002-7584-5038, L.
Marchese0000-0001-6627-8716, C. Martin Perez0000-0003-1581-6152, A.
Mascellani610000-0001-6362-5356, F. Nessi-Tedaldi0000-0002-4721-7966, F.
Pauss0000-0002-3752-4639, V. Perovic0009-0002-8559-0531, S.
Pigazzini0000-0002-8046-4344, M.G. Ratti0000-0003-1777-7855, M.
Reichmann0000-0002-6220-5496, C. Reissel0000-0001-7080-1119, T.
Reitenspiess0000-0002-2249-0835, B. Ristic0000-0002-8610-1130, F.
Riti0000-0002-1466-9077, D. Ruini, D.A. Sanz Becerra0000-0002-6610-4019, R.
Seidita0000-0002-3533-6191, J. Steggemann610000-0003-4420-5510, D.
Valsecchi0000-0001-8587-8266, R. Wallny0000-0001-8038-1613

Universität Zürich, Zurich, Switzerland C. Amsler630000-0002-7695-501X, P.
Bärtschi0000-0002-8842-6027, C. Botta0000-0002-8072-795X, D. Brzhechko, M.F.
Canelli0000-0001-6361-2117, K. Cormier0000-0001-7873-3579, A. De
Wit0000-0002-5291-1661, R. Del Burgo, J.K. Heikkilä0000-0002-0538-1469, M.
Huwiler0000-0002-9806-5907, W. Jin0009-0009-8976-7702, A.
Jofrehei0000-0002-8992-5426, B. Kilminster0000-0002-6657-0407, S.
Leontsinis0000-0002-7561-6091, S.P. Liechti0000-0002-1192-1628, A.
Macchiolo0000-0003-0199-6957, P. Meiring0009-0001-9480-4039, V.M.
Mikuni0000-0002-1579-2421, U. Molinatti0000-0002-9235-3406, I.
Neutelings0009-0002-6473-1403, A. Reimers0000-0002-9438-2059, P. Robmann, S.
Sanchez Cruz0000-0002-9991-195X, K. Schweiger0000-0002-5846-3919, M.
Senger0000-0002-1992-5711, Y. Takahashi0000-0001-5184-2265

National Central University, Chung-Li, Taiwan C. Adloff64, C.M. Kuo, W. Lin,
P.K. Rout0000-0001-8149-6180, P.C. Tiwari390000-0002-3667-3843, S.S.
Yu0000-0002-6011-8516

National Taiwan University (NTU), Taipei, Taiwan L. Ceard, Y.
Chao0000-0002-5976-318X, K.F. Chen0000-0003-1304-3782, P.s. Chen, W.-S.
Hou0000-0002-4260-5118, Y.w. Kao, R. Khurana, G. Kole0000-0002-3285-1497, Y.y.
Li0000-0003-3598-556X, R.-S. Lu0000-0001-6828-1695, E.
Paganis0000-0002-1950-8993, A. Psallidas, J. Thomas-
Wilsker0000-0003-1293-4153, H.y. Wu, E. Yazgan0000-0001-5732-7950

High Energy Physics Research Unit, Department of Physics, Faculty of Science,
Chulalongkorn University, Bangkok, Thailand C.
Asawatangtrakuldee0000-0003-2234-7219, N. Srimanobhas0000-0003-3563-2959, V.
Wachirapusitanand0000-0001-8251-5160

Çukurova University, Physics Department, Science and Art Faculty, Adana,
Turkey D. Agyel0000-0002-1797-8844, F. Boran0000-0002-3611-390X, Z.S.
Demiroglu0000-0001-7977-7127, F. Dolek0000-0001-7092-5517, I.
Dumanoglu650000-0002-0039-5503, E. Eskut0000-0001-8328-3314, Y.
Guler660000-0001-7598-5252, E. Gurpinar Guler660000-0002-6172-0285, C.
Isik0000-0002-7977-0811, O. Kara, A. Kayis Topaksu0000-0002-3169-4573, U.
Kiminsu0000-0001-6940-7800, G. Onengut0000-0002-6274-4254, K.
Ozdemir670000-0002-0103-1488, A. Polatoz0000-0001-9516-0821, B.
Tali680000-0002-7447-5602, U.G. Tok0000-0002-3039-021X, S.
Turkcapar0000-0003-2608-0494, E. Uslan0000-0002-2472-0526, I.S.
Zorbakir0000-0002-5962-2221

Middle East Technical University, Physics Department, Ankara, Turkey K.
Ocalan690000-0002-8419-1400, M. Yalvac700000-0003-4915-9162

Bogazici University, Istanbul, Turkey B. Akgun0000-0001-8888-3562, I.O.
Atakisi0000-0002-9231-7464, E. Gülmez0000-0002-6353-518X, M.
Kaya710000-0003-2890-4493, O. Kaya720000-0002-8485-3822, S.
Tekten730000-0002-9624-5525

Istanbul Technical University, Istanbul, Turkey A. Cakir0000-0002-8627-7689,
K. Cankocak650000-0002-3829-3481, Y. Komurcu0000-0002-7084-030X, S.
Sen740000-0001-7325-1087

Istanbul University, Istanbul, Turkey O. Aydilek0000-0002-2567-6766, S.
Cerci680000-0002-8702-6152, V. Epshteyn0000-0002-8863-6374, B.
Hacisahinoglu0000-0002-2646-1230, I. Hos750000-0002-7678-1101, B.
Isildak760000-0002-0283-5234, B. Kaynak0000-0003-3857-2496, S.
Ozkorucuklu0000-0001-5153-9266, H. Sert0000-0003-0716-6727, C.
Simsek0000-0002-7359-8635, D. Sunar Cerci680000-0002-5412-4688, C.
Zorbilmez0000-0002-5199-061X

Institute for Scintillation Materials of National Academy of Science of
Ukraine, Kharkiv, Ukraine A. Boyaryntsev0000-0001-9252-0430, B.
Grynyov0000-0003-1700-0173

National Science Centre, Kharkiv Institute of Physics and Technology, Kharkiv,
Ukraine L. Levchuk0000-0001-5889-7410

University of Bristol, Bristol, United Kingdom D. Anthony0000-0002-5016-8886,
J.J. Brooke0000-0003-2529-0684, A. Bundock0000-0002-2916-6456, E.
Clement0000-0003-3412-4004, D. Cussans0000-0001-8192-0826, H.
Flacher0000-0002-5371-941X, M. Glowacki, J. Goldstein0000-0003-1591-6014, H.F.
Heath0000-0001-6576-9740, L. Kreczko0000-0003-2341-8330, B.
Krikler0000-0001-9712-0030, S. Paramesvaran0000-0003-4748-8296, S. Seif El
Nasr-Storey, V.J. Smith0000-0003-4543-2547, N. Stylianou770000-0002-0113-6829,
K. Walkingshaw Pass, R. White0000-0001-5793-526X

Rutherford Appleton Laboratory, Didcot, United Kingdom A.H. Ball, K.W.
Bell0000-0002-2294-5860, A. Belyaev780000-0002-1733-4408, C.
Brew0000-0001-6595-8365, R.M. Brown0000-0002-6728-0153, D.J.A.
Cockerill0000-0003-2427-5765, C. Cooke0000-0003-3730-4895, K.V. Ellis, K.
Harder0000-0002-2965-6973, S. Harper0000-0001-5637-2653, M.-L.
Holmberg790000-0002-9473-5985, Sh. Jain0000-0003-1770-5309, J.
Linacre0000-0001-7555-652X, K. Manolopoulos, D.M. Newbold0000-0002-9015-9634,
E. Olaiya, D. Petyt0000-0002-2369-4469, T. Reis0000-0003-3703-6624, G.
Salvi0000-0002-2787-1063, T. Schuh, C.H. Shepherd-
Themistocleous0000-0003-0551-6949, I.R. Tomalin0000-0003-2419-4439, T.
Williams0000-0002-8724-4678

Imperial College, London, United Kingdom R. Bainbridge0000-0001-9157-4832, P.
Bloch0000-0001-6716-979X, C.E. Brown0000-0002-7766-6615, O. Buchmuller, V.
Cacchio, C.A. Carrillo Montoya0000-0002-6245-6535, V.
Cepaitis0000-0002-4809-4056, G.S. Chahal800000-0003-0320-4407, D.
Colling0000-0001-9959-4977, J.S. Dancu, P. Dauncey0000-0001-6839-9466, G.
Davies0000-0001-8668-5001, J. Davies, M. Della Negra0000-0001-6497-8081, S.
Fayer, G. Fedi0000-0001-9101-2573, G. Hall0000-0002-6299-8385, M.H.
Hassanshahi0000-0001-6634-4517, A. Howard, G. Iles0000-0002-1219-5859, J.
Langford0000-0002-3931-4379, L. Lyons0000-0001-7945-9188, A.-M.
Magnan0000-0002-4266-1646, S. Malik, A. Martelli0000-0003-3530-2255, M.
Mieskolainen0000-0001-8893-7401, J. Nash810000-0003-0607-6519, M. Pesaresi,
B.C. Radburn-Smith0000-0003-1488-9675, A. Richards, A.
Rose0000-0002-9773-550X, C. Seez0000-0002-1637-5494, R.
Shukla0000-0001-5670-5497, A. Tapper0000-0003-4543-864X, K.
Uchida0000-0003-0742-2276, G.P. Uttley0009-0002-6248-6467, L.H. Vage, T.
Virdee280000-0001-7429-2198, M. Vojinovic0000-0001-8665-2808, N.
Wardle0000-0003-1344-3356, D. Winterbottom0000-0003-4582-150X

Brunel University, Uxbridge, United Kingdom K. Coldham, J.E.
Cole0000-0001-5638-7599, A. Khan, P. Kyberd0000-0002-7353-7090, I.D.
Reid0000-0002-9235-779X

Baylor University, Waco, Texas, USA S. Abdullin0000-0003-4885-6935, A.
Brinkerhoff0000-0002-4819-7995, B. Caraway0000-0002-6088-2020, J.
Dittmann0000-0002-1911-3158, K. Hatakeyama0000-0002-6012-2451, J.
Hiltbrand0000-0003-1691-5937, A.R. Kanuganti0000-0002-0789-1200, B.
McMaster0000-0002-4494-0446, M. Saunders0000-0003-1572-9075, S.
Sawant0000-0002-1981-7753, C. Sutantawibul0000-0003-0600-0151, M.
Toms0000-0002-7703-3973, J. Wilson0000-0002-5672-7394

Catholic University of America, Washington, DC, USA R.
Bartek0000-0002-1686-2882, A. Dominguez0000-0002-7420-5493, C. Huerta
Escamilla, A.E. Simsek0000-0002-9074-2256, R. Uniyal0000-0001-7345-6293, A.M.
Vargas Hernandez0000-0002-8911-7197

The University of Alabama, Tuscaloosa, Alabama, USA R.
Chudasama0009-0007-8848-6146, S.I. Cooper0000-0002-4618-0313, S.V.
Gleyzer0000-0002-6222-8102, C.U. Perez0000-0002-6861-2674, P.
Rumerio820000-0002-1702-5541, E. Usai0000-0001-9323-2107, C.
West0000-0003-4460-2241

Boston University, Boston, Massachusetts, USA A. Akpinar0000-0001-7510-6617,
A. Albert0000-0003-2369-9507, D. Arcaro0000-0001-9457-8302, C.
Cosby0000-0003-0352-6561, Z. Demiragli0000-0001-8521-737X, C.
Erice0000-0002-6469-3200, E. Fontanesi0000-0002-0662-5904, D.
Gastler0009-0000-7307-6311, J. Rohlf0000-0001-6423-9799, K.
Salyer0000-0002-6957-1077, D. Sperka0000-0002-4624-2019, D.
Spitzbart0000-0003-2025-2742, I. Suarez0000-0002-5374-6995, A.
Tsatsos0000-0001-8310-8911, S. Yuan0000-0002-2029-024X

Brown University, Providence, Rhode Island, USA G. Benelli0000-0003-4461-8905,
X. Coubez23, D. Cutts0000-0003-1041-7099, M. Hadley0000-0002-7068-4327, U.
Heintz0000-0002-7590-3058, J.M. Hogan830000-0002-8604-3452, T.
Kwon0000-0001-9594-6277, G. Landsberg0000-0002-4184-9380, K.T.
Lau0000-0003-1371-8575, D. Li0000-0003-0890-8948, J. Luo0000-0002-4108-8681,
M. Narain0000-0002-7857-7403, N. Pervan0000-0002-8153-8464, S.
Sagir840000-0002-2614-5860, F. Simpson0000-0001-8944-9629, W.Y. Wong, X.
Yan0000-0002-6426-0560, D. Yu0000-0001-5921-5231, W. Zhang

University of California, Davis, Davis, California, USA S.
Abbott0000-0002-7791-894X, J. Bonilla0000-0002-6982-6121, C.
Brainerd0000-0002-9552-1006, R. Breedon0000-0001-5314-7581, M. Calderon De La
Barca Sanchez0000-0001-9835-4349, M. Chertok0000-0002-2729-6273, J.
Conway0000-0003-2719-5779, P.T. Cox0000-0003-1218-2828, R.
Erbacher0000-0001-7170-8944, G. Haza0009-0001-1326-3956, F.
Jensen0000-0003-3769-9081, O. Kukral0009-0007-3858-6659, G.
Mocellin0000-0002-1531-3478, M. Mulhearn0000-0003-1145-6436, D.
Pellett0009-0000-0389-8571, B. Regnery0000-0003-1539-923X, W.
Wei0000-0003-4221-1802, Y. Yao0000-0002-5990-4245, F. Zhang0000-0002-6158-2468

University of California, Los Angeles, California, USA M.
Bachtis0000-0003-3110-0701, R. Cousins0000-0002-5963-0467, A.
Datta0000-0003-2695-7719, J. Hauser0000-0002-9781-4873, M.
Ignatenko0000-0001-8258-5863, M.A. Iqbal0000-0001-8664-1949, T.
Lam0000-0002-0862-7348, E. Manca0000-0001-8946-655X, W.A.
Nash0009-0004-3633-8967, D. Saltzberg0000-0003-0658-9146, B.
Stone0000-0002-9397-5231, V. Valuev0000-0002-0783-6703

University of California, Riverside, Riverside, California, USA R.
Clare0000-0003-3293-5305, M. Gordon, G. Hanson0000-0002-7273-4009, W.
Si0000-0002-5879-6326, S.
Wimpenny${}^{\textrm{\textdagger}}$0000-0003-0505-4908

University of California, San Diego, La Jolla, California, USA J.G.
Branson0009-0009-5683-4614, S. Cittolin0000-0002-0922-9587, S.
Cooperstein0000-0003-0262-3132, D. Diaz0000-0001-6834-1176, J.
Duarte0000-0002-5076-7096, R. Gerosa0000-0001-8359-3734, L.
Giannini0000-0002-5621-7706, J. Guiang0000-0002-2155-8260, R.
Kansal0000-0003-2445-1060, V. Krutelyov0000-0002-1386-0232, R.
Lee0009-0000-4634-0797, J. Letts0000-0002-0156-1251, M.
Masciovecchio0000-0002-8200-9425, F. Mokhtar0000-0003-2533-3402, M.
Pieri0000-0003-3303-6301, M. Quinnan0000-0003-2902-5597, B.V. Sathia
Narayanan0000-0003-2076-5126, V. Sharma0000-0003-1736-8795, M.
Tadel0000-0001-8800-0045, E. Vourliotis0000-0002-2270-0492, F.
Würthwein0000-0001-5912-6124, Y. Xiang0000-0003-4112-7457, A.
Yagil0000-0002-6108-4004

University of California, Santa Barbara - Department of Physics, Santa
Barbara, California, USA L. Brennan0000-0003-0636-1846, C.
Campagnari0000-0002-8978-8177, M. Citron0000-0001-6250-8465, G.
Collura0000-0002-4160-1844, A. Dorsett0000-0001-5349-3011, J.
Incandela0000-0001-9850-2030, M. Kilpatrick0000-0002-2602-0566, J.
Kim0000-0002-2072-6082, A.J. Li0000-0002-3895-717X, P.
Masterson0000-0002-6890-7624, H. Mei0000-0002-9838-8327, M.
Oshiro0000-0002-2200-7516, J. Richman0000-0002-5189-146X, U.
Sarica0000-0002-1557-4424, R. Schmitz0000-0003-2328-677X, F.
Setti0000-0001-9800-7822, J. Sheplock0000-0002-8752-1946, D.
Stuart0000-0002-4965-0747, S. Wang0000-0001-7887-1728

California Institute of Technology, Pasadena, California, USA A.
Bornheim0000-0002-0128-0871, O. Cerri, A. Latorre, J.M.
Lawhorn0000-0002-8597-9259, J. Mao0009-0002-8988-9987, H.B.
Newman0000-0003-0964-1480, T. Q. Nguyen0000-0003-3954-5131, M.
Spiropulu0000-0001-8172-7081, J.R. Vlimant0000-0002-9705-101X, C.
Wang0000-0002-0117-7196, S. Xie0000-0003-2509-5731, R.Y.
Zhu0000-0003-3091-7461

Carnegie Mellon University, Pittsburgh, Pennsylvania, USA J.
Alison0000-0003-0843-1641, S. An0000-0002-9740-1622, M.B.
Andrews0000-0001-5537-4518, P. Bryant0000-0001-8145-6322, V.
Dutta0000-0001-5958-829X, T. Ferguson0000-0001-5822-3731, A.
Harilal0000-0001-9625-1987, C. Liu0000-0002-3100-7294, T.
Mudholkar0000-0002-9352-8140, S. Murthy0000-0002-1277-9168, M.
Paulini0000-0002-6714-5787, A. Roberts0000-0002-5139-0550, A.
Sanchez0000-0002-5431-6989, W. Terrill0000-0002-2078-8419

University of Colorado Boulder, Boulder, Colorado, USA J.P.
Cumalat0000-0002-6032-5857, W.T. Ford0000-0001-8703-6943, A.
Hassani0009-0008-4322-7682, G. Karathanasis0000-0001-5115-5828, E. MacDonald,
N. Manganelli0000-0002-3398-4531, F. Marini0000-0002-2374-6433, A.
Perloff0000-0001-5230-0396, C. Savard0009-0000-7507-0570, N.
Schonbeck0009-0008-3430-7269, K. Stenson0000-0003-4888-205X, K.A.
Ulmer0000-0001-6875-9177, S.R. Wagner0000-0002-9269-5772, N.
Zipper0000-0002-4805-8020

Cornell University, Ithaca, New York, USA J. Alexander0000-0002-2046-342X, S.
Bright-Thonney0000-0003-1889-7824, X. Chen0000-0002-8157-1328, D.J.
Cranshaw0000-0002-7498-2129, J. Fan0009-0003-3728-9960, X.
Fan0000-0003-2067-0127, D. Gadkari0000-0002-6625-8085, S.
Hogan0000-0003-3657-2281, J. Monroy0000-0002-7394-4710, J.R.
Patterson0000-0002-3815-3649, J. Reichert0000-0003-2110-8021, M.
Reid0000-0001-7706-1416, A. Ryd0000-0001-5849-1912, J.
Thom0000-0002-4870-8468, P. Wittich0000-0002-7401-2181, R.
Zou0000-0002-0542-1264

Fermi National Accelerator Laboratory, Batavia, Illinois, USA M.
Albrow0000-0001-7329-4925, M. Alyari0000-0001-9268-3360, O.
Amram0000-0002-3765-3123, G. Apollinari0000-0002-5212-5396, A.
Apresyan0000-0002-6186-0130, L.A.T. Bauerdick0000-0002-7170-9012, D.
Berry0000-0002-5383-8320, J. Berryhill0000-0002-8124-3033, P.C.
Bhat0000-0003-3370-9246, K. Burkett0000-0002-2284-4744, J.N.
Butler0000-0002-0745-8618, A. Canepa0000-0003-4045-3998, G.B.
Cerati0000-0003-3548-0262, H.W.K. Cheung0000-0001-6389-9357, F.
Chlebana0000-0002-8762-8559, G. Cummings0000-0002-8045-7806, J.
Dickinson0000-0001-5450-5328, I. Dutta0000-0003-0953-4503, V.D.
Elvira0000-0003-4446-4395, Y. Feng0000-0003-2812-338X, J.
Freeman0000-0002-3415-5671, A. Gandrakota0000-0003-4860-3233, Z.
Gecse0009-0009-6561-3418, L. Gray0000-0002-6408-4288, D. Green, S.
Grünendahl0000-0002-4857-0294, D. Guerrero0000-0001-5552-5400, O.
Gutsche0000-0002-8015-9622, R.M. Harris0000-0003-1461-3425, R.
Heller0000-0002-7368-6723, T.C. Herwig0000-0002-4280-6382, J.
Hirschauer0000-0002-8244-0805, L. Horyn0000-0002-9512-4932, B.
Jayatilaka0000-0001-7912-5612, S. Jindariani0009-0000-7046-6533, M.
Johnson0000-0001-7757-8458, U. Joshi0000-0001-8375-0760, T.
Klijnsma0000-0003-1675-6040, B. Klima0000-0002-3691-7625, K.H.M.
Kwok0000-0002-8693-6146, S. Lammel0000-0003-0027-635X, D.
Lincoln0000-0002-0599-7407, R. Lipton0000-0002-6665-7289, T.
Liu0009-0007-6522-5605, C. Madrid0000-0003-3301-2246, K.
Maeshima0009-0000-2822-897X, C. Mantilla0000-0002-0177-5903, D.
Mason0000-0002-0074-5390, P. McBride0000-0001-6159-7750, P.
Merkel0000-0003-4727-5442, S. Mrenna0000-0001-8731-160X, S.
Nahn0000-0002-8949-0178, J. Ngadiuba0000-0002-0055-2935, D.
Noonan0000-0002-3932-3769, V. Papadimitriou0000-0002-0690-7186, N.
Pastika0009-0006-0993-6245, K. Pedro0000-0003-2260-9151, C.
Pena850000-0002-4500-7930, F. Ravera0000-0003-3632-0287, A. Reinsvold
Hall860000-0003-1653-8553, L. Ristori0000-0003-1950-2492, E. Sexton-
Kennedy0000-0001-9171-1980, N. Smith0000-0002-0324-3054, A.
Soha0000-0002-5968-1192, L. Spiegel0000-0001-9672-1328, J.
Strait0000-0002-7233-8348, L. Taylor0000-0002-6584-2538, S.
Tkaczyk0000-0001-7642-5185, N.V. Tran0000-0002-8440-6854, L.
Uplegger0000-0002-9202-803X, E.W. Vaandering0000-0003-3207-6950, I.
Zoi0000-0002-5738-9446

University of Florida, Gainesville, Florida, USA P. Avery0000-0003-0609-627X,
D. Bourilkov0000-0003-0260-4935, L. Cadamuro0000-0001-8789-610X, P.
Chang0000-0002-2095-6320, V. Cherepanov0000-0002-6748-4850, R.D. Field, E.
Koenig0000-0002-0884-7922, M. Kolosova0000-0002-5838-2158, J.
Konigsberg0000-0001-6850-8765, A. Korytov0000-0001-9239-3398, K.H. Lo, K.
Matchev0000-0003-4182-9096, N. Menendez0000-0002-3295-3194, G.
Mitselmakher0000-0001-5745-3658, A. Muthirakalayil Madhu0000-0003-1209-3032,
N. Rawal0000-0002-7734-3170, D. Rosenzweig0000-0002-3687-5189, S.
Rosenzweig0000-0002-5613-1507, K. Shi0000-0002-2475-0055, J.
Wang0000-0003-3879-4873

Florida State University, Tallahassee, Florida, USA T.
Adams0000-0001-8049-5143, A. Al Kadhim0000-0003-3490-8407, A.
Askew0000-0002-7172-1396, N. Bower0000-0001-8775-0696, R.
Habibullah0000-0002-3161-8300, V. Hagopian0000-0002-3791-1989, R.
Hashmi0000-0002-5439-8224, R.S. Kim0000-0002-8645-186X, S.
Kim0000-0003-2381-5117, T. Kolberg0000-0002-0211-6109, G. Martinez, H.
Prosper0000-0002-4077-2713, P.R. Prova, O. Viazlo0000-0002-2957-0301, M.
Wulansatiti0000-0001-6794-3079, R. Yohay0000-0002-0124-9065, J. Zhang

Florida Institute of Technology, Melbourne, Florida, USA B. Alsufyani, M.M.
Baarmand0000-0002-9792-8619, S. Butalla0000-0003-3423-9581, T.
Elkafrawy160000-0001-9930-6445, M. Hohlmann0000-0003-4578-9319, R. Kumar
Verma0000-0002-8264-156X, M. Rahmani, F. Yumiceva0000-0003-2436-5074

University of Illinois Chicago, Chicago, USA, Chicago, USA M.R.
Adams0000-0001-8493-3737, C. Bennett, R. Cavanaugh0000-0001-7169-3420, S.
Dittmer0000-0002-5359-9614, O. Evdokimov0000-0002-1250-8931, C.E.
Gerber0000-0002-8116-9021, D.J. Hofman0000-0002-2449-3845, J.h.
Lee0000-0002-5574-4192, D. S. Lemos0000-0003-1982-8978, A.H.
Merrit0000-0003-3922-6464, C. Mills0000-0001-8035-4818, S.
Nanda0000-0003-0550-4083, G. Oh0000-0003-0744-1063, D.
Pilipovic0000-0002-4210-2780, T. Roy0000-0001-7299-7653, S.
Rudrabhatla0000-0002-7366-4225, M.B. Tonjes0000-0002-2617-9315, N.
Varelas0000-0002-9397-5514, X. Wang0000-0003-2792-8493, Z.
Ye0000-0001-6091-6772, J. Yoo0000-0002-3826-1332

The University of Iowa, Iowa City, Iowa, USA M. Alhusseini0000-0002-9239-470X,
D. Blend, K. Dilsiz870000-0003-0138-3368, L. Emediato0000-0002-3021-5032, G.
Karaman0000-0001-8739-9648, O.K. Köseyan0000-0001-9040-3468, J.-P. Merlo, A.
Mestvirishvili880000-0002-8591-5247, J. Nachtman0000-0003-3951-3420, O. Neogi,
H. Ogul890000-0002-5121-2893, Y. Onel0000-0002-8141-7769, A.
Penzo0000-0003-3436-047X, C. Snyder, E. Tiras900000-0002-5628-7464

Johns Hopkins University, Baltimore, Maryland, USA B.
Blumenfeld0000-0003-1150-1735, L. Corcodilos0000-0001-6751-3108, J.
Davis0000-0001-6488-6195, A.V. Gritsan0000-0002-3545-7970, L.
Kang0000-0002-0941-4512, S. Kyriacou0000-0002-9254-4368, P.
Maksimovic0000-0002-2358-2168, M. Roguljic0000-0001-5311-3007, J.
Roskes0000-0001-8761-0490, S. Sekhar0000-0002-8307-7518, M.
Swartz0000-0002-0286-5070, T.Á. Vámi0000-0002-0959-9211

The University of Kansas, Lawrence, Kansas, USA A. Abreu0000-0002-9000-2215,
L.F. Alcerro Alcerro0000-0001-5770-5077, J. Anguiano0000-0002-7349-350X, P.
Baringer0000-0002-3691-8388, A. Bean0000-0001-5967-8674, Z.
Flowers0000-0001-8314-2052, J. King0000-0001-9652-9854, G.
Krintiras0000-0002-0380-7577, M. Lazarovits0000-0002-5565-3119, C. Le
Mahieu0000-0001-5924-1130, C. Lindsey, J. Marquez0000-0003-3887-4048, N.
Minafra0000-0003-4002-1888, M. Murray0000-0001-7219-4818, M.
Nickel0000-0003-0419-1329, M. Pitt0000-0003-2461-5985, S.
Popescu910000-0002-0345-2171, C. Rogan0000-0002-4166-4503, C.
Royon0000-0002-7672-9709, R. Salvatico0000-0002-2751-0567, S.
Sanders0000-0002-9491-6022, C. Smith0000-0003-0505-0528, Q.
Wang0000-0003-3804-3244, G. Wilson0000-0003-0917-4763

Kansas State University, Manhattan, Kansas, USA B. Allmond0000-0002-5593-7736,
S. Duric, A. Ivanov0000-0002-9270-5643, K. Kaadze0000-0003-0571-163X, A.
Kalogeropoulos0000-0003-3444-0314, D. Kim, Y. Maravin0000-0002-9449-0666, T.
Mitchell, K. Nam, J. Natoli0000-0001-6675-3564, D. Roy0000-0002-8659-7762

Lawrence Livermore National Laboratory, Livermore, California, USA F.
Rebassoo0000-0001-8934-9329, D. Wright0000-0002-3586-3354

University of Maryland, College Park, Maryland, USA E.
Adams0000-0003-2809-2683, A. Baden0000-0002-6159-3861, O. Baron, A.
Belloni0000-0002-1727-656X, A. Bethani0000-0002-8150-7043, Y.m.
Chen0000-0002-5795-4783, S.C. Eno0000-0003-4282-2515, N.J.
Hadley0000-0002-1209-6471, S. Jabeen0000-0002-0155-7383, R.G.
Kellogg0000-0001-9235-521X, T. Koeth0000-0002-0082-0514, Y.
Lai0000-0002-7795-8693, S. Lascio0000-0001-8579-5874, A.C.
Mignerey0000-0001-5164-6969, S. Nabili0000-0002-6893-1018, C.
Palmer0000-0002-5801-5737, C. Papageorgakis0000-0003-4548-0346, L.
Wang0000-0003-3443-0626, K. Wong0000-0002-9698-1354

Massachusetts Institute of Technology, Cambridge, Massachusetts, USA J.
Bendavid0000-0002-7907-1789, W. Busza0000-0002-3831-9071, I.A.
Cali0000-0002-2822-3375, Y. Chen0000-0003-2582-6469, M.
D’Alfonso0000-0002-7409-7904, J. Eysermans0000-0001-6483-7123, C.
Freer0000-0002-7967-4635, G. Gomez-Ceballos0000-0003-1683-9460, M. Goncharov,
P. Harris, D. Hoang, D. Kovalskyi0000-0002-6923-293X, J.
Krupa0000-0003-0785-7552, L. Lavezzo0000-0002-1364-9920, Y.-J.
Lee0000-0003-2593-7767, K. Long0000-0003-0664-1653, C.
Mironov0000-0002-8599-2437, C. Paus0000-0002-6047-4211, D.
Rankin0000-0001-8411-9620, C. Roland0000-0002-7312-5854, G.
Roland0000-0001-8983-2169, S. Rothman0000-0002-1377-9119, Z.
Shi0000-0001-5498-8825, G.S.F. Stephans0000-0003-3106-4894, J. Wang, Z.
Wang0000-0002-3074-3767, B. Wyslouch0000-0003-3681-0649, T. J.
Yang0000-0003-4317-4660

University of Minnesota, Minneapolis, Minnesota, USA R.M. Chatterjee, B.
Crossman0000-0002-2700-5085, B.M. Joshi0000-0002-4723-0968, C.
Kapsiak0009-0008-7743-5316, M. Krohn0000-0002-1711-2506, D.
Mahon0000-0002-2640-5941, J. Mans0000-0003-2840-1087, M.
Revering0000-0001-5051-0293, R. Rusack0000-0002-7633-749X, R.
Saradhy0000-0001-8720-293X, N. Schroeder0000-0002-8336-6141, N.
Strobbe0000-0001-8835-8282, M.A. Wadud0000-0002-0653-0761

University of Mississippi, Oxford, Mississippi, USA L.M.
Cremaldi0000-0001-5550-7827

University of Nebraska-Lincoln, Lincoln, Nebraska, USA K.
Bloom0000-0002-4272-8900, M. Bryson, D.R. Claes0000-0003-4198-8919, C.
Fangmeier0000-0002-5998-8047, F. Golf0000-0003-3567-9351, C.
Joo0000-0002-5661-4330, I. Kravchenko0000-0003-0068-0395, I.
Reed0000-0002-1823-8856, J.E. Siado0000-0002-9757-470X, G.R.
Snow${}^{\textrm{\textdagger}}$, W. Tabb0000-0002-9542-4847, A.
Wightman0000-0001-6651-5320, F. Yan0000-0002-4042-0785, A.G.
Zecchinelli0000-0001-8986-278X

State University of New York at Buffalo, Buffalo, New York, USA G.
Agarwal0000-0002-2593-5297, H. Bandyopadhyay0000-0001-9726-4915, L.
Hay0000-0002-7086-7641, I. Iashvili0000-0003-1948-5901, A.
Kharchilava0000-0002-3913-0326, C. McLean0000-0002-7450-4805, M.
Morris0000-0002-2830-6488, D. Nguyen0000-0002-5185-8504, J.
Pekkanen0000-0002-6681-7668, S. Rappoccio0000-0002-5449-2560, H. Rejeb Sfar,
A. Williams0000-0003-4055-6532

Northeastern University, Boston, Massachusetts, USA G.
Alverson0000-0001-6651-1178, E. Barberis0000-0002-6417-5913, Y.
Haddad0000-0003-4916-7752, Y. Han0000-0002-3510-6505, A.
Krishna0000-0002-4319-818X, J. Li0000-0001-5245-2074, G.
Madigan0000-0001-8796-5865, B. Marzocchi0000-0001-6687-6214, D.M.
Morse0000-0003-3163-2169, V. Nguyen0000-0003-1278-9208, T.
Orimoto0000-0002-8388-3341, A. Parker0000-0002-9421-3335, L.
Skinnari0000-0002-2019-6755, A. Tishelman-Charny0000-0002-7332-5098, B.
Wang0000-0003-0796-2475, D. Wood0000-0002-6477-801X

Northwestern University, Evanston, Illinois, USA S.
Bhattacharya0000-0002-0526-6161, J. Bueghly, Z. Chen0000-0003-4521-6086, A.
Gilbert0000-0001-7560-5790, K.A. Hahn0000-0001-7892-1676, Y.
Liu0000-0002-5588-1760, D.G. Monk0000-0002-8377-1999, M.H.
Schmitt0000-0003-0814-3578, A. Taliercio0000-0002-5119-6280, M. Velasco

University of Notre Dame, Notre Dame, Indiana, USA R. Band0000-0003-4873-0523,
R. Bucci, M. Cremonesi, A. Das0000-0001-9115-9698, R.
Goldouzian0000-0002-0295-249X, M. Hildreth0000-0002-4454-3934, K. Hurtado
Anampa0000-0002-9779-3566, C. Jessop0000-0002-6885-3611, K.
Lannon0000-0002-9706-0098, J. Lawrence0000-0001-6326-7210, N.
Loukas0000-0003-0049-6918, L. Lutton0000-0002-3212-4505, J. Mariano, N.
Marinelli, I. Mcalister, T. McCauley0000-0001-6589-8286, C.
Mcgrady0000-0002-8821-2045, K. Mohrman0009-0007-2940-0496, C.
Moore0000-0002-8140-4183, Y. Musienko120009-0006-3545-1938, H.
Nelson0000-0001-5592-0785, R. Ruchti0000-0002-3151-1386, A.
Townsend0000-0002-3696-689X, M. Wayne0000-0001-8204-6157, H. Yockey, M.
Zarucki0000-0003-1510-5772, L. Zygala0000-0001-9665-7282

The Ohio State University, Columbus, Ohio, USA B. Bylsma, M.
Carrigan0000-0003-0538-5854, L.S. Durkin0000-0002-0477-1051, C.
Hill0000-0003-0059-0779, M. Joyce0000-0003-1112-5880, A.
Lesauvage0000-0003-3437-7845, M. Nunez Ornelas0000-0003-2663-7379, K. Wei,
B.L. Winer0000-0001-9980-4698, B. R. Yates0000-0001-7366-1318

Princeton University, Princeton, New Jersey, USA F.M.
Addesa0000-0003-0484-5804, H. Bouchamaoui0000-0002-9776-1935, P.
Das0000-0002-9770-1377, G. Dezoort0000-0002-5890-0445, P.
Elmer0000-0001-6830-3356, A. Frankenthal0000-0002-2583-5982, B.
Greenberg0000-0002-4922-1934, N. Haubrich0000-0002-7625-8169, S.
Higginbotham0000-0002-4436-5461, G. Kopp0000-0001-8160-0208, S.
Kwan0000-0002-5308-7707, D. Lange0000-0002-9086-5184, A.
Loeliger0000-0002-5017-1487, D. Marlow0000-0002-6395-1079, I.
Ojalvo0000-0003-1455-6272, J. Olsen0000-0002-9361-5762, D.
Stickland0000-0003-4702-8820, C. Tully0000-0001-6771-2174

University of Puerto Rico, Mayaguez, Puerto Rico, USA S.
Malik0000-0002-6356-2655

Purdue University, West Lafayette, Indiana, USA A.S.
Bakshi0000-0002-2857-6883, V.E. Barnes0000-0001-6939-3445, S.
Chandra0009-0000-7412-4071, R. Chawla0000-0003-4802-6819, S.
Das0000-0001-6701-9265, A. Gu0000-0002-6230-1138, L. Gutay, M.
Jones0000-0002-9951-4583, A.W. Jung0000-0003-3068-3212, D.
Kondratyev0000-0002-7874-2480, A.M. Koshy, M. Liu0000-0001-9012-395X, G.
Negro0000-0002-1418-2154, N. Neumeister0000-0003-2356-1700, G.
Paspalaki0000-0001-6815-1065, S. Piperov0000-0002-9266-7819, A.
Purohit0000-0003-0881-612X, J.F. Schulte0000-0003-4421-680X, M.
Stojanovic170000-0002-1542-0855, J. Thieman0000-0001-7684-6588, F.
Wang0000-0002-8313-0809, W. Xie0000-0003-1430-9191

Purdue University Northwest, Hammond, Indiana, USA J.
Dolen0000-0003-1141-3823, N. Parashar0009-0009-1717-0413, A.
Pathak0000-0001-9861-2942

Rice University, Houston, Texas, USA D. Acosta0000-0001-5367-1738, A.
Baty0000-0001-5310-3466, T. Carnahan0000-0001-7492-3201, S.
Dildick0000-0003-0554-4755, K.M. Ecklund0000-0002-6976-4637, P.J. Fernández
Manteca0000-0003-2566-7496, S. Freed, P. Gardner, F.J.M.
Geurts0000-0003-2856-9090, A. Kumar0000-0002-5180-6595, W.
Li0000-0003-4136-3409, O. Miguel Colin0000-0001-6612-432X, B.P.
Padley0000-0002-3572-5701, R. Redjimi, J. Rotter0009-0009-4040-7407, S.
Yang0000-0002-2075-8631, E. Yigitbasi0000-0002-9595-2623, Y.
Zhang0000-0002-6812-761X

University of Rochester, Rochester, New York, USA A. Bodek0000-0003-0409-0341,
P. de Barbaro0000-0002-5508-1827, R. Demina0000-0002-7852-167X, J.L.
Dulemba0000-0002-9842-7015, C. Fallon, A. Garcia-Bellido0000-0002-1407-1972,
O. Hindrichs0000-0001-7640-5264, A. Khukhunaishvili0000-0002-3834-1316, P.
Parygin0000-0001-6743-3781, E. Popova0000-0001-7556-8969, R.
Taus0000-0002-5168-2932, G.P. Van Onsem0000-0002-1664-2337

The Rockefeller University, New York, New York, USA K.
Goulianos0000-0002-6230-9535

Rutgers, The State University of New Jersey, Piscataway, New Jersey, USA B.
Chiarito, J.P. Chou0000-0001-6315-905X, Y. Gershtein0000-0002-4871-5449, E.
Halkiadakis0000-0002-3584-7856, A. Hart0000-0003-2349-6582, M.
Heindl0000-0002-2831-463X, D. Jaroslawski0000-0003-2497-1242, O.
Karacheban260000-0002-2785-3762, I. Laflotte0000-0002-7366-8090, A.
Lath0000-0003-0228-9760, R. Montalvo, K. Nash, M. Osherson0000-0002-9760-9976,
H. Routray0000-0002-9694-4625, S. Salur0000-0002-4995-9285, S. Schnetzer, S.
Somalwar0000-0002-8856-7401, R. Stone0000-0001-6229-695X, S.A.
Thayil0000-0002-1469-0335, S. Thomas, J. Vora0000-0001-9325-2175, H.
Wang0000-0002-3027-0752

University of Tennessee, Knoxville, Tennessee, USA H. Acharya, A.G.
Delannoy0000-0003-1252-6213, S. Fiorendi0000-0003-3273-9419, T.
Holmes0000-0002-3959-5174, N. Karunarathna0000-0002-3412-0508, L.
Lee0000-0002-5590-335X, E. Nibigira0000-0001-5821-291X, S.
Spanier0000-0002-7049-4646

Texas A&M University, College Station, Texas, USA M. Ahmad0000-0001-9933-995X,
O. Bouhali920000-0001-7139-7322, M. Dalchenko0000-0002-0137-136X, A.
Delgado0000-0003-3453-7204, R. Eusebi0000-0003-3322-6287, J.
Gilmore0000-0001-9911-0143, T. Huang0000-0002-0793-5664, T.
Kamon930000-0001-5565-7868, H. Kim0000-0003-4986-1728, S.
Luo0000-0003-3122-4245, S. Malhotra, R. Mueller0000-0002-6723-6689, D.
Overton0009-0009-0648-8151, D. Rathjens0000-0002-8420-1488, A.
Safonov0000-0001-9497-5471

Texas Tech University, Lubbock, Texas, USA N. Akchurin0000-0002-6127-4350, J.
Damgov0000-0003-3863-2567, V. Hegde0000-0003-4952-2873, A.
Hussain0000-0001-6216-9002, Y. Kazhykarim, K. Lamichhane0000-0003-0152-7683,
S.W. Lee0000-0002-3388-8339, A. Mankel0000-0002-2124-6312, T. Mengke, S.
Muthumuni0000-0003-0432-6895, T. Peltola0000-0002-4732-4008, I.
Volobouev0000-0002-2087-6128, A. Whitbeck0000-0003-4224-5164

Vanderbilt University, Nashville, Tennessee, USA E. Appelt0000-0003-3389-4584,
S. Greene, A. Gurrola0000-0002-2793-4052, W. Johns0000-0001-5291-8903, R.
Kunnawalkam Elayavalli0000-0002-9202-1516, A. Melo0000-0003-3473-8858, F.
Romeo0000-0002-1297-6065, P. Sheldon0000-0003-1550-5223, S.
Tuo0000-0001-6142-0429, J. Velkovska0000-0003-1423-5241, J.
Viinikainen0000-0003-2530-4265

University of Virginia, Charlottesville, Virginia, USA B.
Cardwell0000-0001-5553-0891, B. Cox0000-0003-3752-4759, J.
Hakala0000-0001-9586-3316, R. Hirosky0000-0003-0304-6330, A.
Ledovskoy0000-0003-4861-0943, A. Li0000-0002-4547-116X, C.
Neu0000-0003-3644-8627, C.E. Perez Lara0000-0003-0199-8864

Wayne State University, Detroit, Michigan, USA P.E. Karchin0000-0003-1284-3470

University of Wisconsin - Madison, Madison, Wisconsin, USA A. Aravind, S.
Banerjee0000-0001-7880-922X, K. Black0000-0001-7320-5080, T.
Bose0000-0001-8026-5380, S. Dasu0000-0001-5993-9045, I. De
Bruyn0000-0003-1704-4360, P. Everaerts0000-0003-3848-324X, C. Galloni, H.
He0009-0008-3906-2037, M. Herndon0000-0003-3043-1090, A.
Herve0000-0002-1959-2363, C.K. Koraka0000-0002-4548-9992, A. Lanaro, R.
Loveless0000-0002-2562-4405, J. Madhusudanan Sreekala0000-0003-2590-763X, A.
Mallampalli0000-0002-3793-8516, A. Mohammadi0000-0001-8152-927X, S. Mondal, G.
Parida0000-0001-9665-4575, D. Pinna, A. Savin, V. Shang0000-0002-1436-6092, V.
Sharma0000-0003-1287-1471, W.H. Smith0000-0003-3195-0909, D. Teague, H.F.
Tsoi0000-0002-2550-2184, W. Vetens0000-0003-1058-1163, A.
Warden0000-0001-7463-7360

Authors affiliated with an institute or an international laboratory covered by
a cooperation agreement with CERN S. Afanasiev0009-0006-8766-226X, V.
Andreev0000-0002-5492-6920, Yu. Andreev0000-0002-7397-9665, T.
Aushev0000-0002-6347-7055, M. Azarkin0000-0002-7448-1447, A.
Babaev0000-0001-8876-3886, A. Belyaev0000-0003-1692-1173, V. Blinov94, E.
Boos0000-0002-0193-5073, V. Borshch0000-0002-5479-1982, D.
Budkouski0000-0002-2029-1007, M. Chadeeva940000-0003-1814-1218, V. Chekhovsky,
A. Dermenev0000-0001-5619-376X, T. Dimova940000-0002-9560-0660, D.
Druzhkin950000-0001-7520-3329, M. Dubinin850000-0002-7766-7175, L.
Dudko0000-0002-4462-3192, G. Gavrilov0000-0001-9689-7999, V.
Gavrilov0000-0002-9617-2928, S. Gninenko0000-0001-6495-7619, V.
Golovtcov0000-0002-0595-0297, N. Golubev0000-0002-9504-7754, I.
Golutvin0009-0007-6508-0215, I. Gorbunov0000-0003-3777-6606, A.
Gribushin0000-0002-5252-4645, Y. Ivanov0000-0001-5163-7632, V.
Kachanov0000-0002-3062-010X, L. Kardapoltsev940009-0000-3501-9607, V.
Karjavine0000-0002-5326-3854, A. Karneyeu0000-0001-9983-1004, V.
Kim940000-0001-7161-2133, M. Kirakosyan, D. Kirpichnikov0000-0002-7177-077X,
M. Kirsanov0000-0002-8879-6538, V. Klyukhin0000-0002-8577-6531, O.
Kodolova960000-0003-1342-4251, D. Konstantinov0000-0001-6673-7273, V.
Korenkov0000-0002-2342-7862, A. Kozyrev940000-0003-0684-9235, N.
Krasnikov0000-0002-8717-6492, A. Lanev0000-0001-8244-7321, P.
Levchenko970000-0003-4913-0538, O. Lukina0000-0003-1534-4490, N.
Lychkovskaya0000-0001-5084-9019, V. Makarenko0000-0002-8406-8605, A.
Malakhov0000-0001-8569-8409, V. Matveev94,980000-0002-2745-5908, V.
Murzin0000-0002-0554-4627, A. Nikitenko99,960000-0002-1933-5383, S.
Obraztsov0009-0001-1152-2758, V. Oreshkin0000-0003-4749-4995, A. Oskin, V.
Palichik0009-0008-0356-1061, V. Perelygin0009-0005-5039-4874, S.
Petrushanko0000-0003-0210-9061, S. Polikarpov940000-0001-6839-928X, V.
Popov0000-0001-8049-2583, O. Radchenko940000-0001-7116-9469, V. Rusinov, M.
Savina0000-0002-9020-7384, V. Savrin0009-0000-3973-2485, D.
Selivanova0000-0002-7031-9434, V. Shalaev0000-0002-2893-6922, S.
Shmatov0000-0001-5354-8350, S. Shulha0000-0002-4265-928X, Y.
Skovpen940000-0002-3316-0604, S. Slabospitskii0000-0001-8178-2494, V.
Smirnov0000-0002-9049-9196, A. Snigirev0000-0003-2952-6156, D.
Sosnov0000-0002-7452-8380, V. Sulimov0009-0009-8645-6685, E.
Tcherniaev0000-0002-3685-0635, A. Terkulov0000-0003-4985-3226, O.
Teryaev0000-0001-7002-9093, I. Tlisova0000-0003-1552-2015, A.
Toropin0000-0002-2106-4041, L. Uvarov0000-0002-7602-2527, A.
Uzunian0000-0002-7007-9020, A. Vorobyev${}^{\textrm{\textdagger}}$, N.
Voytishin0000-0001-6590-6266, B.S. Yuldashev100, A.
Zarubin0000-0002-1964-6106, I. Zhizhin0000-0001-6171-9682, A.
Zhokin0000-0001-7178-5907

†: Deceased
1Also at Yerevan State University, Yerevan, Armenia
2Also at TU Wien, Vienna, Austria
3Also at Institute of Basic and Applied Sciences, Faculty of Engineering, Arab
Academy for Science, Technology and Maritime Transport, Alexandria, Egypt
4Also at Université Libre de Bruxelles, Bruxelles, Belgium
5Also at Universidade Estadual de Campinas, Campinas, Brazil
6Also at Federal University of Rio Grande do Sul, Porto Alegre, Brazil
7Also at UFMS, Nova Andradina, Brazil
8Also at University of Chinese Academy of Sciences, Beijing, China
9Also at Nanjing Normal University, Nanjing, China
10Now at The University of Iowa, Iowa City, Iowa, USA
11Also at University of Chinese Academy of Sciences, Beijing, China
12Also at an institute or an international laboratory covered by a cooperation
agreement with CERN
13Also at Helwan University, Cairo, Egypt
14Now at Zewail City of Science and Technology, Zewail, Egypt
15Also at British University in Egypt, Cairo, Egypt
16Now at Ain Shams University, Cairo, Egypt
17Also at Purdue University, West Lafayette, Indiana, USA
18Also at Université de Haute Alsace, Mulhouse, France
19Also at Department of Physics, Tsinghua University, Beijing, China
20Also at The University of the State of Amazonas, Manaus, Brazil
21Also at Erzincan Binali Yildirim University, Erzincan, Turkey
22Also at University of Hamburg, Hamburg, Germany
23Also at RWTH Aachen University, III. Physikalisches Institut A, Aachen,
Germany
24Also at Isfahan University of Technology, Isfahan, Iran
25Also at Bergische University Wuppertal (BUW), Wuppertal, Germany
26Also at Brandenburg University of Technology, Cottbus, Germany
27Also at Forschungszentrum Jülich, Juelich, Germany
28Also at CERN, European Organization for Nuclear Research, Geneva,
Switzerland
29Also at Institute of Physics, University of Debrecen, Debrecen, Hungary
30Also at Institute of Nuclear Research ATOMKI, Debrecen, Hungary
31Now at Universitatea Babes-Bolyai - Facultatea de Fizica, Cluj-Napoca,
Romania
32Also at Physics Department, Faculty of Science, Assiut University, Assiut,
Egypt
33Also at HUN-REN Wigner Research Centre for Physics, Budapest, Hungary
34Also at Faculty of Informatics, University of Debrecen, Debrecen, Hungary
35Also at Punjab Agricultural University, Ludhiana, India
36Also at UPES - University of Petroleum and Energy Studies, Dehradun, India
37Also at University of Visva-Bharati, Santiniketan, India
38Also at University of Hyderabad, Hyderabad, India
39Also at Indian Institute of Science (IISc), Bangalore, India
40Also at IIT Bhubaneswar, Bhubaneswar, India
41Also at Institute of Physics, Bhubaneswar, India
42Also at Deutsches Elektronen-Synchrotron, Hamburg, Germany
43Now at Department of Physics, Isfahan University of Technology, Isfahan,
Iran
44Also at Sharif University of Technology, Tehran, Iran
45Also at Department of Physics, University of Science and Technology of
Mazandaran, Behshahr, Iran
46Also at Italian National Agency for New Technologies, Energy and Sustainable
Economic Development, Bologna, Italy
47Also at Centro Siciliano di Fisica Nucleare e di Struttura Della Materia,
Catania, Italy
48Also at Università degli Studi Guglielmo Marconi, Roma, Italy
49Also at Scuola Superiore Meridionale, Università di Napoli ’Federico II’,
Napoli, Italy
50Also at Fermi National Accelerator Laboratory, Batavia, Illinois, USA
51Also at Laboratori Nazionali di Legnaro dell’INFN, Legnaro, Italy
52Also at Università di Napoli ’Federico II’, Napoli, Italy
53Also at Consiglio Nazionale delle Ricerche - Istituto Officina dei
Materiali, Perugia, Italy
54Also at IRFU, CEA, Université Paris-Saclay, Gif-sur-Yvette, France
55Also at Riga Technical University, Riga, Latvia
56Also at Department of Applied Physics, Faculty of Science and Technology,
Universiti Kebangsaan Malaysia, Bangi, Malaysia
57Also at Consejo Nacional de Ciencia y Tecnología, Mexico City, Mexico
58Also at Trincomalee Campus, Eastern University, Sri Lanka, Nilaveli, Sri
Lanka
59Also at INFN Sezione di Pavia, Università di Pavia, Pavia, Italy
60Also at National and Kapodistrian University of Athens, Athens, Greece
61Also at Ecole Polytechnique Fédérale Lausanne, Lausanne, Switzerland
62Also at Universität Zürich, Zurich, Switzerland
63Also at Stefan Meyer Institute for Subatomic Physics, Vienna, Austria
64Also at Laboratoire d’Annecy-le-Vieux de Physique des Particules,
IN2P3-CNRS, Annecy-le-Vieux, France
65Also at Near East University, Research Center of Experimental Health
Science, Mersin, Turkey
66Also at Konya Technical University, Konya, Turkey
67Also at Izmir Bakircay University, Izmir, Turkey
68Also at Adiyaman University, Adiyaman, Turkey
69Also at Necmettin Erbakan University, Konya, Turkey
70Also at Bozok Universitetesi Rektörlügü, Yozgat, Turkey
71Also at Marmara University, Istanbul, Turkey
72Also at Milli Savunma University, Istanbul, Turkey
73Also at Kafkas University, Kars, Turkey
74Also at Hacettepe University, Ankara, Turkey
75Also at Istanbul University - Cerrahpasa, Faculty of Engineering, Istanbul,
Turkey
76Also at Ozyegin University, Istanbul, Turkey
77Also at Vrije Universiteit Brussel, Brussel, Belgium
78Also at School of Physics and Astronomy, University of Southampton,
Southampton, United Kingdom
79Also at University of Bristol, Bristol, United Kingdom
80Also at IPPP Durham University, Durham, United Kingdom
81Also at Monash University, Faculty of Science, Clayton, Australia
82Also at Università di Torino, Torino, Italy
83Also at Bethel University, St. Paul, Minnesota, USA
84Also at Karamanoğlu Mehmetbey University, Karaman, Turkey
85Also at California Institute of Technology, Pasadena, California, USA
86Also at United States Naval Academy, Annapolis, Maryland, USA
87Also at Bingol University, Bingol, Turkey
88Also at Georgian Technical University, Tbilisi, Georgia
89Also at Sinop University, Sinop, Turkey
90Also at Erciyes University, Kayseri, Turkey
91Also at Horia Hulubei National Institute of Physics and Nuclear Engineering
(IFIN-HH), Bucharest, Romania
92Also at Texas A&M University at Qatar, Doha, Qatar
93Also at Kyungpook National University, Daegu, Korea
94Also at another institute or international laboratory covered by a
cooperation agreement with CERN
95Also at Universiteit Antwerpen, Antwerpen, Belgium
96Also at Yerevan Physics Institute, Yerevan, Armenia
97Also at Northeastern University, Boston, Massachusetts, USA
98Now at another institute or international laboratory covered by a
cooperation agreement with CERN
99Also at Imperial College, London, United Kingdom
100Also at Institute of Nuclear Physics of the Uzbekistan Academy of Sciences,
Tashkent, Uzbekistan


# FastSample: Accelerating Distributed Graph Neural Network Training for
Billion-Scale Graphs

Hesham Mostafa  Intel Corporation ,  Adam Grabowski  Intel Corporation ,  Md
Asadullah Turja  UNC Chapel Hill ,  Juan Cervino  University of Pennsylvania ,
Alejandro Ribeiro  University of Pennsylvania and  Nageen Himayat  Intel
Corporation

(2024)

###### Abstract.

Training Graph Neural Networks(GNNs) on a large monolithic graph presents
unique challenges as the graph cannot fit within a single machine and it
cannot be decomposed into smaller disconnected components. Distributed
sampling-based training distributes the graph across multiple machines and
trains the GNN on small parts of the graph that are randomly sampled every
training iteration. We show that in a distributed environment, the sampling
overhead is a significant component of the training time for large-scale
graphs. We propose FastSample which is composed of two synergistic techniques
that greatly reduce the distributed sampling time: 1) a new graph partitioning
method that eliminates most of the communication rounds in distributed
sampling , 2) a novel highly optimized sampling kernel that reduces memory
movement during sampling. We test FastSample on large-scale graph benchmarks
and show that FastSample speeds up distributed sampling-based GNN training by
up to 2x with no loss in accuracy.

Graph neural networks, distributed training, Billion-scale graphs

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX

##  1\. Introduction

Graphs are the natural representation for many types of real-world data.
Recommendation graphs (Ying et al., 2018), knowledge graphs (Bollacker et al.,
2008), and molecular graphs (Tran et al., 2023) are few examples of graph-
structured data where we need to extract high-level semantic information about
the graph nodes, the graph links, or the whole graph. Graph Neural Networks
(GNNs) have emerged as a highly-accurate deep-learning based solution for
solving graph-related tasks (Bronstein et al., 2017; Zhou et al., 2020; Wu et
al., 2020). The nature of graph data, however, introduces many unique
challenges when training GNNs on large graphs. Large graphs with millions of
nodes and billions of edges, together with their node and edge features,
cannot typically fit within the memory of a single machine. Distributed
training is thus the natural approach to handle graphs of this size (Zheng et
al., 2020, 2022; Mostafa, 2022). In domains such as computer vision or natural
language processing, it is often straightforward to split the data across
several machines and train in a distributed data-parallel manner. The same
cannot be easily done for graph data. The interconnected nature of the graph
results in data dependencies between different machines no matter how we
partition the graph (Karypis and Kumar, 1997). Therefore, in addition to
parameter gradients, graph data also has to be periodically communicated
between the different machines in each training iteration (Tripathy et al.,
2020).

Distributed GNN training falls into two main categories: full-graph (or full-
batch) training (Mostafa, 2022; Md et al., 2021; Wan et al., 2023), and
sampling-based (or mini-batch) training (Zheng et al., 2020, 2022). In full-
graph training, the GNN is trained on the entire distributed graph at once,
while in sampling-based training, random subgraphs of the full graph are
sampled each training iteration and the GNN is trained on these graph samples.
While sampling-based training has many advantages such as a reduced memory
footprint that makes it more practical for larger graphs (Yang et al., 2022),
and faster convergence due to the multiple weight updates per epoch (Zeng et
al., 2019; Chiang et al., 2019), the graph sampling operation can often
consume a significant portion of the training time (Zhang et al., 2021).
Unlike mini-batching of i.i.d (Independent and identically distributed) data
sources such as images or segments of text, graph mini-batching is
computationally intensive as graph sampling involves traversing the graph to
construct a random sub-graph (Liu et al., 2021). We will show that the graph
sampling overhead is significant in both single-node and distributed GNN
training scenarios. The sampling overhead is more pronounced when working with
distributed graphs as the sampling routines in each worker would typically
need to traverse the graph across the graph partition boundaries. Distributed
graph sampling thus requires several communication rounds to exchange vertex
neighborhood information between the different machines hosting the graph
partitions, which can slow down the sampling operation (Zheng et al., 2020,
2022).

In this paper, we present two synergistic techniques to address the issues
outlined above and obtain dramatic improvements in time to convergence when
running distributed sampling-based GNN training. Our contributions can be
summarized as follows:

  * •

We present a highly optimized fused graph sampling kernel that is up to 2x
faster than the highly efficient kernels in the popular Deep Graph Library
(DGL)GNN package (Wang et al., 2019).

  * •

We observe that in many practical large-scale graphs, the size of the graph
topology (the adjacency matrix) is often minuscule compared to the size of the
graph’s node features. Motivated by that, we describe a hybrid partitioning
scheme that only partitions the node features across the training machines.
This leads to a significant reduction in the number of communication rounds in
distributed sampling-based training, boosting performance by up to 1.5x.

We built a complete software library, FastSample, that implements these two
techniques. We built the library on top of DGL. The ideas in FastSample,
however, can be used to accelerate any other GNN framework.

##  2\. Related Work

GNN training is typically done using open-source libraries such as DGL (Wang
et al., 2019) or PyG (Fey and Lenssen, 1903). There has been several efforts
that build on top of these libraries to enable scalable distributed GNN
training: DistDGL (Zheng et al., 2020) builds on top of DGL by introducing a
distributed sampling-based training pipeline backed up by a communication
backend that uses RPC(Remote Procedure Calls). DistDGLv2 (Zheng et al., 2022)
improves on DistDGL using better load balancing and mini-batch generation
heuristics. Quiver (Tan et al., 2023) scales up PyG model inference to multi-
GPU settings and uses dynamic heuristics to decide tensor placements and task
allocations between the CPU and GPU. GraphLearn (formerly known as Aligraph)
(Zhu et al., 2019) is a high-performance distributed training system that uses
smart neighbor caching to reduce communication in the distributed sampling
step.

An extensive body of work looks at accelerating sampling-based GNN training in
shared memory systems. These approaches, such as WholeGraph (Yang et al.,
2022), Zero-copy (Min et al., 2021), and NeuGraph (Ma et al., 2019), however,
cannot scale to graphs that cannot realistically fit within the main memory of
a single machine. A related approach is to replicate the full graph data
(topology and features) across all machines and only synchronize gradients in
each iteration (Kaler et al., 2022). This approach also does not scale. A work
that bears some similarities to our hybrid partitioning scheme is P3 (Gandhi
and Iyer, 2021) which splits the node feature between machines along the
feature dimension. Our scheme, instead, splits along the node dimension and
ensures that each machine stores the node features of the seed nodes allocated
to it.

Accelerating graph sampling is an important problem. Several approaches have
been proposed to bias the random neighborhood sampling operator to encourage
node reuse during sampling (Zheng et al., 2022; Chiang et al., 2019). Some
Hybrid CPU-GPU approaches use two-step sampling where a large number of nodes
is sampled on CPUs, which are then further sampled on GPUs to produce the
graph minibatch (Dong et al., 2021; Ramezani et al., 2020). These approaches,
however, do not optimize the core sampling operation itself like what we do in
the present work using our fused sampling kernel.

##  3\. Methods

###  3.1. Preliminaries

Let $G(\mathcal{V},\mathcal{E})$ be a graph where $\mathcal{V}$ is the set of
nodes and $\mathcal{E}\subset\mathcal{V}\times\mathcal{V}$ the set of edges.
The input node features of node $i$ are $h_{i}^{0}$ and the feature vector of
the edge from node $j$ to node $i$ is $e_{ji}$. Let $h_{i}^{l}$ be the feature
vector of node $i$ at layer $l$, the GNN layer equations for $l=1,\ldots,L$,
where $L$ is the number of GNN layers, are given by:

(1) $\displaystyle m_{j\rightarrow
i}^{l}=f_{message}(h_{i}^{l-1},h_{j}^{l-1},e_{ji};\theta^{l})$ (2)
$\displaystyle h_{i}^{l}=f_{node}(h_{i}^{l-1},Agg(\\{m_{j\rightarrow
i}^{l}:j\in\mathcal{N}(i)\\});\phi^{l}),$

where $f_{message}$ and $f_{node}$ are general functions whose learnable
parameters at layer $l$ are $\theta^{l}$ and $\phi^{l}$, respectively.
$\mathcal{N}(i)$ is the neighborhood of node $i$ in the graph, and
$m_{j\rightarrow i}$ are messages from node $j$ to an adjacent node $i$. $Agg$
is a permutation-invariant aggregation operator such as $sum$ or $mean$. In
node classification tasks, a subset of the nodes,
$\mathcal{V}^{C}\subset\mathcal{V}$, is labeled and the labels are given
during training. The goal of training is to minimize the prediction error on
these labeled nodes. More precisely, for an $L$-layer GNN, the goal of
training is to minimize the empirical loss:

(3)
$\mathcal{L}=\sum\limits_{v\in\mathcal{V}^{C}}\ell\left(h^{L}_{v},y_{v})\right),$

where $y_{v}$ is the label of node $v$ and $\ell$ is a standard classification
loss such as the cross-entropy loss.

In an $L$-layer GNN, we need the $L$-hop neighborhood of node $v$ in order to
produce the output at that node: $h^{L}_{v}$. For $L$ as low as 2 or 3, the
sizes of these neighborhoods can become prohibitively large, a phenomenon
known as the neighborhood explosion problem. Graph sampling is a standard
approach to address this issue: instead of working with the full $L$-hop
neighborhood, we sample a subset of the $L$-hop neighborhood and use this
subsampled neighborhood during training. More precisely, for an $L$-layer GNN,
we randomly pick a minibatch of training nodes ${\bf
B}\equiv\mathcal{V}^{L}\subset\mathcal{V}^{C}$ and recursively sample their
neighborhoods as follows for $l=L,\ldots,1$:

(4) $\displaystyle{\mathcal{E}}^{l-1}$
$\displaystyle=\bigcup\limits_{v\in\mathcal{V}^{l}}sample\\_edges(v,N_{l})$
(5) $\displaystyle{\mathcal{V}}^{l-1}$ $\displaystyle=\\{v:\exists u\text{
such that }e_{vu}\in{\mathcal{E}}^{l-1}\\}$

$sample\\_edges(v,N_{l})$ randomly samples $N_{l}$ incoming edges to node $v$.
$N_{l}$ is an important hyper-parameter known as the sampling fanout. There
are other more complex sampling operators (Zou et al., 2019; Zeng et al.,
2019; Chiang et al., 2019). However, in this paper, we focus on the random
neighborhood sampling operator outlined above as it is the one widely used in
practice.

After sampling, we construct $L$ bi-partite graphs, where the graph at layer
$l$ is
$\mathcal{G}^{l}=(\mathcal{V}^{l-1},\mathcal{V}^{l};\mathcal{E}^{l-1})$. The
graph at layer $l$ has edges $\mathcal{E}^{l-1}$ from source nodes
$\mathcal{V}^{l-1}$ to target nodes $\mathcal{V}^{l}$. These bi-partite graphs
are also known as Message Flow Graphs (MFGs). GNN layer $l$ is applied to
$\mathcal{G}^{l}$ and the output features for the nodes $\mathcal{V}^{l}$ are
used as the input features of the source nodes of $\mathcal{G}^{l+1}$

###  3.2. Fused sampling

The sampling operation has to be as efficient as possible as it is repeated
every training iteration. Figure 1 illustrates the steps involved in sampling
the graph for a 2-layer GNN in the DGL library. As described in the previous
section, the sampling operation has to be executed $L$ times for an $L$-layer
GNN. Sampling level $l$ involves two steps:

  1. (1)

Sampling the neighbors of the seed nodes $\mathcal{V}^{l}$

  2. (2)

Casting the resulting graph from step 1 as a bi-partite graph,
$\mathcal{G}^{l}$, and reordering the node indices so that they are
contiguous.

In DGL, both steps produce a graph in COO (COOrdinate) format. The preferred
format for GNNs is CSC (Compressed Sparse Column) as it allows us to fetch a
node’s neighbors in O(1), i.e, independently of the size of the graph. We thus
need an additional step to convert from the sampled graph from the COO to the
CSC format. The two formats are illustrated in Fig. 2.

The conventional 2-step approach that is applied at each sampling level
involves many redundant memory movements to write the output of step 1 in COO
format to memory and then read it again to compact it and convert it to CSC
format. Moreover, some information that is easily accessible in step 1 has to
be re-computed in step 2 such as the number of sampled neighbors of each seed
nodes (which can be less than the sampling fanout if the node has few
neighbors).

A CSC matrix ${\bf A}$ is defined by the 2-tuple ${\bf A}\equiv(R,C)$ where
$R$ and $C$ are the row pointer vector, and column indices vector,
respectively. See Fig. 2 for the definition of $R$ and $C$. Algorithm 1
describes our fused sampling kernel which is applied at every sampling level
to yield the CSC matrix for the bi-partite graph at that level as well as the
seed nodes for the next level down. Our kernel avoids the overhead of creating
the intermediate COO representation. It also constructs half of the CSC
representation (the $R$ vector) practically for free during the actual
sampling loop, and by sampling straight into the CSC format, it avoids the
expensive COO to CSC conversion. Our implementation is able to parallelize the
two For loops in the algorithm and the resulting kernel accelerates sampling
by a large margin compared to vanilla DGL sampling.

Algorithm 1 Fused sampling algorithm

Inputs: Graph adjacency in CSC format: ${\bf A}\equiv(R_{G},C_{G})$

Inputs: Unique seed nodes : $\mathcal{V}^{l}$

Inputs: Sampling fanout : $N_{l}$

$R_{l}=zeros(|\mathcal{V}^{l}|+1)$ $\triangleright$ zero vector of size
$|\mathcal{V}^{l}|+1$

$S_{l}=empty\\_list$

for $i:=0$ to $|\mathcal{V}^{l}|$ do

$v\leftarrow\mathcal{V}^{l}[i]$

$\triangleright$ Choose at most $N_{l}$ neighbors of $v$

$sampled\leftarrow Choose(C_{G}[R_{G}[v]:R_{G}[v+1]];N_{l})$

$S_{l}.extend(sampled)$

$R_{l}[i+1]\leftarrow R_{l}[i]+|Sampled|$

end for

$M=fill(|R_{G}|,-1)$ $\triangleright$ vector of size $|R_{G}|$ filled with -1

$C_{l}=zeros(|S_{l}|)$

$\mathcal{V}^{l-1}=empty\\_list$

$idx=0$

for $i:=0$ to $|S_{l}|$ do

$v\leftarrow S_{l}[i]$

if $M[v]==-1$ then

$\mathcal{V}^{l-1}.append(v)$

$M[v]\leftarrow idx$

$idx\leftarrow idx+1$

end if

$C_{l}[i]\leftarrow M[v]$

end for

return $CSCMatrix(R_{l},C_{l}),\mathcal{V}^{l-1}$

Figure 1. Sampling the graph for a 2-layer GNN. The sampling fanout is $2$
throughout. The sampled neighbors at one level become the seed nodes for the
level below. The bi-partite graph produced at each level is used by the
corresponding GNN layer.  Figure 2. Illustration of the COO and CSC
representations of a sparse adjacency matrix $A$. $nnz$ is the number of non-
zero elements. In COO, $(X[i],Y[i])$ are the row-column position of the
$i^{th}$ non-zero element in $A$. In CSC, $R[k+1]-R[k]$ is the number of non-
zero elements in row $k$ of $A$. The position of the non-zero elements in row
$k$ are $C[R[k]:R[k+1]]$

###  3.3. Distributed sampling and hybrid partitioning

In a distributed setting, we partition the graph across multiple machines. We
consider edge-cut partitioning. Each worker stores one graph partition which
comprises the partition node features as well as all incoming edges to the
partition nodes from all other partitions. As illustrated in Fig. 3, each
machine independently samples seed nodes from the local partition, then
independently samples the neighbors of these seed nodes. This can be done
locally as each machine knows the neighbors of its local nodes (but not their
features). All sampling levels below the top level, however, require machines
to communicate. That is because each machine might need to sample the
neighbors of nodes that are not in its local partition.

In general, for each sampling level after the top level, we would require two
communication rounds, one round where machines submit their sampling requests
to other machines, and one round where machines reply to each other with the
the result of the sampling requests. For $L$-level sampling, we thus need
$2(L-1)$ communication rounds to sample the graph. Since the graph features
are also partitioned, two additional communication rounds are needed at the
end for machines to exchange the input features of the sampled graph, bringing
the total number of communication rounds to $2L$. Note that each machine
creates its own local sampled graph. The GNN is trained on the local sampled
graph in each machine, and the parameter gradients are then synchronized.

One important observation is that for many large-scale graphs, node features
form the bulk of the graph size. This is illustrated in Fig. 4 for two of the
largest open-source graphs currently available. Motivated by this observation,
we propose to duplicate the graph topology across all machines while
partitioning the relatively much larger feature tensor. Sampling the graph
topology can thus proceed independently in all machines. Communication is only
needed to exchange the input node features of the sampled graph, bringing the
number of communication rounds from $2L$ down to $2$. We call this scheme
hybrid partitioning and we show that it significantly speeds up training while
keeping memory consumption per machine within reasonable limits.

Figure 3. Distributed sampling for a 2-layer GNN. The graph topology and node
features are partitioned across three machines. At the top sampling level,
each machine independently samples its local seed nodes and the neighbors of
these seed nodes. For all sampling levels after that ($L-1$ levels),
communication is needed. Communication is also needed to exchange the input
node features of the sampled graphs.  Figure 4. Breakdown of the graph storage
requirements for MAG240M (Hu et al., 2021) and IGBH-full (Khatua et al.,
2023). Graph topology (adjacency matrix) is only a small fraction of total
graph size

##  4\. Results

We benchmark our contributions on two popular benchmark graphs: ogbn-products
and ogbn-papers100M. The graph properties are outlined in Table 1. We present
results for different combinations of our techniques. We run our experiments
on 2-socket machines, each equipped with two 4th Gen Intel Xeon Scalable
Processors. All machines are connected by a 200Gbps Infiniband HDR fabric. In
all experiments we use a 3-layer GraphSage model with hidden layer dimensions
of 256. We use dropout between all layers. We use PyTorch 2.0.1 and DGL 1.1.
We use FP32 numerical precision throughout. We build our FastSample library on
top of DGL.

In distributed training experiments, we use torch_ccl (tor, ) as our
communication backend. torch_ccl is the PyTorch wrapper for Intel’s OneCCL
collective communication library (one, ). Unlike popular distributed GNN
training libraries such as DistDGL (Zheng et al., 2020), we do not use point-
to-point communication, but exclusively use synchronous collective
communication calls such as $all\\_to\\_all$ and $all\\_reduce$. We use the
metis library (Karypis and Kumar, 1997) to partition the graph. Metis
minimizes the number of edges that cross the partition boundaries. It balances
the number of nodes and edges in each partition so that all partitions are
roughly of the same size. We also assign roughly the same number of labeled
nodes to each partition. Since the top level sampling seeds are drawn from the
labeled nodes, equalizing the number of labeled nodes across machines ensures
they all have roughly the same number of seeds and can generate the same
number of graph samples during each training epoch (see Fig. 3). In
distributed training experiments, we always use a batch size of 1000 per
machine. We use a learning rate of 0.006.

Table 1. Graph Datasets |  | ogbn-
---
products
| ogbn-
---
papers100M
# nodes | 2.5M | 111M
# edges | 124M | 3.2B
# input features | 100 | 128
# classes | 47 | 172

###  4.1. Fused sampling kernel

We evaluate the performance of our fused sampling kernel on ogbn-papers100M.
There are two major hyper-parameters of the sampling operation: the batch size
which is the number of top-level seed nodes, and the fan-out values at each
sampling level. For a 3-layer GNN, the later can be written as a 3-tuple
$(N_{3},N_{2},N_{1})$ denoting the number of neighbors sampled at layers 3,2,
and 1, respectively. As shown in Fig. 5, our fused sampling kernel
consistently speeds up the graph sampling operation across a wide range of
batch sizes and sampling fanouts. For the full training time (sampling + GNN
training), we attain speedups of up to $25\%$.

Figure 5. Speedup of the sampling time and the overall training time in single
node training on ogbn-papers100M. Our baseline is the highly optimized
sampling kernels in DGL. We plot the speedups for various mini-batch sizes
($1024,2048,\ldots,10240$) and different sampling fanout values for each of
the three GNN layers in the model. Top panel: Speedup of the sampling
operation. We obtain up to 2x speedup. Bottom panel: Speedup of overall
training time (sampling + GNN training). We obtain speedups that are typically
in the range of $10\%$ to $25\%$.

###  4.2. Hybrid partitioning

To evaluate our hybrid partitioning method, we run distributed training on 8
and 16 machines. Hybrid partitioning synergizes very well with our fused
sampling kernel because the full graph topology is available on each machine
and can thus be fed directly to our fused sampling kernel. Figure 6 shows the
distributed training epoch times when training on ogbn-products and ogbn-
papers100M. Hybrid partitioning leads to a significant reduction in epoch
time. When we combine hybrid partitioning with our fused sampling kernel, we
get an even bigger performance boost. Hybrid partitioning together with fused
sampling reduce the per-epoch training time by almost 2X when training ogbn-
papers100M on 8 machines.

Note that hybrid partitioning and fused sampling have no effect on the
convergence properties of the training. Activating or disabling these two
techniques lead to mathematically equivalent training results. We also note
that for fair comparison, all experiments use the same training and
communication software infrastructure. This ensures that any improvements on
the vanilla baseline are solely due to our new methods.

Figure 6. Epoch times when training ogbn-products and ogbn-papers100M on 4 and
8 machines. We consider three scenarios: 1)vanilla training where we partition
the graph topology and features and do not use fused sampling, 2)hybrid
partitioning where we only partition the graph features and do not use fused
sampling, 3)hybrid+fused where we only partition the graph features and use
our fused sampling kernel.

##  5\. Conclusions

Sampling-based training is becoming the method of choice to train GNNs on
large graphs. As graph sizes continue to increase, distributed sampling-based
training becomes the natural choice to accommodate massive graphs. In this
paper, we have shown that the high-performance low-level CPU sampling kernels
that are part of the DGL library, and that are in common use today, still
leave plenty of performance on the table. Our fused kernel is able to
consistently reduce the sampling time, in some cases accelerating the low-
level graph sampling operations by up to 2X.

To make optimal use of our fused sampling kernel, we used a hybrid
partitioning scheme that duplicates graph topology across all workers. The
increased memory consumption per machine due to the duplicated adjacency
matrix is an acceptable compromise since duplicating graph topology greatly
reduces the number of communication rounds needed for sampling. The
combination of hybrid partitioning and our fused sampling kernel is
particularly attractive as it leaves the training iterations mathematically
unchanged while greatly speeding up the distributed training (by up to 2X as
shown in Fig. 6).

In the future, our work can be extended in several ways. For example, we can
combine our hybrid partitioning scheme with feature caching to cache
frequently accessed remote node features in order to reduce communication
volume. Or we can use an adaptive fanout schedule to dynamically adjust the
sampling fanouts based on the training dynamics.

## References

  * Ying et al. (2018) Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec.  Graph convolutional neural networks for web-scale recommender systems.  In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_, pages 974–983, 2018.
  * Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor.  Freebase: a collaboratively created graph database for structuring human knowledge.  In _Proceedings of the 2008 ACM SIGMOD international conference on Management of data_ , pages 1247–1250, 2008.
  * Tran et al. (2023) Richard Tran, Janice Lan, Muhammed Shuaibi, Brandon M Wood, Siddharth Goyal, Abhishek Das, Javier Heras-Domingo, Adeesh Kolluru, Ammar Rizvi, Nima Shoghi, et al.  The open catalyst 2022 (oc22) dataset and challenges for oxide electrocatalysts.  _ACS Catalysis_ , 13(5):3066–3084, 2023.
  * Bronstein et al. (2017) Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst.  Geometric deep learning: going beyond euclidean data.  _IEEE Signal Processing Magazine_ , 34(4):18–42, 2017.
  * Zhou et al. (2020) Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun.  Graph neural networks: A review of methods and applications.  _AI open_ , 1:57–81, 2020.
  * Wu et al. (2020) Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip.  A comprehensive survey on graph neural networks.  _IEEE transactions on neural networks and learning systems_ , 32(1):4–24, 2020.
  * Zheng et al. (2020) Da Zheng, Chao Ma, Minjie Wang, Jinjing Zhou, Qidong Su, Xiang Song, Quan Gan, Zheng Zhang, and George Karypis.  Distdgl: distributed graph neural network training for billion-scale graphs.  In _2020 IEEE/ACM 10th Workshop on Irregular Applications: Architectures and Algorithms (IA3)_ , pages 36–44. IEEE, 2020.
  * Zheng et al. (2022) Da Zheng, Xiang Song, Chengru Yang, Dominique LaSalle, and George Karypis.  Distributed hybrid cpu and gpu training for graph neural networks on billion-scale heterogeneous graphs.  In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , pages 4582–4591, 2022.
  * Mostafa (2022) Hesham Mostafa.  Sequential aggregation and rematerialization: Distributed full-batch training of graph neural networks on large graphs.  _Proceedings of Machine Learning and Systems_ , 4:265–275, 2022.
  * Karypis and Kumar (1997) George Karypis and Vipin Kumar.  Metis: A software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices.  1997\.
  * Tripathy et al. (2020) Alok Tripathy, Katherine Yelick, and Aydın Buluç.  Reducing communication in graph neural network training.  In _SC20: International Conference for High Performance Computing, Networking, Storage and Analysis_ , pages 1–14. IEEE, 2020.
  * Md et al. (2021) Vasimuddin Md, Sanchit Misra, Guixiang Ma, Ramanarayan Mohanty, Evangelos Georganas, Alexander Heinecke, Dhiraj Kalamkar, N.K. Ahmed, and Sasikanth Avancha.  Distgnn: Scalable distributed training for large-scale graph neural networks.  _arXiv preprin tarXiv:2104.06700_ , 2021.
  * Wan et al. (2023) Xinchen Wan, Kaiqiang Xu, Xudong Liao, Yilun Jin, Kai Chen, and Xin Jin.  Scalable and efficient full-graph gnn training for large graphs.  _Proceedings of the ACM on Management of Data_ , 1(2):1–23, 2023.
  * Yang et al. (2022) Dongxu Yang, Junhong Liu, Jiaxing Qi, and Junjie Lai.  Wholegraph: a fast graph neural network training framework with multi-gpu distributed shared memory architecture.  In _SC22: International Conference for High Performance Computing, Networking, Storage and Analysis_ , pages 1–14. IEEE, 2022.
  * Zeng et al. (2019) Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna.  Graphsaint: Graph sampling based inductive learning method.  _arXiv preprint arXiv:1907.04931_ , 2019.
  * Chiang et al. (2019) Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh.  Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 257–266, 2019.
  * Zhang et al. (2021) Bingyi Zhang, Sanmukh R Kuppannagari, Rajgopal Kannan, and Viktor Prasanna.  Efficient neighbor-sampling-based gnn training on cpu-fpga heterogeneous platform.  In _2021 IEEE High Performance Extreme Computing Conference (HPEC)_ , pages 1–7. IEEE, 2021.
  * Liu et al. (2021) Xin Liu, Mingyu Yan, Lei Deng, Guoqi Li, Xiaochun Ye, and Dongrui Fan.  Sampling methods for efficient training of graph convolutional networks: A survey.  _IEEE/CAA Journal of Automatica Sinica_ , 9(2):205–234, 2021.
  * Wang et al. (2019) Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma, et al.  Deep graph library: Towards efficient and scalable deep learning on graphs.  2019\.
  * Fey and Lenssen (1903) Matthias Fey and Jan Eric Lenssen.  Fast graph representation learning with pytorch geometric. arxiv 2019\.  _arXiv preprint arXiv:1903.02428_ , 1903.
  * Tan et al. (2023) Zeyuan Tan, Xiulong Yuan, Congjie He, Man-Kit Sit, Guo Li, Xiaoze Liu, Baole Ai, Kai Zeng, Peter Pietzuch, and Luo Mai.  Quiver: Supporting gpus for low-latency, high-throughput gnn serving with workload awareness.  _arXiv preprint arXiv:2305.10863_ , 2023.
  * Zhu et al. (2019) Rong Zhu, Kun Zhao, Hongxia Yang, Wei Lin, Chang Zhou, Baole Ai, Yong Li, and Jingren Zhou.  Aligraph: A comprehensive graph neural network platform.  _arXiv preprint arXiv:1902.08730_ , 2019.
  * Min et al. (2021) Seung Won Min, Kun Wu, Sitao Huang, Mert Hidayetoğlu, Jinjun Xiong, Eiman Ebrahimi, Deming Chen, and Wen-mei Hwu.  Large graph convolutional network training with gpu-oriented data communication architecture.  _arXiv preprint arXiv:2103.03330_ , 2021.
  * Ma et al. (2019) Lingxiao Ma, Zhi Yang, Youshan Miao, Jilong Xue, Ming Wu, Lidong Zhou, and Yafei Dai.  Neugraph: parallel deep neural network computation on large graphs.  In _2019 $\\{$USENIX$\\}$ Annual Technical Conference ($\\{$USENIX$\\}$$\\{$ATC$\\}$ 19)_, pages 443–458, 2019.
  * Kaler et al. (2022) Tim Kaler, Nickolas Stathas, Anne Ouyang, Alexandros-Stavros Iliopoulos, Tao Schardl, Charles E Leiserson, and Jie Chen.  Accelerating training and inference of graph neural networks with fast sampling and pipelining.  _Proceedings of Machine Learning and Systems_ , 4:172–189, 2022.
  * Gandhi and Iyer (2021) Swapnil Gandhi and Anand Padmanabha Iyer.  P3: Distributed deep graph learning at scale.  In _15th $\\{$USENIX$\\}$ Symposium on Operating Systems Design and Implementation ($\\{$OSDI$\\}$ 21)_, pages 551–568, 2021.
  * Dong et al. (2021) Jialin Dong, Da Zheng, Lin F Yang, and George Karypis.  Global neighbor sampling for mixed cpu-gpu training on giant graphs.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pages 289–299, 2021.
  * Ramezani et al. (2020) Morteza Ramezani, Weilin Cong, Mehrdad Mahdavi, Anand Sivasubramaniam, and Mahmut Kandemir.  Gcn meets gpu: Decoupling “when to sample” from “how to sample”.  _Advances in Neural Information Processing Systems_ , 33:18482–18492, 2020.
  * Zou et al. (2019) Difan Zou, Ziniu Hu, Yewen Wang, Song Jiang, Yizhou Sun, and Quanquan Gu.  Layer-dependent importance sampling for training deep and large graph convolutional networks.  _Advances in neural information processing systems_ , 32, 2019.
  * Hu et al. (2021) Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec.  Ogb-lsc: A large-scale challenge for machine learning on graphs.  _arXiv preprint arXiv:2103.09430_ , 2021.
  * Khatua et al. (2023) Arpandeep Khatua, Vikram Sharma Mailthody, Bhagyashree Taleka, Tengfei Ma, Xiang Song, and Wen-mei Hwu.  Igb: Addressing the gaps in labeling, features, heterogeneity, and size of public graph datasets for deep learning research.  _arXiv preprint arXiv:2302.13522_ , 2023.
  * (32) torch_ccl.  https://github.com/intel/torch-ccl.  Accessed: 2021-10-5.
  * (33) Oneccl.  https://github.com/oneapi-src/oneCCL.  Accessed: 2021-10-5.