Mariia Demianenko, Igor Chilingarian, Kirill Grishin, Vladimir Goradzhanov,
Victoria Toptun, Ivan Katkov, and Ivan Kuzmin 11institutetext: Moscow
Institute of Physics and Technology (National Research University)
11email<EMAIL_ADDRESS>
22institutetext: Sternberg Astronomical Institute, M.V. Lomonosov Moscow State
University
33institutetext: HSE University
44institutetext: Center for Astrophysics – Harvard and Smithsonian (USA)
55institutetext: Department of Physics, M.V. Lomonosov Moscow State University
66institutetext: New York University Abu Dhabi (UAE)
77institutetext: Center for Astro, Particle, and Planetary Physics, NYU Abu
Dhabi (UAE)

# Optical light curves of light-weight supermassive black holes produced by
the Zwicky Transient Facility Forced Photometry Service

Mariia Demianenko 11 2 2 3 3 Kirill Grishin 22 Victoria Toptun 22 5 5 Igor
Chilingarian 44 2 2 Ivan Katkov 66 7 7 2 2 Vladimir Goradzhanov 22 5 5 Ivan
Kuzmin 22 5 5

###### Abstract

In this paper, we present an algorithm to correct optical light curves
obtained using The Zwicky Transient Facility Forced Photometry Service and its
application to the analysis of optical variability of 136 actvie galactic
nuclei (AGN) powered by “light-weight” supermassive black holes (SMBH;
$M_{BH}<2*10^{6}\sun$) including 24 intermediate-mass black holes (IMBH;
$M_{BH}<2*10^{5}\sun$). We detected variability in nearly all sources and also
analyzed its dependence on the X-ray luminosity for 101 objects. We also
identified a previously unknown candidate tidal disruption event (TDE) in SDSS
J112637.74+513423.0.

###### keywords:

active galactic nucleus, intermediate-mass black holes

## Introduction.

The presence of optical variability in a galaxy center is one of the most
reliable diagnostics of an AGN to date. Optical light curves are used to
search for a statistically significant variability of emission. The size of a
region illuminated by an AGN in optical wavelengths defines the stochastic
variability time scale in the light curves of such objects according to e.g.
Burke et al. (2021). This allows one to use optical light curves to search and
confirm IMBHs (Martínez-Palomera et al. 2020) because $M_{BH}$ limits the AGN
luminosity. $M_{BH}$ can be measured from broad optical emission line profiles
in AGN spectra (Greene & Ho 2004; Bon et al. 2019; Chilingarian et al. 2018).

Figure 1: Top left: The light curve J163159.59+243740.3 obtained using ZTF
Forced Photometry Service. Top right: The light curve of the candidate TDE
before correction. Bottom left: The light curve of J163159.59+243740.3 after
correction, points in one field of one quadrant of one CCD are visualized.
Bottom right: The light curve of the candidate TDE after correction. The
points in the most occurring quadrant of the CCD are shown. Figure 2:
Dependence of the the optical light curve variability amplitude on the X-ray
luminosity the two ZTF photometric bands, ‘ZTF g’ (green) and ‘ZTF r’ (red).

## Data.

Chilingarian et al. (2018) defined a sample of IMBHs from the analysis of
optical spectra with the NBursts full spectral fitting (Chilingarian et al.
2007a, b) in the RCSED catalog (Chilingarian et al. 2017), with $M_{BH}$
estimated between $3*10^{4}M\sun$ and $2*10^{5}M\sun$. 10 of these objects
were confirmed in X-ray in Chilingarian et al. (2018), and another 14 were
confirmed later from new and archival X-ray data. We define light-weight SMBHs
as SMBHs having $M_{BH}<2*10^{6}M\sun$. 112 objects from the expanded sample
of candidates light-weight SMBHs were confirmed by archival X-ray data. In
total, we studies 136 optical light curves for 24 bona fide IMBHs and 112 bona
fide light-weight SMBHs. Out of 136 objects, 101 had photometric points in two
filters, ‘ZTF r’ and ’ZTF g’.

## Analysis of optical variability in light-weight SMBHs and IMBHs.

To estimate the optical variability amplitude and its timescale, we need to
construct calibrated light curves without systematic variations exceeding
10-15% of the host galaxy flux. We use differential images, which are obtained
by subtracting the reference image from a current science image matching the
point-spread-function (PSF). The reference image is a weighted average of all
images for a given field, degraded to the same resolution using a Gaussian
filter. Then, PSF photometry is performed without restrictions on the flux
positivity. This approach is implemented in the ZTF Forced Photometry Service.

The resulting points do not always have the same zero point, because they were
observed in different quadrants of different CCDs in the focal plane of the
telescope. And also, color corrections are performed using zero color from the
PanSTARRS photometric system. This correction is well defined only for those
objects for which zero color measurement is available, otherwise the
correction is estimated by the interpolation of PanSTARRS values. For each
quadrant of the CCD and the field, the higher and lower flux limits of that
sensor are defined, and points with flux measurements outside these limit
derived from a specific observation should be excluded (Masci et al. 2019).
Here, we set the threshold signal-to-noise ratio (SNR) to 3. The distance to
the nearest reference was set to $<$1 arcsec, pixel quality indicators helped
to clear outliers related to uncorrected cosmic ray hits and hot pixels. The
zero-point correction was made for each observation, based on the CCD
quadrant, and the average airmass during an observation. Then we defined the
level of significance according to the $\chi^{2}$ criterion at which the light
curve is not consistent with white noise on top of a constant. Many of the
studied objects showed statistically significant variability on timescales
from days to months.

Two examples of uncorrected ZTF Forced Photometry light curves are shown in
Fig. 1 (top row). The bottom row of Fig. 1 shows the light curves after
correction. In Fig.2 we show the dependence of the amplitude of optical
variability (errors calculated using bootstrap) from the X-ray luminosity for
101 objects from our sample.

Also, a strong flare with an amplitude of 0.25 mag was observed in the object
J112637.74+513423.0 for about 3 months, which can be either a heavily dust
obscured supernova explosion or a star ruptured by tidal forces in the SMBH
vicinity (TDE). We show a ZTF light curve of the TDE candidate in Fig 1 (top
right).A more reliable light curve after the correction is shown in Fig 1
(bottom right): the “hump” is reliably traced by data points even in one CCD
quadrant hence clearly confirming that the flare is real.

The bottleneck that limits the scalability of our analysis and makes it yet
impossible to study thousands of AGN is that the ZTF Forced Photometry Service
can process up to 100 objects at a time and it takes hours to weeks to process
every request.

## Conclusions.

The use of optical variability confirmed the high quality of the selection of
IMBH and light-weight SMBHs candidates using a broad component $H\alpha$ and,
in the future, will allow us to exclude candidates where broad lines have a
transient or star formation related nature. Also, light curve analysis can be
used for independent confirmation or selection of IMBH and light-weight SMBH
candidates, because there are strong correlations of $M_{BH}$ and the optical
timescale according to Burke et al. (2021). Acknowledgement. MD is supported
by the RSF project 17-72-20119.

## References

  * Bon et al. (2019) Bon, E., Marziani, P., Jovanović, P., & Bon, N. 2019, Atoms, 7
  * Burke et al. (2021) Burke, C. J., Shen, Y., Blaes, O., et al. 2021, Science, 373, 789
  * Chilingarian et al. (2007a) Chilingarian, I., Prugniel, P., Sil’Chenko, O., & Koleva, M. 2007a, in Stellar Populations as Building Blocks of Galaxies, ed. A. r. Vazdekis & R. Peletier, Vol. 241, 175–176
  * Chilingarian et al. (2018) Chilingarian, I. V., Katkov, I. Y., Zolotukhin, I. Y., et al. 2018, ApJ, 863, 1
  * Chilingarian et al. (2007b) Chilingarian, I. V., Prugniel, P., Sil’Chenko, O. K., & Afanasiev, V. L. 2007b, MNRAS, 376, 1033
  * Chilingarian et al. (2017) Chilingarian, I. V., Zolotukhin, I. Y., Katkov, I. Y., et al. 2017, ApJS, 228, 14
  * Greene & Ho (2004) Greene, J. E. & Ho, L. C. 2004, ApJ, 610, 722
  * Martínez-Palomera et al. (2020) Martínez-Palomera, J., Lira, P., Bhalla-Ladd, I., Förster, F., & Plotkin, R. M. 2020, ApJ, 889, 113
  * Masci et al. (2019) Masci, F. J., Laher, R. R., Rusholme, B., et al. 2019, PASP, 131, 018003


LaBella and Calabrese 1 2024 Medical Imaging with Deep Learning (MIDL) 2020
Marleen de Bruijne, Tal Arbel, Ismail Ben Ayed, Hervé Lombaert BraTS 2023
Meningioma Challenge AnalysisLaBella and Calabrese 1 Duke University Medical
Center, Department of Radiation Oncology, Durham, NC, USA
2 Division of Computational Pathology, Department of Pathology and Laboratory
Medicine, Indiana University School of Medicine, Indianapolis, IN, USA
3 Center for Federated Learning in Medicine, Indiana University, Indianapolis,
IN, USA
4 Department of Neurosurgery, Thomas Jefferson University, Philadelphia, PA,
USA
5 University of California San Francisco, CA, USA
6 Yale University, New Haven, CT, USA
7 Department of Radiation Oncology, Perelman School of Medicine, University of
Pennsylvania, Philadelphia, PA, USA
8 University of Missouri, Columbia, MO, USA
9 University of Missouri, Columbia, MO, USA
10 Children’s Hospital of Philadelphia, University of Pennsylvania,
Philadelphia, PA, USA
11 Center for AI and Data Science for Integrated Diagnostics (AI2D) and Center
for Biomedical Image Computing and Analytics (CBICA), University of
Pennsylvania, Philadelphia, PA, USA
12 Helmholtz AI, Helmholtz Munich, Germany
13 Department of Informatics, Technical University Munich, Germany
14 TranslaTUM - Central Institute for Translational Cancer Research, Technical
University of Munich, Germany
15 Department of Diagnostic and Interventional Neuroradiology, School of
Medicine, Klinikum rechts der Isar, Technical University of Munich, Germany
16 Center for Intelligent Imaging (ci2), Department of Radiology and
Biomedical Imaging, University of California San Francisco (UCSF), San
Francisco, CA, USA
17 Biomedical Image Analysis and Machine Learning, Department of Quantitative
Biomedicine, University of Zurich, Switzerland
18 Department of Neuroradiology, Technical University of Munich, Munich,
Germany
19 University of San Diego, CA, USA
20 Cornell University, Ithaca, NY, USA
21 Department of Radiology, Northwestern University, Evanston, IL, USA
22 Duke University Medical Center, School of Medicine, Durham, NC, USA
23 Medical Artificial Intelligence (MAI) Lab, Crestview Radiology, Lagos,
Nigeria
24 Sage Bionetworks, USA
25 Montreal Neurological Institute (MNI), McGill University, Montreal, QC,
Canada
26 Children’s National Hospital, Washington DC, USA
27 George Washington University, Washington DC, USA
28 Mayo Clinic, Rochester, MN, USA
29 Center for Global Health, Perelman School of Medicine, University of
Pennsylvania, Philadelphia, PA, USA
30 Athinoula A Martinos Center for Biomedical Imaging, Massachusetts General
Hospital and Harvard Medical School, Boston, MA, USA
31 Centre for Medical Image Computing, University College London, London, UK
32 Computer Science and Artificial Intelligence Laboratory, Massachusetts
Institute of Technology, Cambridge, MA, USA
33 PrecisionFDA, U.S. Food and Drug Administration, Silver Spring, MD, USA
34 Department of Applied Mathematics and Computer Science, Technical
University of Denmark, Denmark
35 Athinoula A Martinos Center for Biomedical Imaging, Massachusetts General
Hospital, Boston, MA, USA
36 University of Zurich, Switzerland
37 Booz Allen Hamilton, McLean, VA, USA
38 Mercy Catholic Medical Center, Darby, PA, USA
39 Penn Statistics in Imaging and Visualization Center, Department of
Biostatistics, Epidemiology, and Informatics
40 Greensboro Radiology, Greensboro, NC, USA
41 Radiology Partners, El Segundo, CA, USA
42 Duke University Medical Center, Department of Radiology, Durham, NC, USA
43 Neosoma Inc. Stanford Medicine, Stanford, CA, USA
44 Lagos University Teaching Hospital, Lagos Nigeria.
45 University of Zürich, Zürich,Switzerland
46 Department of Radiology, Perelman School of Medicine, University of
Pennsylvania, Philadelphia, PA, USA
47 Department of Neuroradiology, Uppsala University, Sweden
48 University of Toronto, Toronto, ON, Canada
49 Cancer Imaging Program, National Cancer Institute, National Institutes of
Health, Bethesda, MD, USA
50 Leidos Biomedical Research, Inc., Frederick National Laboratory for Cancer
Research, Frederick, MD, USA
51 Department of Radiology, Brigham and Women’s Hospital, Boston, MA, USA
52 Institute of Neuroscience and Medicine (INM-4), Research Center Juelich,
Juelich, Germany
53 Department of Nuclear Medicine, University Hospital RWTH Aachen, Aachen,
Germany
54 Artificial Intelligence Lab, Department of Radiology, Mayo Clinic,
Rochester, MN, USA
55 Department of Neurosurgery, Indiana University, Indianapolis, IN, USA
56 Department of Medical Image Computing, German Cancer Research Center
(DKFZ), Heidelberg, Germany
57 Department of Neuroradiology, University Hospital Bonn, Bonn Germany
58 Department of Diagnostic and Interventional Radiology, University Hospital
Ulm, Ulm, Germany
59 Department of Diagnostic and Interventional Neuroradiology, School of
Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich,
Germany
60 TUM-Neuroimaging Center, Klinikum rechts der Isar, Technical University of
Munich, Munich, Germany
61 Fort Worth, TX, USA
62 Department of Diagnostic and Interventional Radiology, All India Institute
of Medical Sciences, Rishikesh, India
63 Department of Neuroradiology, Washington University, St. Louis, MO, USA
64 University of Rochester Medical Center, Rochester, NY, USA
65 Faculty of Medicine, Jena University Hospital, Friedrich Schiller
University Jena, Jena, Germany
66 vRad (Radiology Partners), Minneapolis, MN, USA
67 Ross University School of Medicine, Bridgetown, Barbados
68 Department of Radiology, New York University Grossman School of Medicine,
New York, NY, USA
69 Department of Radiology, Arad Hospital, Tehran, Iran
70 Department of Radiology, Faculty of Medicine, University of Ottawa, Cananda
71 Department of Neuroradiology, JDMI, University of Toronto, TO, Canada
72 Department of Radiology Northwestern University, Chicago, IL
73 Department of Radiology, Azienda Ospedaliero Universitaria of Cagliari-Polo
di Monserrato, Cagliari, Italy
74 Department of Radiology, Leeds General Infirmary, Leeds, United Kingdom
75 Ludwig Maximilians University, Munich, Bavaria, Germany
76 Rush University Medical Center, Chicago, IL, USA
77 Department of Neurosurgery, University Hospital of Ioannina, Ioannina,
Greece
78 Department of Neuroradiology, University Hospital Essen, Essen, North
Rhine-Westphalia, Germany
79 Egyptian Ministry of Health, Cairo, Egypt
80 Department of Radiology, Kasr Alainy, Cairo University, Cairo, Egypt
81 Faculty of Medicine, Medical University of Sofia, Sofia, Bulgaria
82 Department of Radiology, American University of Beirut Medical center,
Beirut, Lebanon
83 Radiology Department, King Faisal Medical City, Abha, Saudi Arabia
84 Department of Diagnostic and Interventional Radiology, Cairo University,
Cairo, Egypt
85 Department of Radiology and Medical Imaging, University of Virginia Health,
Charlottesville, VA, USA
86 Department of Radiology and Radiologic Sciences, Vanderbilt University
Medical Center, TN, USA
87 Faculty of Medicine, Hamburg University, Hamburg, Germany
88 Neuroimaging Unit, ASST Ovest Milanese, Legnano, Milan, Italy
89 University of Manitoba, Manitoba, Canada
90 Department of Radiology, School of Medicine, Iran University of Medical
Sciences, Iran, Tehran
91 Department of Radiology, University Hospitals Bristol and Weston NHS
Foundation Trust, Bristol, United Kingdom
92 Department of Radiology, University of Texas Southwestern Medical Center,
TX, USA
93 CMH Lahore Medical College, Lahore, Pakistan
94 MLCommons
95 Factored AI
96 Center For Federated Learning in Medicine, Indiana University,
Indianapolis, IN, USA
97 Medical Working Group, MLCommons, San Fransisco, CA, USA
98 Intel
99 Department of Radiology, Thomas Jefferson University, Philadelphia, PA, USA
100 Department of Radiology and Imaging Sciences, Indiana University School of
Medicine, Indianapolis, IN, USA
101 Department of Neurological Surgery, Indiana University School of Medicine,
Indianapolis, IN, USA
102 Department of Biostatistics and Health Data Science, Indiana University
School of Medicine, Indianapolis, IN, USA

# Analysis of the BraTS 2023 Intracranial Meningioma Segmentation Challenge

Dominic LaBella10000-0003-1713-9538 Ujjwal Baid2,3 Omaditya Khanna4 Shan
McBurney-Lin5 Ryan McLean6 Pierre Nedelec5 Arif Rashid7 Nourel Hoda Tahon8
Talissa Altes4 Radhika Bhalerao5 Yaseen Dhemesh9 Devon Godfrey1 Fathi Hilal8
Scott Floyd1 Anastasia Janas6 Anahita Fathi Kazerooni10,11 John Kirkpatrick1
Collin Kent1 Florian Kofler12,13,14,15 Kevin Leu16 Nazanin Maleki6 Bjoern
Menze17,18 Maxence Pajot5 Zachary J. Reitman1 Jeffrey D. Rudie19,5 Rachit
Saluja20 Yury Velichko21 Chunhao Wang1 Pranav Warman22 Maruf Adewole23 Jake
Albrecht24 Udunna Anazodo25 Syed Muhammad Anwar26,27 Timothy Bergquist24 Sully
Francis Chen22 Verena Chung24 Gian-Marco Conte28 Farouk Dako29 James Eddy24
Ivan Ezhov13,14 Nastaran Khalili10 Juan Eugenio Iglesias30,31,32 Zhifan
Jiang26,27 Elaine Johanson33 Koen Van Leemput34 Hongwei Bran Li35,18,36 Marius
George Linguraru26,27 Xinyang Liu26,27 Aria Mahtabfar4 Zeke Meier37 Ahmed W
Moawad38 John Mongan5 Marie Piraud12 Russell Takeshi Shinohara39,11 Walter F.
Wiggins40,41,42 Aly H. Abayazeed43 Rachel Akinola44 András Jakab45 Michel
Bilello11,46 Maria Correia de Verdier47 Priscila Crivellaro48 Christos
Davatzikos11,46 Keyvan Farahani49 John Freymann50,49 Christopher Hess5 Raymond
Huang51 Philipp Lohmann52,53 Mana Moassefi54 Matthew W. Pease55 Phillipp
Vollmuth56,57 Nico Sollmann58,59,60 David Diffley61 Khanak K. Nandolia62
Daniel I Warren63 Ali Hussain64 Pascal Fehringer65 Yulia Bronstein66 Lisa
Deptula67 Evan G. Stein68 Mahsa Taherzadeh69 Eduardo Portela de Oliveira70
Aoife Haughey71 Marinos Kontzialis72 Luca Saba73 Benjamin Turner74 Melanie M.
T. Brüßeler75 Shehbaz Ansari76 Athanasios Gkampenis77 David Maximilian Weiss78
Aya Mansour79 Islam H. Shawali80 Nikolay Yordanov81 Joel M. Stein46 Roula
Hourani82 Mohammed Yahya Moshebah83 Ahmed Magdy Abouelatta84 Tanvir Rizvi85
Klara Willms6 Dann C. Martin86 Abdullah Okar87 Gennaro D’Anna88 Ahmed Taha89
Yasaman Sharifi90 Shahriar Faghani28 Dominic Kite91 Marco Pinho92 Muhammad
Ammar Haider93 Alejandro Aristizabal94,95 Alexandros Karargyris94 Hasan
Kassem94 Sarthak Pati96,2,97 Micah Sheller98,94 Michelle Alonso-Basanta7
Javier Villanueva-Meyer5 Andreas M Rauschecker5 Ayman Nada8 Mariam Aboian10
Adam E. Flanders99 Benedikt Wiestler18 Spyridon
Bakas2,100,101,1020000-0001-8734-6482 Evan Calabrese42,5

###### Abstract

We describe the design and results from the BraTS 2023 Intracranial Meningioma
Segmentation Challenge. The BraTS Meningioma Challenge differed from prior
BraTS Glioma challenges in that it focused on meningiomas, which are typically
benign extra-axial tumors with diverse radiologic and anatomical presentation
and a propensity for multiplicity. Nine participating teams each developed
deep-learning automated segmentation models using image data from the largest
multi-institutional systematically expert annotated multilabel multi-sequence
meningioma MRI dataset to date, which included 1000 training set cases, 141
validation set cases, and 283 hidden test set cases. Each case included T2,
T2/FLAIR, T1, and T1Gd brain MRI sequences with associated tumor compartment
labels delineating enhancing tumor, non-enhancing tumor, and surrounding non-
enhancing T2/FLAIR hyperintensity. Participant automated segmentation models
were evaluated and ranked based on a scoring system evaluating lesion-wise
metrics including dice similarity coefficient (DSC) and 95% Hausdorff
Distance. The top ranked team had a lesion-wise median dice similarity
coefficient (DSC) of 0.976, 0.976, and 0.964 for enhancing tumor, tumor core,
and whole tumor, respectively and a corresponding average DSC of 0.899, 0.904,
and 0.871, respectively. These results serve as state-of-the-art benchmarks
for future pre-operative meningioma automated segmentation algorithms.
Additionally, we found that 1286 of 1424 cases (90.3%) had at least 1
compartment voxel abutting the edge of the skull-stripped image edge, which
requires further investigation into optimal pre-processing face anonymization
steps.

###### keywords:

Meningioma, BraTS, Machine Learning, Segmentation, BraTS-Meningioma, Image
Analysis Challenge, artificial intelligence, AI

##  1 Introduction and Related Works

Meningiomas are the most common primary brain tumors, and several common
treatment modalities, including surgical resection and radiation therapy,
require accurate delineation of tumor components (Ogasawara et al., 2021;
Rogers et al., 2017, 2020). When used clinically, meningioma MRI segmentation
is often performed using T1-weighted (T1), T2-weighted (T2), T2-FLAIR (FLAIR),
and T1 post-contrast (T1Gd) multi-sequence brain magnetic resonance image
(MRI) (Martz et al., 2022). Meningioma segmentation on brain MRI can be
challenging due to the diverse morphology and location of meningiomas within
the brain. Anatomically, meningiomas arise from the arachnoid layer of the
meninges between the dura mater and pia mater and commonly present at
supratentorial sites of dural reflection, along the sphenoid sinus, and the
skull base. Less commonly, meningiomas occur in intraventricular and
suprasellar regions, the olfactory groove, and in the posterior fossa along
the petrous bone (LaBella et al., 2023). Their extra-axial location can
frequently lead to their exclusion, in whole or in part, from brain MRI skull-
stripping pre-processing steps. Radiographically, meningioma can present with
a wide range of presentations which contributes to the difficulty in creating
accurate generalizable meningioma automated segmentation models (Watts et al.,
2014). Commonly encountered radiographical variants and findings include en
plaque meningioma, which is a plaque-like sessile extension of tumor along the
meninges, cystic meningioma components, dural tail involvement extension,
peri-tumoral edema, and numerous distinct lesions (Watts et al., 2014).

Recent advancements in segmentation techniques for brain tumors, particularly
gliomas, through the application of deep learning and convolutional neural
networks (CNNs), have shown promise in overcoming these challenges, offering
increased accuracy and reproducibility compared to traditional methods
(Pereira et al., 2016; Havaei et al., 2017; Bouget et al., 2022). Since the
inception of the Brain Tumor Segmentation (BraTS) challenge in 2012, they have
been instrumental in propelling forward the field of brain tumor imaging
segmentation by providing comprehensive datasets that facilitate the
development and benchmarking of segmentation algorithms (Menze et al., 2014;
Bakas et al., 2017). The inaugural 2012 challenge had 35 training cases and 15
test cases focused solely on glioma, and the glioma dataset most recently
increased to over 2000 cases included in the 2023 challenge. Studies by Menze
et al. and Bakas et al. underscore the importance of the BraTS dataset in
improving the segmentation accuracy for gliomas, leveraging multi-sequence MR
images to improve the delineation of tumor tissues from non-tumorous brain
matter (Menze et al., 2014; Bakas et al., 2017; Gordillo et al., 2013; Işın et
al., 2016).

In 2023, the BraTS organizing committee hosted new automated segmentation
challenges to additionally focus on pediatric tumors, gliomas diagnosed in
sub-Saharan Africa, brain metastasis, and meningioma (bra, 2023; LaBella et
al., 2023; Baid et al., 2021; Kazerooni et al., 2023; Moawad et al., 2023).
Building on the foundation of prior BraTS challenges, the BraTS 2023
Intracranial Meningioma Segmentation Challenge aims to establish a community
standard and benchmark for intracranial meningioma segmentation (Bakas et al.,
2017; LaBella et al., 2023; Calabrese and LaBella, 2023). We present a
comprehensive analysis of segmentation performance across nine teams
participating in the challenge focusing on key metrics: Enhancing tumor (ET)
dice similarity coefficient (DSC), tumor core (TC) DSC, and whole tumor (WT)
DSC, ET 95% Hausdorff Distance (95HD), TC 95HD, and WT 95HD. These metrics
were evaluated on a lesion-wise basis to account for the possibility of
multiple lesions.

By evaluating each of the competing teams’ automated segmentation algorithms
performance, we can identify state-of-the-art machine learning algorithm
techniques. By doing so, we anticipate extension beyond the technical realm,
to impacting patient outcomes, surgical approaches, radiation therapy
planning, and understanding tumor behavior such as the propensity for an
extra-axial location. As such, this study contributes to the technical field
of medical imaging analysis and to the broader understanding of meningioma
treatment and management strategies.

##  2 Methods

###  2.1 Challenge Data

BraTS Meningioma Challenge image data was contributed from 6 different United
States academic medical centers: Duke University, Yale University, Thomas
Jefferson University, University of California San Francisco, Missouri
University, and University of Pennsylvania. Image data consisted of T1, T2,
FLAIR, and T1Gd brain MRI sequences from patients with radiographic or
pathologic diagnosis of intracranial meningioma. All data preprocessing was
conducted using the FeTS tool, (Pati et al., 2022) and included conversion to
NIfTI file format, co-registration, $1\,\text{mm}^{3}$ isotropic resampling to
the SRI24 atlas space, and automated skull stripping (Schwarz et al., 2019;
Thakur et al., 2019, 2020; Juluru et al., 2020). After image data pre-
processing, tumor compartment labels were created using a comprehensive pre-
segmentation, manual correction, and expert revision process, and included
enhancing tumor, non-enhancing tumor, and surrounding non-enhancing T2/FLAIR
hyperintensity (SNFH) as seen in Figure 1, which is an unmodified figure by
LaBella et al. (LaBella et al., 2023, 2024). All participating institutions
received Institutional Review Board and Data Transfer Agreement approvals
before contributing data, ensuring compliance with relevant regulatory
authorities.

Figure 1: Meningioma sub-compartments considered in the BraTS Pre-operative
Meningioma Dataset. Image panels A-C denote the different tumor sub-
compartments included in manual annotations; (A) enhancing tumor (blue)
visible on a T1-weighted post-contrast image; (B) the non-enhancing tumor core
(red) visible on a T1-weighted post-contrast image; (C) the surrounding FLAIR
hyperintensity (green) visible on a T2/FLAIR-weighted image; (D) combined
segmentations generating the final tumor sub-compartment labels provided in
the BraTS Pre-operative Meningioma Dataset.

###  2.2 Challenge Procedures and Timeline

The BraTS 2023 Intracranial Meningioma Segmentation Challenge was hosted on
the Synapse platform using the BraTS Pre-operative Meningioma Dataset (bra,
2023). To access the challenge dataset and to be eligible for submission of
automated segmentation models, participants were required to register as a
participating team on the Synapse platform. Registered teams developed
automated segmentation algorithms that trained on multi-sequence MRI of pre-
treatment intracranial meningioma with associated ground truth labels that
were released to the participating teams in May 2023.

In June 2023, each of the participating teams had access to additional
validation data consisting of multi-sequence MRI cases. For validation data,
teams were able to assess segmentation performance of their models by
submitting predicted labels through the Synapse platform, but individual
ground truth segmentations were not made publicly available. From July until
August 2023, participating teams utilized the validation dataset to fine tune
their segmentation models and compose short paper manuscripts. At the end of
the validation phase, each participating team uploaded their optimal automated
segmentation model and respective manuscript as an MLCube container, which was
used for evaluation in the testing phase. During the testing phase, the BraTS
organizing committee internally evaluated each of the participating team’s
automated segmentation models on the hidden test set of pre-operative
meningioma cases with ground truth labels.

###  2.3 Algorithm Evaluation

During the testing phase, the BraTS organizing committee evaluated metrics on
three regions of interest including ET, TC, and WT. ET was solely the
enhancing tumor compartment label. TC was the combination of enhancing tumor
and non-enhancing tumor compartment labels. WT was the combination of
enhancing tumor, non-enhancing tumor, and SNFH compartment labels. Note that
the term “whole tumor” was used across the BraTS 2023 cluster of challenges
for consistency; however, this term is not entirely accurate for meningioma,
where SNFH typically does not contain any tumor but rather represents
associated vasogenic edema. Metrics used for evaluation included DSC and the
95HD and were evaluated on a lesion-wise level. The DSC is a measure used to
quantify the similarity between two samples, which, in this context, refers to
the overlap between automated segmentation and the expert annotated ground
truth labels for each respective tumor compartment. The 95HD observed in the
segmentation results. The 95HD was used in lieu of the standard 100% Hausdorff
Distance to account for smaller lesions that may suffer from overestimates of
the standard 100% Hausdorff Distance. For previous BraTS challenges, a global
DSC was used for challenge rankings. However, lesion-wise metrics were adopted
for the 2023 challenge as there was greater potential for multiple distinct
lesions in a single patient image (most notably for the metastasis and
meningioma sub-challenges). Distinct lesions were identified by performing a 1
voxel symmetric dilation on the ground truth WT masks, and then evaluating a
26-connectivity 3D connected component analysis to determine if overlap
between distinct lesions exists (Rudie, 2023). A case’s lesion-wise DSC and
95HD scores are calculated based on equations (1) and (2) respectively
(Saluja, 2023). The lesion-wise scoring system assigned a specific lesion’s
region of interest a DSC score of 0 and a 95HD score of 374 for false positive
or false negatives. The scoring system also excluded evaluation for ground
truth lesions smaller than 50 voxels to avoid evaluation of false ground truth
lesions missed in dataset review. This threshold was discussed and decided by
fellowship trained neuroradiologists after ground truth label review (Saluja,
2023; Rudie, 2023). Evaluation of submissions was performed on MLCommons’
MedPerf, an open federated AI/ML evaluation platform (Karargyris et al.,
2023). MedPerf automated the evaluating pipeline by running the participants’
models on the testing datasets of each contributing site’s data and
calculating evaluation metrics on the resulting predictions. Finally, the
Synapse platform retrieved the metrics results from the MedPerf server and
ranked them to determine the winner (MLCommons Association, 2024; Pati et al.,
2023; Karargyris et al., 2023).

$\text{Lesion-wise Dice
Score}=\frac{\sum_{i}^{L}\text{Dice}(I_{i})}{\text{TP}+\text{FN}+\text{FP}}$
(1) $\text{Lesion-wise
HD95}=\frac{\sum_{i}^{L}\text{HD}_{95}(I_{i})}{\text{TP}+\text{FN}+\text{FP}}$
(2)

###  2.4 Participant Ranking and Workshop Proceedings

The BraTS organizing committee internally evaluated each of the participating
team’s automated segmentation models on the hidden test set of pre-operative
meningioma cases to determine lesion-wise metrics for both DSC and 95HD for
each of the three regions of interest. The participants were ranked against
each other for each lesion-wise metric independently. Then a “BraTS
segmentation score” was calculated based on the average of each independent
lesion-wise metric ranking. The BraTS segmentation score was used to determine
the final participant rankings relative to one another. The three top-ranked
teams were invited to present their findings at the BraTS workshop at the 2023
MICCAI Annual Meeting held in Vancouver, Canada; although final rank was
hidden until the workshop. At the BraTS workshop, the BraTS organizing
committee announced the final placement of the three top-ranked teams.
Monetary awards of $1400, $1000, and $800 were presented to the three top-
ranked teams, respectively.

###  2.5 Challenge Results Analysis

Overall participant and individual team statistical analysis of DSC and 95HD
lesion-wise performance was performed using Python and Microsoft Excel
(Excel). Analysis included calculation of participant average, standard
deviation, and median DSC and 95HD for each region of interest; overall
average and median DSC and 95HD across all participants for each region of
interest; volume calculations of each lesion; and number of abutting voxels of
lesions compared to the pre-processed brain MRI.

###  2.6 Analysis of Tumor Abutment of Brain Masks

Given the extra-axial location of meningiomas, we sought to evaluate the
proportion of meningiomas that were potentially cropped or excluded by the
automated skull stripping process. To determine the volume of each compartment
label and the number of tumor compartment voxels that were directly abutting
the edge of the skull-stripped images for each case, the
NumberOfEdgeNeighbors.py script was internally run by BraTS organizers, which
evaluated each of the 1483 meningioma MRI cases (LaBella, 2024). This analysis
was performed internally due to restricted access to the hidden test dataset.
To determine significance of association of WT volume compared to abutting
voxels, the Pearson Correlation Coefficient was calculated with associated
p-value with a significance level of 0.05.

##  3 Results

A total of 1000 training (70%) multi-sequence pre-operative meningioma MRI
cases, 141 validation cases (10%), and 283 test cases (20%) were utilized
within the BraTS Meningioma Challenge (Table 1) in adherence with standard
machine learning protocols.

A total of 9 participating teams submitted automated segmentation models to
MLCube for the BraTS Challenge 2023: Intracranial Meningioma. The statistical
summary of the teams’ performances is outlined in Tables 2, 3, and 4; which
list the calculated DSCs, and in Tables 5, 6, and 7; which list the calculated
95HD. The maximum recorded average DSC for ET, TC, and WT are 0.899, 0.904,
and 0.871, respectively; and the minimum recorded average 95HD for ET, TC, and
WT are 23.9, 21.8, and 31.4, respectively; highlighting the upper bounds of
team performance within the challenge. The overall challenge summary
statistics across all participating teams are listed in Tables 8 and 9 for DSC
and 95HD respectively.

Table 1: This table presents the total number of cases in each of the training, validation, and testing sets. Note that the training data was released with ground truth labels. Note that the validation data was released without ground truth labels. The included institutions are Duke University (Duke), Thomas Jefferson University (TJU), Missouri (Miss), University of Pennsylvania (Penn), University of California San Francisco (UCSF), and Yale University (Yale). | Train | Validation | Test
---|---|---|---
Total Count | 1000 | 141 | 283
Duke | 315 | 46 | 91
TJU | 236 | 34 | 68
Miss | 132 | 16 | 33
Penn | 31 | 4 | 9
UCSF | 126 | 18 | 35
Yale | 160 | 23 | 47
Release Date | May 2023 | June 2023 | Never released
Table 2: Team DSC, average $\pm$ SD (median), for the ET region of interest. Relative team rankings for each respective metric. Team Name | ET DSC | Rank
---|---|---
NVAUTO | 0.899 $\pm$ 0.189 (0.976) | 1
CNMC_PMI2023 | 0.876 $\pm$ 0.217 (0.968) | 2
blackbean | 0.870 $\pm$ 0.222 (0.969) | 3
Sherlock | 0.854 $\pm$ 0.234 (0.958) | 4
huilin | 0.830 $\pm$ 0.276 (0.959) | 5
i_sahajmistry | 0.799 $\pm$ 0.291 (0.954) | 6
Kurtlab-UW | 0.790 $\pm$ 0.237 (0.896) | 7
MIA | 0.775 $\pm$ 0.305 (0.940) | 8
UMNiverse | 0.007 $\pm$ 0.084 (0.000) | 9
Table 3: Team DSC, average $\pm$ SD (median), for the TC region of interest. Relative team rankings for each respective metric. Team Name | TC DSC | Rank |
---|---|---|---
NVAUTO | 0.904 $\pm$ 0.180 (0.976) | 1 |
blackbean | 0.879 $\pm$ 0.206 (0.969) | 2 |
CNMC_PMI2023 | 0.867 $\pm$ 0.227 (0.968) | 3 |
Sherlock | 0.850 $\pm$ 0.239 (0.959) | 4 |
huilin | 0.820 $\pm$ 0.258 (0.958) | 5 |
Kurtlab-UW | 0.774 $\pm$ 0.250 (0.892) | 6 |
i_sahajmistry | 0.773 $\pm$ 0.303 (0.949) | 7 |
MIA | 0.757 $\pm$ 0.307 (0.941) | 8 |
UMNiverse | 0.027 $\pm$ 0.078 (0.006) | 9 |
Table 4: Team DSC, average $\pm$ SD (median), for the WT region of interest. Relative team rankings for each respective metric. Team Name | WT DSC | Rank |
---|---|---|---
NVAUTO | 0.871 $\pm$ 0.198 (0.964) | 1 |
CNMC_PMI2023 | 0.851 $\pm$ 0.231 (0.953) | 2 |
blackbean | 0.845 $\pm$ 0.226 (0.957) | 3 |
Sherlock | 0.831 $\pm$ 0.244 (0.945) | 4 |
i_sahajmistry | 0.764 $\pm$ 0.296 (0.932) | 5 |
huilin | 0.761 $\pm$ 0.297 (0.897) | 6 |
MIA | 0.751 $\pm$ 0.306 (0.916) | 7 |
Kurtlab-UW | 0.745 $\pm$ 0.257 (0.872) | 8 |
UMNiverse | 0.241 $\pm$ 0.290 (0.092) | 9 |
Table 5: Team 95% Hausdorff distances, average $\pm$ SD (median), for ET regions of interest. Relative team rankings for each respective metric. Team Name | ET 95HD | Rank
---|---|---
NVAUTO | 23.9 $\pm$ 68.5 (0.96) | 1
CNMC_PMI2023 | 30.0 $\pm$ 80.9 (1.0) | 2
blackbean | 34.3 $\pm$ 82.0 (1.0) | 3
Sherlock | 34.3 $\pm$ 87.5 (1.07) | 4
Kurtlab-UW | 39.9 $\pm$ 90.2 (2.0) | 5
huilin | 46.9 $\pm$ 104.2 (1.0) | 6
i_sahajmistry | 56.5 $\pm$ 108.9 (1.0) | 7
MIA | 61.4 $\pm$ 117.6 (1.41) | 8
UMNiverse | 371.4 $\pm$ 314.9 (374.0) | 9
Table 6: Team 95% Hausdorff distances, average $\pm$ SD (median), for TC regions of interest. Relative team rankings for each respective metric. Team Name | TC 95HD | Rank
---|---|---
NVAUTO | 21.8 $\pm$ 64.6 (1.0) | 1
blackbean | 29.9 $\pm$ 75.0 (1.0) | 2
CNMC_PMI2023 | 31.7 $\pm$ 83.5 (1.0) | 3
Sherlock | 35.1 $\pm$ 88.3 (1.93) | 4
Kurtlab-UW | 45.9 $\pm$ 95.8 (2.0) | 5
huilin | 47.7 $\pm$ 105.2 (1.41) | 6
MIA | 61.6 $\pm$ 118.0 (1.73) | 7
i_sahajmistry | 64.1 $\pm$ 112.5 (1.41) | 8
UMNiverse | 150.1 $\pm$ 151.5 (81.5) | 9
Table 7: Team 95% Hausdorff distances, average $\pm$ SD (median), for WT regions of interest. Relative team rankings for each respective metric. Team Name | WT 95HD | Rank
---|---|---
NVAUTO | 31.4 $\pm$ 71.8 (1.0) | 1
CNMC_PMI2023 | 35.2 $\pm$ 86.8 (1.62) | 2
Sherlock | 39.7 $\pm$ 93.1 (1.94) | 3
blackbean | 41.2 $\pm$ 84.3 (1.0) | 4
huilin | 55.9 $\pm$ 106.9 (3.61) | 5
Kurtlab-UW | 56.0 $\pm$ 100.4 (1.05) | 6
MIA | 64.2 $\pm$ 119.5 (2.83) | 7
i_sahajmistry | 66.2 $\pm$ 111.0 (2.0) | 8
UMNiverse | 158.8 $\pm$ 146.9 (189.1) | 9
Table 8: This table presents participating team summary statistics for DSC across three categories: ET, TC, and WT, reflecting the performance of 9 participating teams in a segmentation task. The average represents the overall participants median of all the team’s average DSC scores. The median (Q1, Q3) represents the overall challenge median and 1st and 3rd quartile, respectively, of all the team’s median DSC scores. The max average and max median reflect the top performing team’s respective average or median DSC for each label. | ET | TC | WT
---|---|---|---
Average | 0.830 | 0.820 | 0.764
Std | 0.234 | 0.239 | 0.257
Median | 0.958 | 0.958 | 0.933
(Q1, Q3) | [0.872, 0.981] | [0.850, 0.979] | [0.630, 0.972]
Max Avg | 0.899 | 0.904 | 0.871
Max Med | 0.976 | 0.976 | 0.964
Table 9: This table presents participating team summary statistics for 95HD across three categories: ET, TC, and WT, reflecting the performance of 9 participating teams in a segmentation task. The average represents the overall participants median of all the team’s average 95HD scores. The median (Q1, Q3) represents the overall challenge median and 1st and 3rd quartile, respectively, of all the team’s median 95HD scores. The min average and min median reflect the top performing team’s respective average or median 95HD for each label. | ET | TC | WT
---|---|---|---
Average | 39.9 | 46.0 | 55.9
Std | 87.5 | 95.76 | 100.4
Median | 1.00 | 1.414 | 2.00
(Q1, Q3) | [1.00, 4.24] | [1.00, 6.27] | [1.00, 14.5]
Min Avg | 23.9 | 21.8 | 31.4
Min Med | 1.00 | 1.00 | 1.00

The top 3 ranked teams for the BraTS Meningioma Challenge included NVAUTO,
blackbean, and CNMC_PMI2023 (Capellan-Martin, 2023; Myronenko et al., 2023;
Huang et al., 2023). Each of these teams were invited to give an oral
presentation of their findings and methods at MICCAI 2023 in Vancouver,
Canada. Their lesion-wise DSC and 95HD and median lesion-wise DSC, averaged
over ET, TC, and WT are listed in Table 10. The team leaders and submitted
short paper titles are listed below:

  1. 1.

NVAUTO: Andriy Myronenko et al., Auto3DSeg for Brain Tumor Segmentation from
3D MRI in BraTS 2023 Challenge (Myronenko et al., 2023)

  2. 2.

blackbean: Ziyan Huang et al., Evaluating STU-Net for Brain Tumor Segmentation
(Huang et al., 2023)

  3. 3.

CNMC_PMI2023: Daniel Capellán-Martín et al., Model Ensemble for Brain Tumor
Segmentation in Magnetic Resonance Imaging (Capellan-Martin, 2023)

Table 10: This table shows the lesion-wise metrics for each of the top 3 ranked teams in the BraTS Challenge 2023: Intracranial Meningioma challenge. Note that a precision of 4 decimals is used due to the close final results amongst top participating teams. | Average | Median | Average
---|---|---|---
Team Name | DSC | DSC | 95HD
NVAUTO | 0.8909 | 0.9855 | 25.70
Blackbean | 0.8643 | 0.9861 | 34.84
CNMC_PMI2023 | 0.8638 | 0.9855 | 32.30

Of note, NVAUTO placed in the top 2 for each of the five distinct BraTS 2023
automated segmentation challenges. NVAUTO came in first place for Meningioma,
BraTS-Africa, Brain Metastases; and came in second place for Adult Glioma and
Pediatric Tumors. For the meningioma challenge, NVAUTO had a total of 228 of
283 testing phase cases with a tumor core DSC $\geq$ 0.90. Additionally,
during the public validation phase, as described in their in-person oral
presentation at MICCAI, NVAUTO reported achieving an average composite DSC of
0.935, which is substantially higher than their testing phase top score of
0.891, which suggests some degree of overfitting (Myronenko et al., 2023).

###  3.1 Notable Challenge Cases and Statistics

The top median DSC across all participants for a specific case was a perfect
score of 1.00 for enhancing tumor and tumor core as shown in Figure 2. Note
that there was a total of 33 enhancing tumor voxels, and 23 were abutting the
edge of the MR image. The cases with the next highest overall median and
average DSC are shown in Figure 3. Note that the ET volumes are qualitatively
much greater in these particularly high DSC cases.

There were two meningioma cases with somewhat unusual imaging appearance as
shown in Figures 4 and 5 which had poor performance for test performance
metrics across all participants. Notably, they had a majority of non-enhancing
tumor making up the TC and WT. These lesions correspond to heavily calcified
lesions with little or no visible enhancement and low signal intensity on all
provided sequences.

Figure 2: Image panels of the top scored individual test case with a median
participant DSC of 1.00, 1.00, and 1.00 and average participant DSC of 0.993,
0.881, and 0.882 for enhancing tumor, tumor core, and whole tumor;
respectively. Tumor ground truth sub-compartment labels annotated on axial
(A), sagittal (B), and coronal (C) views of a T1Gd MRI head case. Panel D
depicts the tumor abutting the edge of the skull-stripped brain without a
label. Figure 3: Image panels that depict participants median enhancing tumor
and tumor core DSC of (A) 0.993, (B) 0.991, (C) 0.991, and (D) 0.991 (0.990
for TC). Note that the averages were 0.876, 0.878, 0.864, 0.875 respectively,
which is due to a single team having a DSC of approximately 0.002 for each
case, whereas every other team had an ET DSC above 0.95 for each case. Figure
4: Image panels depicting a completely calcified extra-axial non-enhancing
meningioma that had a median participant DSC of 1.00, 0.00, and 0.00 and
average participant DSC of 0.888, 0.175, and 0.174 for enhancing tumor, tumor
core, and whole tumor; respectively. Figure 5: MRI study demonstrating the
worst performing meningioma case for NVAUTO with an ET DSC of 0.00, TC DSC of
0.00, and WT DSC of 0.338. The blue and red labels represent ground truth
enhancing tumor and non-enhancing tumor, respectively, which combined make up
the tumor core region of interest. Figure 6: Scatterplot graph depicting the
overall participant median DSC for each region of interest to the WT volume
for each of the test cases. Figure 7: Scatterplot graph depicting the overall
participant median DSC for each region of interest to the logarithmic WT
volume for each of the test cases.

Each region of interest’s DSC was compared to WT volume for each respective
case as shown in Figures 6 and 7. There was a nonsignificant positive linear
correlation between DSC vs WT volume for each of the three regions of
interest, ET, TC, and WT; with p values of 0.696, 0.689, and 0.741,
respectively. There was a nonsignificant logarithmic correlation between DSC
vs WT volume for each of the three regions of interest, ET, TC, and WT; with p
values of 0.102, 0.093, and 0.200, respectively.

Skull-stripping resulted in 908 of 1000 training cases, 129 of 141 validation
cases, and 257 of 283 test cases’ meningioma labels (1286 of 1424, 90.3%
overall) having at least 1 compartment voxel abutting the edge of the skull-
stripped image edge as seen in Figure 8. Of the 257 aforementioned cases, the
average and median number of abutting voxels was 628.7 and 394 voxels
respectively. Figure 9 shows the relationship between the number of abutting
voxels and the WT volume ($\text{R}^{2}$ = 0.190 and p = 0.002).

Figure 8: Image panels with the tumor sub-regions annotated on sagittal views
of T1Gd MRI head cases. Panels A1 and A2 represent a meningioma tumor with
4514 voxels abutting the skull-stripped boundary. Panels B1 and B2 represent a
meningioma tumor with 3911 voxels abutting the skull-stripped boundary. Figure
9: Scatterplot depicting the relationship between WT size and the number of
abutting voxels.

While global metrics (i.e. those used in previous BraTS challenges) were not
used for ranking of the 2024 challenge, we have included Figure 10 to show the
relationship between the complete tumor volume and each of the global metrics
for each of the regions of interest. Note that there is a noticeable trend
towards improved global DSC, global 95HD, and global sensitivity as complete
volume increases.

Figure 10: Plots of cumulative average global DSC, global 95HD, and
sensitivity for each of the ET, TC, and WT regions of interest for the subset
of BraTS testing cases that only had 1 ground truth lesion. The number of
subjects with a single lesion is ET (N=257), TC (N=261), and WT (N=261), for
each respective label.

##  4 Discussion

###  4.1 Challenge Summary

The BraTS 2023 Intracranial Meningioma Segmentation Challenge provided
unprecedented insight into the state-of-the-art performance in pre-operative
meningioma segmentation, leveraging the largest multi-institutional
systematically expert annotated multilabel meningioma MRI dataset to date
(Calabrese and LaBella, 2023; LaBella et al., 2024). The challenge saw
remarkable performances, particularly from the NVAUTO team, in DSC and 95HD
across ET, TC, and WT region of interest segmentation tasks. It was notable
that DSC scores were, on average, higher for the meningioma challenge compared
to all other BraTS 2023 segmentation sub-challenges, which may be due to the
relatively lower complexity of meningiomas compared to other tumor types
and/or the relatively high quality and consistency of the meningioma dataset.
These lesion-wise DSC and 95HD scores should be considered as benchmarks for
future pre-operative meningioma segmentation evaluation. Note that lesion-wise
metrics are essential for segmentation tasks with potential for more than one
distinct lesion as global metrics may still remain relatively high even if a
single, smaller lesion is completely missed by the segmentation algorithm.

###  4.2 The Best Algorithm and Caveats

The superior performance of NVAUTO, with lesion-wise median DSC values of
0.976, 0.976, and 0.964 lesion-wise average DSC values of 0.899, 0.904, and
0.871, for ET, TC, and WT respectively, signifies a notable advancement in
automated meningioma segmentation algorithms. These results not only
demonstrate the feasibility of achieving high accuracy in meningioma
segmentation, but also suggest that deep learning models can effectively adapt
to the diverse morphology and anatomical locations of meningiomas. NVAUTO’s
base algorithm, known as AutoSeg3D, an open-source and Pytorch-based
framework, is particularly adaptable to a variety of medical imaging automated
segmentation challenges (Myronenko et al., 2023). Auto3DSeg allows for auto-
scaling to available GPUs; 5-fold training with SegResNet, DiNTS, and
SwinUNETR models; and performing inference and ensembling using each of the
multiple trained models. For the challenges, NVAUTO found the SegResNet model
to be the most accurate and was ultimately the optimal model architecture
selected by Auto3DSeg for each of the automated segmentation challenges
(Myronenko et al., 2023). Additionally, Auto3DSeg, allows for use of a variety
of image modalities and is not limited to just MRI. Auto3DSeg is advertised to
run with a GPU RAM $\geq$ 8GB. However, for the 2023 BraTS challenges, NVAUTO
used 8 x 16GB NVIDIA V100 machines (Myronenko et al., 2023). Future studies
should assess the ability to use Auto3DSeg on more widely available, consumer
grade GPUs and non-NVIDIA branded GPUs.

###  4.3 Overall Segmentation Performance

The challenge results revealed a broad range of performances among the 9
participating teams, with DSC scores for the ET ranging from 0.899 to 0.007.
Such variability underscores the challenge’s complexity and the diverse
approaches to segmentation. When comparing NVAUTO’s meningioma segmentation
performance compared to their performance on other challenges, it was found
that they performed the best on meningioma’s DSC for ET and TC regions of
interest but performed only second best for DSC for WT as shown in Table 11.
NVAUTO’s algorithm showed that Sub-Saharan Africa gliomas had a higher DSC for
WT than all other tumor types. Since meningioma tend to have a smaller overall
whole tumor volume and a higher ET or TC to WT ratio compared to glioblastoma,
it can be hypothesized that there is less available training information to
create as accurate SNFH compartment labels; and thereafter, the WT region of
interest (Ogasawara et al., 2021; Gilard et al., 2021; Khandwala et al.,
2021).

Table 11: Average DSC for ET, TC, and WT for the top performing team, NVAUTO, using their algorithm, Auto3DSeg, across each of the different BraTS 2023 segmentation challenges. Note that 3 decimal precision is shown for meningioma to emphasize the small increase in DSC for non-ET compared to ET. | ET | TC | WT
---|---|---|---
| (Avg DSC) | (Avg DSC) | (Avg DSC)
Meningioma | 0.899 | 0.904 | 0.870
Glioma | 0.810 | 0.830 | 0.840
Sub-Saharan | 0.790 | 0.840 | 0.910
Africa |  |  |
Metastasis | 0.600 | 0.650 | 0.620
Pediatric | 0.550 | 0.780 | 0.840

###  4.4 Limitations of the BraTS Meningioma Benchmark

Our analysis revealed that all participating teams performed relatively poorly
on heavily calcified meningiomas with little or no enhancing tumor. For
example, teams had an average TC DSC of only 0.156 for one particular non-
enhancing meningioma case. This relatively poor performance is presumably
related to the relative rarity of this imaging appearance of meningioma, and
the fact that only a small number of such cases were included in the training
dataset. Future datasets should include more cases of exclusively non-
enhancing tumor to allow for more generalizable automated segmentation models.
It is important to consider that radiotherapy plans only consider a single GTV
represented by the TC, and whether these indeterminate TC regions are labeled
as non-enhancing vs enhancing regions would not impact the resulting treatment
volumes (Rogers et al., 2017, 2020).

Note that in Figures 6 and 7, each of the lines of best fit for ET, TC, and WT
trend towards improved DSC with increased WT volume. In an automated
segmentation challenge, this could cause a higher perceived test set overall
DSC score if a larger proportion of larger tumors are included in the test set
compared to the overall population. Therefore, it is important to ensure
balance within the training, validation, and test sets regarding tumor size,
which was not explicitly done for this iteration of the challenge.

###  4.5 Future Directions

Future studies involving meningioma automated segmentation should focus on the
most important clinical volumes, particularly the tumor core which comprises
the radiotherapy gross tumor volume (GTV). Additionally, future studies should
focus on the segmentation of meningioma along the intra-axial and extra-axial
border with various face anonymization pre-processing techniques, due to the
high frequency for meningioma to be excluded by skull-stripping as
demonstrated by our results (Watts et al., 2014; Schwarz et al., 2019, 2021).

This study performed an analysis of the propensity of meningioma to abut the
skull-stripped image, thereby having the potential to exclude portions of the
meningioma within the skull-stripped image. Due to skull-stripping resulting
in 1286 of 1483 meningioma having at least 1 compartment voxel abutting the
boundary of the skull-stripped image, future studies should evaluate a
different pre-processing anonymization technique that will allow for inclusion
of more volume of intracranial meningioma. Schwarz et al. describes a
promising mri_reface technique that performs face anonymization by modifying
the MR head image face and ear regions to represent an average human face and
ear, thereby preserving the vast majority of the MR head image (Schwarz et
al., 2019, 2021). Bischoff et al. describes mri_deface, a defacing tool that
removes facial features by assigns a probability of voxel being “face” or
“brain” and removes voxels that have non-zero probability of being “face” but
zero probability of being “brain” (Theyers et al., 2021; Bischoff-Grethe et
al., 2007).

RTOG 0539, a phase II trial of observation for low-risk meningiomas and of
radiotherapy for intermediate- and high-risk meningiomas, describes radiation
treatment planning and target volume protocols that should be used for
meningiomas (Rogers et al., 2017, 2020). For radiation planning, they only
required use of pre-operative and post-operative contrast-enhanced MRI. RTOG
0539 defines the GTV to encompass the tumor bed on postoperative-enhanced MRI
and to include any residual nodular enhancement. They also state that trailing
linear dural tail and cerebral edema should not be specifically included
within the GTV, since there is no evidence that recurrence is more likely
within the dural tail (Rogers et al., 2017, 2020).

Therefore, future studies that focus on automated segmentation of meningioma
for radiation therapy planning should place emphasis on evaluation of the
enhancing tumor and post-op resection bed volumes on post-operative T1Gd
treatment planning MRI, while reducing emphasis on the surrounding non-
enhancing T2/FLAIR hyperintensity compartment and the small trailing linear
dural tail enhancement.

However, in the post-treatment follow-up setting, RTOG 0539 still required the
use of T1, T2, FLAIR, and T1Gd series, which is similar to the imaging used in
the BraTS 2023 Challenge: Intracranial Meningioma. For this challenge’s pre-
operative meningioma dataset, the enhancing and non-enhancing tumor labels
compose the tumor-core. The SNFH label representing edema, which was included
in the WT region of interest is not typically included within radiation
therapy meningioma target volumes.

##  5 Conclusion

The BraTS 2023 Intracranial Meningioma Segmentation Challenge has marked a
significant step forward in the segmentation of meningioma tumors,
highlighting both the potential and limitations of current methodologies. As
the field moves forward, a focus on enhancing dataset diversity, refining pre-
processing techniques, and tailoring segmentation tasks to specific clinical
needs will be crucial in translating these advancements into clinical
practice.

Research reported in this publication was partly supported by the National
Institutes of Health (NIH) under award numbers: NCI K08CA256045, NCI/ITCR
U24CA279629, and NCI/ITCR U01CA242871. The content of this publication is
solely the responsibility of the authors and does not represent the official
views of the NIH.

Developing large and well curated mpMRI datasets for automated segmentation
model development requires significant time and expertise from neuro-radiology
experts. We are grateful to everyone who contributed to the development and
review of the tumor volume labels including volunteer annotators/approvers
from the American Society of Neuroradiology.

The work follows appropriate ethical standards in conducting research and
writing the manuscript, following all applicable laws and regulations
regarding treatment of animals and human subjects. All participating sites had
institutional review board (IRB) approval. A waiver for informed consent was
provided by each institution’s respective IRB.

We declare we don’t have conflicts of interest.

The BraTS Meningioma Pre-operative Dataset training (1,000/1,424, 70%) and
validation (141/1,424, 10%) data are publicly available on Synapse (Calabrese
and LaBella, 2023; LaBella et al., 2024). The testing dataset (283/1,424, 20%)
will be kept private for the foreseeable future to allow for the unbiased
assessment of future segmentation algorithms.

## References

  * bra (2023) Synapse: Brain tumor segmentation (brats) cluster of challenges.  https://www.synapse.org/#!Synapse:syn51156910/wiki/, 2023.
  * Baid et al. (2021) Ujjwal Baid, Satyam Ghodasara, Suyash Mohan, Michel Bilello, Evan Calabrese, Errol Colak, Keyvan Farahani, Jayashree Kalpathy-Cramer, Felipe C Kitamura, Sarthak Pati, et al.  The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification.  _arXiv preprint arXiv:2107.02314_ , 2021.
  * Bakas et al. (2017) Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin S Kirby, John B Freymann, Keyvan Farahani, and Christos Davatzikos.  Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features.  _Scientific data_ , 4(1):1–13, 2017.
  * Bischoff-Grethe et al. (2007) Amanda Bischoff-Grethe, I Burak Ozyurt, Evelina Busa, Brian T Quinn, Christine Fennema-Notestine, Camellia P Clark, Shaunna Morris, Mark W Bondi, Terry L Jernigan, Anders M Dale, et al.  A technique for the deidentification of structural brain mr images.  _Human brain mapping_ , 28(9):892–903, 2007.
  * Bouget et al. (2022) David Bouget, André Pedersen, Asgeir S Jakola, Vasileios Kavouridis, Kyrre E Emblem, Roelant S Eijgelaar, Ivar Kommers, Hilko Ardon, Frederik Barkhof, Lorenzo Bello, et al.  Preoperative brain tumor imaging: Models and software for segmentation and standardized reporting.  _Frontiers in neurology_ , 13:932219, 2022.
  * Calabrese and LaBella (2023) E. Calabrese and D. LaBella.  2023 brats meningioma dataset.  https://www.synapse.org/#!Synapse:syn51514106, 2023.
  * Capellan-Martin (2023) D. Capellan-Martin.  Brats challenge 2023: Model ensemble for brain tumor segmentation in mri.  In _MICCAI_ , Vancouver, Canada, 2023.
  * Gilard et al. (2021) Vianney Gilard, Abdellah Tebani, Ivana Dabaj, Annie Laquerrière, Maxime Fontanilles, Stéphane Derrey, Stéphane Marret, and Soumeya Bekri.  Diagnosis and management of glioblastoma: A comprehensive perspective.  _Journal of personalized medicine_ , 11(4):258, 2021.
  * Gordillo et al. (2013) Nelly Gordillo, Eduard Montseny, and Pilar Sobrevilla.  State of the art survey on mri brain tumor segmentation.  _Magnetic resonance imaging_ , 31(8):1426–1438, 2013.
  * Havaei et al. (2017) Mohammad Havaei, Axel Davy, David Warde-Farley, Antoine Biard, Aaron Courville, Yoshua Bengio, Chris Pal, Pierre-Marc Jodoin, and Hugo Larochelle.  Brain tumor segmentation with deep neural networks.  _Medical image analysis_ , 35:18–31, 2017.
  * Huang et al. (2023) Z. Huang et al.  Exploring model size and patch size for brats23 challenge.  In _MICCAI_ , Vancouver, Canada, 2023.
  * Işın et al. (2016) Ali Işın, Cem Direkoğlu, and Melike Şah.  Review of mri-based brain tumor image segmentation using deep learning methods.  _Procedia Computer Science_ , 102:317–324, 2016.
  * Juluru et al. (2020) Krishna Juluru, Eliot Siegel, and Jan Mazura.  Identification from mri with face-recognition software.  _The New England Journal of Medicine_ , 382(5):489–490, 2020.
  * Karargyris et al. (2023) Alexandros Karargyris, Renato Umeton, Micah J Sheller, Alejandro Aristizabal, Johnu George, Anna Wuest, Sarthak Pati, Hasan Kassem, Maximilian Zenk, Ujjwal Baid, et al.  Federated benchmarking of medical artificial intelligence with medperf.  _Nature Machine Intelligence_ , 5(7):799–810, 2023.
  * Kazerooni et al. (2023) Anahita Fathi Kazerooni, Nastaran Khalili, Xinyang Liu, Debanjan Haldar, Zhifan Jiang, Syed Muhammed Anwar, Jake Albrecht, Maruf Adewole, Udunna Anazodo, Hannah Anderson, et al.  The brain tumor segmentation (brats) challenge 2023: Focus on pediatrics (cbtn-connect-dipgr-asnr-miccai brats-peds).  _ArXiv_ , 2023.
  * Khandwala et al. (2021) Kumail Khandwala, Fatima Mubarak, and Khurram Minhas.  The many faces of glioblastoma: Pictorial review of atypical imaging features.  _The Neuroradiology Journal_ , 34(1):33–41, 2021.
  * LaBella (2024) Dominic LaBella.  MeningiomaAnalysis.  https://doi.org/10.5281/zenodo.10685213, 2024.
  * LaBella et al. (2023) Dominic LaBella, Maruf Adewole, Michelle Alonso-Basanta, Talissa Altes, Syed Muhammad Anwar, Ujjwal Baid, Timothy Bergquist, Radhika Bhalerao, Sully Chen, Verena Chung, et al.  The asnr-miccai brain tumor segmentation (brats) challenge 2023: Intracranial meningioma.  _arXiv preprint arXiv:2305.07642_ , 2023.
  * LaBella et al. (2024) Dominic LaBella, Omaditya Khanna, Shan McBurney-Lin, Ryan Mclean, Pierre Nedelec, Arif S. Rashid, Nourel hoda Tahon, Talissa Altes, Ujjwal Baid, Radhika Bhalerao, Yaseen Dhemesh, Scott Floyd, Devon Godfrey, Fathi Hilal, Anastasia Janas, Anahita Kazerooni, Collin Kent, John Kirkpatrick, Florian Kofler, Kevin Leu, Nazanin Maleki, Bjoern Menze, Maxence Pajot, Zachary J. Reitman, Jeffrey D. Rudie, Rachit Saluja, Yury Velichko, Chunhao Wang, Pranav I. Warman, Nico Sollmann, David Diffley, Khanak K. Nandolia, Daniel I Warren, Ali Hussain, John Pascal Fehringer, Yulia Bronstein, Lisa Deptula, Evan G. Stein, Mahsa Taherzadeh, Eduardo Portela de Oliveira, Aoife Haughey, Marinos Kontzialis, Luca Saba, Benjamin Turner, Melanie M. T. Brüßeler, Shehbaz Ansari, Athanasios Gkampenis, David Maximilian Weiss, Aya Mansour, Islam H. Shawali, Nikolay Yordanov, Joel M. Stein, Roula Hourani, Mohammed Yahya Moshebah, Ahmed Magdy Abouelatta, Tanvir Rizvi, Klara Willms, Dann C. Martin, Abdullah Okar, Gennaro D’Anna, Ahmed Taha, Yasaman Sharifi, Shahriar Faghani, Dominic Kite, Marco Pinho, Muhammad Ammar Haider, Michelle Alonso-Basanta, Javier Villanueva-Meyer, Andreas M. Rauschecker, Ayman Nada, Mariam Aboian, Adam Flanders, Spyridon Bakas, and Evan Calabrese.  A multi-institutional meningioma mri dataset for automated multi-sequence image segmentation.  _Scientific Data_ , 11:496, 2024.  10.1038/s41597-024-03350-9.  URL https://doi.org/10.1038/s41597-024-03350-9.
  * Martz et al. (2022) N Martz, J Salleron, F Dhermain, G Vogin, JF Daisne, R Mouttet Audouard, R Tanguy, G Noel, M Peyre, I Lecouillard, et al.  Anocef consensus guideline on target volume delineation for meningiomas radiotherapy.  _International Journal of Radiation Oncology, Biology, Physics_ , 114(3):e46, 2022.
  * Menze et al. (2014) Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al.  The multimodal brain tumor image segmentation benchmark (brats).  _IEEE transactions on medical imaging_ , 34(10):1993–2024, 2014.
  * MLCommons Association (2024) MLCommons Association.  Mlcube: Standardizing ml deployment.  https://mlcommons.org/working-groups/data/mlcube/, 2024.  Accessed: 2024-05-08.
  * Moawad et al. (2023) Ahmed W Moawad, Anastasia Janas, Ujjwal Baid, Divya Ramakrishnan, Leon Jekel, Kiril Krantchev, Harrison Moy, Rachit Saluja, Klara Osenberg, Klara Wilms, et al.  The brain tumor segmentation (brats-mets) challenge 2023: Brain metastasis segmentation on pre-treatment mri.  _ArXiv_ , 2023.
  * Myronenko et al. (2023) A. Myronenko, D. Yang, Y. He, and D. Xu.  Auto3dseg for brain tumor segmentation from 3d mri in brats 2023 challenge.  In _MICCAI_ , Vancouver, Canada, 2023.
  * Ogasawara et al. (2021) Christian Ogasawara, Brandon D Philbrick, and D Cory Adamson.  Meningioma: a review of epidemiology, pathology, diagnosis, treatment, and future directions.  _Biomedicines_ , 9(3):319, 2021.
  * Pati et al. (2022) Sarthak Pati, Ujjwal Baid, Brandon Edwards, Micah J Sheller, Patrick Foley, G Anthony Reina, Siddhesh Thakur, Chiharu Sako, Michel Bilello, Christos Davatzikos, et al.  The federated tumor segmentation (fets) tool: an open-source solution to further solid tumor research.  _Physics in Medicine & Biology_, 67(20):204002, 2022.
  * Pati et al. (2023) Sarthak Pati, Siddhesh P Thakur, İbrahim Ethem Hamamcı, Ujjwal Baid, Bhakti Baheti, Megh Bhalerao, Orhun Güley, Sofia Mouchtaris, David Lang, Spyridon Thermos, et al.  Gandlf: the generally nuanced deep learning framework for scalable end-to-end clinical workflows.  _Communications Engineering_ , 2(1):23, 2023.
  * Pereira et al. (2016) Sérgio Pereira, Adriano Pinto, Victor Alves, and Carlos A Silva.  Brain tumor segmentation using convolutional neural networks in mri images.  _IEEE transactions on medical imaging_ , 35(5):1240–1251, 2016.
  * Rogers et al. (2020) C Leland Rogers, Minhee Won, Michael A Vogelbaum, Arie Perry, Lynn S Ashby, Jignesh M Modi, Anthony M Alleman, James Galvin, Shannon E Fogh, Emad Youssef, et al.  High-risk meningioma: initial outcomes from nrg oncology/rtog 0539.  _International Journal of Radiation Oncology* Biology* Physics_ , 106(4):790–799, 2020.
  * Rogers et al. (2017) Leland Rogers, Peixin Zhang, Michael A Vogelbaum, Arie Perry, Lynn S Ashby, Jignesh M Modi, Anthony M Alleman, James Galvin, David Brachman, Joseph M Jenrette, et al.  Intermediate-risk meningioma: initial outcomes from nrg oncology rtog 0539.  _Journal of neurosurgery_ , 129(1):35–47, 2017.
  * Rudie (2023) J. Rudie.  Brats 2023 segmentation metrics: Clinical relevance.  In _Proceedings of the Medical Image Computing and Computer Assisted Intervention – MICCAI_ , Vancouver, Canada, 2023.
  * Saluja (2023) R. Saluja.  Lesion-wise performance metrics for brats-2023 segmentation challenges.  In _Proceedings of the Medical Image Computing and Computer Assisted Intervention – MICCAI_ , Vancouver, Canada, 2023.
  * Schwarz et al. (2019) Christopher G Schwarz, Walter K Kremers, Terry M Therneau, Richard R Sharp, Jeffrey L Gunter, Prashanthi Vemuri, Arvin Arani, Anthony J Spychalla, Kejal Kantarci, David S Knopman, et al.  Identification of anonymous mri research participants with face-recognition software.  _New England Journal of Medicine_ , 381(17):1684–1686, 2019.
  * Schwarz et al. (2021) Christopher G Schwarz, Walter K Kremers, Heather J Wiste, Jeffrey L Gunter, Prashanthi Vemuri, Anthony J Spychalla, Kejal Kantarci, Aaron P Schultz, Reisa A Sperling, David S Knopman, et al.  Changing the face of neuroimaging research: comparing a new mri de-facing technique with popular alternatives.  _NeuroImage_ , 231:117845, 2021.
  * Thakur et al. (2020) Siddhesh Thakur, Jimit Doshi, Sarthak Pati, Saima Rathore, Chiharu Sako, Michel Bilello, Sung Min Ha, Gaurav Shukla, Adam Flanders, Aikaterini Kotrotsou, et al.  Brain extraction on mri scans in presence of diffuse glioma: Multi-institutional performance evaluation of deep learning methods and robust modality-agnostic training.  _NeuroImage_ , 220:117081, 2020.
  * Thakur et al. (2019) Siddhesh P Thakur, Jimit Doshi, Sarthak Pati, Sung Min Ha, Chiharu Sako, Sanjay Talbar, Uday Kulkarni, Christos Davatzikos, Guray Erus, and Spyridon Bakas.  Skull-stripping of glioblastoma mri scans using 3d deep learning.  In _International MICCAI Brainlesion Workshop_ , pages 57–68. Springer, 2019.
  * Theyers et al. (2021) Athena E Theyers, Mojdeh Zamyadi, Mark O’Reilly, Robert Bartha, Sean Symons, Glenda M MacQueen, Stefanie Hassel, Jason P Lerch, Evdokia Anagnostou, Raymond W Lam, et al.  Multisite comparison of mri defacing software across multiple cohorts.  _Frontiers in psychiatry_ , 12:617997, 2021.
  * Watts et al. (2014) J Watts, G Box, A Galvin, P Brotchie, N Trost, and T Sutherland.  Magnetic resonance imaging of meningiomas: a pictorial review.  _Insights into imaging_ , 5(1):113–122, 2014.


For entropy dissipation, we first show that the integrability condition holds
for the entropy/entropy-flux pair $h^{\gamma}$ and $\mathbf{j}^{\gamma}$ with
the flux term
$\mathbf{F}(\mathbf{u})=\left<\mathbf{v}\otimes\mathbf{m}f^{\gamma}_{\mathbf{u}}\right>$.
For $i=1,2,3$, let
$j_{i}^{\gamma}:=\left<v_{i}\eta(f^{\gamma}_{\mathbf{u}})\right>$ and
$F_{i}(\mathbf{u}):=\left<v_{i}\,\mathbf{m}f^{\gamma}_{\mathbf{u}}\right>$,
then

$\nabla_{\mathbf{u}}j^{\gamma}_{i}(\mathbf{u})=\left<v_{i}\nabla_{\mathbf{u}}\eta(f^{\gamma}_{\mathbf{u}})\right>=\left<v_{i}\eta^{\prime}(f^{\gamma}_{\mathbf{u}})\nabla_{\mathbf{u}}f^{\gamma}_{\mathbf{u}}\right>=\left<v_{i}\eta^{\prime}(\eta_{*}^{\prime}(\mathbf{g}^{\gamma}_{\mathbf{u}}\cdot\mathbf{m}))\nabla_{\mathbf{u}}f^{\gamma}_{\mathbf{u}}\right>=\mathbf{g}^{\gamma}_{\mathbf{u}}\cdot\left<v_{i}\mathbf{m}\,\nabla_{\mathbf{u}}f^{\gamma}_{\mathbf{u}}\right>=\nabla_{\mathbf{u}}h^{\gamma}(\mathbf{u})\cdot\nabla_{\mathbf{u}}F_{i}(\mathbf{u}),$
(110)

where we have used the chain rule, the definition of $f^{\gamma}_{\mathbf{u}}$
in Eq. (37), the Legendre duality, and the fact that
$\mathbf{g}^{\gamma}_{\mathbf{u}}=\nabla_{\mathbf{u}}h^{\gamma}(\mathbf{u})$.
To derive the entropy dissipation law, we multiply the moment system from the
left with $\nabla_{\mathbf{u}}h^{\gamma}(\mathbf{u})$ and obtain

$\nabla_{\mathbf{u}}h^{\gamma}(\mathbf{u})\cdot\partial_{t}\mathbf{u}+\nabla_{\mathbf{u}}h^{\gamma}(\mathbf{u})\cdot\sum_{i=1}^{3}\partial_{{x}_{i}}F_{i}(\mathbf{u})=\nabla_{\mathbf{u}}h^{\gamma}(\mathbf{u})\cdot\left<\mathbf{m}Q({f}_{\mathbf{u}}^{\gamma})\right>\>.$
(111)

From the reverse chain rule and Eq. (110), this leads to

$\partial_{t}h^{\gamma}(\mathbf{u})+\nabla_{\mathbf{x}}\cdot\bm{j}^{\gamma}(\mathbf{u})=\nabla_{\mathbf{u}}h^{\gamma}(\mathbf{u})\cdot\left<\mathbf{m}Q({f}_{\mathbf{u}}^{\gamma})\right>.$
(112)

Entropy dissipation requires non-positivity of right-hand side term, which is
given by

$\nabla_{\mathbf{u}}h^{\gamma}(\mathbf{u})\cdot\left<\mathbf{m}Q({f}_{\mathbf{u}}^{\gamma})\right>=\left<\mathbf{g}^{\gamma}_{\mathbf{u}}\cdot\mathbf{m}Q({f}_{\mathbf{u}}^{\gamma})\right>=\left<\eta^{\prime}(\eta_{*}^{\prime}(\mathbf{g}^{\gamma}_{\mathbf{u}}\cdot\mathbf{m}))Q({f}_{\mathbf{u}}^{\gamma})\right>=\left<\eta^{\prime}({f}_{\mathbf{u}}^{\gamma})Q({f}_{\mathbf{u}}^{\gamma})\right>\leq
0,$ (113)

where the equalities follow from similar properties used in Eq. (110) in the
reversed order, and the inequality is from the entropy dissipation law in Eq.
(6).

We next show that the closed moment system is hyperbolic. Note that the
hyperbolicity follows from the entropy dissipation property of the moment
system. Consider again the flux function of the moment system

$\displaystyle\mathbf{F}(\mathbf{u})=\left<\mathbf{v}\otimes\mathbf{m}f^{\gamma}_{\mathbf{u}}\right>=\left<\mathbf{v}\otimes\mathbf{m}\,\eta^{\prime}_{*}\left(\mathbf{g}_{\mathbf{u}}^{\gamma}\cdot\mathbf{m}\right)\right>.$
(114)

We define
$j_{i,*}(\mathbf{g})=\left<v_{i}\eta_{*}(\mathbf{g}\cdot\mathbf{m})\right>$,
$i=1,2,3$, such that
$\nabla_{\mathbf{g}}j_{i,*}(\mathbf{g}^{\gamma}_{\mathbf{u}})=F_{i}(\mathbf{u})$,
and denote

$\displaystyle
J_{i}={\nabla_{\mathbf{g}}^{2}}j_{i,*}(\mathbf{g}^{\gamma}_{\mathbf{u}}),\quad
i=1,2,3,\quad\text{and}\quad
K=\nabla_{\mathbf{u}}^{2}h^{\gamma}(\mathbf{u})=\nabla_{\mathbf{u}}\mathbf{g}^{\gamma}_{\mathbf{u}},$
(115)

where $K$ is symmetric positive definite by the strict convexity of
$h^{\gamma}(\mathbf{u})$. By using chain rule, the flux term in Eq. (56) can
be written as

$\nabla_{\mathbf{x}}\left<\mathbf{v}\otimes\mathbf{m}f^{\gamma}_{\mathbf{u}}\right>=\sum_{i=1}^{3}\partial_{x_{i}}F_{i}(\mathbf{u})=\sum_{i=1}^{3}\partial_{x_{i}}(\nabla_{\mathbf{g}}j_{i,*}(\mathbf{g}^{\gamma}_{\mathbf{u}}))=\sum_{i=1}^{3}{\nabla_{\mathbf{g}}^{2}}j_{i,*}(\mathbf{g}^{\gamma}_{\mathbf{u}})\cdot\nabla_{\mathbf{u}}\mathbf{g}^{\gamma}_{\mathbf{u}}\cdot\partial_{x_{i}}\mathbf{u}=\sum_{i=1}^{3}J_{i}K\partial_{x_{i}}\mathbf{u}\>.$
(116)

The symmetric positive definite matrix $K$ then symmetrizes [19] the closed
moment system (56); that is, after multiplication on the left by $K$,

$K\partial_{t}\mathbf{u}+\sum_{i=1}^{3}KJ_{i}K\partial_{x_{i}}\mathbf{u}\>=K\left<\mathbf{m}Q({f}_{\mathbf{u}}^{\gamma})\right>,$
(117)

where the matrix $KJ_{i}K$ is symmetric for each $i\in\\{1,2,3\\}$.

∎