# A two-stage formalism for common-envelope phases of massive stars

Ryosuke Hirai (平井遼介) School of Physics and Astronomy, Monash University, VIC
3800, Australia  OzGrav: The Australian Research Council Centre of Excellence
for Gravitational Wave Discovery, Clayton, VIC 3800, Australia  Ilya Mandel
School of Physics and Astronomy, Monash University, VIC 3800, Australia
OzGrav: The Australian Research Council Centre of Excellence for Gravitational
Wave Discovery, Clayton, VIC 3800, Australia

###### Abstract

We propose a new simple formalism to predict the orbital separations after
common-envelope phases with massive star donors. We focus on the fact that
massive red supergiants tend to have a sizeable radiative layer between the
dense helium core and the convective envelope. Our formalism treats the
common-envelope phase in two stages: dynamical in-spiral through the outer
convective envelope and thermal timescale mass transfer from the radiative
intershell. With fiducial choices of parameters, the new formalism typically
predicts much wider separations compared to the classical energy formalism.
Moreover, our formalism predicts that final separations strongly depend on the
donor evolutionary stage and companion mass. Our formalism provides a
physically-motivated alternative option for population synthesis studies to
treat common-envelope evolution. This treatment will impact on predictions for
massive-star binaries, including gravitational-wave sources, X-ray binaries
and stripped-envelope supernovae.

Common envelope evolution (2154) — Massive stars (732) — Low-mass X-ray binary
stars (939) — Core-collapse supernovae (304) — Gravitational wave sources
(677)

††software: MESA (Paxton et al., 2011, 2013, 2015, 2018, 2019), MESA SDK
(Townsend, 2021), gnuplot (Williams et al., 2021)

##  1 Introduction

ce evolution remains one of the most uncertain phases of binary evolution (see
Ivanova et al., 2013, 2020, for an overview on common-envelope (CE)
evolution). In this phase, one of the stars in a binary system evolves to
initiate unstable mass transfer, engulfing the companion star. The companion
star will spiral in to orbit the core of the primary star inside a shared
envelope, releasing a large amount of orbital energy. Part of the released
energy can be transferred to the envelope to eject it. As a result, the binary
is left with the companion star and the core of the primary in a very tight
orbit.

The CE phase was initially introduced to explain the origin of cataclysmic
variables (Paczynski, 1976). Today, the idea is extended to higher-mass stars
as well, and is considered responsible for explaining the origin of many other
systems such as type Ia and stripped-envelope supernova progenitors, X-ray
binaries and gravitational-wave sources, to name just a few. The main goal of
CE studies is to establish a way to predict the outcome, namely the post-CE
orbital separation, provided the pre-CE binary properties. Despite the
extensive efforts made via hydrodynamical modelling in 1D (e.g. Ivanova et
al., 2015; Clayton et al., 2017; Fragos et al., 2019; Klencki et al., 2021;
Marchant et al., 2021) and 3D (e.g. Rasio & Livio, 1996; Ricker & Taam, 2008;
Passy et al., 2012; Ivanova & Nandez, 2016; Iaconi et al., 2017; Law-Smith et
al., 2020; Glanz & Perets, 2021; Moreno et al., 2021; Lau et al., 2022a, b;
Gonzalez-Bolivar et al., 2022), there is not yet a sufficient understanding of
the physics to be able to provide accurate predictions.

Many binary population synthesis studies instead adopt a simple
parameterization to CE phases based on energy balance (the so-called
$\alpha$-formalism). This parameterization enables us to predict the outcome
given an appropriate choice for the value of the parameter
$\alpha_{\mathrm{CE}}$. However, there is growing evidence both
observationally and theoretically that the value for $\alpha_{\mathrm{CE}}$
may not be universal across different systems (Politano, 2004; Iaconi & De
Marco, 2019), questioning the suitability of the parameterization itself. In
particular, there seems to be a qualitative discrepancy between low-mass and
high-mass donors (e.g. Wilson & Nordhaus, 2022), necessitating the use of
different values of $\alpha_{\mathrm{CE}}$ for different mass regimes.

There is an alternative formalism based on angular momentum conservation,
often called the $\gamma$-formalism (Nelemans et al., 2000; Nelemans & Tout,
2005). However, various studies point out that the ratio of the initial and
final orbital angular momenta are so large that even a small change in model
parameters will predict wildly different final separations (Webbink, 2008;
Ivanova et al., 2013, 2020). Therefore, there seems to be little advantage of
the $\gamma$ formalism over the $\alpha$ formalism as an outcome predictor.

In this letter, we propose a new framework to parameterize the CE phase, with
particular emphasis on massive star donors. In Section 2, we review the
classical $\alpha$-formalism and then explain our new proposed framework. We
demonstrate some examples in Section 3 and conclude our study in Section 4.

##  2 Method

###  2.1 The classical energy formalism

The orbital separations after CE phases are often estimated with a simple
framework based on energy conservation (van den Heuvel, 1976; Tutukov &
Yungelson, 1979; Webbink, 1984). The main idea is that the change in orbital
energy of the binary is used to provide the energy required to unbind the
envelope. This implicitly assumes that the orbital energy is fully efficiently
converted into kinetic energy of the envelope, just enough to accelerate it to
escape velocity. In reality the envelope can have outflow velocities above the
escape velocity, or it can radiate away energy from the photosphere, causing
it to “waste” part of the injected energy; on the other hand, there could be
extra energy sources such as feedback from the accretor (e.g. Fryer et al.,
1996; Soker, 2004; Chamandy et al., 2018; López-Cámara et al., 2019), or
magnetic fields (Ohlmann et al., 2016). To account for possible energy sources
and sinks, a single efficiency factor is often introduced in the following
form (Tutukov & Yungelson, 1979; Iben & Tutukov, 1985; de Kool, 1990):

$E_{\mathrm{env}}=\alpha_{\mathrm{CE}}\left(-\frac{GM_{\mathrm{d}}M_{2}}{2a_{i}}+\frac{GM_{\mathrm{core}}M_{2}}{2a_{f}}\right),$
(1)

where $E_{\mathrm{env}}$ is the binding energy of the envelope, $G$ is the
gravitational constant, $M_{\mathrm{d}}$ is the mass of the pre-CE donor,
$M_{2}$ is the mass of the secondary, $M_{\mathrm{core}}$ is the mass of the
donor’s core, $a_{i},a_{f}$ are the pre- and post-CE orbital separations and
$\alpha_{\mathrm{CE}}$ is the efficiency factor. In the absence of any extra
energy sources, the efficiency should be $\alpha_{\mathrm{CE}}\leq 1$.

The envelope binding energy is generally expressed as

$E_{\mathrm{env}}=\int^{M_{\mathrm{core}}}_{M_{\mathrm{d}}}\left(-\frac{Gm}{r}+\alpha_{\mathrm{th}}\epsilon\right)dm,$
(2)

where $m$ is the mass coordinate, $r=r(m)$ is the radius at a given mass
coordinate and $\epsilon$ is the specific internal energy. The internal energy
term contains several sub-components depending on the study. The envelope
material contains a large amount of thermal energy which is required to
maintain hydrostatic equilibrium. Once the equilibrium breaks down, the
thermal energy can do work to accelerate the gas. For this reason, thermal
energy is often considered as an extra energy source that can be used to eject
the envelope and this is accounted for by applying that contribution to
$E_{\mathrm{env}}$. Another commonly discussed energy source is the energy
provided by recombination. This is a latent energy source that is only
released when the gas expands and cools, but the total energy released can be
significant especially in the outer parts of the envelope. To take into
account this extra energy source, the ionization potentials for each ion are
sometimes included in the internal energy term. In a recent study, we showed
that while helium recombination may have a direct influence on the post-plunge
separation, the influence of hydrogen recombination is limited (Lau et al.,
2022b). In the following, we assume that $\epsilon$ includes the recombination
energy of helium as well as the thermal energy of the gas and radiation. The
coefficient $\alpha_{\mathrm{th}}$ is introduced to be able to tune the
efficiency of initial internal energy in ejecting the envelope (Han et al.,
1995).

Instead of directly using the binding energy itself in Eq. (1), it is commonly
parameterized as

$E_{\mathrm{env}}=\frac{GM_{\mathrm{d}}M_{\mathrm{env}}}{\lambda
R_{\mathrm{d}}},$ (3)

where $M_{\mathrm{env}}\equiv M_{\mathrm{d}}-M_{\mathrm{core}}$ is the
envelope mass, $R_{\mathrm{d}}$ is the donor radius and $\lambda$ is an order
unity parameter that expresses how centrally concentrated the density
distribution of the envelope is. Population synthesis studies often employ
either a fixed value for $\lambda$ or tabulated fits based on detailed stellar
evolution calculations (e.g. Dewi & Tauris, 2000; Xu & Li, 2010a, b).

Given all the pre-CE binary properties
($E_{\mathrm{env}},M_{\mathrm{d}},M_{2},a_{i}$) and the core mass of the
primary ($M_{\mathrm{core}}$), the post-CE separation ($a_{f}$) can be
estimated from Eq. (1). In many binary population synthesis studies, a fixed
universal value is assumed for $\alpha_{\mathrm{CE}}$ to model the population
of various observed systems. Calculations with various choices of
$\alpha_{\mathrm{CE}}$ are carried out to quantify the uncertainties. However,
both observations of post-CE systems and 3D hydrodynamical simulations seem to
suggest that the value of $\alpha_{\mathrm{CE}}$ is not universal and can
depend on many factors such as donor mass, mass ratio, evolutionary stage,
etc. (Politano, 2004; Iaconi & De Marco, 2019). In particular, massive donors
have fundamentally different structures from low-mass donors (see next
section) and hence are expected to behave qualitatively differently (e.g.
Fragos et al., 2019; Wilson & Nordhaus, 2022; Lau et al., 2022a). There seems
to be a building notion that corrections to the $\alpha$-formalism are
required to explain various observations simultaneously (e.g. Politano, 2004,
2021; Politano & Weiler, 2007).

###  2.2 A “two-stage” formalism

The most obvious way to implement system-dependent CE efficiencies is to
create a functional form for $\alpha_{\mathrm{CE}}$ instead of a constant
value. In principle, this can provide the correct post-CE separations as long
as the function is correct. However, having higher-order corrections to model
parameters is often a sign that the model itself is not suitable. Here, we
propose a new simple framework for treating CE evolution that naturally
behaves differently between low-mass and high-mass donors and also has a
strong mass ratio dependence.

The main assumption that goes into the classical $\alpha$-formalism is that
the whole CE process is sufficiently shorter than the thermal timescale of the
star so that it can be treated as an adiabatic process. This may be true for
lower mass donors, as the thermal timescales of the envelopes are relatively
long compared to the inspiral timescales. At higher masses, the thermal
timescales are much shorter, and can become comparable to the inspiral
timescale. It starts to become questionable whether the adiabatic treatment is
adequate.

There is also a structural difference between low-mass red giants and high-
mass red supergiants. In Figure 1, we compare the entropy distributions inside
a $1~{}M_{\odot}$ RG and a $12~{}M_{\odot}$ RSG computed with the public
stellar evolution code MESA (Paxton et al., 2011). While for low-mass RGs the
envelope is almost fully convective, high-mass RSGs have a radiative layer
between the core and a convective outer layer. This intermediate radiative
layer is much more tightly bound to the core, usually having an order of
magnitude higher binding energy compared to the convective layer despite the
contained mass being significantly less. Tabulated fits to the $\lambda$
parameter are therefore often quite small ($\lambda\sim\mathcal{O}(0.1)$) for
higher mass stars (Dewi & Tauris, 2000; Xu & Li, 2010a, b).

Figure 1: Entropy distribution in a $1~{}M_{\odot}$ RG and $12~{}M_{\odot}$
RSG. The $1~{}M_{\odot}$ RG is chosen at a time when it has reached a radius
of $100~{}R_{\odot}$, and the $12~{}M_{\odot}$ RSG is chosen when it has
reached $600~{}R_{\odot}$, just before the onset of core helium burning. The
mass coordinate is normalized by the total mass. Solid parts of the curve
indicate the convective layers while the dotted parts are radiative. The
helium cores are colored with a lighter shade.

An important feature of convective layers is that they have flat or slightly
negative entropy gradients, while radiative layers have positive entropy
gradients (Figure 1). This difference is crucial, as it determines the
dynamical response of the stellar radius to mass loss (Soberman et al., 1997).
Roughly speaking, the stellar radius will contract in response to mass loss
when the surface is radiative (Ivanova, 2011). This is because as the surface
layers are removed, lower entropy material is exposed at the surface and
therefore it cannot re-expand to its original size. On the other hand,
convective envelopes can dynamically expand in response to mass loss due to
the opposite argument. Thus, for a given mass ratio, mass transfer will be
more unstable when the surface is convective and would likely lead to a CE
phase.

Once a CE phase is initiated, the companion rapidly spirals in due to loss of
corotation and dynamical friction with the envelope gas. According to 3D
hydrodynamical simulations, the rapid plunge-in abruptly transitions into a
slower spiral-in when the envelope is mostly ejected and corotation is
achieved. For massive stars, the stalling can occur outside the radiative part
of the envelope depending on the companion mass (Lau et al., 2022a, b).

After losing its convective envelope, the remaining radiative portion of the
envelope does not dynamically expand but is still out of thermal equilibrium,
so expands on a thermal timescale (Ivanova, 2011; Vigna-Gómez et al., 2022).
This may initiate a more stable mass transfer to the companion star, which can
further alter the orbit until most of the envelope is lost and the core
detaches from its Roche lobe. At this point, energy conservation is no longer
valid because by definition of the thermal timescale, the energy generated
through nuclear burning during this phase becomes comparable to the binding
energy. Therefore, it is more appropriate to model this phase with angular
momentum conservation rather than energy balance arguments.

Building on these previous findings, we construct an alternative formalism for
predicting CE outcomes by splitting the whole process into two stages: I. the
rapid spiral-in through the convective envelope, and II. the stable mass
transfer of the radiative intershell. A schematic picture of our formalism
compared to the classical $\alpha$-formalism is illustrated in Figure 2.
(Stage I) We assume that the inspiral is much shorter than the thermal
timescale and proceeds until the entire convective envelope is ejected. To
estimate the orbital separation after stage I, we adopt the traditional
$\alpha$-formalism while treating the radiative intershell as part of the
core. Given the absence of extra energy sources on the dynamical timescale, we
set $\alpha_{\mathrm{CE}}=1$ as a conservative estimate for the efficiency of
stage I.
(Stage II) We assume the mass transfer is stable and non-conservative, and the
transferred matter takes away some angular momentum from the system. In
general, the CE phase is considered to occur when the companion mass is
relatively small compared to the donor. Therefore, the companion should have a
considerably longer thermal timescale compared to the donor and would not be
able to accrete significant amounts of mass. In this paper, we assume for
simplicity that the mass transfer is fully non-conservative, although some
amount of accretion may in principle occur during this phase (Ivanova, 2011).
We estimate the final orbital separation by solving for the system angular
momentum evolution

$\frac{\dot{a}}{a}=\left(-2+\frac{M_{\mathrm{d}}}{M_{\mathrm{d}}+M_{2}}\right)\frac{\dot{M_{\mathrm{d}}}}{M_{\mathrm{d}}}+\frac{\dot{J}_{\mathrm{orb}}}{J_{\mathrm{orb}}},$
(4)

where $J_{\mathrm{orb}}$ is the orbital angular momentum (Postnov & Yungelson,
2014). The value of $\dot{J}_{\mathrm{orb}}$ depends on the details of how the
transferred material escapes the system. As a first attempt, we assume the
matter is isotropically re-emitted from the accretor

$\frac{\dot{J}_{\mathrm{orb}}}{J_{\mathrm{orb}}}=\frac{\dot{M_{\mathrm{d}}}M_{\mathrm{d}}}{M_{2}(M_{\mathrm{d}}+M_{2})}.$
(5)

Other choices could be made depending on the situation such as angular
momentum loss from the $L_{2}$ point or from a circumbinary disk or some
intermediate point similar to the $\gamma$-formalism as explored by MacLeod et
al. (2018a). With any choice of angular momentum loss mode that is greater
than isotropic re-emission, we would expect a tighter post-CE separation or
higher fraction of mergers. We leave such exploration for future work. We
additionally assume that the entire envelope will be lost in this second
stage. Strictly speaking, the amount of envelope mass left on the core should
depend on the size of the Roche lobe, but generally the envelope expands to
large sizes unless it is stripped down to the last $\sim 1/4$–$1/3$ of the
radiative intershell (Ivanova, 2011; Vigna-Gómez et al., 2022). Indeed, some
detailed 1D CE simulations find that the donor detaches from its Roche lobe
only when the remaining envelope mass is extremely small (Eldridge & Stanway,
2009; Eldridge et al., 2017; Fragos et al., 2019; Marchant et al., 2021). The
small amount of remaining envelope can strongly influence the observable
features of the remaining star (e.g. Hall et al., 2013; Götberg et al., 2017),
its later evolution (e.g. Laplace et al., 2020), and the eventual supernova
(e.g. Dessart et al., 2011; Gilkis & Arcavi, 2022). However, the retained
envelope mass is small enough, except perhaps at low metallicities (e.g.
Götberg et al., 2017), that it should not significantly affect subsequent
orbital evolution and final separation even if it leads to another mass
transfer episode.

Figure 2: Schematic diagram of the traditional $\alpha$-formalism and the two-
stage formalism we propose in this paper.

To demonstrate the predictions for the post-CE orbits with our new formalism,
we evolve a representative star with $M_{\mathrm{d}}=12~{}M_{\odot}$ through
the RSG phase using MESA. For simplicity, we ignore any mass loss that may
occur via stellar winds or pulsations. We compute the core mass
$M_{\mathrm{core}}$, convective envelope mass $M_{\mathrm{conv}}$, radiative
intershell mass $M_{\mathrm{int}}$ and the binding energy of the convective
envelope $E_{\mathrm{conv}}$ at each time. Inlists, extra subroutines and the
output file used in this study are made available at
https://doi.org/10.5281/zenodo.7066430 (catalog doi:10.5281/zenodo.7066430).
We use the above quantities to estimate the final post-CE separation for
various companion star masses. To provide a pre-CE orbital separation, we
assume that the donor has just filled its Roche lobe. The resulting separation
is computed using the formula provided in Eggleton (1983). The true onset
separation of the CE phase is still heavily debated. In some cases it may be
initiated via Darwin instability (Darwin, 1879), especially for the more
extreme mass ratio systems. In other cases the outflow through the second
Lagrangian point may carry away angular momentum to drive the system into
coalescence (e.g. MacLeod et al., 2018b). When exactly the CE commences and
whether there are significant structural changes in the donor before the
plunge-in is still unclear. In any case, the exact choice of the pre-CE
orbital separation has little influence on our demonstration as it only has a
minor contribution in stage I through Eq. (1). We set $\alpha_{\mathrm{CE}}=1$
and $\alpha_{\mathrm{th}}=1$ for stage I of our formalism and assume isotropic
re-emission for stage II.

##  3 Results

We show the evolution of key quantities for a representative model in Figure 3
with a $M_{\mathrm{d}}=12~{}M_{\odot}$ donor star. The upper panel shows how
the convective envelope mass grows as the star evolves. Once the star reaches
a certain threshold radius ($R_{\mathrm{d}}\sim 250~{}R_{\odot}$), a
convective envelope develops and quickly grows up to when it reaches
$R_{\mathrm{d}}\sim 400~{}R_{\odot}$ and then slowly grows up to its maximal
value at the maximum radius ($R_{\mathrm{d}}\sim 1000~{}R_{\odot}$). The core
mass stays almost constant while the star expands and crosses the Hertzsprung
gap up to $R_{\mathrm{d}}\sim 700~{}R_{\odot}$. There is a steep jump in core
mass beyond that radius, as the larger radii are only reached in later stages
of the evolution when helium is depleted in the core (we do not display
intermediate timesteps, when the star contracts and re-expands during core
helium burning, as such stars will not generally initiate mass transfer in
isolated binaries). If a CE phase is initiated at radii within the pink shaded
region, the interaction can be classified as case B mass transfer, whereas if
it is initiated in the light blue region it can be classified as case C mass
transfer. While the core mass and radiative intershell mass show a jump
between case B and case C, the convective envelope mass seems to connect more
smoothly.

Figure 3: Key stellar quantities of a $12~{}M_{\odot}$ star as a function of
the stellar radius. Values are plotted when the star reaches the given radius
for the first time, which corresponds to when the donor starts transferring
mass to the companion for the first time. (Upper panel) Masses of the
radiative intershell $M_{\mathrm{rad}}$, convective outer envelope
$M_{\mathrm{conv}}$ and helium core $M_{\mathrm{core}}$. (Lower panel) Binding
energies of the radiative intershell $E_{\mathrm{env,rad}}$ (blue curves) and
the convective outer envelope $E_{\mathrm{env,conv}}$ (magenta curves). The
energies were computed with (solid) and without (dashed) internal energy. As a
visual guide, we overplot simple estimates of the binding energy with fixed
values of $\lambda$ (dotted curves).

The binding energies of the radiative intershell (blue curves) and the
convective outer envelope (magenta curves) are shown in the lower panel. The
radiative intershell has more than an order of magnitude higher binding energy
throughout most of the evolution. Similar to the masses, the radiative
intershell binding energy shows jumps between case B and case C CEs, but the
convective envelope binding energy shows a smooth transition. The latter is
almost perfectly consistent with having a constant value of $\lambda$
throughout, with $\lambda=0.5$ for the gravitational binding energy
($\alpha_{\mathrm{th}}=0$) and $\lambda\sim 1.5$ when internal energy is
included ($\alpha_{\mathrm{th}}=1$). This agreement was observed in all the
masses we explored ($M_{\mathrm{d}}=8,12,20~{}M_{\odot}$), which could be a
useful relation when generalizing our formalism.

Figure 4: Post-CE orbital separations predicted from our new formalism for
1.4, 3, and 5 $M_{\odot}$ companions as a function of the radius of the 12
$M_{\odot}$ donor when it first overflows its Roche lobe. Dot-dashed curves
show the pre-CE orbital separation, dashed curves show the orbital separations
after stage I and solid curves show the final post-CE separations. For
comparison, we overplot the core radius $R_{\mathrm{core}}$ (black curve) and
radius at the top of the intershell $R_{\mathrm{rad}}$ (grey curve), along
with the post-CE separations predicted by the classical $\alpha$-formalism
with $\alpha_{\mathrm{CE}}=1$ (dotted curves). The blue rectangle indicates
the Fragos et al. (2019) result for a $12+1.4~{}M_{\odot}$ system.

Figure 4 displays the final separations predicted from our two-stage
formalism. After stage I, the orbital separation shrinks roughly by an order
of magnitude to $a\sim 20$–$200~{}R_{\odot}$ depending on the companion mass
(dashed curves). The amount of shrinkage depends only weakly on the
evolutionary stage of the donor when the CE phase is initiated. In some cases
when the companion mass is small, the separation after stage I becomes smaller
than the outer radius of the radiative intershell $R_{\mathrm{rad}}$. It is
not trivial what should happen in such cases. There is little mass in the
outer $\sim 2/3$ of the radiative intershell, so it may still be possible to
smoothly transition into a sufficiently stable mass transfer phase if it only
ducks into the radiative intershell by a small amount. Even if it dives deep
into the radiative intershell, it may still be possible to eject the envelope
via adiabatic processes as explored by Law-Smith et al. (2020). Conversely,
the dynamical friction within the radiative intershell may be strong enough to
cause the companion to merge with the donor core. Or the transferred mass may
develop into a contact binary configuration in which the angular momentum is
taken away from the $L_{2}$ point. In this paper, we assume that it smoothly
transitions to stage II in all cases. However, it may be sensible in future
studies to consider alternative treatments for the cases where the companion
dives into the radiative intershell after stage I.

The following stage II creates a dramatic difference in the final separations
depending on the companion mass. The orbit shrinks with stable mass transfer
when the companion mass is smaller than the core mass
($M_{2}<M_{\mathrm{core}}$), whereas the orbit expands when the companion is
heavier than the core ($M_{2}>M_{\mathrm{core}}$). When the core is similar to
the companion mass ($M_{2}\sim M_{\mathrm{core}}$), the separation hardly
changes after stage II. There is also little change for all case C CEs,
because the intershell mass is much smaller at these stages. We note that our
separation predicted for a $12+1.4~{}M_{\odot}$ system with a
$R_{\mathrm{d}}=600~{}R_{\odot}$ donor is within a factor $\lesssim 1.5$ of
the results obtained from 1D CE simulations by Fragos et al. (2019) despite
the drastically different approaches.

Figure 5: The value of $\alpha_{\mathrm{CE}}$ required to reproduce the final
separations predicted from our new formalism.

Our formalism generally predicts rather large post-CE separations, ranging
from a few tens to a few hundred $R_{\odot}$, except for accretors with the
mass of a neutron star, which can yield much smaller separations. In fact,
most systems seem to avoid either a merger with the core or a Roche lobe
overflow from a companion. This is in stark contrast with the post-CE
separations predicted from the classical $\alpha$-formalism (dotted curves),
which are orders of magnitude smaller throughout most of the parameter space
when we set $\alpha_{\mathrm{CE}}=1$.

In Figure 5, we plot the values of $\alpha_{\mathrm{CE}}$ required in the
classical $\alpha$-formalism to reproduce the final separations from our new
formalism. In most cases, our new formalism is equivalent to setting
$\alpha_{\mathrm{CE}}$ to very high values ($\alpha_{\mathrm{CE}}>10$). The
only exception is for early case B CEs with lower-mass companions. Such high
values of $\alpha_{\mathrm{CE}}$ have not typically been considered in binary
population synthesis studies. There are no direct observational constraints on
$\alpha_{\mathrm{CE}}$ for massive stars, but similar ranges of
$\alpha_{\mathrm{CE}}$ are often assumed as those inferred from observations
and simulations of low-mass stars, which are generally low (Iaconi & De Marco,
2019). We note that the BPASS code (Eldridge & Stanway, 2009; Eldridge et al.,
2017) takes an original approach to CE phases and Eldridge et al. (2017)
report that they have equivalent $\alpha_{\mathrm{CE}}\lambda$ values ranging
between 2 to 100, with the majority between 2 to 30. Assuming that their
$\lambda$ values are similar to the typical values including the intershell
($\lambda\sim 0.1$–$0.5$), that corresponds to $\alpha_{\mathrm{CE}}\sim
4$–$300$, which is similar to what we predict in our new formalism. However,
they also report that the CE efficiency is highest when the mass ratio is
smallest, an opposite trend from what we find. Thus the similarity may be
coincidental.

There is a narrow range with the larger companion mass ($M_{2}=5~{}M_{\odot}$)
where we predict a final separation which is impossible to achieve in the
$\alpha$-formalism no matter what value of $\alpha_{\mathrm{CE}}(>0)$ is used
(when $a_{f}/a_{i}\geq M_{\mathrm{core}}/M_{\mathrm{d}}$). This only occurs in
regions when the convective envelope is still shallow and therefore the whole
process is more like fully stable mass transfer with mass ratio reversal than
a CE phase. We caution that in these regimes where the star is still close to
being fully radiative, our assumption of stable mass transfer may break down
due to the so-called delayed dynamical instability (Hjellming & Webbink,
1987).

Most importantly, our new formalism has highly variable equivalent
$\alpha_{\mathrm{CE}}$’s depending on the evolutionary stage and companion
masses (see Figure 5). In the classical $\alpha$-formalism, the post-CE
separation scales linearly with the companion mass (Figure 4). This suggests
that using universal values for $\alpha_{\mathrm{CE}}$ in rapid binary
population synthesis studies will not achieve even qualitatively accurate
predictions.

Our illustrative examples used a particular set of MESA models where we
assumed mostly default settings with $Z=0.014$ for the metallicity, along with
a convective step overshooting parameter of $\alpha_{\mathrm{ov}}=0.33$ and a
semiconvection parameter of $\alpha_{\mathrm{sc}}=0.01$. Other assumptions can
yield markedly different relationships between convective envelope binding
energies, radiative intershell masses, and stellar radii. These variations
include the treatment of mixing processes (Schootemeijer et al., 2019; Klencki
et al., 2021), and the effect of previous accretion onto the donor (Renzo et
al., 2022). While the specific CE outcomes will depend on the details of
stellar evolution models, the fundamental physical considerations that
motivate the two-stage approach proposed here still hold.

##  4 Conclusion and discussion

We propose a new formalism to predict the outcomes of CE phases with massive
star donors. In our new formalism, we split the whole CE process into two
stages: (I) dynamical inspiral through the outer convective part of the
envelope, and (II) thermal timescale mass transfer of the inner radiative part
of the envelope. We treat stage I as an adiabatic process and apply the
commonly used $\alpha$-formalism (or energy formalism). Stage II is treated as
a stable mass transfer, focusing on the angular momentum evolution of the
system. The combined outcome gives the final post-CE separation.

We demonstrated the predictions for the final separations with our formalism
with some fiducial choices of parameters ($\alpha_{\mathrm{CE}}=1$ for stage I
and isotropic re-emission for stage II). Overall, the predicted post-CE
orbital separations seem to be rather wide compared to what would be predicted
from classical energy formalisms. There is a qualitative difference depending
on whether the companion mass is larger or smaller than the donor core mass.
When the companion is lighter than the core, the orbit can shrink
substantially down to values similar to those predicted by traditional
formalisms. On the other hand when the companion is heavier than the core, the
orbit can widen after the initial plunge-in, ending up at large orbital
separations.

The generally wide post-CE separations can have broad implications. One
example is that we may predict a high fraction of stripped-envelope supernova
progenitors to have low-mass companions, which previously would have been
considered unable to survive CEs. In a supernova called SN2006jc, we estimated
from the current companion photometry that the progenitor system experienced a
CE phase with a $M_{\mathrm{d}}\sim 12~{}M_{\odot}$ donor and $M_{2}\sim
3$–$4~{}M_{\odot}$ companion, ending up with a $a\sim 40~{}R_{\odot}$
separation (Sun et al., 2020; Ogata et al., 2021). Our new formalism predicts
$a\sim 40$–$100~{}R_{\odot}$ for $M_{2}=3~{}M_{\odot}$, which is roughly in
agreement with this observational constraint. Another example may be iPTF13bvn
(Cao et al., 2013). Based on its classification as a type Ib supernova and on
the progenitor radius inferred from pre-supernova photometry, we argued that
it should have experienced a CE phase with $\alpha_{\mathrm{CE}}\gtrsim 20$
(Hirai, 2017). Such high efficiencies are easily achievable in our new
formalism, implying that iPTF13bvn may have a low-mass companion star. Many
other stripped-envelope supernova progenitors may also have dim low-mass
companions, which may explain the lack of companion detections.

The increase in the post-CE separation predicted by our two-stage formalism
may explain how binaries with relatively extreme mass ratios may survive mass
transfer, addressing a puzzle in the formation of low-mass X-ray binaries
(Podsiadlowski & Rappaport, 2000; Podsiadlowski et al., 2002). The wide
separations predicted for massive companions also indicate that CEs are
unlikely to contribute significantly to the formation of binary black holes as
gravitational-wave sources, since extreme mass ratios are unlikely when
transferring mass onto a black hole formed from the primary and $30M_{\odot}$
black hole binaries with separations above $\sim 50R_{\odot}$ will not merge
in the age of Universe. On the other hand, our formalism can reproduce very
tight Galactic double neutron star systems observed as radio pulsars (e.g.
Hulse & Taylor, 1975), provided a CE stage initiated by an early Hertzsprung-
gap donor is survivable.

One of the main features of our formalism is that the CE efficiency has a
strong dependence on the system parameters such as the evolutionary stage of
the donor and the companion mass. When translated to the traditional
$\alpha$-formalism, our final separations correspond to an extremely wide
range of equivalent $\alpha_{\mathrm{CE}}$ values from $\sim 0.1$ to
$>10^{4}$. This should have a significant impact on the results of population
synthesis studies, which currently typically employ a universal value of
$\alpha_{\mathrm{CE}}$. Frequently, the uncertainties are explored by varying
the value of the universal $\alpha_{\mathrm{CE}}$, but that cannot reproduce
the qualitative sensitivity to the donor’s evolutionary stage and companion
mass. We leave it for future work to explore how the predictions of the
populations of various objects such as binary black hole and binary neutron
star mergers, X-ray binaries, double pulsars and stripped-envelope supernovae
will be impacted with our new formalism.

We stress that we are not trying to establish a complete model, but rather a
physically-motivated framework as a basis for further research. Many of the
assumptions going into this formalism should be examined in future work. Some
other intricacies such as pre-CE evolution, onset of CE phases, envelope
fallback, etc have been ignored to maintain simplicity for our formalism.
However, we believe that our model captures the key features that distinguish
massive star donors from low-mass donors, and provides a useful guideline for
how to treat CEs in population synthesis studies.

The authors thank Orsola De Marco, Mike Lau, and Daniel Price for discussions,
and Lewis Picker for comments on the manuscript. This work was performed on
the OzSTAR national facility at Swinburne University of Technology. The OzSTAR
program receives funding in part from the Astronomy National Collaborative
Research Infrastructure Strategy (NCRIS) allocation provided by the Australian
Government. IM is a recipient of the Australian Research Council Future
Fellowship FT190100574.

## References

  * Cao et al. (2013) Cao, Y., Kasliwal, M. M., Arcavi, I., et al. 2013, ApJ, 775, L7, doi: 10.1088/2041-8205/775/1/L7
  * Chamandy et al. (2018) Chamandy, L., Frank, A., Blackman, E. G., et al. 2018, MNRAS, 480, 1898, doi: 10.1093/mnras/sty1950
  * Clayton et al. (2017) Clayton, M., Podsiadlowski, Ph., Ivanova, N., & Justham, S. 2017, MNRAS, 470, 1788, doi: 10.1093/mnras/stx1290
  * Darwin (1879) Darwin, G. H. 1879, Proc. R. Soc. Lond., 29, 168, doi: 10.1098/rspl.1879.0028
  * de Kool (1990) de Kool, M. 1990, ApJ, 358, 189, doi: 10.1086/168974
  * Dessart et al. (2011) Dessart, L., Hillier, D. J., Livne, E., et al. 2011, MNRAS, 414, 2985, doi: 10.1111/j.1365-2966.2011.18598.x
  * Dewi & Tauris (2000) Dewi, J. D. M., & Tauris, T. M. 2000, A&A, 360, 1043.  https://arxiv.org/abs/astro-ph/0007034
  * Eggleton (1983) Eggleton, P. P. 1983, ApJ, 268, 368, doi: 10.1086/160960
  * Eldridge & Stanway (2009) Eldridge, J. J., & Stanway, E. R. 2009, MNRAS, 400, 1019, doi: 10.1111/j.1365-2966.2009.15514.x
  * Eldridge et al. (2017) Eldridge, J. J., Stanway, E. R., Xiao, L., et al. 2017, PASA, 34, e058, doi: 10.1017/pasa.2017.51
  * Fragos et al. (2019) Fragos, T., Andrews, J. J., Ramirez-Ruiz, E., et al. 2019, ApJ, 883, L45, doi: 10.3847/2041-8213/ab40d1
  * Fryer et al. (1996) Fryer, C. L., Benz, W., & Herant, M. 1996, ApJ, 460, 801, doi: 10.1086/177011
  * Gilkis & Arcavi (2022) Gilkis, A., & Arcavi, I. 2022, MNRAS, 511, 691, doi: 10.1093/mnras/stac088
  * Glanz & Perets (2021) Glanz, H., & Perets, H. B. 2021, MNRAS, 507, 2659, doi: 10.1093/mnras/stab2291
  * Gonzalez-Bolivar et al. (2022) Gonzalez-Bolivar, M., De Marco, O., Lau, M. Y. M., Hirai, R., & Price, D. J. 2022, arXiv e-prints, arXiv:2205.09749.  https://arxiv.org/abs/2205.09749
  * Götberg et al. (2017) Götberg, Y., de Mink, S. E., & Groh, J. H. 2017, A&A, 608, A11, doi: 10.1051/0004-6361/201730472
  * Hall et al. (2013) Hall, P. D., Tout, C. A., Izzard, R. G., & Keller, D. 2013, MNRAS, 435, 2048, doi: 10.1093/mnras/stt1422
  * Han et al. (1995) Han, Z., Podsiadlowski, Ph., & Eggleton, P. P. 1995, MNRAS, 272, 800, doi: 10.1093/mnras/272.4.800
  * Hirai (2017) Hirai, R. 2017, MNRAS, 466, 3775, doi: 10.1093/mnras/stw3321
  * Hjellming & Webbink (1987) Hjellming, M. S., & Webbink, R. F. 1987, ApJ, 318, 794, doi: 10.1086/165412
  * Hulse & Taylor (1975) Hulse, R. A., & Taylor, J. H. 1975, ApJ, 195, L51, doi: 10.1086/181708
  * Iaconi & De Marco (2019) Iaconi, R., & De Marco, O. 2019, MNRAS, 490, 2550, doi: 10.1093/mnras/stz2756
  * Iaconi et al. (2017) Iaconi, R., Reichardt, T., Staff, J., et al. 2017, MNRAS, 464, 4028, doi: 10.1093/mnras/stw2377
  * Iben & Tutukov (1985) Iben, I., J., & Tutukov, A. V. 1985, ApJS, 58, 661, doi: 10.1086/191054
  * Ivanova (2011) Ivanova, N. 2011, ApJ, 730, 76, doi: 10.1088/0004-637X/730/2/76
  * Ivanova et al. (2015) Ivanova, N., Justham, S., & Podsiadlowski, Ph. 2015, MNRAS, 447, 2181, doi: 10.1093/mnras/stu2582
  * Ivanova et al. (2020) Ivanova, N., Justham, S., & Ricker, P. 2020, Common Envelope Evolution, doi: 10.1088/2514-3433/abb6f0
  * Ivanova & Nandez (2016) Ivanova, N., & Nandez, J. L. A. 2016, MNRAS, 462, 362, doi: 10.1093/mnras/stw1676
  * Ivanova et al. (2013) Ivanova, N., Justham, S., Chen, X., et al. 2013, A&A Rev., 21, 59, doi: 10.1007/s00159-013-0059-2
  * Klencki et al. (2021) Klencki, J., Nelemans, G., Istrate, A. G., & Chruslinska, M. 2021, A&A, 645, A54, doi: 10.1051/0004-6361/202038707
  * Laplace et al. (2020) Laplace, E., Götberg, Y., de Mink, S. E., Justham, S., & Farmer, R. 2020, A&A, 637, A6, doi: 10.1051/0004-6361/201937300
  * Lau et al. (2022a) Lau, M. Y. M., Hirai, R., González-Bolívar, M., et al. 2022a, MNRAS, 512, 5462, doi: 10.1093/mnras/stac049
  * Lau et al. (2022b) Lau, M. Y. M., Hirai, R., Price, D. J., & Mandel, I. 2022b, arXiv e-prints, arXiv:2206.06411.  https://arxiv.org/abs/2206.06411
  * Law-Smith et al. (2020) Law-Smith, J. A. P., Everson, R. W., Ramirez-Ruiz, E., et al. 2020, arXiv e-prints, arXiv:2011.06630.  https://arxiv.org/abs/2011.06630
  * López-Cámara et al. (2019) López-Cámara, D., De Colle, F., & Moreno Méndez, E. 2019, MNRAS, 482, 3646, doi: 10.1093/mnras/sty2959
  * MacLeod et al. (2018a) MacLeod, M., Ostriker, E. C., & Stone, J. M. 2018a, ApJ, 868, 136, doi: 10.3847/1538-4357/aae9eb
  * MacLeod et al. (2018b) —. 2018b, ApJ, 863, 5, doi: 10.3847/1538-4357/aacf08
  * Marchant et al. (2021) Marchant, P., Pappas, K. M. W., Gallegos-Garcia, M., et al. 2021, A&A, 650, A107, doi: 10.1051/0004-6361/202039992
  * Moreno et al. (2021) Moreno, M. M., Schneider, F. R. N., Roepke, F. K., et al. 2021, arXiv e-prints, arXiv:2111.12112.  https://arxiv.org/abs/2111.12112
  * Nelemans & Tout (2005) Nelemans, G., & Tout, C. A. 2005, MNRAS, 356, 753, doi: 10.1111/j.1365-2966.2004.08496.x
  * Nelemans et al. (2000) Nelemans, G., Verbunt, F., Yungelson, L. R., & Portegies Zwart, S. F. 2000, A&A, 360, 1011.  https://arxiv.org/abs/astro-ph/0006216
  * Ogata et al. (2021) Ogata, M., Hirai, R., & Hijikawa, K. 2021, MNRAS, 505, 2485, doi: 10.1093/mnras/stab1439
  * Ohlmann et al. (2016) Ohlmann, S. T., Röpke, F. K., Pakmor, R., Springel, V., & Müller, E. 2016, MNRAS, 462, L121, doi: 10.1093/mnrasl/slw144
  * Paczynski (1976) Paczynski, B. 1976, in Structure and Evolution of Close Binary Systems, ed. P. Eggleton, S. Mitton, & J. Whelan, Vol. 73, 75
  * Passy et al. (2012) Passy, J.-C., De Marco, O., Fryer, C. L., et al. 2012, ApJ, 744, 52, doi: 10.1088/0004-637X/744/1/52
  * Paxton et al. (2011) Paxton, B., Bildsten, L., Dotter, A., et al. 2011, ApJS, 192, 3, doi: 10.1088/0067-0049/192/1/3
  * Paxton et al. (2013) Paxton, B., Cantiello, M., Arras, P., et al. 2013, ApJS, 208, 4, doi: 10.1088/0067-0049/208/1/4
  * Paxton et al. (2015) Paxton, B., Marchant, P., Schwab, J., et al. 2015, ApJS, 220, 15, doi: 10.1088/0067-0049/220/1/15
  * Paxton et al. (2018) Paxton, B., Schwab, J., Bauer, E. B., et al. 2018, ApJS, 234, 34, doi: 10.3847/1538-4365/aaa5a8
  * Paxton et al. (2019) Paxton, B., Smolec, R., Schwab, J., et al. 2019, ApJS, 243, 10, doi: 10.3847/1538-4365/ab2241
  * Podsiadlowski & Rappaport (2000) Podsiadlowski, Ph., & Rappaport, S. 2000, ApJ, 529, 946, doi: 10.1086/308323
  * Podsiadlowski et al. (2002) Podsiadlowski, Ph., Rappaport, S., & Pfahl, E. D. 2002, ApJ, 565, 1107, doi: 10.1086/324686
  * Politano (2004) Politano, M. 2004, ApJ, 604, 817, doi: 10.1086/381958
  * Politano (2021) —. 2021, A&A, 648, L6, doi: 10.1051/0004-6361/202140442
  * Politano & Weiler (2007) Politano, M., & Weiler, K. P. 2007, ApJ, 665, 663, doi: 10.1086/518997
  * Postnov & Yungelson (2014) Postnov, K. A., & Yungelson, L. R. 2014, Living Reviews in Relativity, 17, 3, doi: 10.12942/lrr-2014-3
  * Rasio & Livio (1996) Rasio, F. A., & Livio, M. 1996, ApJ, 471, 366, doi: 10.1086/177975
  * Renzo et al. (2022) Renzo, M., Zapartas, E., Justham, S., et al. 2022, arXiv e-prints, arXiv:2206.15338.  https://arxiv.org/abs/2206.15338
  * Ricker & Taam (2008) Ricker, P. M., & Taam, R. E. 2008, ApJ, 672, L41, doi: 10.1086/526343
  * Schootemeijer et al. (2019) Schootemeijer, A., Langer, N., Grin, N. J., & Wang, C. 2019, A&A, 625, A132, doi: 10.1051/0004-6361/201935046
  * Soberman et al. (1997) Soberman, G. E., Phinney, E. S., & van den Heuvel, E. P. J. 1997, A&A, 327, 620.  https://arxiv.org/abs/astro-ph/9703016
  * Soker (2004) Soker, N. 2004, New A, 9, 399, doi: 10.1016/j.newast.2004.01.004
  * Sun et al. (2020) Sun, N.-C., Maund, J. R., Hirai, R., Crowther, P. A., & Podsiadlowski, Ph. 2020, MNRAS, 491, 6000, doi: 10.1093/mnras/stz3431
  * Townsend (2021) Townsend, R. 2021, MESA SDK for Linux, 21.4.1, Zenodo, doi: 10.5281/zenodo.5802444
  * Tutukov & Yungelson (1979) Tutukov, A., & Yungelson, L. 1979, in Mass Loss and Evolution of O-Type Stars, ed. P. S. Conti & C. W. H. De Loore, Vol. 83, 401–406
  * van den Heuvel (1976) van den Heuvel, E. P. J. 1976, in Structure and Evolution of Close Binary Systems, ed. P. Eggleton, S. Mitton, & J. Whelan, Vol. 73, 35
  * Vigna-Gómez et al. (2022) Vigna-Gómez, A., Wassink, M., Klencki, J., et al. 2022, MNRAS, 511, 2326, doi: 10.1093/mnras/stac237
  * Webbink (1984) Webbink, R. F. 1984, ApJ, 277, 355, doi: 10.1086/161701
  * Webbink (2008) Webbink, R. F. 2008, in Astrophysics and Space Science Library, Vol. 352, Astrophysics and Space Science Library, ed. E. F. Milone, D. A. Leahy, & D. W. Hobill, 233, doi: 10.1007/978-1-4020-6544-6_13
  * Williams et al. (2021) Williams, T., Kelley, C., & many others. 2021, Gnuplot 5.4: an interactive plotting program, http://www.gnuplot.info
  * Wilson & Nordhaus (2022) Wilson, E. C., & Nordhaus, J. 2022, arXiv e-prints, arXiv:2203.06091.  https://arxiv.org/abs/2203.06091
  * Xu & Li (2010a) Xu, X.-J., & Li, X.-D. 2010a, ApJ, 716, 114, doi: 10.1088/0004-637X/716/1/114
  * Xu & Li (2010b) —. 2010b, ApJ, 722, 1985, doi: 10.1088/0004-637X/722/2/1985

  *[CE]: common-envelope
  *[RG]: red giant
  *[RSG]: red supergiant
  *[RGs]: red giant
  *[RSGs]: red supergiant
  *[CEs]: common-envelope


# Histopathology DatasetGAN: Synthesizing Large-Resolution Histopathology
Datasets

S.A. Rizvi1, P. Cicalese1, S.V. Seshan3, S. Sciascia4, J.U.Becker2 and H.V.
Nguyen1 1\. Department of Electrical and Computer Engineering, University of
Houston, Houston, TX, USA
2\. Institute of Pathology, University Hospital of Cologne, Cologne, Germany
3\. Weill Cornell Medicine, Cornell University, New York City, New York
4\. Center of Research of Immunopathology and Rare Diseases and SCDU
Nephrology
and Dialysis, Department of Clinical and Biological Sciences, University of
Turin, Turin, Italy
{srizvi10, pcicalese<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Self-supervised learning (SSL) methods are enabling an increasing number of
deep learning models to be trained on image datasets in domains where labels
are difficult to obtain. These methods, however, struggle to scale to the high
resolution of medical imaging datasets, where they are critical for achieving
good generalization on label-scarce medical image datasets. In this work, we
propose the Histopathology DatasetGAN (HDGAN) framework, an extension of the
DatasetGAN semi-supervised framework for image generation and segmentation
that scales well to large-resolution histopathology images. We make several
adaptations from the original framework, including updating the generative
backbone, selectively extracting latent features from the generator, and
switching to memory-mapped arrays. These changes reduce the memory consumption
of the framework, improving its applicability to medical imaging domains. We
evaluate HDGAN on a thrombotic microangiopathy high-resolution tile dataset,
demonstrating strong performance on the high-resolution image-annotation
generation task. We hope that this work enables more application of deep
learning models to medical datasets, in addition to encouraging more
exploration of self-supervised frameworks within the medical imaging domain.

###### Index Terms:

medical image segmentation, deep learning, semi-supervised learning,
generative adversarial networks.

## I. Introduction

Deep learning is powering an increasing number of medical image segmentation
applications, accelerating the field past previous statistical and machine
learning methods [1]. Semantic segmentation, which aims to make structural
features more apparent in images, is a crucial part of aiding clinicians in
making more accurate diagnoses and improving computer-physician interaction.
Acquiring the large medical image datasets required for training deep models,
however, poses difficult challenges. Medical datasets are 1) expensive to
gather, 2) require very time-consuming annotation, often at expert-level, and
3) often come in large file formats, making it computationally challenging to
train large models on.

Semi-supervised learning (SSL) allows for learning under these limitations,
utilizing more widely available unlabeled data in order to augment datasets
and allow the training of larger, more data-hungry models. Current SSL
literature includes pseudo-labeling methods [2] which attempt to create labels
on unlabeled data, and consistency regularization methods which encourage
networks to output consistent predictions for augmented data. Contrastive
methods [3] define an unsupervised loss term on unlabeled examples which
allows for the training of feature extractors. These can then be trained on
labeled examples, reducing the overall number of labels required for training.

Generative models have been shown to learn a powerful and detailed latent
space representation [4] while learning to generate images following the
original data distribution. [4] proposes a semi-supervised framework for
pixel-wise classification in images, utilizing a Multilayer Perceptron (MLP)
to classify the GAN’s latent features into a segmentation map. This framework,
however, requires large memory and computational resources which limits its
scalability to domains with larger image sizes. We aim to extend this
framework and increase its applicability to medical images by reducing the
computational requirements needed to learn off of the features learned by the
generative models.

In this work, we propose Histopathology DatasetGAN (HDGAN), a framework for
generating high-resolution image-annotation pairs for augmenting medical image
datasets. We focus on the scalability of the framework to high-resolution
images to accommodate the large images sizes typically seen in the field of
medical imaging. By utilizing memory-mapped arrays and reducing the number of
latent features extracted from the generative model we drastically reduce the
memory consumption and computational requirements for the framework, allowing
us to generate image-latent pairs at a much higher resolution. We demonstrate
the effectiveness of our framework on a Thrombotic Microangiopathy (TMA)
large-resolution tile dataset, and expect that this work will open the doors
for further exploration and adoption of semi-supervised training strategies
for medical image analysis tasks.

## II. Related Works

Semi-supervised learning aims to train more powerful and data-hungry networks
using a mix of labeled and unlabeled data, taking advantage of the large
quantities of unlabeled data available in domains where obtaining labels for
data is expensive. Pseudo-labeling [2, 5] and consistency regularization [3,
5] have shown success on semantic segmentation tasks, utilizing unlabeled data
by first training a network on the small labeled dataset, and then retraining
on both real labels and highly confident pseudolabels predicted on the
unlabeled data.

Generative networks learn a latent featurespace while learning to synthesize
images, resulting in latent features containing rich semantic information
which can aid in semi-supervised tasks. Several works have explored using GAN
latent features in order to aid in semi-supervised learning tasks for semantic
segmentation. [6] proposes a framework for projecting the latent feature maps
of a generative network onto semantic segmentation maps using a lightweight
decoder built off of convolutional and residual blocks. DatasetGAN [4]
proposed a pixel-level framework that used an ensemble of MLPs to directly
classify pixel-level concatenated features from the entire latent space of the
generative network into a segmentation mask. Further works building off of
DatasetGAN focused on expanding the diversity of the generated dataset [7] and
augmenting the generator with a label synthesis branch to directly have the
generator predict segmentation masks [8].

Semi-supervised semantic segmentation has been explored in many works, due to
the high annotative cost and expertise needed for annotation within the
medical domain. In [9], the authors predicted pseudo-labels on unlabeled
images and set up an optimization problem to alternately update network
parameters and predict segmentations on unlabeled medical images. Other works
pursued adversarial [10] or transformation-consistency training [11] in order
to utilized unlabeled medical images. In [8], the authors evaluated their
image-annotation generative framework on chest X-ray and skin lesion
segmentation, showing strong generalization capabilities with their fully
generative approach. We focus back on the original DatasetGAN framework,
focusing on improving the memory consumption of the framework. [12] presented
a preliminary analysis of the applicability of DatasetGAN to the medical
imaging domain, making three recommendations to the framework in order to make
its application to the medical image more tractable. The first two
recommendations involved upgrading the backbone generative network to
StyleGAN2-ADA to improve training in limited data regimes, and the third
recommendation involved reducing the number of latent features extracted from
the generative network while maintaining segmentation mask quality. All three
recommendations are satisfied in our extension to the DatasetGAN framework, in
addition to our use of memory-mapped arrays to make GPU-based training more
feasible.

Figure 1: Histopathology DatasetGAN framework overview

## III. Data

The dataset for our experiments consisted of 1,577 whole slide images (WSIs)
taken from 100 native kidney biopsies; 50 with a diagnosis of thrombotic
microangiopathy (TMA) and 50 mimickers of TMA, with similar or overlapping
histopathological features in which a different diagnosis was made. These
mimicker diagnoses included necrotising small vessel vasculitis,
cryoglobulinemic glomerulopathy, light chain deposition disease, severe benign
nephrosclerosis, collapsing type focal and segmental glomerulosclerosis and
Bevacizumab-associated obliterative glomerulopathy. The WSIs are taken from
the three medical centers (Cologne, Weill-Cornell and Turin) and included a
combination of at least three of the four diverse histopathological stainings
hematoxylin-eosin (HE), periodic-acid Schiff (PAS), Jones silver and
trichrome, all scanned with a x40 objective. In this work, we pre-process the
WSIs by first tiling each WSI and extracting any 4096 x 4096 tiles which
contained morphological compartments of interest. We then filtered out tiles
which consisted of mostly whitespace, resulting in a dataset of 32,732 tiles.
We split the tiles into 5 folds and train our StyleGAN2-ADA backbone on 4 of
the folds, leaving out one fold as a holdout.

## IV. Methodology

Our method builds off of the original DatasetGAN framework [4], with
modifications which make it more suited for high-resolution medical images.
The core idea is, given a latent vector z, we can extract the latent features
from a generative network when the latent vector is passed through the
generator. These feature maps contain semantic information about the image
being generated, and therefore are useful for pixel-wise segmentation tasks.
The feature maps which are extracted at different resolutions from the
generator network are upsampled to match the output image size, yielding a
feature map tensor with the same height and width dimensions as the output
image. With annotations on a small number of generated images, a smaller
classifier can then be trained to predict segmentation maps in a pixel-wise
fashion from the extracted feature maps. This provides the framework for
generating image-latent pairs.

We extend this framework to work with high-resolution histopathology images by
making several adjustments. First, for our generative backbone we use
StyleGAN2-ADA [13], which has been proven to work well in limited data
settings with histopathology images. Second, unlike [4], we choose to only
extract and upsample latent features from the last 5 blocks in the synthesis
module of the generator network, as shown in Figure 1. This yields feature
maps which contain more fine-grained semantic information. These later
features are sufficient for our pixel-wise segmentation task, and drastically
cut the amount of computation and upsampling required. The computational
burden is still too high, however, to fit the entire feature map dataset into
memory while training when dealing with large histopathology images. We
therefore use memory-mapped arrays during training, which allows for only
those portions of the feature map tensor which are currently in use to be
loaded from disk and into memory, reducing the memory requirement of the
framework. This allows us to scale our framework to 4096 x 4096 TMA tile
segmentation, with samples shown in Figure 2.

## V. Experimental Design

We evaluate the HDGAN framework on a 5-class semantic segmentation task on the
TMA tile dataset. First, we generate 500 images with a truncation of 0.7 using
our trained StyleGAN2-ADA network, saving the corresponding latent vectors. 36
of the 500 images were then selected for our dataset by an expert
nephropathologist, who provided pixel-wise annotations of each morphological
compartment for these 36 images. We randomly chose 16 images as the training
set for the pixel-level classifiers, leaving 4 images for validation and
another 16 images as the test set. For baseline experiments, we refer readers
to the original DatasetGAN paper [4] for comparisons to semi-supervised
baselines.

For our classifier, we use a simple 3-layer Multilayer Perceptron (MLP) with
ReLU activation, Batch Normalization [14], and Dropout [15] following each
linear layer. We train the classifier using vanilla stochastic gradient
descent with a learning rate of 0.0001. For our loss function, we use
CrossEntropy loss.

## VI. Results

In Table 1 we report the class-wise pixel-level accuracy of the pixel-
classifier on our test set. As expected, the pixel-level classifiers are able
to achieve high classification accuracy on several morphological compartments,
with an average Dice coefficient of 0.92 on the test set images. The
classifiers struggle, however, when morphological structures in generated
images resemble two different compartments, often misclassifying arteriole
pixels as either glomeruli or artery. This semantic blending was confirmed by
our expert annotator, sometimes forcing the annotator to choose between
different class labels when a generated compartment had features of both
present.

Figure 2: Examples of synthesized images and annotations from the Histopathology DatasetGAN framework. Row a) represents generated TMA 4096x4096 tiles, with row b) and row c) showing the corresponding predicted segmentation map and ground truth segmentation map, respectively. TABLE 1: Class-wise Pixel Accuracy Class |  Accuracy (%)
---|---
Whitespace |  68.42
Cortical Tubulointerstitium |  66.57
Glomerulus |  82.89
Arteriole |  52.09
Artery |  77.82

## VII. Summary

In this paper, we proposed an extension of the DatasetGAN framework with
several adaptations making the framework more computationally tractable for
large medical images. By utilizing only a subset of latent features from the
generative network and using memory-mapped arrays, we greatly reduce the
memory requirements of the framework. We also update the backbone of the
framework to the StyleGAN2 architecture with adaptive discriminator
augmentation, which allows for more stable training in limited data settings
as is often encountered in medical imaging domains.

## VIII. Future Work

The DatasetGAN framework suffers limitations when the generative backbone is
not trained well on the original dataset. This manifested itself in the
produced segmentation maps as noisy contours around compartments, where the
semantic meaning held in the feature space of the generative networks was not
decisively one compartment or another. Future work exploring the latent space
of generative networks in relation to semantic meaning of morphological
compartments could help disentangle the feature space and result in cleaner
segmentation maps. In addition, more works exploring generative frameworks
that directly generate the segmentation map such as in [8] can help advance
the frameworks for dataset augmentation through semi-supervised learning.

## Acknowledgements

The research in this work was supported by the National Science Foundation and
the National Institute of Health. The content represents the views of the
authors, not necessarily the views of the funding agencies.

## References

  * [1] T. Lei, R. Wang, Y. Wan, X. Du, H. Meng, and A. Nandi, “Medical image segmentation using deep learning: A survey.” 2020.
  * [2] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. Raffel, “Mixmatch: A holistic approach to semi-supervised learning,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [3] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” _Advances in neural information processing systems_ , vol. 30, 2017\.
  * [4] Y. Zhang, H. Ling, J. Gao, K. Yin, J. Lafleche, A. Barriuso, A. Torralba, and S. Fidler, “Datasetgan: Efficient labeled data factory with minimal human effort,” _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 10 145–10 155.
  * [5] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. Raffel, E. Cubuk, A. Kurakin, and C. Li, “Fixmatch: Simplifying semi-supervised learning with consistency and confidence,” _Advances in neural information processing systems_ , vol. 33, pp. 596–608, 2020.
  * [6] D. Galeev, K. Sofiiuk, D. Rukhovich, M. Romanov, O. Barinova, and A. Konushin, “Learning high-resolution domain-specific representations with a gan generator,” _Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR)_. Springer, 2021, pp. 108–118.
  * [7] D. Li, H. Ling, S. Kim, K. Kreis, S. Fidler, and A. Torralba, “Bigdatasetgan: Synthesizing imagenet with pixel-wise annotations,” _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 21 330–21 340.
  * [8] D. Li, J. Yang, K. Kreis, A. Torralba, and S. Fidler, “Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization,” _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 8300–8311.
  * [9] W. Bai, O. Oktay, M. Sinclair, H. Suzuki, M. Rajchl, G. Tarroni, B. Glocker, A. King, P. Matthews, and D. Rueckert, “Semi-supervised learning for network-based cardiac mr image segmentation,” _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2017, pp. 253–260.
  * [10] A. Mondal, J. Dolz, and C. Desrosiers, “Few-shot 3d multi-modal medical image segmentation using generative adversarial learning,” _arXiv preprint arXiv:1810.12241_ , 2018.
  * [11] X. Li, L. Yu, H. Chen, C. Fu, L. Xing, and P. Heng, “Transformation-consistent self-ensembling model for semisupervised medical image segmentation,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, no. 2, pp. 523–534, 2020.
  * [12] Z. Fan, V. Kelkar, M. Anastasio, and H. Li, “Application of datasetgan in medical imaging: preliminary studies,” _Medical Imaging 2022: Image Processing_ , vol. 12032. SPIE, 2022, pp. 452–458.
  * [13] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila, “Training generative adversarial networks with limited data,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 104–12 114, 2020.
  * [14] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” _International conference on machine learning_. PMLR, 2015, pp. 448–456.
  * [15] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” _The journal of machine learning research_ , vol. 15, no. 1, pp. 1929–1958, 2014.