# Understanding the Salt Effects on the Liquid-Liquid Phase Separation of
Proteins

Chao Duan  Department of Chemical and Biomolecular Engineering, University of
California Berkeley, CA 94720, USA  Rui Wang<EMAIL_ADDRESS>Department of Chemical and Biomolecular Engineering, University of California
Berkeley, CA 94720, USA  Materials Sciences Division, Lawrence Berkeley
National Lab, Berkeley, CA 94720, USA

(August 28, 2024)

###### Abstract

Protein aggregation via liquid-liquid phase separation (LLPS) is ubiquitous in
nature and intimately connects to many human diseases. Although it is widely
known that the addition of salt has crucial impacts on the LLPS of protein,
full understanding of the salt effect remains an outstanding challenge. Here,
we develop a molecular theory which systematically incorporates the self-
consistent field theory for charged macromolecules into the solution
thermodynamics. The electrostatic interaction, hydrophobicity, ion solvation
and translational entropy are included in a unified framework. Our theory
fully captures the long-standing puzzles of the non-monotonic salt
concentration dependence and the specific ion effect. We find that proteins
show salting-out at low salt concentrations due to ionic screening. The
solubility follows the inverse Hofmeister series. In the high salt
concentration regime, protein remains salting-out for small ions but turns to
salting-in for larger ions, accompanied by the reversal of the Hofmeister
series. We reveal that the solubility at high salt concentrations is
determined by the competition between the solvation energy and translational
entropy of ion. Furthermore, we derive an analytical criterion for determining
the boundary between the salting-in and salting-out regimes. The theoretical
prediction is in quantitative agreement with experimental results for various
proteins and salt ions without any fitting parameters.

##  I Introduction

Protein aggregation is ubiquitous in living cells, through which plenty of
biomolecular condensates can be assembled [1, 2]. These biomolecular
condensates play a vital role in cellular organization and functions, such as
the formation of nucleoli [3], heterochromatin and ribonucleoprotein granule
[4, 5] as well as signal transduction within the cytoplasm [6, 7, 8]. In
addition, the aggregation of various misfolded proteins intimately linked to
many neurodegenerative diseases including Alzheimer’s, Parkinson’s, diabetes,
and prion diseases [9, 10]. Evidence is mounting that protein aggregation
proceeds via a liquid-liquid phase separation (LLPS), which is manifested as
the formation of a dense phase often resembling liquid droplets and a
coexisting dilute phase [11, 12, 13, 14]. Revealing the essential physical
chemistry of the LLPS-driven aggregation will help delineate the functions of
biomolecular condensates and provides useful guidance for the therapy of
diseases [15, 16, 17]. In spite of increasing academic interests,
understanding and regulating LLPS of protein remains a big challenge [18].

Salt effect on LLPS of protein is one of the most long-standing puzzles. It is
well-known that the ionic environment has critical impacts on the LLPS;
besides, the addition of salt also provides an effective tool to modulate it
[19]. However, this salt effect is very complicated: the LLPS of protein has
non-trivial dependence on both the salt concentration and the chemical
identity of ions (usually known as the specific ion effect) [20, 21, 22, 23,
24]. Zhang and Cremer measured the cloud point of positively-charged lysozyme
solutions [26]. At low salt concentrations, they found that the solubility of
lysozyme decreases as salt concentration increases, i.e. protein salting-out.
The increase of solubility follows the inverse Hofmeister series of anion. In
contrast, at high salt concentrations, lysozyme remains to show salting-out
for some anions (e.g. Cl-), whereas other anions (e.g. Br- and I-) enhance the
lysozyme solubility, i.e. protein salting-in. The solubility increase follows
the direct Hofmeister series in the high salt concentration regime. Neither
the non-monotonic salt concentration effect nor the specific ion effect can be
explained, even qualitatively, by the standard mean-field Poisson-Boltzmann
(PB) theory [25]. Similar salt-dependent behaviors have also been observed in
other protein solutions [27, 28, 29, 30] and soft matter systems such as
synthetic polymers [31, 32, 33] and colloidal dispersions [34, 35], implying
the universality of the salt effects on LLPS.

Figure 1: (A) Schematic of the total system consisting of coexisting dilute
phase (D) and concentrated phase (C). The dilute phase is an assemble of
protein aggregates with different aggregation number. (B) A subsystem
containing one isolated aggregate in the presence of salt ions. (C) A
representative phase diagram plotting the equilibrium volume fractions of the
two coexisting phases ($\phi_{D}$ and $\phi_{C}$) as a function of bulk salt
concentration $c_{b}$. $a_{+}=a_{-}=2.5\mathring{\rm A}$, $z_{+}=z_{-}=1$,
$\epsilon_{P}=30$, and $\epsilon_{S}=80$.

Many theoretical and computational efforts have been made to explain these
salt effects. Kastelic et al. assumed a phenomenological model for the
interaction energy between proteins, where the well depths in the presence of
different alkali-halide salts were fitted to experimental data [36]. They
suggested that the salt effect on LLPS is majorly attributed to the ionic
screening, but the salting-in behavior and the reversal of Hofmeister series
observed at high salt concentrations have not been captured. Zhang and Cremer
developed a modified binding isotherm model [26]. The model parameters
representing the effectiveness and equilibrium constant for the association of
a specific anion to protein surface were fitted to the measured cloud point.
They found that the salt effect in the high salt concentration regime is
correlated to the interfacial tension of protein surrounded by anions with
different polarizability. Furthermore, using a modified PB theory to account
for ion size and polarizability, Bostr$\ddot{\rm o}$m et al. suggested that
the reversal of the Hofmeister series at high salt concentrations originates
from the inversion of effective surface charge of proteins [37]. However,
there is no theory up to now that can unify the description of the salt
effects on LLPS of proteins for the entire salt concentration regime. The
underlying physical chemistry, particularly for the counterintuitive behaviors
observed at high salt concentrations, is still unclear.

To uncover the salt effect on LLPS of protein, we develop a molecular theory
which systematically includes the electrostatics, hydrophobic interaction, ion
solvation and translational entropy of protein in a unified framework.
Compared to the existing theories, we have made the following two major
improvements. First, we explicitly account for the highly localized density
fluctuation of proteins in the dilute phase rather than assuming random mixing
as invoked in the Flory-Huggins (F-H) theory [38, 39]. This enables the
accurate treatment of ionic screening effect on a charged protein aggregate.
Second, we include the self-energy of ions as a result of electrostatic
fluctuation, which captures the salt effects beyond the mean-field PB level
[40, 41]. Our theory predicts that protein salting-out at low salt
concentrations is attributed to the screening effect, whereas protein
solubility at high salt concentrations is determined by the competition
between the solvation energy and translational entropy of ions. Furthermore,
we derive an analytical criterion for determining the boundary between the
salting-in and salting-out regimes for different proteins and ions. The
theoretical prediction is in quantitative agreement with experimental data
reported in literature without any fitting parameters.

##  II Theory

The solubility of protein in a salt solution is built upon the equilibrium
between a dilute phase and a protein-rich concentrated phase as illustrated in
Fig. 1A. The concentrated solution can be modeled by a homogeneous liquid-like
condensate due to the negligible density fluctuation and the surface
contribution. However, the description of the dilute phase is nontrivial
because of the large localized density fluctuation. An instantaneous picture
of the dilute protein solution has localized high concentrations where the
proteins are located and pure salt solutions elsewhere. This is an exactly
different scenario compared to that envisioned in the random mixing picture of
F-H theory used in existing work [15, 16, 17, 42, 43]. To account for this
large localized density fluctuation in the dilute phase, we focus on the
subvolume of the entire solution containing only one isolated protein or one
multi-protein aggregate (see Fig. 1B). The density profile and free energy of
the protein/aggregate is obtained by applying the self-consistent field theory
(SCFT) in the subvolume. This information is then incorporated into the
framework of dilute solution thermodynamics to reconstruct the solution
behavior of the entire dilute phase.

### Self-Consistent Field Theory for an Isolated Protein/Aggregate

As shown in Fig. 1B, we consider a subvolume consisting of an isolated
aggregate of $m$ proteins and $n_{S}$ solvent molecules in the presence of
$n_{\pm}$ mobile ions with valency $z_{\pm}$. $m=1$ specifies the case of an
isolated protein. The subvolume is taken as a semicanonical ensemble: the
number of proteins is fixed whereas solvent and mobile ions are connected with
a bulk salt solution of ion concentration $c^{b}_{\pm}$ that maintains the
chemical potentials of the solvent $\mu_{S}$ and ions $\mu_{\pm}$ [44, 39].
The proteins considered here are assumed to be unfolded or intrinsic
disordered, where the widely-adopted charged macromolecular model is invoked
to describe these proteins [17, 45, 46]. This model is also general for
synthetic polyelectrolytes and other biomacromolecules [47]. The charged
macromolecule is assumed to be a Gaussian chain of $N$ Kuhn segments with Kuhn
length $b$. The smeared charge model is adopted to describe the backbone
charge distribution with the charge density $\alpha$ [48]. For For simplicity,
the volume of the chain segment and the solvent molecule are assumed to be the
same $v_{0}$. The local hydrophobic interaction between protein and solvent is
described by the Flory parameter $\chi$. The key results of the SCFT are the
following set of equations for protein density $\rho_{P}({\bf r})$, conjugate
fields $\omega_{P}(\bf r)$ and $\omega_{S}(\bf r)$, electrostatic potential
$\psi({\bf r})$ and ion concentration $c_{\pm}({\bf r})$ (see SI, Section 1
for the detailed derivation):

$\displaystyle\omega_{P}({\bf r})$ $\displaystyle-\omega_{S}({\bf
r})=\chi[1-2\rho_{P}({\bf r})]-\frac{v_{0}}{2}\frac{\partial\epsilon({\bf
r})}{\partial\rho_{P}({\bf r})}[\nabla\psi({\bf r})]^{2}$
$\displaystyle+\alpha\psi({\bf r})+v_{0}\left[c_{+}({\bf r})\frac{\partial
u_{+}({\bf r})}{\partial\rho_{P}({\bf r})}+c_{-}({\bf r})\frac{\partial
u_{-}({\bf r})}{\partial\rho_{P}({\bf r})}\right]$ (1a)
$\displaystyle\rho_{P}({\bf r})$
$\displaystyle=\frac{m}{Q_{P}}\int_{0}^{N}{\rm d}sq({\bf r},s)q({\bf r},N-s)$
(1b) $\displaystyle 1-\rho_{P}({\bf r})$ $\displaystyle={\rm e}^{\mu_{S}}{\rm
exp}[-\omega_{S}({\bf r})]$ (1c) $\displaystyle-\nabla\cdot[\epsilon({\bf
r})\nabla\psi({\bf r})]$ $\displaystyle=z_{+}c_{+}({\bf r})-z_{-}c_{-}({\bf
r})+\frac{\alpha}{v_{0}}\rho_{P}({\bf r})$ (1d) $\displaystyle c_{\pm}({\bf
r})$ $\displaystyle=\lambda_{\pm}{\rm exp}[\mp z_{\pm}\psi({\bf
r})-u_{\pm}({\bf r})]$ (1e)

where $\epsilon({\bf r})=kT\epsilon_{0}\epsilon_{r}({\bf r})/e^{2}$ is the
scaled permittivity with $\epsilon_{0}$ the vacuum permittivity, $e$ the
elementary charge and $\epsilon_{r}({\bf r})$ the local dielectric constant.
$\epsilon_{r}({\bf r})$ can be evaluated based on the local composition [49,
50]. Here a linear mixing rule is adopted which leads to $\epsilon_{r}({\bf
r})=\epsilon_{P}\rho_{P}({\bf r})+\epsilon_{S}(1-\rho_{P}({\bf r}))$, with
$\epsilon_{P}$ and $\epsilon_{S}$ the dielectric constant of the pure protein
and solvent, respectively [49, 50, 51]. $\lambda_{\pm}=e^{\mu_{\pm}}/v_{\pm}$
is the fugacity of the ions controlled by the bulk salt concentration. $Q_{P}$
is the single-chain partition function given by $Q_{P}=(1/v_{0})\int{\rm
d}{\bf r}$ $q({\bf r},N)$, whereas $q({\bf r},s)$ is the chain propagator
determined by the diffusion equation

$\displaystyle\frac{\partial q({\bf r},s)}{\partial s}$
$\displaystyle=\frac{b^{2}}{6}\nabla^{2}q({\bf r},s)-\omega_{P}({\bf r})q({\bf
r},s)$ (2)

$u_{\pm}(\bf r)$ in Eq. 1e in the self energy of ions resulting from the
fluctuation of the electrostatic field [40, 41]. If the nonuniversal
contribution of the fluctuation in the length scale of the ion size is
retained, $u_{\pm}(\bf r)$ reduces to the local Born energy as:

$\displaystyle u_{\pm}({\bf r})=$ $\displaystyle\frac{z_{\pm}^{2}e^{2}}{8\pi
a_{\pm}\epsilon({\bf r})}$ (3)

with $a_{\pm}$ the Born radius of ions. The Born solvation energy accounts for
the electrostatic interaction between the ion and the local dielectric medium
[49, 52]. It captures the fact that ions are more preferable to be distributed
in the medium with higher dielectric constant. For systems with spatially
varying dielectric permittivity, $u_{\pm}$ is not a constant, and cannot be
adsorbed into the redefinition of the chemical potential. It will thus affect
both the ion distribution and protein density profile as indicated in Eqs. 1a
and 1e. The non-local contributions of electrostatic fluctuation, such as ion
correlation and image force, can be rigorously included into the self energy
through Gaussian variational approach [40, 41]. We refer interested readers to
the relevant literature for more details. The free energy of the subsystem is
then

$\displaystyle F_{m}$ $\displaystyle=-m{\rm ln}Q_{P}+{\rm ln}(m!)-{\rm
e}^{\mu_{S}}Q_{S}$ $\displaystyle+\frac{1}{v_{0}}\int{\rm d}{\bf
r}\left[\chi\rho_{P}(1-\rho_{P})-\omega_{P}\rho_{P}-\omega_{S}(1-\rho_{P})\right]$
$\displaystyle+\int{\rm d}{\bf
r}\left[\frac{\alpha}{v_{0}}\rho_{P}\psi-\frac{\epsilon}{2}(\nabla\psi)^{2}-c_{+}-c_{-}+c^{b}_{+}+c^{b}_{-}\right]$
(4)

### Phase Equilibrium

The protein solution in the dilute phase can be reconstructed by incorporating
the density profile and free energy of the $m$-aggregate obtained from SCFT
into the framework of dilute solution thermodynamics [39]. The free energy
density of the entire dilute solution with volume $V$, including the
translational entropy of aggregates, can be written as

$\displaystyle\frac{F_{D}}{V}=$
$\displaystyle\sum_{m=1}^{\infty}\\{C_{m}F_{m}+C_{m}[{\rm
ln}(C_{m}v_{m})-1]\\}$ (5)

where $C_{m}$ is the concentration of the $m$-aggregate. $v_{m}$ is a
reference volume which for simplicity can be taken as the volume of the
$m$-aggregate. $C_{m}v_{m}$ thus becomes the corresponding volume fraction
$\phi_{m}$ of the $m$-aggregate. In Eq. 5, the interaction between different
aggregates is ignored under the assumption of sufficiently dilute solution.
The equilibrium concentration of $m$-aggregate can be obtained by minimization
of the free energy density in Eq. 5 subject to fixed total protein
concentration $\sum^{\infty}_{m=1}mC_{m}$, which results in the following
distribution:

$\displaystyle\phi_{m}=$ $\displaystyle\phi^{m}_{1}{\rm exp}(-\Delta F_{m})$
(6)

Here $\Delta F_{m}=F_{m}-mF_{1}$ is the free energy of formation of the
$m$-aggregate from $m$ single isolated proteins.

The protein solution in the concentrated phase can be modeled as an infinitely
large aggregate with uniform protein density. The free energy density is
directly obtained by applying SCFT to a homogeneous system and Eq. 4 becomes:

$\displaystyle\frac{F_{C}}{V}$ $\displaystyle=\frac{\phi_{P}}{N}{\rm
ln}\left(\frac{\phi_{P}}{N}-1\right)+(1-\phi_{P})[{\rm ln}(1-\phi_{P})-1)]$
$\displaystyle+\chi\phi_{P}(1-\phi_{P})+\frac{\alpha}{v_{0}}\phi_{P}\psi-
c_{+}-c_{-}+c_{+}^{b}+c_{-}^{b}$ (7)

where $\psi$ is the electrostatic potential difference between the
concentrated phase and the dilute phase usually known as Donnan potential or
Galvani potential [41, 49]. $\psi$ is obtained by applying the charge
neutrality constrain to the homogeneous concentrated phase.

The equilibrium between the protein dilute phase and the protein concentrated
phase is determined by the respective equality of the chemical potential of
the protein and the solvent in the two coexisting phases, which results in

$\displaystyle F_{1}+{\rm ln}\phi_{1}$ $\displaystyle={\rm
ln}\frac{\phi_{C}}{N}-1+(1-N)(1-\phi_{C})+\chi N(1-\phi_{C})^{2}$
$\displaystyle+{\mu}^{elec}_{P}$ (8a)
$\displaystyle-\sum_{m=1}\frac{\phi_{m}}{mN}$
$\displaystyle=\left(1-\frac{1}{N}\right)\phi_{C}+{\rm ln}(1-\phi_{C})+\chi
N{\phi_{C}}^{2}+{\mu}^{elec}_{S}$ (8b)

where $\phi_{C}$ is the equilibrium volume fraction of protein in the
concentrated phase and $\phi_{m}$ is the equilibrium volume fraction of the
$m$-aggregate in the dilute phase given by Eq. 6. The total volume fraction of
protein in the dilute phase is thus $\phi_{D}=\sum_{m=1}\phi_{m}$. It should
be noted that the sum on the left-hand side of Eq. 8b is the dimensionless
osmotic pressure in dilute phase (in accordanace with Van’t Hoff law) as
expected for an ideal solution [53]. ${\mu}^{elec}_{P}$ and ${\mu}^{elec}_{S}$
are the electrostatic contributions in the chemical potentials of protein and
solvent, respectively, which are given by

$\displaystyle{\mu}^{elec}_{P}$ $\displaystyle=\alpha N\psi-
Nv_{0}[(c_{+}-c^{b}_{+})+(c_{-}-c^{b}_{-})]$
$\displaystyle+Nv_{0}(u_{+}c_{+}+u_{-}c_{-})\left(\frac{\epsilon_{S}-\epsilon_{P}}{\epsilon}\right)(1-\phi_{C})$
(9a) $\displaystyle{\mu}^{elec}_{S}$
$\displaystyle=-v_{0}[(c_{+}-c^{b}_{+})+(c_{-}-c^{b}_{-})]$
$\displaystyle+v_{0}(u_{+}c_{+}+u_{-}c_{-})\left(\frac{\epsilon_{S}-\epsilon_{P}}{\epsilon}\right)\phi_{C}$
(9b)

It is worth noting that the three terms on the right hand side of Eq. 9a
represent the contributions from the energy of a charged protein in the
electrostatic field, translational entropy and the solvation energy of salt
ions, respectively. For each salt concentration $c_{b}$ in the bulk salt
solution (i.e. reservoir), the equilibrium volume fractions in the coexisting
dilute and concentrated phases $\phi_{D}$ and $\phi_{C}$ are obtained by
solving Eqs. 8a and 8b simultaneously, from which the phase diagram as
illustrated in Fig. 1C can be obtained.

##  III Results

In the current work, we focus on the salt concentration effect and the
specific ion effect. The number of Kuhn segments in the protein is set as
$N=50$ with $b=1.0$nm. We use the simple system of a homogeneous chain with
uniform backbone charge distribution to illustrate the fundamental physical
chemistry. The backbone charge density $\alpha=+0.05$, where positive $\alpha$
is adopted to facilitate the comparison with the corresponding proteins
studied in experiments [26, 27, 28, 30]. The volume of the chain segment and
the solvent molecule is assumed to be the same as $v_{0}=1.0$nm3. The
temperature is set to be 298K with the Flory parameter $\chi=1.2$.

Figure 2: Protein solubility $\phi_{D}$ as a function of salt concentration
$c_{b}$ for anions with different radius $a_{-}$. $a_{+}=2.5\mathring{\rm A}$,
$z_{+}=z_{-}=1$, $\epsilon_{S}=80$. (A) $\epsilon_{P}=30$ and (B)
$\epsilon_{P}=10$.

### Salt Effects on the Protein Solubility

The salt effects on LLPS of protein observed in experiments show complicated
dependence on both the salt concentration and the chemical identity of ions.
We theoretically investigate the protein solubility for different salt
concentrations and various anion radii. Here, the solubility is represented by
$\phi_{D}$, the equilibrium volume fraction of the dilute phase on the
coexistence curve (see Fig. 1C). Figure 2A shows that the solubility decreases
as $c_{b}$ increases in the low salt concentration regime ($c_{b}<0.2$M),
indicating protein salting-out. At the same $c_{b}$, the solubility decreases
with the increase of anion radius, consistent with the trend of inverse
Hofmeister series. In contrast, in the high salt concentration regime
($c_{b}>0.2$M), protein remains salting-out for small anion
($a_{-}=2.0\mathring{\rm A}$), but turns to salting-in for larger ions. The
solubility increases with the increase of anion radius, indicating the direct
Hofmeister series. The dependence of LLPS on both the salt concentration and
the specific ions predicted by our theory is in good agreement with the
solubility measurements of lysozyme in Zhang and Cremer’s experiments [26].
Particularly, they found salting-out behavior at high salt concentrations only
for small Cl-, whereas all other larger anions show salting-in behavior,
exactly captured by Fig. 2A.

Figure 3: The salt effect on the electrostatic double layer structure around a
positively-charged protein in the low salt concentration regime. (A) anion
concentration profile $c_{-}(r)$ and (B) electrostatic potential $\psi(r)$.
$a_{+}=2.5\mathring{\rm A}$, $z_{+}=z_{-}=1$, $\epsilon_{P}=10$,
$\epsilon_{S}=80$, and $c_{b}=20$mM. (C) Schematics of different screening
effects for small anion and large anion.

The salt effects on the solubility in the high salt concentration regime also
depends on the property of protein. If a protein with lower dielectric
constant ($\epsilon_{P}=10$) is adopted as shown in Fig. 2B, it exhibits
salting-out in the entire salt concentration regime for all the anions with
$a_{-}\leq 3.5\mathring{\rm A}$. This is in stark contrast with the behavior
predicted for proteins with high $\epsilon_{P}$. It is interesting to note
that the same trend has also been reported in experiments. Cho et al. measured
the solubility of elastin-like polypeptide which has lower dielectric constant
than lysozyme [27]. All anions investigated in their work show salting-out at
high salt concentrations. Similar all salting-out behavior has been observed
by Zhang et al. in the synthetic poly(N-isopropylacrylamide) (PNIPAM)
system[31]. The dielectric constant of PNIPAM is less than 5 as reported in
the literature [54]. These experimental results are in good agreement with our
theoretical prediction.

As elucidated in Eq. 8, the LLPS of protein is determined by the interplay
between the hydrophobic attraction, Coulomb repulsion, as well as the
solvation energy and the translational entropy of ions. The solubility is
directly controlled by the effective two-body interaction between proteins:
attractive contributions to the interaction favor condensation, whereas
repulsive contributions prefer dissolution. The impacts of the aforementioned
four contributions on the two-body interaction and their salt-concentration
dependence are summarized in Table 1. The hydrophobicity of protein backbone
always leads to effective attraction and is independent of $c_{b}$, which thus
can be neglected when considering salt effects. Coulomb interaction between
likely-charged proteins is repulsive and decays exponentially with $c_{b}$ as
a result of ionic screening. Furthermore, the contribution of ion solvation is
effectively attractive. Ions prefer to be dissolved in the medium with higher
dielectric constant as indicated by the Born solvation model (Eq. 3). This
selective partition leads to depletion of ions from proteins and thus drives
phase separation. Lastly, the translational entropy of ions favors a uniform
distribution in the entire solution, which suppresses the aggregation of
proteins and thus provides an effective repulsion. As illustrated in Eq. 9,
the contributions of both the ionic solvation and translational entropy depend
linearly on $c_{b}$. In the following two subsections, we will provide more
detailed analysis on the salt effects in the low and high salt concentration
regimes, respectively.

Table 1: The impacts of different contributions and their salt-concentration dependence on the effective two-body interaction between proteins  Contribution | Effective interaction |  $\rm c_{b}$-dependence
---|---|---
hydrophobicity | attractive | $\sim c^{0}_{b}$
Coulomb interaction | repulsive |  $\sim e^{-\kappa r}/r$ ($\kappa\sim c^{1/2}_{b}$)
ion solvation | attractive | $\sim c_{b}$
entropy of ion | repulsive | $\sim c_{b}$

### Ionic Screening at Low Salt Concentrations

In the low salt concentration regime, the Coulomb repulsion between proteins
dominates compared to the contributions from ionic solvation and translational
entropy. Thus, the key factor that determines the salt effects on LLPS is how
the Coulomb repulsion are screened by salt ions. The screening effect gets
stronger as $c_{b}$ increases, which leads to the reduction of the effective
charge of protein and thus weakens the two-body repulsion. Therefore, the
solubility of protein decreases as $c_{b}$ increases, indicating a salting-out
behavior (see Fig. 2).

While the salting-out behavior is universal for all ions in the low salt
concentration regime, its degree exhibits specific ion effect because of the
different efficacy of anions in screening the Coulomb repulsion. Based on the
Born solvation model, ions are more preferable to be distributed in the
solvent region than the protein region as $\epsilon_{S}>\epsilon_{P}$ in most
cases. This selective partition becomes more pronounced for smaller anions.
Figure 3 shows the electrostatic double layer structure around a positively-
charged protein. Anions with smaller radius are repelled more from the protein
center, resulting in a less screened Coulomb potential. Therefore, protein
solubility decreases with the increase of the anion radius, in agreement with
the trend of inverse Hofmeister series observed in experiments at low salt
concentrations. Zhang and Cremer suggested that the specific ion effect on
LLPS in the low salt concentration regime is mainly originating from the
effectiveness of anions with different sizes in associating with the
positively-charged protein [26]. Their explanation is consistent with the
mechanism revealed in our results.

### Competition between Ion Solvation and Translational Entropy at High Salt
Concentrations

In the high salt concentration regime, the charges carried by proteins are
largely screened, and hence the Coulomb repulsion becomes less significant.
The LLPS of protein is mainly determined by the competition between the
solvation and translational entropy of ions as illustrated in Fig. 4. The
tendency for ions to be preferentially solvated by medium with higher
dielectric constant leads to a driving force for the separation of proteins
from the solvent phase. This reduces the solubility, i.e. salting out. On the
contrary, the translational entropy of ions favors a uniform distribution in
the entire system, which enhances the miscibility between protein and solvent,
i.e. salting in.

Figure 4: Schematics of salt effects on the LLPS of proteins in the high salt
concentration regime. (A) Ion solvation dominates for the case of small ions,
which favors salting-out. (B) Translational entropy of ion dominates for the
case of large ions, which favors salting-in.

Based on the electrostatic contributions to the chemical potential in Eq. 9,
the competition between ion solvation and translational entropy can be
quantified by:

$\displaystyle\Delta{\mu}^{elec}_{P}=$
$\displaystyle{\mu}^{elec}_{P}(D)-{\mu}^{elec}_{P}(C)$ $\displaystyle\approx$
$\displaystyle(z_{+}+z_{-})Nv_{0}c_{b}\left[\frac{l_{B,0}}{2}\left(\frac{\epsilon_{S}-\epsilon_{P}}{\epsilon^{2}_{S}}\right)\frac{1}{\bar{a}}-1\right]$
(10)

where $\bar{a}$ is the valency-weighted harmonic average radius of cation and
anion given by
$(z_{+}+z_{-})/\bar{a}=z^{2}_{+}z_{-}/a_{+}+z^{2}_{-}z_{+}/a_{-}$.
$l_{B,0}=e^{2}/(4\pi\epsilon_{0}kT)$ is the Bjerrum length in vacuum. The
detailed derivation of Eq. 10 is provided in SI, Section 2.
$\Delta{\mu}^{elec}_{P}$ represents the driving force for a single protein to
transfer from the concentrated phase (Phase C) to the dilute phase (Phase D).
$\Delta{\mu}^{elec}_{P}>0$, ion solvation dominates, and protein prefers to
stay in the concentrated phase rather than the dilute phase, which indicates
salting-out. $\Delta{\mu}^{elec}_{P}<0$, translational entropy dominates,
indicating salting-in. Equation 10 shows that the solvation effect becomes
less pronounced as $\bar{a}$ increases. This explains our numcerical results
in Fig. 2 and the experimental observations that protein salting-in occurs for
larger ions. This can also explain the specific ion effect that protein
solubility increases with the anion radius, consistent with the trend of
direct Hofmeister series observed in the high salt concentration regime [26,
27, 28, 30, 31, 32, 33]. Furthermore, the solvation energy depends on ion
valency as well. From the expression of $\bar{a}$, ions with higher valency
can be equivalently interpreted as monovalent ions with smaller effective
radius. Therefore, multivalent ions promote salting-out. It explains the
experimental findings in various protein and polymer solutions that
SO${}^{2-}_{4}$ shows much stronger tendency of salting-out even than Cl-,
although $a_{\rm{SO^{2-}_{4}}}$ is larger than $a_{\rm{Cl^{-}}}$ [27, 30, 31].

As indicated by Eq. 10, the solubility at high salt concentrations also
depends on the dielectric constant of protein $\epsilon_{P}$.
$\Delta{\mu}^{elec}_{P}$ decreases with the increase of $\epsilon_{P}$,
preferring salting-in. This is consistent with the experimental observation
that lysozyme with higher $\epsilon_{P}$ has stronger tendency of salting-in
than elastin-like polypeptide with lower $\epsilon_{P}$. Baldwin measured the
solubility of peptide and observed that salting-out becomes more pronounced as
the number of hydrocarbon side groups increases [55]. More hydrocarbon side
groups leads to the reduction of the dielectric constant of peptide.
Furthermore, Shimada et al. recently investigated the LLPS of ureido-
derivatized polymers [33]. They found that the solubility behavior turns from
salting-out to salting-in as more ureido groups are grafted the polymer. The
ureido group is highly polar and hence expected to increase the dielectric
constant of the polymer [56, 57]. Their experimental results can be well
captured by our theory.

Figure 5: Comparison of the solubility behavior predicted by our theory with
experimental results for various proteins and polymers in concentrated sodium
solutions with different anions. The dash diagonal line is the universal
criterion given by Eq. 11 for determining the boundary between the salting-in
and salting-out regimes. Scattering data points represent the experimental
results reported in literature, where open and filled symbols denote salting-
in and salting-out behaviors, respectively. The Born radii of Cl-,
NO${}^{-}_{3}$, Br-, I-, ClO${}^{-}_{4}$, SO${}^{2-}_{4}$ and Na+ are 1.91,
1.98, 2.05, 2.26, 2.83, 3.79 and 2.5 Å, respectively [59]. The dielectric
constant of water $\epsilon_{S}$=80. The dielectric constants of lysozyme
[60], elastin-like polypeptide (ELP)[61], PNIPAM[54] and
poly(allylamine)-copoly(allylurea) (PAU)[62] are 30, 10, 4.2, 56,
respectively.

Our theory provides a simple analytical criterion for determining the
solubility behavior, i.e. salting-in versus salting-out. From
$\Delta{\mu}^{elec}_{P}=0$ in Eq. 10, the boundary between the salting-in and
salting-out regimes is given by the following universal line:

$\displaystyle\frac{l_{B,0}}{2}\left(\frac{\epsilon_{S}-\epsilon_{P}}{\epsilon^{2}_{S}}\right)=$
$\displaystyle\bar{a}$ (11)

This analytical result in Eq. 11 is confirmed by the numerical calculations
(Fig. S1). To directly compare our theoretical predictions with experimental
measurements, Fig. 5 shows the solubility behaviors of two proteins (lysozyme
[26] and elastin-like polypeptide [27]) and two synthetic polymers (PNIPAM
[31] and poly(allylamine)-copoly(allylurea) [33]) in the solutions of sodium
salts with various anions. For a specific pair of protein and anion, salting-
in result observed in experiment is denoted by open symbol whereas salting-out
result is denoted by filled symbol. These two types of data points locate
almost exactly within the corresponding regimes separated by the universal
line predicted by Eq. 11. The protein solubility increases following
SO${}^{2-}_{4}$, Cl-, NO${}^{-}_{3}$, Br-, I- and ClO${}^{-}_{4}$, precisely
the direct Hofmeister series [20, 21, 58]. It is clear that our theoretical
result is quite universal, which captures the known salt concentration effect
and specific ion effect on LLPS of different proteins and polymers.

It should be noted that our prediction is in quantitative agreement with
experimental results without any fitting parameters. Only intrinsic parameters
of ions such as the valency and radius as well as the dielectric constant of
protein are needed, which can either be adopted from literature or measured in
experiments. It is also interesting to note that our theory captures the salt
effects on LLPS by only considering the contribution of Born energy in the ion
solvation, indicating its dominant role for simple ions like halogen anions.
However, for ions with more complex constitutions and structures, other
contributions such as hydration, dispersion and polarization should also be
taken into account [59, 58, 52, 35]. However, the existence and relative
importance of these higher order effects on LLPS can only be evaluated when
the essential Born solvation energy and translational entropy of ions are
systematically treated as in our theory.

### Protein Solubility vs. Charge Inversion

The preferable distribution of ions in the high dielectric medium also leads
to counterintuitive surface properties of protein. Salis et al. observed that
the electrophoritic mobility of a single lysozyme changes sign as salt
concentration increases, indicating an inversion of the effective surface
charge of the protein [63]. Bostr$\ddot{\rm o}$m et al., suggested that charge
inversion can explain the specific ion effect on protein solubility: reversal
of the Hofmester series from the inverse sequence at low salt concentrations
to the direct sequence at high salt concentrations [37]. Similar charge
inversion can also be predicted by our theory. Figure 6A shows the
electrostatic potential profiles $\psi(r)$ of a single protein in the presence
of various anions. As anion radius increases, $\psi(r)$ turns from its
original positive value to negative, which indicates the occurrence of charge
inversion for larger anions. This is in agreement with previous experimental
results and theoretical predictions. The charge inversion can be explained by
the induced Galvani potential and local charge separation at protein surface
due to different Born solvation energy between cation and anion [49]. The
induced negative charge becomes stronger with the increase of $a_{-}$, which
can even overcompensate the original positive charge of protein as the net
charge density shown in the inset of Fig. 6A. It is worth noting that charge
inversion is usually observed in the presence of multivalent counterions as a
result of strong ionic correlation. Here, it occurs in monovalent salt
systems, which is driven by the unequal solvation energy between cation and
anion.

Figure 6: The relation between charge inversion and solubility. (A)
Electrostatic potentials $\psi(r)$ profile of a single protein in the presence
of various anions. $a_{+}=2.5\mathring{\rm A}$, $z_{+}=z_{-}=1$,
$\epsilon_{P}=30$, and $c_{b}=0.7$M. The inset shows the profiles of the net
charge density $c_{\rm
net}(r)=z_{+}c_{+}(r)-z_{-}c_{-}(r)+\alpha{\rho}_{P}(r)/v_{0}$. (B)
Electrostatic potentials at the protein surface $\psi_{S}$ as a function of
salt concentration $c_{b}$. Filled symbols in Fig. 6B locate the corresponding
turning points from salting-out to salting-in shown in Fig. 2A.

We further compare the charge inversion for a single protein and the
collective solubility behavior for the entire protein solution to elucidate
their relation. Figure 6B plots the electrostatic potential at protein surface
$\psi_{S}$ as a function of salt concentration. Charge inversion can only been
observed for larger anion with $a_{-}=3.5\text{\AA}$. By comparing the results
of Fig. 2A and Fig. 6B, it can been clearly seen that the turning point from
salting-out to salting in for $a_{-}=3.5\text{\AA}$ appears at $c_{b}=0.16$M,
which is different from the charge inversion point $c_{b}=0.55$M. The
inconsistency is more obvious for smaller ions ($a_{-}=2.5\text{\AA}$ and
$a_{-}=3.0\text{\AA}$), where the turning points of solubility can be observed
in Fig. 2A but the charge inversion is absent within the given range of salt
concentration. Similar inconsistency can also been found from the comparison
between the electrophoritic mobility of a single protein and the solubility of
protein solutions measured by experiments [26, 63]. For the same lysozyme
protein, the charge inversion occurs for all anions, whereas on the contrary,
the turning of the solubility from salting-out to salting-has not been
observed for Cl-. Furthermore, charge inversion is also inconsistent with the
reversal of Hofmeister series. Figure 2A shows that the reversal of Hofmeister
series from the inverse sequence to the direct sequence occurs around
$c_{b}=0.25$M, largely different from the charge inversion point shown in Fig.
6B. Therefore, our theoretical results suggest that the counterintuitive
behaviors of turning solubility and reversal of Hofmeister series observed at
high salt concentrations are not related to charge inversion. They are
actually attributed to the competition between the ion solvation and
translational entropy as elucidated in the above subsection.

##  IV Conclusions and Discussion

We develop a self-consistent theory to study salt effect on LLPS of protein
solutions by systematically incorporating electrostatic interaction,
hydrophobicity, ion solvation and transnational entropy into a unified
framework. Our theory has made important improvements compared to previous
mean-field work. Both the highly localized density fluctuation of proteins in
the dilute phase and the electrostatic fluctuation (manifested by the self-
energy of ions) are explicitly accounted for. The long-standing puzzles of the
non-monotonic salt concentration dependence and the specific ion effect are
fully captured by our theory. We find that proteins show salting-out at low
salt concentrations due to ionic screening. The solubility decreases with the
increase of anion radius, following the inverse Hofmeister series. On the
other hand, in the high salt concentration regime, protein remains salting-out
for small ions but turns to salting-in for larger ions. The Hofmeister series
is reversed to the direct sequence. We reveal that both the turning of
solubility from salting-out to salting-in and the reversal of the Hofmeister
series are attributed to the competition between the solvation energy and
translational entropy of ions, but are not related to charge inversion of a
single protein. Furthermore, we derive an analytical criterion for determining
the boundary between the salting-in and salting-out regimes. Without any
fitting parameters, the theoretical prediction is in quantitative agreement
with experimental results for various proteins and polymers in sodium
solutions with a broad range of anions.

Our theory reveals the essential physical chemistry of salt effects on LLPS
using a simple charged macromolecular model, which can also be applied to
other soft matter systems. The theory can be generalized to macromolecules
with more complicated structures (e.g., chain architecture, heterogenous
composition and charge distribution, local rigidity, helicity, etc.) and
interactions that better represent real proteins. Although the charged
macromolecular model seems only applicable to unfolded or intrinsic disordered
proteins, the salt concentration effect and specific ion effect elucidated
here is universal for both the unfolded and folded proteins. This is because
the description of a giant liquid-like condensate is not sensitive to the
folding details of a single protein. Furthermore, our theory captures the salt
effects on LLPS by only considering the contribution of Born energy in the ion
solvation, indicating its dominant role for simple ions like halogen anions.
However, other contributions such as hydration, dispersion and polarization
should also be taken into account for ions with more complex structures. These
effects can be straightforwardly incorporated into the current theoretical
framework. The existence and relative importance of these higher order effects
on LLPS can only be evaluated when the essential Born energy and translational
entropy of ions are accurately treated as in our work. The fundamental insight
revealed here provides important guidance for modulating the LLPS of proteins
via the addition of salt as an effective tool, which helps understand the
functions of cellular organization and rationally design therapy for diseases.

##  V Materials and Methods

We use spherical coordinate in the numerical calculations of self-consistent
field theory based on the symmetry of spherical aggregate. Both the protein
density and electrostatic potential are set to be zero at the boundary of the
spherical simulation box. The Crank–Nicolson method is used to solve the
modified diffusion equation of chain propagator (Eq. (2)) [64]. The number of
points that the chain contour has been discretized is set to be $N_{s}=1000$.
The grid lattices are set such that the lattice spacings are smaller than
0.1$b$. $\mu_{S}$ is set to be $-1$, such that the free energy of the
reservoir of pure salt solution outside of the subvolume is 0. The equilibrium
structure and the free energy can be obtained by solving Eqs. (1)-(3)
iteratively until convergence. To accelerate the convergence, we use the
following strategy to update the fields. Fields conjugate to the density of
protein and solvent molecules are updated by a simple mixing rule, i. e.,
$\omega^{new}_{P,S}\leftarrow\lambda\omega^{new}_{P,S}+(1-\lambda)\omega^{old}_{P,S}$.
The same rule is adopted for updating electrostatic potential $\psi$ and Born
energy $u_{\pm}$. The field conjugated to the incompressibility condition is
updated by $\eta^{new}\leftarrow\eta^{old}+\kappa(\rho_{P}+\rho_{S}-1)$, where
the second term on the r.h.s is adopted to reinforce the incompressibility.
$\lambda$=0.01 and $\kappa$=2.0 are chosen in our calculations. The relative
errors for the free energy and the incompressibility condition are set to be
below $10^{-11}$ and $10^{-7}$, respectively.

##  VI Acknowledgment

Acknowledgment is made to the donors of the American Chemical Society
Petroleum Research Fund for partial support of this research. This research
used the computational resources provided by the Kenneth S. Pitzer Center for
Theoretical Chemistry.

##  VII References

## References

  * [1] S. Alberti, A. A. Hyman, Biomolecular condensates at the nexus of cellular stress, protein aggregation disease and ageing. Nat. Rev. Mol. Cell Biol. 22, 196-213 (2021).
  * [2] A. S. Lyon, W. B. Peeples, M. K. Rosen, A framework for understanding the functions of biomolecular condensates across scales. Nat. Rev. Mol. Cell Biol. 22, 215-235 (2021).
  * [3] D. L. J. Lafontaine, J. A. Riback, R. Bascetin, C. P. Brangwynne, The nucleolus as a multiphase liquid condensate. Nat. Rev. Mol. Cell Biol. 22, 165-182 (2021).
  * [4] C. P. Brangwynne et al., Germline P granules are liquid droplets that localize by controlled dissolution/condensation. Science 324, 1729-1732 (2009).
  * [5] A. R. Strom, et al., Phase separation drives heterochromatin domain formation. Nature 547, 241-245 (2017).
  * [6] C. T. Pawson, J. D. Scott, Signal integration through blending, bolstering and bifurcating of intracellular information. Nat. Struct. Mol. Biol. 17, 653–658 (2010).
  * [7] P. A. Chong, J. D. Forman-Kay, Liquid–liquid phase separation in cellular signaling systems. Curr. Opin. Struct. Biol. 41, 180–186 (2016).
  * [8] A. C. Woerner et al., Cytoplasmic protein aggregates interfere with nucleocytoplasmic transport of protein and RNA.Science 351, 173-176 (2016).
  * [9] T. P. J. Knowles, M. Vendruscolo, C. M. Dobson, The amyloid state and its association with protein misfolding diseases. Nat. Rev. Mol. Cell Biol. 15, 384-396 (2014).
  * [10] C. Chiti, C. M. Dobson, Protein misfolding, amyloid formation, and human disease: A summary of progress over the last decade. Annu. Rev. Biochem. 86, 27-68 (2017).
  * [11] S. Ray et al., $\alpha$-Synuclein aggregation nucleates through liquid–liquid phase separation. Nat. Chem. 12, 705-716 (2020).
  * [12] J. A. Riback et al., Composition-dependent thermodynamics of intracellular phase separation. Nature 581, 209–214 (2020).
  * [13] M. Fuxreiter, M. Vendruscolo, Generic nature of the condensed states of proteins. Nat. Cell Biol. 23, 587-594 (2021).
  * [14] S. F. Shimobayashi, P. Ronceray, D. W. Sanders, M. P. Haataja, C. P. Brangwynne, Nucleation landscape of biomolecular condensates. Nature 599, 503–506 (2021).
  * [15] C. P. Brangwynne, P. Tompa, R. V. Pappu, Polymer physics of intracellular phase transitions. Nat. Phys. 11, 899-904 (2015).
  * [16] Y. Shin, C. P. Brangwynne, Liquid phase condensation in cell physiology and disease. Science 357, eaaf4382 (2017).
  * [17] J.-M. Choi, A. S. Holehouse, R. V. Pappu, Physical principles underlying the complex biology of intracellular phase transitions. Annu. Rev. Biophys. 49, 107-133 (2020).
  * [18] S. Alberti, A. Gladfelter, T. Mittag, Considerations and challenges in studying liquid-liquid phase separation and biomolecular condensates. Cell 176, 419-434 (2019).
  * [19] A. E. Posey, A. S. Holehouse, R. V. Pappu, Methods in enzymology (ed. Rhoades, E.) Ch. 1, Vol. 611, 1-30 (Academic Press, 2018).
  * [20] Y. J. Zhang, P. S. Cremer, Chemistry of Hofmeister anions and osmolytes. Annu. Rev. Phys. Chem. 61, 63-83 (2010).
  * [21] H. I. Okur et al., Beyond the Hofmeister series: Ion-specific effects on proteins and their biological functions. J. Phys. Chem. B 105, 1997-2014 (2017).
  * [22] M.-T. Wei et al., Phase behaviour of disordered proteins underlying low density and high permeability of liquid organelles. Nat. Chem. 9, 1118-1125 (2017).
  * [23] G. Krainer et al., Reentrant liquid condensate phase of proteins is stabilized by hydrophobic and non-ionic interactions. Nat. Commun. 12, 1085 (2021).
  * [24] B. A. Rogers et al., Weakly hydrated anions bind to polymers but not monomers in aqueous solutions. Nat. Chem. 14, 40-45 (2022).
  * [25] D. Ben-Yaakov, D. Andelman, D. Harries, R. Podgornik, Beyond standard Poisson–Boltzmann theory: Ion-specific interactions in aqueous solutions. J. Phys.: Condens. Matter 21, 424106 (2009).
  * [26] Y. J. Zhang, P. S. Cremer, The inverse and direct Hofmeister series for lysozyme. Proc. Natl. Acad. Sci. U.S.A. 105,15249-15253 (2009).
  * [27] Y. Cho et al., Effects of Hofmeister anions on the phase transition temperature of elastin-like polypeptides. J. Phys. Chem. B 112, 13765-13771 (2008).
  * [28] B. D. Mason, J. Zhang-van Enk, L. Zhang, R. L. Remmele, J. F. Zhang, Liquid-liquid phase separation of a monoclonal antibody and nonmonotonic influence of Hofmeister anions. Biophys. J. 99, 3792–3800 (2010).
  * [29] A. Patel et al., ATP as a biological hydrotrope. Science 356, 753-756 (2017).
  * [30] J. B. Otis, S. Sharpe, Sequence context and complex Hofmeister salt interactions dictate phase separation propensity of resilin-like polypeptides. Biomacromolecules 23, 5225-5238 (2022).
  * [31] Y. J. Zhang, S. Furyk, D. E. Bergbreiter, P. S. Cremer, Specific ion effects on the water solubility of macromolecules: PNIPAM and the Hofmeister series. J. Am. Chem. Soc. 127, 14505-14510 (2005).
  * [32] K. Weckstr$\ddot{\rm o}$m, A. C. Papageorgiou, Lower consolute boundaries of the nonionic surfactant C8E5 in aqueous alkali halide solutions: An approach to reproduce the effects of alkali halides on the cloud-point temperature. J. Colloid Interface Sci. 310, 151-162 (2007).
  * [33] N. Shimada, H. Ino, K. Maie, M. Nakayama, K. Kano, A. Maruyama, Ureido-derivatized polymers based on both poly(allylurea) and poly(L-citrulline) exhibit UCST-type phase transition behavior under physiologically relevant conditions. Biomacromolecules 12, 3418-3422 (2011).
  * [34] A. Stradner et al., Equilibrium cluster formation in concentrated protein solutions and colloids. Nature 432, 492-495 (2004).
  * [35] A. Salis, B. W. Ninham, Models and mechanisms of Hofmeister effects in electrolyte solutions, and colloid and protein systems revisited. Chem. Soc. Rev. 43, 7358-7377 (2014).
  * [36] M. Kastelica, Y. V. Kalyuzhnyi, B. Hribar-Lee, K. A. Dill, V. Vlachy, Protein aggregation in salt solutions. Proc. Natl. Acad. Sci. U.S.A. 112, 6766-6770 (2015).
  * [37] M. Bostr$\ddot{\rm o}$m, D. F. Parsons, A. Salis, B. W. Ninham, M. Monduzzi, Possible origin of the inverse and direct hofmeister series for lysozyme at low and high salt concentrations. Langmuir 27, 9504-9511 (2011).
  * [38] R. Wang, Z.-G. Wang, Theory of polymers in poor solvent: Phase equilibrium and nucleation behavior. Macromolecules 45, 6266-6271 (2012).
  * [39] C. Duan, R. Wang, Electrostatics-induced nucleated conformational transition of protein aggregation. Phys. Rev. Lett. 130, 158401 (2023).
  * [40] Z.-G. Wang, Fluctuation in electrolyte solutions: The self energy. Phys. Rev. E 81, 021501 (2010).
  * [41] N. R. Agrawal, R. Wang, Self-consistent description of vapor-liquid interface in ionic fluids. Phys. Rev. Lett. 129, 228001 (2022).
  * [42] E. W. Martin et al., Valence and patterning of aromatic residues determine the phase behavior of prion-like domains. Science 367, 694-699 (2020).
  * [43] B. Gouveia et al., Capillary forces generated by biomolecular condensates. Nature 609, 255–264 (2022).
  * [44] C. Duan, W. H. Li, R. Wang, Conformation of a single polyelectrolyte in poor solvents. J. Chem. Phys. 153, 064901 (2020).
  * [45] R. Mezzenga, P. Fischer, The self-assembly, aggregation and phase transitions of food protein systems in one, two and three dimensions. Rep. Prog. Phys. 76, 046601 (2013).
  * [46] R. van der Lee et al., Classification of intrinsically disordered regions and proteins. Chem. Rev. 114, 6589-6631 (2014).
  * [47] M. Muthukumar, 50th anniversary perspective: A perspective on polyelectrolyte solutions. Macromolecules 50, 9528-9560 (2017).
  * [48] G. H. Fredrickson, The equilibrium theory of inhomogeneous polymers (Oxford University Press, New York, 2006).
  * [49] R. Wang, Z.-G. Wang, Effects of ion solvation on phase equilibrium and interfacial tension of liquid mixtures. J. Chem. Phys. 135, 014707 (2011).
  * [50] C. E. Sing, J. W. Zwanikken, M. O. de la Cruz, Electrostatic control of block copolymer morphology. Nat. Mater. 13, 694–698, (2012).
  * [51] B. Zhuang, G. Ramanauskaite, Z. Y. Koa, Z.-G. Wang, Like dissolves like: A first-principles theory for predicting liquid miscibility and mixture dielectric constant. Sci. Adv. 7, eabe7275 (2021).
  * [52] R. Wang, Z.-G. Wang, Continuous self-energy of ions at the dielectric interface. Phys. Rev. Lett. 112, 136101 (2014).
  * [53] M. Doi, Soft matter physics (Oxford University Press, 2013).
  * [54] C. Rullyani et al., Stimuli-responsive polymer as gate dielectric for organic transistor sensors. Org. Electron. 85, 105818 (2020).
  * [55] R. L. Baldwin, How Hofmeister ion interactions affect protein stability. Biophys. J. 71, 2056-2063 (1996).
  * [56] J. Wyman, Dielectric constants: Ethanol-diethyl ether and urea-water solutions between 0 and 50°. J. Am. Chem. Soc. 55, 4116-4121 (1933).
  * [57] I. M. Klotz, Equilibrium constants and free energies in unfolding of proteins in urea solutions. Proc. Natl. Acad. Sci. U.S.A. 93, 14411–14415 (1996).
  * [58] P. L. Nostro, B. W. Ninham, Hofmeister Phenomena: An update on ion specificity in biology. Chem. Rev. 112, 2286-2322 (2012).
  * [59] A. P. dos Santos, A. Diehl, Y. Levin, Surface tensions, surface potentials, and the Hofmeister series of electrolyte solutions. Langmuir 26, 10778-10783 (2010).
  * [60] P. E. Smith, R. M.Brame, A. E. Mark, W. F. van Gunsteren, Dielectric properties of trypsin inhibitor and lysozyme calculated from molecular dynamics simulations. J. Phys. Chem. 97, 2009-2014 (1993).
  * [61] M. Baer, E. Schreiner, A. Kohlmeyer, R. Rousseau, D. Marx, Structure-property relationships of elastin-like polypeptides: A review of experimental and computational studies. J. Phys. Chem. B 110, 3576-3587 (2006).
  * [62] Dielectric constant of poly(allylamine)-copoly(allylurea) (PAU) is determined based on the linear mixing rule [49, 50, 51] of its major constituting components, hydrocarbon group ($\epsilon_{r}\approx 2$) and ureido group ($\epsilon_{r}\approx 100$). The latter is estimated from the reported data of water-urea mixture at 298K [56].
  * [63] A. Salis, F. Cugia, D. F. Parsons, B. W. Ninham, M. Monduzzi, Hofmeister series reversal for lysozyme by change in pH and salt concentration: Insights from electrophoretic mobility measurements. Phys. Chem. Chem. Phys. 14, 4343-4346 (2012).
  * [64] J. D. Hoffman, Numerical methods for engineers and scientists (Marcel Dekker, Inc., New York, 2001).


# End-to-End Video Instance Segmentation with Transformers

Yuqing Wang1 Zhaoliang Xu1 Xinlong Wang2 Chunhua Shen2 Baoshan Cheng1 Hao
Shen1 Corresponding author. Huaxia Xia1
1 Meituan 2 The University of Adelaide  Australia
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Video instance segmentation (VIS) is the task that requires simultaneously
classifying, segmenting and tracking object instances of interest in video.
Recent methods typically develop sophisticated pipelines to tackle this task.
Here, we propose a new video instance segmentation framework built upon
Transformers, termed VisTR, which views the VIS task as a direct end-to-end
parallel sequence decoding/prediction problem. Given a video clip consisting
of multiple image frames as input, VisTR outputs the sequence of masks for
each instance in the video in order directly. At the core is a new, effective
instance sequence matching and segmentation strategy, which supervises and
segments instances at the sequence level as a whole. VisTR frames the instance
segmentation and tracking in the same perspective of similarity learning, thus
considerably simplifying the overall pipeline and is significantly different
from existing approaches.

Without bells and whistles, VisTR achieves the highest speed among all
existing VIS models, and achieves the best result among methods using single
model on the YouTube-VIS dataset. For the first time, we demonstrate a much
simpler and faster video instance segmentation framework built upon
Transformers, achieving competitive accuracy. We hope that VisTR can motivate
future research for more video understanding tasks.

Code is available at: https://git.io/VisTR

##  1 Introduction

Instance segmentation is one of the fundamental tasks in computer vision.
While significant progress has been witnessed in instance segmentation of
images [9, 22, 25, 26, 5, 27], much less effort was spent on segmenting
instances in videos. Here we propose a new video instance segmentation
framework built upon Transformers. Video instance segmentation (VIS), recently
proposed in [30], requires one to simultaneously classify, segment and track
object instances of interest in a video sequence. It is more challenging in
that one needs to perform instance segmentation for each individual frame and
at the same time to establish data association of instances across consecutive
frames, a.k.a., tracking.

State-of-the-art methods typically develop sophisticated pipelines to tackle
this task. Top-down approaches [30, 2] follow the tracking-by-detection
paradigm, relying heavily on image-level instance segmentation models [9, 6]
and complex human-designed rules to associate the instances. Bottom-up
approaches [1] separate object instances by clustering learned pixel
embeddings. Due to heavy reliance on the dense prediction quality, these
methods often need multiple steps to generate the masks iteratively, which
makes them slow. Thus, a simple, end-to-end trainable VIS framework is highly
desirable.

Here, we take a deeper look at the video instance segmentation task. Video
frames contain richer information than single images such as motion patterns
and temporal consistency of instances, offering useful cues for instance
segmentation, and classification. At the same time, the better learned
instance features can help tracking of instances. In essence, the instance
segmentation and instance tracking are both concerned with similarity
learning: instance segmentation is to learn the pixel-level similarity and
instance tracking is to learn the similarity between instances. Thus, it is
natural to solve these two sub-tasks in a single framework and benefit each
other. Here we aim to develop such an end-to-end VIS framework. The framework
needs to be simple and achieves strong performance without whistles and bells.
To this end, we propose to employ the Transformers [23]. Importantly, for the
first time we demonstrate that, as the Transformers provide building blocks,
it enables one to design a simple and clean framework for VIS, and possibly
for a much wider range of video processing tasks in computer vision. Thus
potentially, it is possible to unify most vision tasks of different input
modalities—such as image, video and point clouds processing—into the
Transformer framework. Transformers are widely used for sequence to sequence
learning in NLP [23], and start to show promises in vision [4, 8].
Transformers are capable of modeling long-range dependencies, and thus can be
naturally applied to video for learning temporal information across multiple
frames. In particular, the core mechanism of Transformers, self-attention, is
designed to learn and update the features based on all pairwise similarities
between them. The above characteristics of Transformers make them great
candidates for the VIS task.

In this paper, we propose the Video Instance Segmentation TRansformer (VisTR),
which views the VIS task as a parallel sequence decoding/prediction problem.
Given a video clip that consists of multiple image frames as input, the VisTR
outputs the sequence of masks for each instance in the video in order
directly. The output sequence for each instance is referred to as instance
sequence in this paper. The overall VisTR pipeline is illustrated in fig:1. In
the first stage, given a sequence of video frames, a standard CNN module
extracts features of individual image frames, then the multiple image features
are concatenated in the frame order to form the clip-level feature sequence.
In the second stage, the Transformer takes the clip-level feature sequence as
input, and outputs a sequence of object predictions in order. In fig:1 same
shapes represent predictions for the same image, and the same colors represent
the same instance of different images. The sequence of predictions follow the
order of input images, and the predictions of each image follows the same
instance order. Thus, instance tracking is achieved seamlessly and naturally
in the same framework of instance segmentation.

To achieve this goal, there are two main challenges: 1) how to maintain the
order of outputs and 2) how to obtain the mask sequence for each instance out
of the Transformer network. Correspondingly, we introduce the instance
sequence matching strategy and the instance sequence segmentation module. The
instance sequence matching performs bipartite graph matching between the
output instance sequence and the ground-truth instance sequence, and
supervises the sequence as a whole. Thus, the order can be maintained
directly. The instance sequence segmentation accumulates the mask features for
each instance across multiple frames through self-attention and segments the
mask sequence for each instance through 3D convolutions.

Our main contributions are summarized as follows.

  * •

We propose a new video instance segmentation framework built upon
Transformers, termed VisTR, which views the VIS task as a direct end-to-end
parallel sequence decoding/prediction problem. The framework is significantly
different from existing approaches, considerably simplifying the overall
pipeline.

  * •

VisTR solves the VIS from a new perspective of similarity learning. Instance
segmentation is to learn the pixel-level similarity and instance tracking is
to learn the similarity between instances. Thus, instance tracking is achieved
seamlessly and naturally in the same framework of instance segmentation.

  * •

The key to the success of VisTR is a new strategy for instance sequence
matching and segmentation, which is tailored for our framework. This
carefully-designed strategy enables us to supervise and segment instances at
the sequence level as a whole.

  * •

VisTR achieves strong results on the YouTube-VIS dataset, achieving 40.1% in
mask mAP at the speed of 57.7 FPS , which is the best and fastest among
methods that use a single model.

##  2 Related work

Video object segmentation. VOS [18] is closely related to VIS. Analogue to
object tracking, which is detecting boxes of foreground objects in a class-
agnostic fashion, VOS is segmenting masks of foreground class-agnostic
objects. Same as in tracking, usually one is allowed to use only the first few
frames’ annotations for training. In contrast, VIS requires to segment and
track all instance masks of a fixed category set of objects in a video
sequence.

Video instance segmentation. The VIS task [30] requires classifying,
segmenting instances in each frame and linking the same instance across
frames. State-of-the-art methods typically develop sophisticated pipelines to
tackle it. MaskTrack R-CNN[30] extends the Mask R-CNN [9] with a tracking
branch and external memory that saves the features of instances across
multiple frames. Maskprop [2] builds on the Hybrid Task Cascade Network [6],
and re-uses the predicted masks to crop the extracted features, then
propagates them temporally to improve the segmentation and tracking. STEm-Seg
[1] proposes to model video clips as 3D space-time volumes and then separates
object instances by clustering learned embeddings. Note that the above
approaches either rely on complex heuristic rules to associate the instances
or require multiple steps to generate and optimize the masks iteratively. In
contrast, here we aim to build a simple and end-to-end trainable VIS
framework.

Transformers. Transformers were first proposed in [23] for the sequence-to-
sequence machine translation task, and since then have become the de facto
method in most NLP tasks. The core mechanism of Transformers, self-attention,
makes it particularly suitable for modeling long-range dependencies. Very
recently, Transformers start to show promises in solving computer vision
tasks. DETR [4] builds an object detection systems based on Transformers,
which largely simplifies the traditional detection pipeline, and achieves on
par performances compared with highly-optimized CNN based detectors[19]. Our
work here is inspired by DETR. ViT [8] introduces the Transformer to image
recognition and models an image as a sequence of patches, which attains
excellent results compared to state-of-the-art convolutional networks. The
above works show the effectiveness of Transformers in image understanding
tasks. To our knowledge, thus far there are no prior applications of
Transformers to video instance segmentation. It is intuitive to see that the
Transformers’ advantage of modeling long-range dependencies makes it an ideal
candidate for learning temporal information across multiple frames for video
understanding tasks. Here, we propose the VisTR method and provide an
affirmative answer to that. As the original Transformers are auto-regressive
models, which generate output tokens one by one, for efficiency, VisTR employs
a non-auto-regressive variant of the Transformer to achieve parallel sequence
generation.

##  3 Our Method: VisTR

Figure 1: The overall architecture of VisTR. It contains four main components:
1) a CNN backbone that extracts feature representation of multiple images; 2)
an encoder-decoder Transformer that models the relations of pixel-level
features and decodes the instance-level features; 3) an instance sequence
matching module that supervises the model; and 4) an instance sequence
segmentation module that outputs the final mask sequences (Best viewed on
screen).

We tackle the video instance segmentation task by modeling it as a direct
sequence prediction problem. Given a video clip that consists of multiple
image frames as input, the VisTR outputs the sequence of masks for each
instance in the video in order. To achieve this goal, we introduce the
instance sequence matching and segmentation strategy to supervise and segment
the instances at the sequence level as a whole. In this section, we first
introduce the overall architecture of the proposed VisTR in sec:3.1, then the
details of the instance sequence matching and segmentation module will be
described in sec:3.2 and sec:3.3 respectively.

###  3.1 VisTR Architecture

The overall VisTR architecture is depicted in fig:2. It contains four main
components: a CNN backbone to extract compact feature representations of
multiple frames, an encoder-decoder Transformer to model the similarity of
pixel-level and instance-level features, an instance sequence matching module
for supervising the model, and an instance sequence segmentation module.

Backbone. The backbone extracts the original pixel-level feature sequence of
the input video clip. Assume that the initial video clip with $T$ frames of
resolution $H_{0}\times W_{0}$ is denoted by $x_{clip}\in\mathbb{R}^{T\times
3\times H_{0}\times W_{0}}$. First, a standard CNN backbone generates a lower-
resolution activation map for each frame, then the features for each frame are
concatenated to form the clip level feature map $f_{0}\in\mathbb{R}^{T\times
C\times H\times W}$.

Transformer encoder. The Transformer encoder is employed to model the
similarities among all the pixel level features in the clip. First, a
1$\times$1 convolution is applied to the above feature map, reducing the
dimension from $C$ to $d$ ($d<C$), resulting in a new feature map
$f_{1}\in\mathbb{R}^{T\times d\times H\times W}$. To form a clip level feature
sequence that can be fed into the Transformer encoder, we flatten the spatial
and temporal dimensions of $f_{1}$ into one dimension, resulting in a 2D
feature map of size $d\times(T\cdot H\cdot W)$. Note that the temporal order
is always in accordance with that of the initial input. Each encoder layer has
a standard architecture that consists of a multi-head self-attention module
and a fully connected feed forward network (FFN).

Temporal and spatial positional encoding. The Transformer architecture is
permutation-invariant, while the segmentation task requires precise position
information. To compensate for this, we supplement the features with fixed
positional encodings information that contains the three dimensional
(temporal, horizontal and vertical) positional information in the clip. Here
we adapt the positional encoding in the original Transformer [23] for our 3D
case. Specifically, for the coordinates of each dimension we independently use
$\nicefrac{{d}}{{3}}$ sine and cosine functions with different frequencies:

${\rm PE}({\rm pos},i)=\begin{cases}\sin\bigl{(}{\rm
pos}\cdot\omega_{k}\bigr{)},~{}~{}~{}~{}\text{for }i=2k,\\\ \cos\bigl{(}{\rm
pos}\cdot\omega_{k}\bigr{)},~{}~{}~{}~{}\text{for }i=2k+1;\end{cases}$ (1)

where $\omega_{k}=1/10000^{2k/\frac{d}{3}}$; ‘${\rm pos}$’ is the position in
the corresponding dimension. Note that the $d$ should be divisible by 3, as
the positional encodings of the three dimensions should be concatenated to
form the final $d$ channel positional encoding. These encodings are added to
the input of each attention layer.

Transformer decoder. The Transformer decoder aims to decode the top pixel
features that can represent the instances of each frame, which is called
instance level features. Motivated by DETR [4], we also introduce a fixed
number of input embeddings to query the instance features from pixel features,
termed as instance queries. Suppose that the model decodes $n$ instances each
frame, then for $T$ frames the instance query number is $N=n\cdot T$. The
instance queries are learned by the model and have the same dimension with the
pixel features. Taking the output of encoder $E$ and $N$ instance queries $Q$
as input, the Transformer decoder outputs $N$ instance features, denoted by
$O$ in fig:2. The overall predictions follow the input frame order, and the
order of instance predictions for different images is the same. Thus, the
tracking of instances in different frames could be realized by linking the
items of the corresponding indices directly.

###  3.2 Instance Sequence Matching

VisTR infers a fixed-size sequence of $N$ predictions, in a single pass
through the decoder. One of the main challenges for this framework is to
maintain the relative position of predictions for the same instance in
different images, a.k.a., instance sequence. In order to find the
corresponding ground truth and supervise the instance sequence as a whole, we
introduce the instance sequence matching strategy.

As the VisTR decode $n$ instances each frame, the number of instance sequence
is also $n$. Let us denote by
$\hat{y}=\left\\{{\hat{y}_{i}}\right\\}_{i=1}^{n}$ the predicted instance
sequences, and $y$ the ground truth set of instance sequences. Assuming $n$ is
larger than the number of instances in the video clip, we consider $y$ also as
a set of size $n$ padded with $\emptyset$. In order to find a bipartite graph
matching between the two sets, we search for a permutation of $n$ elements
$\sigma\in S_{n}$ with the lowest cost:

$\hat{\sigma}=\underset{\sigma\in
S_{n}}{\arg\min}\sum_{i}^{n}\mathcal{L}_{\text{match}}\left(y_{i},\hat{y}_{\sigma(i)}\right)$
(2)

where
$\mathcal{L}_{\operatorname{match}}\left(y_{i},\hat{y}_{\sigma(i)}\right)$ is
a pair-wise matching cost between ground truth $y_{i}$ and an instance
sequence prediction with index $\sigma(i)$. The optimal assignment could be
computed efficiently by the Hungarian algorithm[11], following prior work
(e.g., [21]).

As computing the mask sequence similarity directly is computationally
intensive, we find a surrogate, the box sequence to perform the matching. To
obtain the box predictions, we apply a 3-layer feed forward network (FFN) with
ReLU activation function and a linear projection layer to the object
predictions $O$ of Transformer decoder. Following the same practice of DETR
[4], the FFN predicts the normalized center coordinates, height and width of
the box w.r.t. input image, and the linear layer predicts the class label
using a softmax function. We also add a “background” class to represent that
no object is detected.

Given the $N=n\cdot T$ bounding box predictions for the object predictions
sequence, we could associate $n$ box sequences for each instance by their
indices, referred to as ins1 box seq…ins4 box seq in fig:2. The matching loss
takes both the class predictions and the similarity of predicted and ground
truth boxes into account. Each element $i$ of the ground truth set can be seen
as

$y_{i}=\\{(c_{i},c_{i}...,c_{i}),(b_{i,0},b_{i,1}...,b_{i,T})\\}$ (3)

where $c_{i}$ is the target class label (which may be $\emptyset$) for this
instance, and $b_{i,t}\in[0,1]^{4}$ is a vector that defines ground truth box
center coordinates and its relative height and width in the frame $t$. $T$
represent the number of input frames. Thus, for the predictions of instance
with index $\sigma(i)$ we denoted the probability of class $c_{i}$ as

${\hat{p}}_{(\sigma(i))}(c_{i})=\\{{\hat{p}}_{(\sigma(i),0)}(c_{i})...,\hat{p}_{(\sigma(i),T)}(c_{i})\\}$
(4)

and the predicted box sequence as

$\hat{b}_{\sigma(i)}=\left\\{\hat{b}_{(\sigma(i),0)},\hat{b}_{(\sigma(i),1)}...,\hat{b}_{(\sigma(i),T)}\right\\}$
(5)

With the above notation, we define

$\mathcal{L}_{\text{match}}\left(y_{i},\hat{y}_{\sigma(i)}\right)=-\hat{p}_{\sigma(i)}\left(c_{i}\right)+\mathcal{L}_{\mathrm{box}}\bigl{(}b_{i},\hat{b}_{\sigma(i)}\bigr{)},$
(6)

where $c_{i}\neq\emptyset$. Based on the above criterion, we could find the
one-to-one matching of the sequences by the Hungarian algorithm. Given the
optimal assignment, we could compute the loss function, the Hungarian loss for
all pairs matched in the previous step. The loss is a linear combination of a
negative log-likelihood for class prediction, a box loss and mask loss for the
instance sequences:

$\displaystyle\mathcal{L}_{\text{Hung}}(y,\hat{y})$
$\displaystyle=\sum_{i=1}^{N}\Bigl{[}(-\log\hat{p}_{\hat{\sigma}(i)}(c_{i}))+\mathcal{L}_{\text{box
}}(b_{i},\hat{b}_{\hat{\sigma}}(i))$ $\displaystyle+\mathcal{L}_{\text{mask
}}(m_{i},\hat{m}_{\hat{\sigma}}(i))\Bigr{]}.$ (7)

Here $c_{i}\neq\emptyset$, and $\hat{\sigma}$ is the optimal assignment
computed in Eq. (2). The Hungarian loss is used to train the whole framework.

The second part of the matching cost and the Hungarian loss is
$\mathcal{L}_{\rm box}$ that scores the bounding boxes. We use a linear
combination of the sequence level $\mathcal{L}_{1}$ loss and the generalized
IOU loss [20]:

$\displaystyle\mathcal{L}_{\mathrm{box}}\bigl{(}b_{i},\hat{b}_{\sigma(i)}\bigr{)}$
$\displaystyle=\frac{1}{T}\sum_{t=1}^{T}\Bigl{[}\lambda_{\mathrm{iou}}\cdot\mathcal{L}_{\mathrm{iou}}\bigl{(}b_{i,t},\hat{b}_{\sigma(i),t}\bigr{)}$
$\displaystyle+\lambda_{\mathrm{L}1}\left\|b_{i,t}-\hat{b}_{\sigma(i),t}\right\|_{1}\Bigr{]}.$
(8)

Here $\lambda_{\mathrm{iou}},\lambda_{\mathrm{L}1}\in\mathbb{R}$ are hyper-
parameters. These two losses are normalized by the number of instances inside
the batch. In the sequel, we present the details.

###  3.3 Instance Sequence Segmentation

The instance sequence segmentation module aims to predict the mask sequence
for each instance. To realize that, the model needs to accumulate the mask
features of multiple frames for each instance firstly, then the mask sequence
segmentation is performed on the accumulated features.

The mask features are obtained by computing the similarity map between the
object predictions $O$ and the Transformer encoded features $E$. To simplify
the calculation, we only compute with the features of its corresponding frame
for each object prediction. For each frame, the object predictions $O$ and the
corresponding encoded feature maps $E$ are fed into the self-attention module
to obtain the initial attention maps. Then the attention maps will be fused
with the initial backbone features $B$ and the transformed encoded features
$E$ of the corresponding frames, following a similar practice with the
DETR[4]. The last layer of the fusion is a deformable convolution layer[7]. In
this way, the mask features for each instance of different frames are
obtained.

Following the same spirit of taking the instance sequence as a whole, the mask
features of the same instance in different frames should be propagated and
reinforce each other. We propose to utilize the 3D convolution to realize
that. Assume that the mask feature for instance $i$ of frame $t$ is
$g_{i,t}\in\mathbb{R}^{1\times
a\times\nicefrac{{H_{0}}}{{4}}\times\nicefrac{{W_{0}}}{{4}}}$, where $a$ is
the channel number, then we concatenate the features of $T$ frames to form the
$G_{i}\in\mathbb{R}^{1\times a\times
T\times\nicefrac{{H_{0}}}{{4}}\times\nicefrac{{W_{0}}}{{4}}}$. The instance
sequence segmentation module takes the instance sequence mask feature $G_{i}$
as input, and output the mask sequence $m_{i}\in\mathbb{R}^{1\times 1\times
T\times\nicefrac{{H_{0}}}{{4}}\times\nicefrac{{W_{0}}}{{4}}}$ for the instance
directly. This module contains three 3D convolutional layers and Group
Normalization[29] layers with ReLU activation function. No normalization or
activation is performed after the last convolution layer, and the output
channel number of the last layer is 1. In this way, the masks of the instance
for $T$ frames are obtained. The mask loss for supervising the predictions in
Eq. (7) is defined as a combination of the Dice [16] and Focal loss [13]:

$\displaystyle\mathcal{L}_{\text{mask}}\left(m_{i},\hat{m}_{\sigma(i)}\right)$
$\displaystyle=\lambda_{\text{mask}}\frac{1}{T}\sum_{t=1}^{T}\Bigl{[}{{\cal
L}_{\text{Dice}}}(m_{i,t},\hat{m}_{\sigma(i),t})$ $\displaystyle+{{\cal
L}_{\text{Focal}}}(m_{i,t},\hat{m}_{\sigma(i),t})\Bigr{]}.$ (9)

##  4 Experiments

In this section, we conduct experiments on the YouTube-VIS [30] dataset, which
contains 2238 training, 302 validation and 343 test video clips. Each video of
the dataset is annotated with per pixel segmentation mask, category and
instance labels. The object category number is 40. As the test set evaluation
is closed, we evaluate our method in the validation set. The evaluation
metrics are average precision (AP) and average recall (AR), with the video
Intersection over Union (IoU) of the mask sequences as the threshold.

###  4.1 Implementation Details

Model settings. As the largest number of the annotated video length for
YouTube-VIS [30] is 36, we take this value as the default input video clip
length $T$. Thus, no post-processing is needed to associate different clips
from one video, which makes our model totally end-to-end trainable. The model
predicts 10 objects for each frame, thus the total object query number is 360.
For the Transformer we use 6 encoder, 6 decoder layers of width 384 with 8
attention heads. Unless otherwise specified, ResNet-50 [10] is used as our
backbone networks and the same hyper-parameters of DETR [4] are used.

Training. The model is implemented with PyTorch-1.6 [17], trained with AdamW
[15] of initial Transformer’s learning rate being $10^{-4}$ , the backbone’s
learning rate being $10^{-5}$. The models are trained for 18 epochs, with the
learning rate decays by 10x at 12 epochs. We initialize our backbone networks
with the weights of DETR pretrained on COCO [14]. The models are trained on 8
V100 GPUs of 32G RAM, with 1 video clip per GPU. The frame sizes are
downsampled to 300$\times$540 to fit the GPU memory.

Inference. During inference, we follow the same scale setting as training. No
post-processing is needed for associating instances. Instances with scores
larger than 0.001 are kept. The mean score for all the frames is used as the
instance score. For instances that have been classified to different
categories in different frames, we use the most frequently predicted category
as the final instance category.

###  4.2 Ablation Study

In this section we conduct extensive ablation experiments to study the core
factors of VisTR. Comparison results are reported in Table 1.

Length | AP | $\rm AP_{50}$ | $\rm AP_{75}$ | $\rm AR_{1}$ | $\rm AR_{10}$
---|---|---|---|---|---
18 | 29.7 | 50.4 | 31.1 | 29.5 | 34.4
24 | 30.5 | 47.8 | 33.0 | 29.5 | 34.4
30 | 31.7 | 53.2 | 32.8 | 31.3 | 36.0
36 | 33.3 | 53.4 | 35.1 | 33.1 | 38.5
(a) Video sequence length. The performance improves as the sequence length
increases.

| # | AP | $\rm AP_{50}$ | $\rm AP_{75}$ | $\rm AR_{1}$ | $\rm AR_{10}$
---|---|---|---|---|---|---
video level | 1 | 8.4 | 13.2 | 9.5 | 20.0 | 20.8
frame level | 36 | 13.7 | 23.3 | 14.5 | 30.4 | 35.1
ins. level | 10 | 32.0 | 52.8 | 34.0 | 31.6 | 37.2
pred. level | 360 | 33.3 | 53.4 | 35.1 | 33.1 | 38.5
(b) Instance query embedding. Instance-level query is only 1.3% lower in AP
than the prediction-level query with 36$\times$ fewer embeddings.

time order | AP | $\rm AP_{50}$ | $\rm AP_{75}$ | $\rm AR_{1}$ | $\rm AR_{10}$
---|---|---|---|---|---
random | 32.3 | 52.1 | 34.3 | 33.8 | 37.3
in order | 33.3 | 53.4 | 35.1 | 33.1 | 38.5
(c) Video sequence order. Sequence in time order is 1.0% better in AP than
sequence in random order.

| AP | $\rm AP_{50}$ | $\rm AP_{75}$ | $\rm AR_{1}$ | $\rm AR_{10}$
---|---|---|---|---|---
w/o | 28.4 | 50.1 | 29.5 | 29.6 | 33.3
w | 33.3 | 53.4 | 35.1 | 33.1 | 38.5
(d) Position encoding. Position encoding brings about 5% AP gains to VisTR.

| AP | $\rm AP_{50}$ | $\rm AP_{75}$ | $\rm AR_{1}$ | $\rm AR_{10}$
---|---|---|---|---|---
CNN | 32.0 | 54.5 | 31.5 | 31.6 | 37.7
Transformer | 33.3 | 53.4 | 35.1 | 33.1 | 38.5
(e) CNN-encoded feature vs. Transformer-encoded feature for mask prediction.
The transformer improves the feature quality.

| AP | $\rm AP_{50}$ | $\rm AP_{75}$ | $\rm AR_{1}$ | $\rm AR_{10}$
---|---|---|---|---|---
w/o | 33.3 | 53.4 | 35.1 | 33.1 | 38.5
w | 34.4 | 55.7 | 36.5 | 33.5 | 38.9
(f) Instance sequence segmentation module. The module with 3D convolutions
brings 1.1% AP gains.

Table 1: Ablation experiments for VisTR. All models are trained on YouTubeVIS
train for 10 epochs and tested on YouTubeVIS val, using the ResNet-50
backbone.

The main difference between video and image is that video contains temporal
information. How to effectively learn and exploit temporal information is the
key to video understanding. Firstly, we study the importance of temporal
information to VisTR in two dimensions: the amount and the order.

Video sequence length. To evaluate the importance of the amount of temporal
information to VisTR, we experiment with models trained with different input
video sequence lengths. As reported in Table 1(a), with the length varying
from 18 to 36, the AP increases monotonically from 29.7% to 33.3%. This result
shows that more temporal information indeed helps the model learn better. As
the largest video length of the dataset is 36, we argue that, if with a larger
dataset, VisTR can achieve even better results. Note that for this experiment,
if the clip length is less than the video length, instance matching in
overlapping frames is used for associating them from different clips.

Video sequence order. As the movement of objects in real scenes are
continuous, we believe that the order of temporal information is also
important. To evaluate, we perform a comparison of the model trained with
input video sequence in random order vs. time order. Results in Table 1(c)
show that the model learned according to the time order information achieves 1
point higher, which verifies the importance of the temporal order.

Positional encoding. Position information is important for the dense
prediction problem of VIS. As the original feature sequence contains no
positional information, we supplement with the spatial and temporal positional
encodings, which indicate the relative positions in the video sequence.
Experiments of models with and without positional encoding are presented in
Table 1(d). The model without positional encoding manages to achieve 28.4% AP.
Our explanation is that the ordered format of the sequence supervision and the
correspondence between the input and output order of the Transformer provide
some relative positional information implicitly. In the second experiment, the
performance improves by about 5 points, which verifies the necessity of
explicit positional encoding.

Instance queries. The instance queries are learned embeddings for decoding the
representative instance predictions. In this experiment, we study the effect
of instance queries and attempt to exploit the inner connections among them by
varying the embedding number. Suppose the model decode $n$ instances each
frame, and the frame number is $T$. The input instance query number should be
$n\times T$ to decode the same number for predictions. In the default setting,
one embedding is responsible for one prediction, the model directly learns
$n\times T$ unique embeddings, termed as ‘prediction level’ in Table 1(b). In
the ‘video level setting’, one embedding is learned for all the instance
predictions, i.e., the same embedding is repeated $n\times T$ times as the
input of decoder. In the ‘frame-level’ setting, the model only learns $T$
unique embeddings and repeats them by $n$ times. In the ‘instance level’
setting, the model only learns $n$ unique embeddings and repeats them by $T$
times. The n and T corresponds to the value of 10 and 36 in the table
respectively. The result is 8.4% AP and 13.7% AP for ‘video level’ and ‘frame
level’ settings respectively. Surprisingly, the ‘instance level’ queries can
achieve 32.0% AP, which is only 1.3 points lower than the default setting. The
result shows that the queries for one instance can be shared for the VisTR
model, which makes the tracking natural. But the queries for one frame can not
be shared.

Transformers for feature encoding. As illustrated in the ‘instance sequence
segmentation’ module of fig:2. The module takes three types of features as
input: the features ‘B’ from the backbone, the feature ‘E’ from the encoder
and the attention map computed by the feature ‘E’ and ‘O’. To show the
superiority of Transformers in feature encoding, we compare the results of
using the original input ‘O’ vs. output ‘E’ of the encoder for the second
feature, a.k.a., CNN-encoded features vs. Transformer-encoded features. As
reported in Table 1(e), the CNN-encoded features achieves 32.0% AP, and the
Transformer-encoded features achieve 1.3 points higher. This demonstrates that
features are learned better after the Transformer updates them based on all
pairwise similarities between them through self-attention. The result also
shows the superiority of modeling the spatial and temporal features as a
whole.

Instance sequence segmentation. The segmentation process contains both the
instance mask feature accumulation and instance sequence segmentation modules.
The instance sequence segmentation module takes the instance sequence as a
whole. We expect that it can strengthen the mask prediction by learning the
temporal information through 3D convolutions. Thus, when objects are in
challenging situations such as occlusions or motion blurs, the module can
learn to propagate information from other frames to help the segmentation.
Besides, the features of the same instance from multiple frames could help the
network recognize the instance better. In this experiment, we perform a study
of models with or without the 3D instance sequence segmentation module. For
the former case, we apply a 2D convolutional layer with the output channel
being 1 to the mask features for each instance of each frame to obtain the
masks. The comparison is shown in Table 1(f). The instance sequence
segmentation module improves the result by 1.1 points, which verifies the
effectiveness of the proposed module.

With these ablation studies, we conclude that in VisTR design: the temporal
information, positional encodings, instance queries, global self-attention in
the encoder and the instance sequence segmentation module, all play important
roles w.r.t. the final performance.

###  4.3 Main Results

Method | backbone | FPS | AP | $\rm AP_{50}$ | $\rm AP_{75}$ | $\rm AR_{1}$ | $\rm AR_{10}$
---|---|---|---|---|---|---|---
DeepSORT[28] | ResNet-50 | - | 26.1 | 42.9 | 26.1 | 27.8 | 31.3
FEELVOS[24] | ResNet-50 | - | 26.9 | 42.0 | 29.7 | 29.9 | 33.4
OSMN[31] | ResNet-50 | - | 27.5 | 45.1 | 29.1 | 28.6 | 33.1
MaskTrack R-CNN[30] | ResNet-50 | 20.0 | 30.3 | 51.1 | 32.6 | 31.0 | 35.5
STEm-Seg[1] | ResNet-50 | - | 30.6 | 50.7 | 33.5 | 31.6 | 37.1
STEm-Seg[1] | ResNet-101 | 2.1 | 34.6 | 55.8 | 37.9 | 34.4 | 41.6
MaskProp[2] | ResNet-50 | - | 40.0 | - | 42.9 | - | -
MaskProp[2] | ResNet-101 | - | 42.5 | - | 45.6 | - | -
VisTR | ResNet-50 | 30.0/69.9 | 36.2 | 59.8 | 36.9 | 37.2 | 42.4
VisTR | ResNet-101 | 27.7/57.7 | 40.1 | 64.0 | 45.0 | 38.3 | 44.9
Table 2: Video instance segmentation AP (%) on the YouTube-VIS [30] validation
dataset. Note that, for the first three methods, we have cited the results
reported by the re-implementations in [30] for VIS. Other results are adopted
from their original paper. For the speed of VisTR we report the FPS results
with and without the data loading process. Here we naively load the images
serially, taking unnecessarily long time. The data loading process can be much
faster by parallelizing.  Figure 2: Visualization of VisTR on the YouTube-VIS
[30] validation dataset. Each row contains images from the same video. For
each video, here the same colors depict the mask sequences of the same
instances (Best viewed on screen).

We compare VisTR against some state-of-the-art methods in video instance
segmentation in Table 2. The comparison is performed in terms of both accuracy
and speed. The methods in the first three rows are originally proposed for
tracking or VOS. We have cited the results reported by the re-implementations
in [30] for VIS. Other methods including the MaskTrack RCNN, MaskProp [2] and
STEm-Seg [1] are originally proposed for the VIS task in the temporal order.

For the accuracy measured by AP, VisTR achieves the best result among methods
using a single model without any bells and whistles. Using the same backbone
of ResNet-50 [10], VisTR achieves about 6 points higher in AP than the
MaskTrack R-CNN and the recently proposed STEm-Seg method. Besides, we argue
the AP gap between VisTR and MaskProp mainly comes from its combination of
multiple networks, i.e., Spatiotemporal Sampling Network [3], Feature Pyramid
Network [12], Hybrid Task Cascade Network [6] and the High-Resolution Mask
Refinement post-processing. Since our aim is to design a conceptually simple
and end-to-end framework, many improvements methods, such as complex video
data augmentation and multi-stage mask refinement are beyond the scope of this
work. For the speed measured by FPS (frames per second), VisTR shows a
significant advantage among all the reported results, achieving 27.7 FPS with
the ResNet-101 backbone. If excluding the data loading process of multiple
images, the speed can achieve 57.7 FPS. Note that, as we load the images in
serial, the data loading process can be easily parallelized. The fast speed of
VisTR owes to its design of parallel decoding and no post-processing.

The visualization of VisTR on the YouTube-VIS[30] validation dataset is shown
in fig:3, with each row containing images sampled from the same video. VisTR
can track and segment instances well in challenging situations such as: (a)
instances overlapping, (b) changes of relative positions between instance, (c)
confusion by the same category instances that are close together and (d)
instances in various poses.

##  5 Conclusion

In this paper, we have proposed a new video instance segmentation framework
built upon Transformers, which views the VIS task as a direct end-to-end
parallel sequence decoding/prediction problem. In this way, instance tracking
is achieved seamlessly and naturally in the same framework of instance
segmentation, which is significantly different from and simpler than existing
approaches, considerably simplifying the overall pipeline. Without bells and
whistles, VisTR achieves the best result and the highest speed among methods
using a single model on the YouTube-VIS dataset. To our knowledge, our work is
the first one that applies the Transformer to video instance segmentation. We
hope that similar approaches can be applied to many more video understanding
tasks in the future.

Acknowledgements This work was in part supported by Beijing Science and
Technology Project (No. Z181100008918018). CS and his employer received no
financial support for the research, authorship, and/or publication of this
article.

## References

  * [1] Ali Athar, Sabarinath Mahadevan, Aljoša Ošep, Laura Leal-Taixé, and Bastian Leibe.  Stem-seg: Spatio-temporal embeddings for instance segmentation in videos.  In Proc. Eur. Conf. Comp. Vis., 2020.
  * [2] Gedas Bertasius and Lorenzo Torresani.  Classifying, segmenting, and tracking object instances in video with mask propagation.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 9739–9748, 2020\.
  * [3] Gedas Bertasius, Lorenzo Torresani, and Jianbo Shi.  Object detection in video with spatiotemporal sampling networks.  In Proc. Eur. Conf. Comp. Vis., pages 331–346, 2018.
  * [4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In Proc. Eur. Conf. Comp. Vis., 2020.
  * [5] Hao Chen, Kunyang Sun, Zhi Tian, Chunhua Shen, Yongming Huang, and Youliang Yan.  Blendmask: Top-down meets bottom-up for instance segmentation.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 8573–8581, 2020\.
  * [6] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al.  Hybrid task cascade for instance segmentation.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 4974–4983, 2019\.
  * [7] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In Proc. IEEE Int. Conf. Comp. Vis., pages 764–773, 2017.
  * [8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [9] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask R-CNN.  In Proc. IEEE Int. Conf. Comp. Vis., pages 2961–2969, 2017.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 770–778, 2016\.
  * [11] Harold W. Kuhn.  The Hungarian method for the assignment problem.  Naval research logistics quarterly, 2(1-2):83–97, 1955.
  * [12] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 2117–2125, 2017\.
  * [13] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar.  Focal loss for dense object detection.  In Proc. IEEE Int. Conf. Comp. Vis., Oct 2017.
  * [14] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In Proc. Eur. Conf. Comp. Vis., pages 740–755, 2014.
  * [15] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In Proc. Int. Conf. Learn. Representations, 2018.
  * [16] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi.  V-net: Fully convolutional neural networks for volumetric medical image segmentation.  In Proc. Int. Conf. 3D Vis., pages 565–571, 2016.
  * [17] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In Proc. Advances in Neural Inf. Process. Syst., pages 8026–8037, 2019.
  * [18] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung.  A benchmark dataset and evaluation methodology for video object segmentation.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2016.
  * [19] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Proc. Advances in Neural Inf. Process. Syst., pages 91–99, 2015\.
  * [20] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 658–666, 2019\.
  * [21] Russell Stewart, Mykhaylo Andriluka, and Andrew Y Ng.  End-to-end people detection in crowded scenes.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 2325–2333, 2016\.
  * [22] Zhi Tian, Chunhua Shen, and Hao Chen.  Conditional convolutions for instance segmentation.  In Proc. Eur. Conf. Comp. Vis., 2020.
  * [23] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Proc. Advances in Neural Inf. Process. Syst., pages 5998–6008, 2017.
  * [24] Paul Voigtlaender, Yuning Chai, Florian Schroff, Hartwig Adam, Bastian Leibe, and Liang-Chieh Chen.  Feelvos: Fast end-to-end embedding learning for video object segmentation.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 9481–9490, 2019\.
  * [25] Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li.  SOLO: Segmenting objects by locations.  In Proc. Eur. Conf. Comp. Vis., 2020.
  * [26] Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen.  SOLOv2: Dynamic and fast instance segmentation.  In Proc. Advances in Neural Inf. Process. Syst., 2020.
  * [27] Yuqing Wang, Zhaoliang Xu, Hao Shen, Baoshan Cheng, and Lirong Yang.  Centermask: single shot instance segmentation with point representation.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., 2020.
  * [28] Nicolai Wojke, Alex Bewley, and Dietrich Paulus.  Simple online and realtime tracking with a deep association metric.  In Proc. IEEE Int. Conf. Image Process., pages 3645–3649, 2017\.
  * [29] Yuxin Wu and Kaiming He.  Group normalization.  In Proc. Eur. Conf. Comp. Vis., pages 3–19, 2018.
  * [30] Linjie Yang, Yuchen Fan, and Ning Xu.  Video instance segmentation.  In Proc. IEEE Int. Conf. Comp. Vis., 2019.
  * [31] Linjie Yang, Yanran Wang, Xuehan Xiong, Jianchao Yang, and Aggelos K Katsaggelos.  Efficient video object segmentation via network modulation.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 6499–6507, 2018\.