# Generative Semantic Segmentation

Jiaqi Chen1 Jiachen Lu1 Xiatian Zhu2 Li Zhang1
1Fudan University 2University of Surrey
https://github.com/fudan-zvg/GSS Li Zhang<EMAIL_ADDRESS>is the
corresponding author with School of Data Science, Fudan University.

###### Abstract

We present Generative Semantic Segmentation (GSS), a generative learning
approach for semantic segmentation. Uniquely, we cast semantic segmentation as
an image-conditioned mask generation problem. This is achieved by replacing
the conventional per-pixel discriminative learning with a latent prior
learning process. Specifically, we model the variational posterior
distribution of latent variables given the segmentation mask. To that end, the
segmentation mask is expressed with a special type of image (dubbed as
maskige). This posterior distribution allows to generate segmentation masks
unconditionally. To achieve semantic segmentation on a given image, we further
introduce a conditioning network. It is optimized by minimizing the divergence
between the posterior distribution of maskige (i.e. segmentation masks) and
the latent prior distribution of input training images. Extensive experiments
on standard benchmarks show that our GSS can perform competitively to prior
art alternatives in the standard semantic segmentation setting, whilst
achieving a new state of the art in the more challenging cross-domain setting.

##  1 Introduction

The objective of semantic segmentation is to predict a label for every single
pixel of an input image [32]. Conditioning on each pixel’s observation,
existing segmentation methods [4, 58, 52, 9] naturally adopt the
discriminative learning paradigm, along with dedicated efforts on integrating
task prior knowledge (e.g., spatial correlation) [58, 9, 23, 47]. For example,
existing methods [4, 58, 52] typically use a linear projection to optimize the
log-likelihood classification for each pixel. Despite the claim of subverting
per-pixel classification, the bipartite matching-based semantic segmentation
[9, 8] still cannot avoid the per-pixel max log-likelihood.

Figure 1:  Schematic comparison between (a) conventional discriminative
learning and (b) our generative learning based model for semantic
segmentation. Our GSS introduces a latent variable $z$ and, given the
segmentation mask $c$, it learns the posterior distribution of $z$ subject to
the reconstruction constraint. Then, we train a conditioning network to model
the prior of $z$ by aligning with the corresponding posterior distribution.
This formulation can thus generate the segmentation mask for an input image.

In this paper, we introduce a new approach, Generative Semantic Segmentation
(GSS), that formulates semantic segmentation as an image-conditioned mask
generation problem. This conceptually differs from the conventional
formulation of discriminative per-pixel classification learning, based on the
log-likelihood of a conditional probability (i.e. the classification
probability of image pixels). Taking the manner of image generation instead
[24, 45], we generate the whole segmentation masks with an auxiliary latent
variable distribution introduced. This formulation is not only simple and more
task-agnostic, but also facilitates the exploitation of off-the-shelf big
generative models (e.g. DALL$\cdot$E [40] trained by 3 billion iterations on a
300 million open-image dataset, far beyond both the data scale and training
cost of semantic segmentation).

However, achieving segmentation segmentation in a generic generation framework
(e.g. the Transformer architecture [15]) is non-trivial due to drastically
different data format. To address this obstacle, we propose a notion of
maskige that expresses the segmentation mask in the RGB image form. This
enables the use of a pretrained latent posterior distribution (e.g. VQVAE
[40]) of existing generative models. Our model takes a two-stage optimization:
(i) Learning the posterior distribution of the latent variables conditioned on
the semantic segmentation masks so that the latent variables can simulate the
target segmentation masks; To achieve this, we introduce an fixed pre-trained
VQVAE [40] and a couple of lightweight transformation modules, which can be
trained with minimal cost, or they can be manually set up without requiring
any additional training. In either case, the process is efficient and does not
add significant overhead to the overall optimization. (ii) Minimizing the
distance between the posterior distribution and the prior distribution of the
latent variables given input training images and their masks, enabling to
condition the generation of semantic masks on the input images. This can be
realized by a generic encoder-decoder style architecture (e.g. a Transformer).

We summarize the contributions as follows. (i) We propose a Generative
Semantic Segmentation approach that reformulates semantic segmentation as an
image-conditioned mask generation problem. This represents a conceptual shift
from conventional discriminative learning based paradigm. (ii) We realize a
GSS model in an established conditional image generation framework, with
minimal need for task-specific architecture and loss function modifications
while fully leveraging the knowledge of off-the-shelf generative models. (iii)
Extensive experiments on several semantic segmentation benchmarks show that
our GSS is competitive with prior art models in the standard setting, whilst
achieving a new state of the art in the more challenging and practical cross-
domain setting (e.g. MSeg [26]).

##  2 Related work

Semantic segmentation Since the inception of FCN [32], semantic segmentation
have flourished by various deep neural networks with ability to classify each
pixel. The follow-up efforts then shift to improve the limited receptive field
of these models. For example, PSPNet [57] and DeepLabV2 [3] aggregate multi-
scale context between convolution layers. Sequentially, Nonlocal [49], CCNet
[21], and DGMN [56] integrate the attention mechanism in the convolution
structure. Later on, Transformer-based methods (e.g. SETR [58] and Segformer
[52]) are proposed following the introduction of Vision Transformers. More
recently, MaskFormer [9] and Mask2Former [8] realize semantic segmentation
with bipartite matching. Commonly, all the methods adopt the discriminative
pixel-wise classification learning paradigm. This is in contrast to our
generative semantic segmentation.

Image generation In parallel, generative models [15, 40] also excel. They are
often optimized in a two-stage training process: (1) Learning data
representation in the first stage and (2) building a probabilistic model of
the encoding in the second stage. For learning data representation, VAE [24]
reformulates the autoencoder by variational inference. GAN [19] plays a zero-
sum game. VQVAE [45] extends the image representation learning to discrete
spaces, making it possible for language-image cross-model generation. [27]
replaces element-wise errors of VAE with feature-wise errors to capture data
distribution. For probabilistic model learning, some works [41, 14, 51] use
flow for joint probability learning. Leveraging the Transformers to model the
composition between condition and images, Esser et al. [15] demonstrate the
significance of data representation (i.e. the first stage result) for the
challenging high-resolution image synthesis, obtained at high computational
cost. This result is inspiring to this work in the sense that the diverse and
rich knowledge about data representation achieved in the first stage could be
transferable across more tasks such as semantic segmentation.

Generative models for visual perception Image-to-image translation made one of
the earliest attempts in generative segmentation, with far less success in
performance [22]. Some good results were achieved in limited scenarios such as
face parts segmentation and Chest X-ray segmentation [28]. Replacing the
discriminative classifier with a generative Gaussian Mixture model, GMMSeg
[29] is claimed as generative segmentation, but the most is still of
discriminative modeling. The promising performance of Pix2Seq [7] on several
vision tasks leads to the prevalence of sequence-to-sequence task-agnostic
vision frameworks. For example, Unified-I/O [33] supports a variety of vision
tasks within a single model by seqentializing each task to sentences.
Pix2Seq-D [6] deploys a hierarchical VAE (i.e. diffusion model) to generate
panoptic segmentation masks. This method is inefficient due to the need for
iterative denoising. UViM [25] realizes its generative panoptic segmentation
by introducing latent variable conditioned on input images. It is also
computationally heavy due to the need for model training from scratch. To
address these issues, we introduce a notion of maskige for expressing
segmentation masks in the form of RGB images, enabling the adopt of off-the-
shelf data representation models (e.g. VGVAE) already pretrained on vast
diverse imagery. This finally allows for generative segmentation model
training as efficiently as conventional discriminative counterparts.

##  3 Methodology

###  3.1 GSS formulation

Figure 2:  An illustration of our efficient latent posterior learning. Instead
of training a maskige encoder-decoder, we utilize a well pretrained VQVAE [45]
(gray blocks) and optimize the transformations $\mathcal{X}$ and
$\mathcal{X}^{-1}$ (green blocks). To optimize $\mathcal{X}$ and
$\mathcal{X}^{-1}$ with gradient descent, we employ the Gumbel softmax
relaxation technique [35]. “Rec.”: Reconstructed.  Figure 3:  Illustration of
latent prior learning (top) and generative inference pipeline (bottom). During
training, for latent prior learning, we optimize the image encoder
$\mathcal{I}_{\psi}$ while freezing the maskige encoder $\mathcal{E}_{\phi}$.
The objective is to minimize the divergence (e.g. cross entropy loss) between
the prior distribution and the posterior distribution of latent tokens. During
generative inference, we use the prior $z\sim p(z|x)$ inferred by
$\mathcal{I}_{\psi}$ to generate the maskige with maskige decoder
$\mathcal{D}_{\theta}$. “Pred.”: Predicted.

Traditionally, semantic segmentation is formulated as a discriminative
learning problem as

$\max_{\pi}\log p_{\pi}(c|x)$ (1)

where $x\in\mathbb{R}^{H\times W\times 3}$ is an input image,
$c\in\\{0,1\\}^{H\times W\times K}$ is a segmentation mask in $K$ semantic
categories, and $p_{\pi}$ is a discriminative pixel classifier. Focusing on
learning the classification boundary of input pixels, this approach enjoys
high data and training efficiency [38].

In this work, we introduce Generative Semantic Segmentation (GSS) by
introducing a discrete $L$-dimension latent distribution $q_{\phi}(z|c)$ (with
$z\in\mathbb{Z}^{L}$) to the above log-likelihood as:

$\displaystyle\log
p(c|x)\geq\mathbb{E}_{q_{\phi}(z|c)}\left[\log\frac{p(z,c|x)}{q_{\phi}(z|c)}\right],$

which is known as the Evidence Lower Bound (ELBO) [24] (details are given in
the supplementary material). Expanding the ELBO gives us

$\mathbb{E}_{q_{\phi}(z|c)}\left[\log
p_{\theta}(c|z)\right]-D_{KL}\Big{(}q_{\phi}(z|c),p_{\psi}(z|x)\Big{)},$ (2)

where we have three components in our formulation:

• $p_{\psi}$: An image encoder (denoted as $\mathcal{I}_{\psi}$) that models
the prior distribution of latent tokens $z$ conditioned on the input image
$x$.

• $q_{\phi}$: A function that encodes the semantic segmentation mask $c$ into
discrete latent tokens $z$, which includes a maskige encoder (denoted as
$\mathcal{E}_{\phi}$, implemented by a VQVAE encoder [45]) and a linear
projection (denoted as $\mathcal{X}$, which will be detailed in Section 3.3).

• $p_{\theta}$: A function that decodes the semantic segmentation mask $c$
from the discrete latent tokens $z$, which includes a maskige decoder (denoted
$\mathcal{D}_{\theta}$, implemented by a VQVAE decoder [45]) and
$\mathcal{X}^{-1}$ (the inverse process of $\mathcal{X}$).

#### Architecture

The architecture of GSS comprises three components: $\mathcal{I}_{\psi}$,
$\mathcal{E}_{\phi}$ and $\mathcal{D}_{\theta}$. $\mathcal{E}\phi$ and
$\mathcal{D}_{\theta}$ are implemented as VQVAE encoder and decoder [45],
respectively. Meanwhile, $\mathcal{I}_{\psi}$ is composed of an image backbone
R(e.g. esNet [20] or Swin Transformer [31]) and a _Multi-Level Aggregation_
(MLA). As an essential part, MLA is constructed using $D$ shifted window
Transformer layers [31] and a linear projection layer. The resulting output is
a discrete code $z\in\mathbb{Z}^{H/d\times W/d}$, where $d$ denotes the
downsample ratio.

#### Optimization

Compared to the log-likelihood in discriminative models, optimizing the ELBO
of a general model is more challenging [38]. End-to-end training cannot reach
a global optimization. For Eq. (2), often we name the first term
$\mathbb{E}_{q_{\phi}(z|c)}\left[\log p_{\theta}(c|z)\right]$ as a
reconstruction term and the second KL-divergence as the prior term. In the
next section we will introduce the optimization of this ELBO.

###  3.2 ELBO optimization for semantic segmentation

The ELBO optimization process for semantic segmentation involves two main
steps, as described in [40]. The first step is latent posterior learning, also
known as reconstruction (see Figure 2). Here, the ELBO is optimized with
respect to $\theta$ and $\phi$ through training a VQVAE [45] to reconstruct
the desired segmentation masks. The second step is latent prior learning (see
Figure 3). Once $\theta$ and $\phi$ are fixed, an image encoder $\psi$ is
optimized to learn the prior distribution of latent tokens given an input
image.

Typically, the first stage of ELBO optimization is both most important and
most expensive (much more than many discriminative learning counterparts)
[15]. To address this challenge, we propose an efficient latent posterior
learning process.

###  3.3 Stage I: Efficient latent posterior learning

During the first stage (shown in Figure 2), the initial prior $p_{\psi}(z|x)$
is set as the uniform distribution. Conventionally, the first stage latent
posterior training is conducted by

$\min_{\theta,\phi}\mathbb{E}_{q_{\phi}(z|c)}\|p_{\theta}(c|z)-c\|.$ (3)

To optimize Eq. (3) more efficiently, we introduce a random variable
transformation $\mathcal{X}:\mathbb{R}^{K}\to\mathbb{R}^{3}$, its pseudo-
inverse function $\mathcal{X}^{-1}:\mathbb{R}^{3}\to\mathbb{R}^{K}$, and the
maskige decoder $\mathcal{D}_{\theta}$. Applying expectation with transformed
random variable, we have

$\displaystyle\quad\min_{\hat{\phi},\hat{\theta}}\mathbb{E}_{q_{\hat{\phi}}(\hat{z}|\mathcal{X}(c))}\|\mathcal{D}_{\hat{\theta}}(\hat{z})-\mathcal{X}(c)\|$
$\displaystyle+\min_{\mathcal{X}^{-1}}\mathbb{E}_{q_{\hat{\phi}}(\hat{z}|\mathcal{X}(c))}\|\mathcal{X}^{-1}(\mathcal{D}_{\hat{\theta}}(\hat{z}))-c\|.$
(4)

Please refer to supplementary material for more details. Then we find that
$\mathcal{X}(c)=x^{(c)}\in\mathbb{R}^{H\times W\times 3}$ can be regarded as a
kind of RGB image, where each category is represented by a specific color. For
convenience, we term it maskige. Therefore, the optimization can be rewritten
as

$\displaystyle\quad\min_{\hat{\phi},\hat{\theta}}\mathbb{E}_{q_{\hat{\phi}}(\hat{z}|x^{c})}\|\mathcal{D}_{\hat{\theta}}(\hat{z})-x^{(c)}\|$
$\displaystyle+\min_{\mathcal{X}^{-1}}\mathbb{E}_{q_{\hat{\phi}}(\hat{z}|\mathcal{X}(c))}\|\mathcal{X}^{-1}(\hat{x}^{(c)})-c\|,$
(5)

where $\hat{x}^{(c)}=\mathcal{D}_{\hat{\theta}}(\hat{z})$. Now, the first term
of Eq. (5) can be regarded as an image reconstruction task (see Figure 2). In
practice, this has been already well optimized by [15, 40] using million-scale
datasets. This allows us to directly utilize the off-the-shelf pretrained
models. As such, our optimization problem can be simplified as:

$\min_{\mathcal{X}^{-1}}\mathbb{E}_{q_{\hat{\phi}}(\hat{z}|\mathcal{X}(c))}\|\mathcal{X}^{-1}(\hat{x}^{(c)})-c\|.$
(6)

Note that the parameters (0.9K$\sim$466.7K in our designs) of $\mathcal{X}$
and $\mathcal{X}^{-1}$ are far less than $\theta,\phi$ (totally 29.1M
parameters with the VQVAE from [40]), thus more efficient and cheaper to
train. Concretely, we only optimize the small $\mathcal{X}$ while freezing
$\theta,\phi$. Following [25, 6], we use cross-entropy loss instead of MSE
loss in Eq. (6) for a better minimization between segmentation masks.

#### Linear maskige designs

The optimization problem for $\mathcal{X}$ and $\mathcal{X}^{-1}$ is non-
convex, making their joint optimization challenging. To overcome this issue,
we optimize $\mathcal{X}$ and $\mathcal{X}^{-1}$ separately. For simplicity,
we model both $\mathcal{X}$ and $\mathcal{X}^{-1}$ as linear functions (linear
assumption). Specifically, we set $x^{(c)}=c\beta$ where
$\beta\in\mathbb{R}^{K\times 3}$, and $\hat{c}=\hat{x}^{(c)}\beta^{\dagger}$
where $\beta^{\dagger}\in\mathbb{R}^{3\times K}$. Under the linear assumption,
we can transfer Eq. (6) into a least squares problem with an explicit solution
$\beta^{\dagger}=\beta^{\top}(\beta\beta^{\top})^{-1}$, so that
$\mathcal{X}^{-1}$ is free of training.

To enable zero-cost training of $\mathcal{X}$, we can also manually set the
value of $\beta$ properly. We suggest a maximal distance assumption for
selecting the value of $\beta$ to encourage the encoding of $K$ categories to
be as widely dispersed as possible in the three-dimensional Euclidean space
$\mathbb{R}^{3}$. More details are provided in the supplementary material.

#### Non-linear maskige designs

For more generic design, non-linear models (e.g. CNNs or Transformers) can be
also used to express $\mathcal{X}^{-1}$. (non-linear assumption)

#### Concrete maskige designs

We implement four optimization settings for $\mathcal{X}$ and/or
$\mathcal{X}^{-1}$ with varying training budgets. We define the naming
convention of “GSS-[F/T] [F/T] (-$O$)” in the following rules. (i) Basic
settings “-[F/T] [F/T]” on whether $\mathcal{X}$ and/or $\mathcal{X}^{-1}$
require training: “F” stands for Free of training, and “T” for Training
required. (ii) Optional settings “-O” (e.g. “R” or “W”) which will be
explained later on. All GSS variants are described below.

• GSS-FF (training free): Modeling both $\mathcal{X}$ and $\mathcal{X}^{-1}$
using linear functions, with $\beta$ initialized under maximal distance
assumption, and $\beta^{\dagger}$ optimized using least squares. For
comparison, we will experiment with GSS-FF-R, where $\beta$ is Randomly
initialized.

• GSS-FT (training required): Modeling $\mathcal{X}$ using a linear function
but modeling $\mathcal{X}^{-1}$ using a non-linear function (e.g. a three-
layer convolutional neural network). We initialize $\beta$ with maximal
distance assumption and optimize $\mathcal{X}^{-1}$ with gradient descent. A
stronger design GSS-FT-W utilizes a single-layer Shifted Window Transformer
block [31] as the non-linear $\mathcal{X}^{-1}$. Notably, we train
$\mathcal{X}$ and $\mathcal{X}^{-1}$ separately, as described in Section 4.1.

• GSS-TF (training required): Modeling both $\mathcal{X}$ and
$\mathcal{X}^{-1}$ using linear functions. We train $\beta$ with gradient
descent and optimize $\beta^{\dagger}$ with least squares according to
$\beta$.

• GSS-TT (training required): Modeling $\mathcal{X}$ using a linear function
but modeling $\mathcal{X}^{-1}$ using a non-linear function (e.g. a three-
layer CNN). We jointly train both functions using gradient descent.

To perform end-to-end optimization of the $\mathcal{X}$ function using
gradient descent for both GSS-TF&TT, a hard Gumbel-softmax relaxation
technique [35] is used. This involves computing the argmax operation during
the forward step, while broadcasting the gradients during the backward step.
Our linear designs (i.e. GSS-FF&FF-R) is training free with zero cost. Our
non-linear assumption based designs (e.g. GSS-FT&FT-W) has high performance
potential at acceptable training cost (see Section 4.2).

###  3.4 Stage II: Latent prior learning

We show in Figure 3 (Top) the latent prior learning. In this stage, we learn
the prior joint distribution between mask latent representation $z$ and images
$x$, with $\phi,\theta$ both fixed.

#### Objective

The optimization target of this stage is the second term of Eq. (2):

$\min_{\psi}D_{KL}\Big{(}q_{\phi}(z|c),p_{\psi}(z|x)\Big{)},$

where $z$ is in a discrete space of codebook-sized (e.g. 8192 in [40])
integers. The objective is to minimize the distance between the discrete
distribution of $z$ predicted by latent prior encoder $p_{\psi}$ and the $z$
given by VQVAE. Since the entropy of $q_{\phi}$ is fixed (i.e.the ground
truth), we can use the cross-entropy function to measure their alignment.

#### Unlabeled area auxiliary

Due to high labeling cost and challenge, it is often the case that a fraction
of areas per image are unlabeled (i.e. unknown/missing labels). Modeling per-
pixel conditional probability $p(c|x)$ in existing discriminative models, this
issue can be simply tackled by ignoring all unlabeled pixels during training.

In contrast, generative models (e.g. UViM [25] and our GSS) are trained at the
latent token level, without flexible access to individual pixels. As a result,
unlabeled pixels bring about extra challenges, as they can be of objects/stuff
of any categories heterogeneously. Without proper handling, a generative model
may learn to classify difficult pixels as the unlabelled and hurting the final
performance (see Figure 4).

To address this problem, we exploit a pseudo labeling strategy. The idea is to
predict a label for each unlabeled pixel. Specifically, we further introduce
an auxiliary head $p_{\xi}(\bar{c}|z)$ during latent prior learning (i.e.
state II) to label all unlabeled areas. Formally, we form an enhanced ground-
truth mask by $\tilde{c}=M_{u}\cdot\bar{c}+(1-M_{u})\cdot c$ where $M_{u}$
masks out labeled pixels, $\bar{c}$ denotes the pseudo labels, and $\tilde{c}$
denotes the labels after composition. The training objective of this stage
cane be then revised as:

$\min_{\psi}D_{KL}\left(q_{\phi}(z|c),p_{\psi}(z|x)\right)+p_{\xi}(\bar{c}|z).$
(7)

###  3.5 Generative inference

As illustrated in Figure 3 (bottom), we first take the latent tokens $z$ that
are predicted by the image encoder $\mathcal{I}_{\psi}$, and feed them into
the maskige decoder $\mathcal{D}_{\theta}$ to generate the predicted maskige
$\hat{x}^{(c)}$. Next, we apply the inverse transformation $\mathcal{X}^{-1}$
(Section 3.3) to the predicted maskige to obtain the final segmentation mask
$\hat{c}$.

##  4 Experiment

###  4.1 Experimental setup

#### Cityscapes [11]

provides pixel-level annotations for 19 object categories in urban scene
images at a high resolution of $2048\times 1024$. It contains 5000 finely
annotated images, split into 2975, 500 and 1525 images for training,
validation and testing respectively.

#### ADE20K [59]

is a challenging benchmark for scene parsing with 150 fine-grained semantic
categories. It has 20210, 2000 and 3352 images for training, validation and
testing.

#### MSeg [26]

is a composite dataset that unifies multiple semantic segmentation datasets
from different domains. In particular, the taxonomy and pixel-level
annotations are aligned by relabeling more than 220,000 object masks in over
80,000 images. We follow the standard setting: the train split [30, 2, 3, 11,
39, 46, 53, 43] for training a unified semantic segmentation model, the test
split (unseen to model training) [16, 37, 1, 18, 12, 55] for cross-domain
validation.

#### Evaluation metrics

The mean Intersection over Union (mIoU) and pixel-level accuracy (mAcc) are
reported for all categories, following the standard evaluation protocol [11].

#### Implementation details

We operate all experiments on mmsegmentation [10] with 8 NVIDIA A6000 cores.
_(i) Data augmentation_ : Images are resized to $1024\times 2048$ on
Cityscapes, $512\times 2048$ on ADE20K and MSeg, and random cropped
($768\times 768$ on cityscapes and $512\times 512$ on ADE20K and MSeg) and
random horizontal flipped during training. No test time augmentation is
applied. _(ii) Training schedule for latent prior learning_ : The batch size
is 16 on Cityscapes and MSeg and 32 on ADE20K. The total number of iterations
is 80,000 on Cityscapes and 160,000 on ADE20K and MSeg.

————–Image

————————Ground Truth

——————without auxiliary

—————–with auxiliary———

Figure 4: Qualitative results of unlabeled area auxiliary on Cityscapes [11] dataset.  GSS variants | mIoU | Training time
---|---|---
GSS-FF-R | 62.83 |  0
GSS-FF | 84.31 |  0
GSS-FT | 86.10 |  $\leq$20
GSS-TF | 84.37 |  $\leq$5
GSS-TT | 36.11 |  $\leq$5
GSS-FT-W | 87.73 |  $\leq$350
Table 1:  Ablation on the variants of latent posterior learning on the val set of ADE20K. Metrics: The maskige reconstruction performance in mIoU, as well as the training time in GPU hours (i.e., effective single-core hours).  Design |  Maskige? | Cityscapes | ADE20K | Train time
---|---|---|---|---
VQGAN [15] | ✗ | 82.16 | 81.89 |  $\leq$500
VQGAN [15] | ✓ | 75.09 | 42.70 |  $\leq$100
UViM [25] | ✗ | 89.14 | 78.98 |  $\leq$2,000
DALL$\cdot$E [40] | ✓ | 95.17 | 87.73 |  $\leq$350
Table 2:  Ablation on maskige reconstruction by different VQVAE designs on Cityscapes semantic val split and ADE20k val split. In case of no maskige, we directly reconstruct the segmentation mask with $K$ (the number of classes) channels. Unit for training time is GPU hour (i.e. the effective single-core hour).  $d$ | Unlabel | MLA |  mIoU |  mAcc
---|---|---|---|---
1/8 |  |  | 40.64 | 52.55
1/8 | ✓ |  | 43.72 | 56.08
1/4 | ✓ |  | 43.98 | 56.11
1/4 | ✓ | ✓ | 46.29 | 57.84
Table 3:  Ablation on latent prior learning on the val split of ADE20K. “Unlabel” denotes unlabeled area auxiliary, and “MLA” denotes Multi-Level Aggregation. “$d$” is the downsample ratio of discrete mask representation size between input image size.  Method |  Pretrain |  Backbone |  Iteration | mIoU
---|---|---|---|---
_\- Discriminative modeling:_ |
FCN [32] |  1K |  ResNet-101 |  80k | 77.02
PSPNet [57] |  1K |  ResNet-101 |  80k | 79.77
DeepLab-v3+ [5] |  1K |  ResNet-101 |  80k | 80.65
NonLocal [49] |  1K |  ResNet-101 |  80k | 79.40
CCNet [21] |  1K |  ResNet-101 |  80k | 79.45
Maskformer [9] |  1K |  ResNet-101 |  90k | 78.50
Mask2former [8] |  1K |  ResNet-101 |  90k | 80.10
SETR [58] |  22K |  ViT-Large |  80k | 78.10
UperNet [50] |  22K |  Swin-Large |  80k | 82.89
Maskformer [9] |  22K |  Swin-Large |  90k | 78.50
Mask2former [8] |  22K |  Swin-Large |  90k | 83.30
SegFormer [52] |  1K |  MiT-B5 |  160k | 82.25
_\- Generative modeling:_ |  |
UViM† [25] |  22K |  Swin-Large |  160k | 70.77
GSS-FF (Ours) |  1K |  ResNet-101 |  80k | 77.76
GSS-FT-W (Ours) |  1K |  ResNet-101 |  80k | 78.46
GSS-FF (Ours) |  22K |  Swin-Large |  80k | 78.90
GSS-FT-W (Ours) |  22K |  Swin-Large |  80k | 80.05
Table 4: Performance comparison on the Cityscapes val split: UViM† [25] is reproduced by us on PyTorch. “1K” means pretrained on ImageNet 1K [13] while “22K” means pretrained on ImageNet 22K [13]. Method |  Pretrain |  Backbone |  Iteration | mIoU
---|---|---|---|---
_\- Discriminative modeling:_ |  |
FCN [32] |  1K |  ResNet-101 |  160k | 41.40
CCNet [21] |  1K |  ResNet-101 |  160k | 43.71
DANet [17] |  1K |  ResNet-101 |  160k | 44.17
UperNet [50] |  1K |  ResNet-101 |  160k | 43.82
Deeplab V3+ [5] |  1K |  ResNet-101 |  160k | 45.47
Maskformer [9] |  1K |  ResNet-101 |  160k | 45.50
Mask2former [8] |  1K |  ResNet-101 |  160k | 47.80
OCRNet [54] |  1K |  HRNet-W48 |  160k | 43.25
SegFormer [52] |  1K |  MiT-B5 |  160k | 50.08
SETR [58] |  22K |  ViT-Large |  160k | 48.28
_\- Generative modeling:_ |  |
UViM† [25] |  22k |  Swin-Large |  160k | 43.71
GSS-FF (Ours) |  22K |  Swin-Large |  160k | 46.29
GSS-FT-W (Ours) |  22K |  Swin-Large |  160k | 48.54
Table 5: Performance comparison with previous art methods on the ADE20K val split. UViM† [25] is reproduced by ourselves.  Method | Backbone | Iteration |  VOC [16] |  Context [37] |  CamVid [1] |  WildDash [55] |  KITTI [18] |  ScanNet [12] | h. mean
---|---|---|---|---|---|---|---|---|---
_\- Discriminative modeling:_ |  |  |  |  |  |  |
CCSA [36] | HRNet-W48 | 500k | 48.9 | - | 52.4 | 36.0 | - | 27.0 | 39.7
MGDA [42] | HRNet-W48 | 500k | 69.4 | - | 57.5 | 39.9 | - | 33.5 | 46.1
MSeg [26] | HRNet-W48 | 500k | 70.7 | 42.7 | 83.3 | 62.0 | 67.0 | 48.2 | 59.2
MSeg† [26] | HRNet-W48 | 160k | 63.8 | 39.6 | 73.9 | 60.9 | 65.1 | 43.5 | 54.9
MSeg† [26] | Swin-Large | 160k | 78.7 | 47.5 | 75.1 | 66.1 | 68.1 | 49.0 | 61.7
_\- Generative modeling:_ |  |  |  |  |  |  |
GSS-FF (Ours) | HRNet-W48 | 160k | 64.1 | 37.1 | 72.3 | 59.3 | 62.0 | 40.6 | 52.6
GSS-FT-W (Ours) | HRNet-W48 | 160k | 65.2 | 38.8 | 75.2 | 62.5 | 66.2 | 43.1 | 55.2
GSS-FF (Ours) | Swin-Large | 160k | 78.7 | 45.8 | 74.2 | 61.8 | 65.4 | 46.9 | 59.5
GSS-FT-W (Ours) | Swin-Large | 160k | 79.5 | 47.7 | 75.9 | 65.3 | 68.0 | 49.7 | 61.9
Table 6: Cross-domain semantic segmentation performance on MSeg dataset test split. “h. mean” is the harmonic mean [26]. MSeg† [26] is reproduced by us on MMSegmentation [10] Sharing $\mathcal{I}_{\psi}$ |  Sharing maskiage | GSS-FF | GSS-FT-W
---|---|---|---
|  | 78.9 | 80.5
|  ✓ | 78.0 | 79.5
✓ |  ✓ | 76.6 | 78.4
Table 7: Transferring the maskiage and image encoder $\mathcal{I}_{\psi}$ from
MSeg to Cityscapes (val split). Metric: mIoU.

###  4.2 Ablation studies

Latent posterior learning We evaluate the variants of latent posterior
learning as described in Section 3.3. We observe from Table 1 that: (i) GSS-FF
comes with no extra training cost. Whilst UViM consumes nearly 2K GPU hours
for training a VQVAE with similar performance achieved [25]. (ii) The randomly
initialized $\beta$ (i.e. GSS-FF-R) leads to considerable degradation on the
least square optimization. (iii) With our maximal distance assumption, the
regularized $\beta$ of GSS-FF brings clear improvement, suggesting the
significance of initialization and its efficacy of our strategy. (iv) With
three-layer conv network with activation function for $\mathcal{X}^{-1}$, GSS-
FT achieves good reconstruction. Further equipping with a two-layer Shifted
Window Transformer block [31] for $\mathcal{X}^{-1}$ (i.e. GSS-FT-W) leads to
the best result at a cost of extra 329.5 GPU hours. This is due to more
accurate translation from predicted maskige to segmentation mask. (v)
Interestingly, with automatic $\beta$ optimization, GSS-TF brings no benefit
over GSS-FF. (vi) Further, joint optimization of both $\mathcal{X}$ and
$\mathcal{X}^{-1}$ (i.e. GSS-TT) fails to achieve the best performance. (vii)
In conclusion, GSS-FF is most efficient with reasonable accuracy, whilst GSS-
FT-W is strongest with good efficiency.

VQVAE design We examine the effect of VQVAE in the context of maskige. We
compare three designs: (1) UViM-style [25]: Using images as auxiliary input to
reconstruct a segmentation mask in form of $K$-channels ($K$ is the class
number) in a ViT architecture. In this no maskige case, the size of
segmentation mask may vary across different datasets, leading to a need for
dataset-specific training. This scheme is thus more expensive in compute. (2)
VQGAN-style [15]: Using a CNN model for reconstructing natural images (maskige
needed for segmentation mask reconstruction) or $K$-channel segmentation masks
(no maskige case) separately, both optimized in generative adversarial
training manner with a smaller codebook. (3) DALL$\cdot$E-style [40]: The one
we adopt, as discussed earlier. We observe from Table 2 that: (i) Due to the
need for dataset specific training, UViM-style is indeed more costly than the
others. This issue can be well mitigated by our maskige with the first stage
training cost compressed dramatically, as evidenced by DALL$\cdot$E-style and
VQGAN-style. Further, the inferiority of UViM over DALL$\cdot$E suggests that
our maskige is a favored strategy than feeding image as auxiliary input. (ii)
In conclusion, using our maskige and DALL$\cdot$E pretrained VQVAE yields the
best performance in terms of both accuracy and efficiency.

Latent prior learning We ablate the second training stage for learning latent
joint prior. The baseline is GSS-FF without the unlabeled area auxiliary and
Multi-Level Aggregation (MLA, including a 2-layer Swin block [31]), under 1/8
downsample ratio. We observe from Table 3 that: (i) Our unlabeled area
auxiliary boosts the accuracy by $3.1\%$, suggesting the importance of
complete labeling which however is extremely costly in semantic segmentation.
(ii) Increasing the discrete mask representation resolution is slightly
useful. (iii) The MLA plays another important role, e.g. giving a gain of
$2.3\%$.

—Image

—————-Ground Truth

————–VQGAN [15]

————–UViM† [25]

————-GSS (ours)

Figure 5: Qualitative results of maskige reconstruction on ADE20K [59]
dataset. Note that the black areas in the Ground Truth correspond to unlabeled
regions, and thus no impact on mIoU measurement.

———–Image

————Ground Truth

———Prediction

————-Image

————Ground Truth

———Prediction——-

Figure 6: Qualitative results of semantic segmentation on Cityscapes [11] and
ADE20K [59] datasets.

###  4.3 Single-domain semantic segmentation

We compare our GSS with prior art discriminative methods and the latest
generative model (UViM [25], a replicated version for semantic segmentation
task). We report the results in Table 4 for Cityscapes [11] and Table 5 for
ADE20K [59]. _(i) In comparison to discriminative methods_: Our GSS yields
competitive performance with either Transformers (Swin) or CNNs (e.g.
ResNet-101). For example, under the same setting, GSS matches the result of
Maskformer [9]. Also, GSS-FT-W is competitive to the Transformer-based SETR
[58] on both datasets. _(ii) In comparison to generative methods_: GSS-FF
surpasses UViM [25] by a large margin whilst enjoying higher training
efficiency. Specifically, UViM takes 1,900 TPU-v3 hours for the first training
stage and 900 TPU-v3 hours for the second stage. While the first stage takes
only 329.5 GPU hours with GSS-FT-W, and zero time with GSS-FF. The second
stage of GSS-FF requires approximately 680 GPU hours. This achievement is due
to our maskige mechanism for enabling the use of pretrained data
representation and a series of novel designs for joint probability
distribution modeling.

###  4.4 Cross-domain semantic segmentation

We evaluate cross-domain zero-shot benchmark [26]. We compare the proposed GSS
with MSeg [26], a domain generalization algorithm (CCSA) [36] and a multi-task
learning algorithm (MGDA) [42]. We test both HRNet-W48 [44] and Swin-Large
[31] as backbone. As shown in Table 6, our GSS is superior to all competitors
using either backbone. This suggests that generative learning could achieve
more domain-generic representation than conventional discriminative learning
counterparts.

#### Domain generic maskige

Being independent to the visual appearance of images, maskige is intrinsically
domain generic. To evaluate this, we transfer the maskige from MSeg to
Cityscapes. As shown in Table 7, GSS can still achieve 79.5 mIoU (1% drop). In
comparison, image representation transfer would double the performance
decrease.

###  4.5 Qualitative evaluation

We evaluate the first-stage reconstruction quality of our GSS, UViM [25] and
VQGAN [15]. As shown in Figure 5, GSS produces almost error-free
reconstruction with clear and precise edges, and UViM fails to recognize some
small objects while yielding distorted segmentation. VQGAN [15] achieves
better classification accuracy but produces more ambiguous edge segmentation.
As shown in Figure 6, GSS produces fine edge segmentation for interior
furniture divisions on ADE20K [59] and accurately segments distant pedestrians
and slender poles on Cityscapes [11].

##  5 Conclusion

In this paper, we have presented a Generative Semantic Segmentation (GSS)
approach. Casting semantic segmentation as an image-conditioned mask
generation problem, our formulation is drastically distinctive to conventional
discriminative learning based alternatives. This is established on a novel
notion of maskige and an efficient optimization algorithm in two stages: (i)
Learning the posterior distribution of the latent variables for segmentation
mask reconstruction, and (ii) minimizing the distance between posterior
distribution and the prior distribution of latent variables for enabling input
images to be conditioned on. Extensive experiments on standard benchmarks
demonstrate that our GSS achieves competitive performance in comparison to
prior art discriminative counterparts, whilst establishing new state of the
art in the more challenging cross-domain evaluation setting.

#### Acknowledgments

This work was supported in part by National Natural Science Foundation of
China (Grant No. 62106050), Lingang Laboratory (Grant No. LG-QS-202202-07) and
Natural Science Foundation of Shanghai (Grant No. 22ZR1407500).

## References

  * [1] Gabriel J Brostow, Julien Fauqueur, and Roberto Cipolla.  Semantic object classes in video: A high-definition ground truth database.  Pattern Recognition Letters, 2009.
  * [2] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari.  Coco-stuff: Thing and stuff classes in context.  In CVPR, 2018.
  * [3] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Semantic image segmentation with deep convolutional nets and fully connected crfs.  arXiv preprint, 2014.
  * [4] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint, 2017.
  * [5] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In ECCV, 2018.
  * [6] Ting Chen, Lala Li, Saurabh Saxena, Geoffrey Hinton, and David J Fleet.  A generalist framework for panoptic segmentation of images and videos.  arXiv preprint, 2022.
  * [7] Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Geoffrey Hinton.  Pix2seq: A language modeling framework for object detection.  In ICLR, 2021.
  * [8] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  In CVPR, 2022.
  * [9] Bowen Cheng, Alex Schwing, and Alexander Kirillov.  Per-pixel classification is not all you need for semantic segmentation.  In NeurIPS, 2021.
  * [10] MMSegmentation Contributors.  Openmmlab semantic segmentation toolbox and benchmark.  https://github.com/open-mmlab/mmsegmentation, 2020.
  * [11] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In CVPR, 2016.
  * [12] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In CVPR, 2017.
  * [13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [14] Patrick Esser, Robin Rombach, and Bjorn Ommer.  A disentangling invertible interpretation network for explaining latent representations.  In CVPR, 2020.
  * [15] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In CVPR, 2021.
  * [16] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  IJCV, 2010.
  * [17] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu.  Dual attention network for scene segmentation.  In CVPR, 2019.
  * [18] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  The International Journal of Robotics Research, 2013.
  * [19] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 2020.
  * [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [21] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu.  Ccnet: Criss-cross attention for semantic segmentation.  In ICCV, 2019.
  * [22] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In CVPR, 2017.
  * [23] Ge-Peng Ji, Keren Fu, Zhe Wu, Deng-Ping Fan, Jianbing Shen, and Ling Shao.  Full-duplex strategy for video object segmentation.  In ICCV, 2021.
  * [24] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint, 2013.
  * [25] Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby.  Uvim: A unified modeling approach for vision with learned guiding codes.  arXiv preprint, 2022.
  * [26] John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun.  Mseg: A composite dataset for multi-domain semantic segmentation.  In CVPR, 2020.
  * [27] Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther.  Autoencoding beyond pixels using a learned similarity metric.  In ICML, 2016.
  * [28] Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler.  Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization.  In CVPR, 2021.
  * [29] Chen Liang, Wenguan Wang, Jiaxu Miao, and Yi Yang.  Gmmseg: Gaussian mixture based generative semantic segmentation models.  In NeurIPS, 2022.
  * [30] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [31] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, 2021.
  * [32] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In CVPR, 2015.
  * [33] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi.  Unified-io: A unified model for vision, language, and multi-modal tasks.  arXiv preprint, 2022.
  * [34] Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang.  Soft: softmax-free transformer with linear complexity.  In NeurIPS, 2021.
  * [35] Chris J Maddison, Andriy Mnih, and Yee Whye Teh.  The concrete distribution: A continuous relaxation of discrete random variables.  arXiv preprint, 2016.
  * [36] Saeid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto.  Unified deep supervised domain adaptation and generalization.  In ICCV, 2017.
  * [37] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille.  The role of context for object detection and semantic segmentation in the wild.  In CVPR, 2014.
  * [38] Kevin P Murphy.  Machine learning: a probabilistic perspective.  MIT press, 2012.
  * [39] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder.  The mapillary vistas dataset for semantic understanding of street scenes.  In ICCV, 2017.
  * [40] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In ICML, 2021.
  * [41] Robin Rombach, Patrick Esser, and Björn Ommer.  Making sense of cnns: Interpreting deep representations and their invariances with inns.  In ECCV, 2020.
  * [42] Ozan Sener and Vladlen Koltun.  Multi-task learning as multi-objective optimization.  NeurIPS, 2018.
  * [43] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao.  Sun rgb-d: A rgb-d scene understanding benchmark suite.  In CVPR, 2015.
  * [44] Ke Sun, Yang Zhao, Borui Jiang, Tianheng Cheng, Bin Xiao, Dong Liu, Yadong Mu, Xinggang Wang, Wenyu Liu, and Jingdong Wang.  High-resolution representations for labeling pixels and regions.  arXiv preprint, 2019.
  * [45] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  NeurIPS, 2017.
  * [46] Girish Varma, Anbumani Subramanian, Anoop Namboodiri, Manmohan Chandraker, and CV Jawahar.  Idd: A dataset for exploring problems of autonomous navigation in unconstrained environments.  In WACV, 2019.
  * [47] Qiang Wan, Zilong Huang, Jiachen Lu, YU Gang, and Li Zhang.  Seaformer: Squeeze-enhanced axial transformer for mobile semantic segmentation.  In ICLR.
  * [48] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  In CVPR, 2021.
  * [49] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In CVPR, 2018.
  * [50] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun.  Unified perceptual parsing for scene understanding.  In ECCV, 2018.
  * [51] Zhisheng Xiao, Qing Yan, and Yali Amit.  Generative latent flow.  arXiv preprint, 2019.
  * [52] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  In NeurIPS, 2021.
  * [53] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell.  Bdd100k: A diverse driving dataset for heterogeneous multitask learning.  In CVPR, 2020.
  * [54] Yuhui Yuan, Xilin Chen, and Jingdong Wang.  Object-contextual representations for semantic segmentation.  In ECCV, 2020.
  * [55] Oliver Zendel, Katrin Honauer, Markus Murschitz, Daniel Steininger, and Gustavo Fernandez Dominguez.  Wilddash-creating hazard-aware benchmarks.  In ECCV, 2018.
  * [56] Li Zhang, Dan Xu, Anurag Arnab, and Philip HS Torr.  Dynamic graph message passing networks.  In CVPR, 2020.
  * [57] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In CVPR, 2017.
  * [58] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In CVPR, 2021.
  * [59] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Semantic understanding of scenes through the ade20k dataset.  IJCV, 2019.

##  Appendix A Proofs

###  A.1 Derivation of GSS ELBO

We provide the proof of Eq. (2) in the main paper here. We rewrite the log-
likelihood of semantic segmentation $\log{p(c|x)}$ by introducing a discrete
L-dimension latent distribution $q(z|c)$ (with $z\in\mathbb{Z}^{L}$).

$\displaystyle\log{p(c|x)}$ $\displaystyle=\log{\int p(c,z|x)}\text{\ d}z$
$\displaystyle=\log{\int p(c,z|x)}\frac{q(z|c)}{q(z|c)}\text{\ d}z$
$\displaystyle=\log\mathbb{E}_{q(z|c)}\left[\frac{p(c,z|x)}{q(z|c)}\right]$
$\displaystyle\geq\mathbb{E}_{q(z|c)}\left[\log\frac{p(c,z|x)}{q(z|c)}\right]$
$\displaystyle\left(\text{as}\right.-log(\cdot)$ is convex, by Jensen’s
Inequality: $\displaystyle f(\sum_{i}\lambda_{i}x_{i})$
$\displaystyle\leq\sum_{i}\lambda_{i}f(x_{i})\text{, where }\lambda_{i}\geq
0,\sum_{i}\lambda_{i}=1\left.\right)$
$\displaystyle=\mathbb{E}_{q(z|c)}\left[\log\frac{p(c|z)p(z|x)}{q(z|c)}\right]$
$\displaystyle=\mathbb{E}_{q(z|c)}\left[\log
p(c|z)\right]+\mathbb{E}_{q(z|c)}\left[\log\frac{p(z|x)}{q(z|c)}\right]$
$\displaystyle=\mathbb{E}_{q(z|c)}$ $\displaystyle\left[\log
p(c|z)\right]-D_{KL}(q(z|c),p(z|x))$
$\displaystyle=\mathbb{E}_{q_{\phi}(z|c)}$ $\displaystyle\left[\log
p_{\theta}(c|z)\right]-D_{KL}(q_{\phi}(z|c),p_{\psi}(z|x)).$

Different from ELBO [24] in VAE, the latent variable we introduce here is
$q(z|c)$, rather than $q(z)$ to solve the conditioned mask generation problem.

###  A.2 Derivation of latent posterior learning

We provide the proof of Eq. (4) in the main paper here. As stated in main
paper, the first stage latent posterior training is conducted by a MSE loss

$\min_{\theta,\phi}\sum_{c}\mathbb{E}_{q_{\phi}(z|c)}\|p_{\theta}(c|z)-c\|.$

Let us denote $\hat{c}=p_{\theta}(c|z)$ is the reconstructed mask. Then, we
define a linear transform $x^{(c)}=\mathcal{X}_{\beta}(c)=c\beta$, where
$\beta\in\mathbb{R}^{K\times 3}$ and an arbitrary inverse transform
$\hat{c}=\mathcal{X}^{-1}_{\gamma}(\hat{x}^{(c)})$. Noted that the parameter
$\gamma$ can be non-linear. $x^{(c)}$ is called maskige and $\hat{x}^{(c)}$ is
the reconstructed maskige produced by the maskige decoder
$\hat{x}^{(c)}=\mathcal{D}_{{\theta}}(\hat{z})$. The transformed latent
parameter $\hat{z}$ preserves the probability for the linear transformation,

Figure 7:  An illustration of our transformed objective. Rec. stands for
reconstruction.

$q_{\hat{\phi}}(\hat{z}|x^{(c)})=q_{\hat{\phi}}(\hat{z}|c\beta)=q_{\phi}(z|c).$
(8)

Then, we have

$\displaystyle\min_{\theta,\phi}\sum_{c}\mathbb{E}_{q_{\phi}(z|c)}2\|\hat{c}-c\|$
$\displaystyle=$
$\displaystyle\min_{\theta,\phi,\gamma}\sum_{c}\mathbb{E}_{q_{\phi}(z|c)}\left[\|\mathcal{X}^{-1}(\hat{x}^{(c)})-c\|+\|\hat{c}-c\|\right].$

For the first term, since
$\mathcal{X}^{-1}(\hat{x}^{(c)})=\mathcal{X}^{-1}(\mathcal{D}_{{\theta}}(\hat{z}))$
which is not related to $\theta$ and $\phi$. Therefore, we have

$\displaystyle\min_{\theta,\phi,\gamma}\sum_{c}\mathbb{E}_{q_{\phi}(z|c)}\|\mathcal{X}^{-1}(\hat{x}^{(c)})-c\|$
$\displaystyle=$
$\displaystyle\min_{\gamma}\sum_{c}\mathbb{E}_{q_{\phi}(z|c)}\|\mathcal{X}^{-1}(\mathcal{D}_{\theta}(\hat{z}))-c\|$
$\displaystyle=$
$\displaystyle\min_{\gamma,\beta}\sum_{c}\mathbb{E}_{q_{\hat{\phi}}(\hat{z}|x^{(c)})}\|\mathcal{X}^{-1}(\mathcal{D}_{{\theta}}(\hat{z}))-c\|\
(\text{Eq.~{}\eqref{equ:qhatz}}).$ (9)

Figure 8:  Encoder-decoder style architecture of GSS. “Pred.” stands for
prediction.

For the second term, $\hat{c}=\mathcal{D}_{\theta}(z)$ is not related to
$\gamma$, which can be rewritten as

$\displaystyle\min_{\theta,\phi}\sum_{c}\mathbb{E}_{q_{\phi}(z|c)}\|\hat{c}-c\|$
$\displaystyle=\min_{\theta,\phi,s.t.\|\beta\|=1}\sum_{c}\mathbb{E}_{q_{\phi}(z|c)}\|\hat{c}-c\|\|\beta\|$
$\displaystyle=\min_{\theta,\phi,s.t.\|\beta\|=1}\sum_{c}\mathbb{E}_{q_{\phi}(z|c)}\|\hat{c}\beta-c\beta\|$
$\displaystyle=\min_{\theta,\phi,s.t.\|\beta\|=1}\sum_{c}\mathbb{E}_{q_{\phi}(z|c)}\|(\hat{c}\beta-\hat{x}^{(c)})\
(\text{equal to 0 by def.})$
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad+(\hat{x}^{(c)}-x^{(c)})$
$\displaystyle\quad\quad\quad\quad\quad\quad\quad\quad+(x^{(c)}-c\beta)\|\
(\text{equal to 0 by def.})$
$\displaystyle=\min_{\theta,\phi,s.t.\|\beta\|=1}\sum_{x^{(c)}}\mathbb{E}_{q_{\phi}(z|c)}\|\hat{x}^{(c)}-x^{(c)}\|\
(\text{not related to }\theta)$
$\displaystyle=\min_{{\theta},\phi,s.t.\|\beta\|=1}\sum_{x^{(c)}}\mathbb{E}_{q_{\phi}(z|c)}\|\mathcal{D}_{{\theta}}(\hat{z})-x^{(c)}\|$
$\displaystyle=\min_{{\theta},\hat{\phi},\beta,s.t.\|\beta\|=1}\sum_{x^{(c)}}\mathbb{E}_{q_{\hat{\phi}}(\hat{z}|x^{(c)})}\|\mathcal{D}_{{\theta}}(\hat{z})-x^{(c)}\|\
(\text{Eq.~{}\eqref{equ:qhatz}}).$

Combining Eq. (9) and Eq. (A.2), our final objective is

$\displaystyle\min_{\hat{\phi},{\theta},\beta,s.t.\|\beta\|=1}\sum_{x^{(c)}}\mathbb{E}_{q_{\hat{\phi}}(\hat{z}|x^{(c)})}\|\mathcal{D}_{{\theta}}(\hat{z})-x^{(c)}\|$
$\displaystyle+$
$\displaystyle\quad\min_{\gamma,\beta}\sum_{c}\mathbb{E}_{q_{\hat{\phi}}(\hat{z}|x^{(c)})}\|\mathcal{X}^{-1}(\mathcal{D}_{{\theta}}(\hat{z}))-c\|.$
(10)

For the first term, it is a VQVAE [45] reconstruction objective for maskige.
Therefore, a VQVAE pretrained by DALL$\cdot$E [40] with a large-scale
OpenImage dataset can readily offer a good lower bound for the first term.

As such, only the second term is left for optimization. We can optimize the
$\gamma$ with gradient descent, corresponding to GSS-FT&TT. Besides, we can
solve this problem more efficiently by a linear assumption, i.e.
$\mathcal{X}^{-1}(\hat{x}^{(c)})=\hat{x}^{(c)}\gamma$ where
$\gamma\in\mathbb{R}^{3\times K}$. We denote the $\hat{X}^{(c)}$ is a matrix
with each row an reconstructed maskige and $C$ is a matrix with each row an
input mask. We solve the optimization with least square error

$\displaystyle\|\mathcal{X}^{-1}(\hat{X}^{(c)})-C\|^{2}$ $\displaystyle=$
$\displaystyle\|\hat{X}^{(c)}\gamma-C\|^{2}$ $\displaystyle=$
$\displaystyle\|(\hat{X}^{(c)}-X^{(c)}+X^{(c)})\gamma-C\|^{2}$
$\displaystyle\leq$
$\displaystyle\left(\|\hat{X}^{(c)}-C\beta\|\|\gamma\|+\|X^{(c)}\gamma-C\|\right)^{2}.$
(11)

The optimization over both $\beta$ and $\gamma$ is non-convex (as shown by the
poor performance with GSS-TT), so we optimize them sequentially in GSS-
FF&FT&TF. For GSS-FF, we use a hand-crafted optimized $\beta$.

$\displaystyle\left(\|\hat{X}^{(c)}-C\beta\|\|\gamma\|+\|X^{(c)}\gamma-C\|\right)^{2}$
$\displaystyle\leq$
$\displaystyle\left(\tau\|\gamma\|+\|X^{(c)}\gamma-C\|\right)^{2}$
$\displaystyle=$
$\displaystyle\left(\tau\|\gamma\|+\|C\beta\gamma-C\|\right)^{2}$
$\displaystyle\leq$
$\displaystyle\left(\tau\|\gamma\|+\|C\|\|\beta\gamma-\mathbb{1}\|\right)^{2}.$
(12)

where $\tau=\|\hat{X}^{(c)}-C\beta\|$ is bounded and unrelated to $\gamma$ by
provided VQVAE and $\beta$. Our objective then changes to minimize the upper
bound.

$\displaystyle\min_{\gamma,s.t.\|\gamma\|=1}\text{RSS}(\gamma)$
$\displaystyle=\min_{\gamma,s.t.\|\gamma\|=1}\|\beta\gamma-\mathbb{1}\|^{2}$
$\displaystyle=\min_{\gamma,s.t.\|\gamma\|=1}(\beta\gamma-\mathbb{1})^{\top}(\beta\gamma-\mathbb{1}).$
(13)

We take the derivative of Eq. (13), then

$\frac{\partial\text{RSS}}{\partial\gamma}=2\beta^{\top}(\beta\gamma-\mathbb{1})=0.$
(14)

The unique solution of Eq. (14) is

$\displaystyle\beta^{\top}\beta\gamma$ $\displaystyle=\beta^{\top}$
$\displaystyle(\Rightarrow)\quad(\beta^{\top}\beta)^{-1}(\beta^{\top}\beta)\gamma$
$\displaystyle=(\beta^{\top}\beta)^{-1}\beta^{\top}$
$\displaystyle(\Rightarrow)\quad\quad\quad\quad\quad\quad\quad\ \ \gamma$
$\displaystyle=(\beta^{\top}\beta)^{-1}\beta^{\top}.$ (15)

For the special design GSS-TF, we use a cascaded optimization to automatically
optimize $\beta$ and $\gamma$.

$\displaystyle\beta_{t+1}$
$\displaystyle=\mathop{\arg\min}\limits_{\beta}\left(\|\hat{X}^{(c)}-C\beta\|\|\gamma_{t}\|+\|C\beta\gamma_{t}-C\|\right)^{2}$
$\displaystyle\gamma_{t+1}$
$\displaystyle=(\beta_{t+1}^{\top}\beta_{t+1})^{-1}\beta_{t+1}^{\top}.$

The $\beta_{t+1}$ is optimized by one mini-batch step of gradient descent.

Method | Iteration |  VOC [16] |  Context [37] |  CamVid [1] |  WildDash [55] |  KITTI [18] |  ScanNet [12] | h. mean
---|---|---|---|---|---|---|---|---
_\- Discriminative modeling:_ |  |  |  |  |  |  |  |
CCSA [36] | 500k | 48.9 | - | 52.4 | 36.0 | - | 27.0 | 39.7
MGDA [42] | 500k | 69.4 | - | 57.5 | 39.9 | - | 33.5 | 46.1
MSeg-w/o relabel [26] | 500k | 70.2 | 42.7 | 82.0 | 62.7 | 65.5 | 43.2 | 57.6
MSeg [26] | 500k | 70.7 | 42.7 | 83.3 | 62.0 | 67.0 | 48.2 | 59.2
MSeg-480p [26] | 1,500k | 76.4 | 45.9 | 81.2 | 62.7 | 68.2 | 49.5 | 61.2
MSeg-720p [26] | 1,500k | 74.7 | 44.0 | 83.5 | 60.4 | 67.9 | 47.7 | 59.8
MSeg-1080p [26] | 1,500k | 72.0 | 44.0 | 84.5 | 59.9 | 66.5 | 49.5 | 59.8
_\- Generative modeling:_ |  |  |  |  |  |  |  |
GSS-FF (Ours) | 160k | 78.7 | 45.8 | 74.2 | 61.8 | 65.4 | 46.9 | 59.5
GSS-FT-W (Ours) | 160k | 79.5 | 47.7 | 75.9 | 65.3 | 68.0 | 49.7 | 61.9
Table 8: Additional cross-domain semantic segmentation performance on MSeg
dataset test split [26]: We add performance of Mseg-480p, Mseg-720p and
Mseg-1080p [26] results to Table 6 of the main paper. No improved versions of
our methods are included. “480p”, “720p” and “1080p” mean all test images are
resized to 480p (the shorter side is 480 pixel), 720p and 1080p, respectively,
when MSeg model inference.

##  Appendix B Maskige optimization designs

As illustrated above, $\beta$ is a linear projection applied on the ground-
truth mask, i.e. $x^{(c)}=c\beta$. As is shown in Eq. (11) and Eq. (12), the
quality of $\|\hat{X}^{(c)}-C\beta\|$ directly affects the quality of the
following $\gamma$ optimization, so the optimization of $\beta$ will affect
the difficulty degree of our learning of $\mathcal{X}_{\gamma}^{-1}$. The
cascaded optimization in GSS-TF (84.37% reconstruction mIoU) provides an
upper-bound for $\beta$ optimization. However, the gradient descent
optimization costs another $5$ GPU hours according to Table 1 of main paper.
Therefore, we propose a hand-crafted optimization of $\beta$ in GSS-FF&FT&FT-W
that achieves satisfying performance without requiring extra training time,
based on the _maximal distance assumption_.

To understand this assumption, we can consider the linear projection parameter
$\beta\in\mathbb{R}^{K\times 3}$ as a colorization process, where each
category is assigned an rgb color. The idea behind the maximal distance
assumption is to maximize the color difference between the encoding of each
category. For instance, if two different categories are assigned similar
colors, the model may struggle to differentiate between them. Therefore, by
maximizing the distance between color embeddings, we can improve the model’s
ability to distinguish between categories. We interpret the parameter $\beta$
as R, G, B color sequences $\mathcal{A}^{r},\mathcal{A}^{g},\mathcal{A}^{b}$
assigned to each category. To better satisfy the _maximal distance assumption_
, we will try different ways to construct these sequences, i.e.., assigning
colors to each category.

_(i) Arithmetic sequence on R/G/B channels_: Designing three arithmetic
sequences $\mathcal{A}^{r},\mathcal{A}^{g},\mathcal{A}^{b}$ for R/G/B channels
respectively. Then we have

$\mathcal{A}^{m}=\left\\{a_{1}^{m},a_{2}^{m},\dots,a_{i}^{m},\dots,a_{n}^{m}\right\\},m\in\left\\{r,g,b\right\\}.$
(16)

For the $i$-th color value,

$a_{i}^{m}=a_{1}^{m}+(i-1)\cdot k^{m},k^{m}\in N^{+},$ (17)

where color channel $m\in\left\\{r,g,b\right\\}$, the interval of arithmetic
sequence $k^{m}$ can be difference between channels, $a_{1}^{m}$ default is
$0$. The set of colors is the Cartesian product of these three series,

$\mathcal{C}=\mathcal{A}^{r}\times\mathcal{A}^{g}\times\mathcal{A}^{b}.$ (18)

E.g., if the interval of R, G, B channel $k=45$, the color set $\mathcal{C}$
will be $\left\\{(0,0,0)\right.$, $(0,0,45),\dots,$
$\left.(225,225,225)\right\\}$.

_(ii) Misalignment start points_: The original starting point of the
arithmetic sequence is 0, 0, 0 for R/G/B respectively. In order to avoid
duplication of values, we let R/G/B have different starting points,

$a_{1}^{r}\neq a_{1}^{g}\neq a_{1}^{b}.$ (19)

In practice, we simply set to $a_{1}^{r}=0,a_{1}^{g}=1,a_{1}^{b}=2$.

_(iii) Random additive factors_: Adding three independent random factors
$t\in[0,T]$ on the R/G/B arithmetic sequence respectively, to avoid repetition
of several same values,

$a_{i}^{m}=a_{1}^{m}+(i-1)\cdot k^{m}+t_{i}^{m}.$ (20)

E.g., a color sequence with random additive factors: $\left\\{(1,7,3)\right.$,
$(4,2,45)$, $\dots$, $\left.(235,215,232)\right\\}$. In practical terms, $T$
is set to $15$.

_(iv) Category-specific refinement_: We equip the lower IoU categories with
values where the R/G/B values vary at large degrees (e.g., we replace
$(128,128,128)$ with $(0,128,255)$). In addition, we keep the color away from
gray as possible, because gray is located in the center of the color space,
thus being close to many categories and giving rise to a harder learning
problem. Such an category-specific refinement allows each category to be
possibly furthest from the others as possible.

Colorization technique | mIoU | aAcc
---|---|---
Arithmetic sequence | 85.99 | 94.37
\+ Misalignment start points | 86.12 | 94.45
\+ Random additive factors | 87.42 | 95.08
\+ Category-specific refinement | 87.73 | 95.29
Table 9:  Ablation on _maximal distance assumption_ : The maskige
reconstruction performance (mIoU and aAcc) of GSS-FF on ADE20K val split under
different Mask-to-maskige transformations $\mathcal{X}$.

#### Results

As shown in Table 9, it is evident that the colorization design for maskige
generation presents a good amount of impact on the reconstruction performance.
In particular, the last design category-specific refinement yields the best
results, conforming our intuition and design consideration.

#### Visualization

For visual understanding, in Figure 13 and Figure 14 we visualize the 150
colors corresponding to all the categories of ADE20K [59] generated by the
_maximal distance assumption_ (hand-designed) and gradient descent
optimization (learned), respectively. We observe that the hand-designed method
produces the colors with enhanced contrast and greater vibrancy. Instead, the
colors learned are vibrant for the more frequent categories and relatively
dark for the less frequent categories.

##  Appendix C Overall architecture

Following [15, 40], the modeling of latent prior learning is formulated by an
encoder-decoder architecture (See Figure 8). For the image encoder
$\mathcal{I}_{\psi}$, we take the advantage of hierarchical shifted window
transformer [31] for extracting the multi-scale information [48] and memory
efficiency [34]. This is different from UViM [25], which uses a single-scale
and full-range Transformer as the encoder. To implement the image encoder, we
use the Swin-Large architecture [31], pre-trained on ImageNet-22K [13], as the
backbone. As shown in Figure 8, we use four-scale feature maps ($1/4$, $1/8$,
$1/16$, $1/32$) and upsample all the lower-resolution features to $1/4$ scale,
then concatenate four features across the channel [52]. The multi-level
aggregation consists of an MLP and $D$ layers of hierarchical shifted-window
Transformer [31], with the swin window size set to 7, the number of attention
heads to 16, the embedding dimension to 512, and the FFN dimension to 1024.
For the implementation version with resnet as the backbone, $D=6$. However,
for models with strong Swin Transformer backbones, fewer MLA layers are
needed, and thus $D=2$. We implement the maskige decoder
$\mathcal{D}_{\theta}$ as a fixed VQVAE decoder [45].

##  Appendix D More training details

_(i) Latent posterior learning_: As illustrated before, the latent posterior
learning is simplified as:

$\min_{\mathcal{X}^{-1}}\mathbb{E}_{q_{\hat{\phi}}(\hat{z}|\mathcal{X}(c))}\|\mathcal{X}^{-1}(\hat{x}^{(c)})-c\|.$
(21)

The target can be interpreted as minimizing the distance between a ground-
truth segmentation mask and the predicted mask. Following [25, 6], we use
cross-entropy loss instead of euclidean distance for a better minimization
between segmentation masks.

_(ii) Latent posterior learning for $\mathcal{X}$_: For GSS variants whose
$\mathcal{X}$ dose not require training, such as GSS-FF&FT&FT-W, we assign a
3-channel encoding to each category directly based on the maximum distance
assumption. For the GSS variants that require training, including GSS-TF&TF&,
we freeze the parameters of the VQVAE of the DALL-E pretrain and train
$\mathcal{X}$ for 4,000 iterations using SGD optimizer with a batch size of
16. By the way, the training process of GSS-TT also optimizes the
$\mathcal{X}^{-1}$ function.

_(iii) Latent posterior learning for $\mathcal{X}^{-1}$_: We propose a method
for training an $\mathcal{X}^{-1}$ that is more robust to noise (used in GSS-
FT&FT-W). We found that training $\mathcal{X}^{-1}$ with noisy maskige helps
it learn to be more robust. To provide noisy maskige, we can use the trained
$\mathcal{I}_{\psi}$, DALL$\cdot$E pre-trained $\mathcal{D}_{\theta}$, and
$\mathcal{X}^{-1}$ to directly predict noisy maskige predictions. In practice,
we use $\mathcal{I}_{\psi}$ that trained up to the middle checkpoint (e.g.,
32,000 iterations) or final checkpoint in latent prior learning.
$\mathcal{X}^{-1}$ is trained using cross-entropy loss and optimized with
AdamW, with a batch size of 16. We trained GSS-FT-W for 40,000 iterations,
while GSS-FT was trained for only 3,000 iterations due to its fast
convergence. The non-linear function $\mathcal{X}^{-1}$ is implemented using
either a convolutional or Swin block structure. Specifically, for GSS-FT, the
structure comprises two conv $1\times 1$ layers enclosing a conv $3\times 3$
layer. However, this approach is superseded by GSS-FT-W, the final model,
which employs a group of Swin blocks with a number of heads of 4, a Swin
window size of 7, and an embedding channel of 128 to realize
$\mathcal{X}^{-1}$. Regardless of the specific implementation,
$\mathcal{X}^{-1}$ relies on local RGB information in the predicted mask to
deduce the category of each pixel.

_(iv) Latent prior learning_: For the optimization of $\mathcal{I}_{\psi}$, we
use the AdamW optimizer and implement a polynomial learning rate decay
schedule[57] with a minimum learning rate of $0.0$. We set the initial
learning rate to $1.5\times 10^{-3}$ for Cityscapes and $1.2\times 10^{-4}$
for ADE20K and Mseg.

##  Appendix E Domain generic maskige and image encoder

We did two tests by deriving a general maskige on MSeg [26].

(i) As shown in Table 7, we applied our general maskige to the Cityscapes
dataset and achieved a mIoU score of 79.5, which is only slightly lower than
the mIoU score of 80.5 obtained using the Cityspaces specific maskige. This
result demonstrates the versatility of our maskige across different datasets.

(ii) To further evaluate the effectiveness of our domain-generic approach, we
shared the image encoder $\mathcal{I}_{\psi}$ between MSeg and Cityscapes and
trained our model on the training split of MSeg. We then evaluated the model
on the zero-shot test split consisting of 6 unseen datasets. As shown in Table
6, our GSS outperforms other state-of-the-art methods on the MSeg dataset.
These experiments demonstrate that our maskige is domain-generic and has the
potential for open-world settings.

##  Appendix F Additional quantitative results

We additionally compare the improved versions of MSeg [26] with 1,500k longer
training on the cross-domain benchmark. As shown in Table 8, despite using
short training, our model still achieves better performance. This verifies the
advantage of our method in terms of training efficiency, in addition to the
accuracy.

##  Appendix G Additional qualitative results

For further qualitative evaluation, we visualize the prediction results of our
GSS on both single-domain segmentation datasets [11, 59] and cross-domain
segmentation dataset [26].

As shown in Figure 9, our GSS has an accurate perception of buses, trucks and
pedestrians in distance, whilst also splitting the dense and slim poles. In
Figure 10, we see that GSS correctly recognises a wide range of furniture such
as curtains, cabinets, murals, doors and toilets; This suggests that our
maskige generative approach can accurately represent a wide range of semantic
entities. Figure 11 and Figure 12 show the cross-domain segmentation
performance on images from previously unseen domains (Mseg test datasets). It
can be seen that GSS performs well in all five datasets in the MSeg test split
[26], further validating that our generative algorithm has strong cross-domain
generalization capabilities.

##  Appendix H Reproduced semantic segmentation version of UViM [25]

We reproduce UViM with mmsegmentation and follow the hyperparameter and
structure in the paper [25]. To achieve a fair comparison with our approach,
we have made some modifications: (i) we implement Swin-Large [31] pretrained
on ImageNet 22K [13] as the Language model $LM$ as ours; (ii) we generate the
Guiding code straightforwardly in a non-autoregressive manner; (iii) we
trained 80k iterations in the first stage of UViM [25] and 160k iterations in
the second stage. These modifications are necessary to ensure a fair
comparison.

##  Appendix I Societal impact

Given that the strong cross-domain generalization capability, we consider our
model has the potential to be used in a wide range of visual scenarios. This
is desired in practical applications due to the benefits of reducing the
demands of per-domain model training and easier deployment and system
management. This is meaningful and advantageous in both economics and
environment. On the other hand, there exist the potential to spawn abuse of
our algorithm and unexpected undesirable uses. Therefore, it is necessary to
strengthen the regulation and supervision of algorithm applications, in order
to guarantee that new algorithms including ours can be used safely and
responsibly for the good of the humanity and society.

##  Appendix J Limitations and Future Work

While our study represents a significant step forward for generative
segmentation, our models still fall short of the performance achieved by top
discriminative models. One contributing factor is that decision boundaries for
generative models are often less precise than those of discriminative models,
resulting in less accurate object edges in segmentation. Another drawback is
that generative models require larger amounts of data to achieve good
performance, because discriminative models only learn decision boundaries,
while generative models need to learn the distribution of the entire sample
space. In our experiments, the performance of MSeg is better compared to
Cityscapes and ADE20K, which roughly indicates this point.

Additionally, since we convert all categories to colors, the color space is
limited, and as the number of categories increases, the colors become more
crowded. This can lead to confusion when using $\mathcal{X}^{-1}$ to query and
predict the closest pre-defined color for each category from maskige,
especially near object edges. Therefore, it is worth trying to expand this
space to higher dimensions.

Looking ahead, there are several avenues for future research in generative
semantic segmentation. One promising direction is instance-level segmentation,
which would enable more precise identification and separation of individual
objects within an image. Additionally, we believe that it would be valuable to
explore a unified model that can perform multiple vision tasks, such as
segmentation, 2D object detection, depth prediction, 3D detection, and more.

Given that the second stage training of GSS focuses on latent prior learning,
new vision tasks could be inclusively added by incorporating a new posterior
distribution of latent variables, without requiring any changes to the model
architecture. By pursuing these directions, we believe that significant
advances can be made in the field of generative semantic segmentation.

—Image

————————————Ground Truth

———————————-Prediction

Figure 9: Qualitative results of semantic segmentation on Cityscapes val split
[11].

——Image

————————————–Ground Truth

——————————–Prediction

Figure 10: Qualitative results of semantic segmentation on ADE20K val split
[59].

———Image

————-Ground Truth

———Prediction

—————Image

———–Ground Truth

———–Prediction——-

Figure 11: Qualitative results of semantic segmentation on MSeg test datasets
[26]. From top to bottom: Pascal VOC [16], Pascal Context [37], ScanNet-20
[12], CamVid [1] and WildDash [55] (the last two rows).

Figure 12: Qualitative results of semantic segmentation on MSeg test dataset
[26] (KITTI dataset [18]). The $1^{st}$ row is input image, the $2^{rd}$ row
is Ground Truth, and the $3^{rd}$ row is prediction result. Figure 13:
Visualization of maskige for each category in ADE20K [59] dataset under
_maximal distance assumption_.  Figure 14:  Visualization of maskige for each
category in ADE20K [59] dataset under gradient descent optimization.