# Distortion-Aware Network Pruning and
Feature Reuse for Real-time Video Segmentation

Hyunsu Rhee1, Dongchan Min1, Sunil Hwang1,
Bruno Andreis1, Sung Ju Hwang1,2
KAIST1, AITRICS2, South Korea
{ryanrhee, alsehdcks95, sunilhoho, andries<EMAIL_ADDRESS>

###### Abstract

Real-time video segmentation is a crucial task for many real-world
applications such as autonomous driving and robot control. Since state-of-the-
art semantic segmentation models are often too heavy for real-time
applications despite their impressive performance, researchers have proposed
lightweight architectures with speed-accuracy trade-offs, achieving real-time
speed at the expense of reduced accuracy. In this paper, we propose a novel
framework to speed up any architecture with skip-connections for real-time
vision tasks by exploiting the temporal locality in videos. Specifically, at
the arrival of each frame, we transform the features from the previous frame
to reuse them at specific spatial bins. We then perform partial computation of
the backbone network on the regions of the current frame that captures
temporal differences between the current and previous frame. This is done by
dynamically dropping out residual blocks using a gating mechanism which
decides which blocks to drop based on inter-frame distortion. We validate our
Spatial-Temporal Mask Generator (STMG) on video semantic segmentation
benchmarks with multiple backbone networks, and show that our method largely
speeds up inference with minimal loss of accuracy.

##  1 Introduction

Semantic segmentation [50, 51, 48, 4, 26, 8, 17] is a fundamental task in
computer vision that is crucial for many real-world applications including
autonomous driving, robot control, augmented reality, surveillance system,
aerial imagery, drone image analysis, and medical diagnosis [38, 6, 53, 2, 1,
34, 23]. Recently, much focus has been placed on developing segmentation
models for streaming videos. This is a challenging task that requires not only
high accuracy but also real-time inference speed [18, 42]. However, majority
of semantic segmentation models still rely on computationally heavy backbone
networks [13] to achieve state-of-the-art performance. These heavy backbones,
in most cases, become computational bottlenecks in the real-time video
semantic segmentation pipeline.

Figure 1: Conceptual Overview. Our proposed Spatial-Temporal Mask Generator
(STMG) captures temporally evolving regions (highlighted by the red box)
between two consecutive frames using spatial masks. This allows us to reuse
the computed features from the previous frames for static regions.
Additionally, conditioned on the distortion between the two frames, we
generate a block pruning mask that determines which residual blocks to drop
when running the segmentation model on the current frame. The transformed
features from the previous frame are then combined with the output of the
partial network computation using a feature aggregation module (FAM).

One approach to tackle this challenge is to identify the key frames which are
processed with heavy backbone networks and to propagate the pre-computed
features of key frames for other frames [54, 21, 44, 55, 47] in the video
stream. However, this approach is based on the assumption that flow estimation
and feature propagation are faster than running the backbone networks [54].
Since state-of-the-art lightweight architectures achieve real-time speed, this
approach is not applicable to modern segmentation models for real-time speed
up. Further, the approach sacrifices the segmentation accuracy of non-key
frames even with accurate optical flow-based feature propagation, thus
maintaining the accuracy of non-key frames in this pipeline still remains a
problem to be addressed. Hu et al. [18] exploit temporal information in videos
to tackle this problem via a temporally distributed network architecture.
However, this strategy is suboptimal in that the sub-networks are designed
independently of individual frames with no consideration of temporal locality
across frames.

To tackle these issues, we propose a novel method to achieve practical speed-
ups for real-time video semantic segmentation networks, exploiting spatial-
temporal locality to create input-dependent sub-networks that preserves
accuracy on both key and non-key frames at inference time. Specifically, we
propose an input-dependent block-wise pruning mechanism to obtain inference
time dynamic sub-networks tailored to each frame by exploiting the resiliency
of residual networks to the removal of residual blocks at inference time due
to their skip-connections. That is, we learn a masking mechanism that decides
which residual blocks to drop and which to keep, conditioned on a given input
frame and spatial-temporal information across previous frames.

While the partial computation of residual blocks produces real-time speed-ups,
it still suffers from the speed-accuracy trade-off problem. To compensate for
this, we further exploit the temporal locality of video frames as depicted in
Figure 1. In particular, we transform the features from previous frames for
reuse in the static regions across adjacent frames. We do this by generating
spatial masks that capture the distortion between adjacent frames. However in
most video semantic segmentation datasets such as Cityscapes [6], not all
frames are labelled hence we utilize knowledge distillation by employing a
strong image segmentation model as a teacher network to train the spatial mask
generator.

Figure 2: Performance on Cityscapes. Our proposed method, Spatial-Temporal
Mask Generator, provides a better trade-off between mIoU and FPS.

In addition, to avoid information loss due to repeated partial computations
for sequential frames, we introduce a distortion-aware scheduling policy which
determines non-key frames to apply the proposed model. Specifically, we
consider frames with distortions greater than some set threshold as key
frames, and invoke the full network to compute their features. Crucially, when
a frame is determined to be a non-key frame, we transform the feature from the
previous frame, and perform partial computation of the backbone network only
in spatial regions that change from the previous frame. The resulting model,
Spatial-Temporal Mask Generator (STMG) achieves a better speed-accuracy trade-
offs compared to semantic and video semantic segmentation models as shown in
Figure 2. Our contributions are three fold:

  * •

We propose a framework to speed up the inference of residual backbone networks
for real-time video semantic segmentation by performing partial network
evaluation and reusing temporally consistent features across frames.

  * •

We propose a method to capture spatial-temporal information across frames
based on inter-frame distortion without heavy optical flow computations.

  * •

We validate our method on two benchmarks against multiple state-of-the-art
real-time video segmentation models and show that our method can speed up
models with residual backbones at inference time with marginal loss of
accuracy.

Figure 3: The overall process for video segmentation. Our framework exploits
spatial-temporal properties by extracting features from images of adjacent
frames and utilizing them to compute both spatial mask and block pruning mask.
In detail, a 2D mask is generated by calculating the cosine similarity between
two features and utilized as a ratio to blend the features pixel-wisely
(spatial mask). These spatial-temporal properties are transferred to learn the
block dropping behavior by providing the extracted features to generate the
block pruning mask by concatenation (block pruning mask).

##  2 Related Work

#### Semantic Segmentation

While the literature on semantic segmentation is vast, we only describe few
recent works that are relevant to ours. Deep convolutional neural networks
have been widely used in semantic segmentation tasks by adopting a data-driven
approach to improve accuracy. By providing an effective global contextual
prior and fusing local and global context features, the networks can learn
details and semantics simultaneously [50]. With urban scene properties where
each row of the image has different statistics in terms of category
distribution, each row of the feature map can be individually more focused on
a specific channel using a height-driven attention map [5]. Explicitly
exploiting visual dependency relations, such as intra-class, inter-class and
global dependence among semantic entities, networks can improve the
generalization ability [31].

#### Real-time Semantic Segmentation

Modern approaches to real-time semantic segmentation usually adopt a strategy
of reducing the size of an input image provided to the residual backbone, as
residual networks are considered to be bottlenecks in the overall computation
[51]. Reducing the size of an input image generally degrades spatial
resolution, which leads to a degradation of performance. Hence recent
approaches adopt a strategy of designing a dual-resolution path consisting of
a spatial path and a context path [48, 49]. The spatial path with a high-
resolution feature adopts a small stride to preserve the spatial information
while a context path with a low-resolution feature adopts a fast downsampling
strategy to obtain sufficient receptive field. Bilateral connections between
dual resolutions are continuously repeated as the network deepens for
efficient information fusion to deliver the state-of-the-art performance of
the latest models [17].

#### Video Semantic Segmentation

Zhu et al. [54] and Li et al. [28] introduce scheduling policies to identify
key frames to apply heavy neural networks and propagate the feature across
multiple frames sequentially. That is, features are reused and this reduces
the computational requirements on non-key frames. Adaptive scheduling policies
that select key frames based on changing video dynamics instead of heuristic
policies have been recently proposed in Xu et al. [47], Wang et al. [44] and
He et al. [12]. However, these policies are still suboptimal in that the
candidates for sub-networks are limited to a subset of image segmentation
network with feature propagation module which are pre-defined before the
training stages resulting in a non-dynamic neural network architecture.

#### Dynamic Network Pruning

Residual networks [13] are composed of residual blocks and skip connections
which enable them to behave like ensembles of relatively shallow networks
[41]. These networks have been shown to be resilient to layer dropping,
usually with minimal loss of accuracy. In Wang et al. [43] and Wu et al. [46],
this resilience to layer dropping is exploited using reinforcement learning to
prevent over-parameterization in residual networks. Li et al. [27] exploits
contextual information to develop dynamic routing methods that adapt to the
scale of the input image for the semantic segmentation task. Furthermore, He
et al. [14] focuses on utilizing spatial information for channel-wise pruning
[15, 29] of the semantic segmentation networks. However, for video semantic
segmentation, such methods that exploit spatial information are insufficient
since they do not consider the temporal nature of videos. Hence, to fully
utilize contextual information in the video semantic segmentation task, the
spatial-temporal properties have to be taken into account.

#### Dependent Beta-Bernoulli Process

In latent feature model, the Indian buffet process [11] defines a prior
distribution on the binary feature indicators with potentially infinite number
of features. Dependent Indian buffet process and its finite-dimensional beta-
Bernoulli approximation [45, 52, 37] extend the Indian buffet process (IBP) to
incorporate input covariates as follows:

$\displaystyle\varphi_{k}\sim p(\varphi_{k}),\quad
z_{n,k}|\varphi_{k},\mathbf{x}_{n}\sim Ber(g(\varphi_{k},\mathbf{x}_{n})),$
(1)

where $g$ is an arbitrary function mapping $\varphi_{k}$ and $\mathbf{x}_{n}$
to a probability. The input covariates $\mathbf{x}_{n}$ corresponds to the
inputs of a certain layer and dropout probabilities are adjusted according to
$\mathbf{x}_{n}$ in the dependent beta-Bernoulli dropout.

##  3 Method

In this section, we first introduce our dynamic block pruning mechanism based
on spatial-temporal locality. We then describe a spatial-temporal mask
generator for feature transformation and reuse based on the distortion between
adjacent frames. The overall framework consists of two networks, the
segmentation network and the mask generator. Note that the proposed method is
model-agnostic and applicable to any semantic segmentation model with a
residual network.

###  3.1 Input-dependent Block Pruning

In this section, we describe the input-dependent block pruning mechanism. Let
$\widehat{\mathbf{W}}$ be the parameters of a residual neural network with
$\mathbf{x}_{n}$ as input and let $\mathbf{z}_{n}\in\\{0,1\\}^{K}$ be a binary
mask vector generated for the input $\mathbf{x}_{n}$ where $K$ is the number
of prunable residual blocks. Note that the first block in each residual layer
is not prunable. Conditioned on the input $\mathbf{x}_{n}$, we define the
input-dependent residual block pruning probability
$\bm{\varphi}(\mathbf{x}_{n})\in[0,1]^{K}$ for $\mathbf{x}_{n}$ and generate a
binary mask vector $\mathbf{z}_{n}$ for each block as follows:

$z_{n,k}\sim Ber(\varphi_{k}(\mathbf{x}_{n}))\text{ for }k=1,\dots,K.$ (2)

That is, given a residual network with $K$ prunable blocks, we model the block
pruning probability of each individual block as a Bernoulli random process
follows:

$\psi(\mathbf{u}|\mathbf{x}_{n})=\prod_{k=1}^{K}\varphi_{k}(\mathbf{x}_{n})^{u_{k}}(1-\varphi_{k}(\mathbf{x}_{n}))^{1-u_{k}},$
(3)

where $u_{k}\in\\{0,1\\}$ indicates whether to prune or keep the $k$-th
residual block.

In this paper, we model the transformation process from images into pruning
probabilities with a Bayesian approach by inducing sparsity with the batch
normalization [20] layer of the pruning model inspired by Lee et al. [25]. To
obtain a sparse pruning mask, we impose a sparsity inducing prior on
$\varphi_{k}(\mathbf{x}_{n})$ and independently compute the residual block
pruning probabilities as follows:

$\varphi_{k}(\mathbf{x}_{n},\beta_{k})=\mathrm{clamp}\bigg{(}\gamma_{k}\frac{g_{k}(\mathbf{x}_{n})-\mu_{k}}{\sigma_{k}}+\beta_{k},\tau\bigg{)},$
(4)

where $g_{k}(\mathbf{x}_{n})$ denotes a transformation of $\mathbf{x}_{n}$
using the non-linear function $g$ for the $k$-th residual block, $\mu_{k}$ and
$\sigma_{k}$ are the estimates of the $k$-th component’s mean and standard
deviation of the transformed inputs, $\gamma_{k}$ and $\beta_{k}$ are scaling
and shifting parameters, and $\tau>0$ is some tolerance to prevent overflow,
which is set as $1e^{-10}$ while
$\mathrm{clamp}(x,\tau)=\mathrm{min}(1-\tau,\mathrm{max}(\tau,x))$.

Figure 4: Illustration of input-dependent block pruning. We visualize how the
extracted features $\bf t$ and $\bf e$ are utilized by STMG for generating the
residual block pruning probabilities.

During the optimization phase, $z_{n,k}$ is sampled from the relaxed Bernoulli
distribution. Specifically, by passing $\bm{\delta}_{uc}\in\mathbb{R}^{K}$ to
a softplus function and sampling
$\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ to provide a variability
during the optimization stage, the sparsity inducing prior on
$\bm{\varphi}(\mathbf{x}_{n})$, which is
$\mathcal{N}(\bm{\beta}_{p},\rho\mathbf{I})$, and the trainable parameters of
$\bm{\beta}$ are defined as follows:

$\bm{\beta}\sim\mathcal{N}(\bm{\beta}_{bn},\bm{\delta}),\,\,\bm{\beta}_{p}\sim\mathcal{N}(\bm{\beta}_{p},\rho\mathbf{I}),$
(5)

while $\bm{\beta}_{bn}$ is obtained with batch normalization layer and
$\delta_{k}=\epsilon_{k}\cdot\mathrm{softplus}(\delta_{uc,k})$ is obtained
with $\bm{\delta}_{uc}$ and $\bm{\epsilon}$ during the optimization phase.

That is, $D_{KL}[\bm{\beta}\|\bm{\beta}_{p}]$, the KL divergence loss for
sparsity inducing prior on $\bm{\varphi}(\mathbf{x}_{n})$ is defined as
follows:

$D_{KL}[\bm{\beta}\|\bm{\beta}_{p}]=\log\frac{\rho}{\delta}+\frac{\delta^{2}+(\bm{\beta}_{p}-\bm{\beta}_{bn})^{2}}{2\rho^{2}},$
(6)

where $\rho$ and $\bm{\beta}_{p}$ are constants for sparsity inducing prior
and $\delta$ and $\bm{\beta}_{bn}$ are trainable parameters of the pruning
mechanism. An illustration of the pruning method is depicted in Figure 4.

###  3.2 Spatial-Temporal Mask Generator

We describe the Spatial-Temporal Mask Generator (STMG) which generates two
types of mask: a block pruning mask and a spatial mask, based on spatial-
temporal information. An overview of the mask generator is depicted in Figure
3.

#### Block Pruning Mask

Based on the input-dependent block pruning method previously described, we
develop a block mask generator for prunable residual blocks. Specifically, the
mask generator takes as input two adjacent frames, ${{\bf I}_{pre}}$ and
${{\bf I}_{cur}}$, and processes each frame using a feature extractor,
$Enc_{f}$, with shared weights across all frames. We then concatenate and pass
the outputs of the feature extractor to another convolution layer to generate
a pruning mask with the same dimensionality as the number of residual blocks
in the backbone network as follows:

${\bf t}=Enc_{f}({{\bf I}_{pre}}),\;\,{\bf e}=Enc_{f}({{\bf
I}_{cur}}),\>\,{\bf m_{b}}={\bf{H}}({\bf t}\oplus{\bf e}),$ (7)

where $\bf t$ and $\bf e$ are extracted features from the previous and current
frame, $\bf m_{b}$ denotes a block pruning mask and $\bf H$ denotes overall
transformation of the input in the pruning mechanism.

#### Spatial Mask

In addition, to compensate for information loss due to the partial computation
of residual blocks, we propose a spatial masking scheme which exploits the
temporal locality of videos. Specifically, we generate a spatial mask, ${\bf
m_{sp}}$, which predicts unchanged regions between adjacent frames together
with the block pruning mask. By applying the spatial mask on the features of
the previous frame, we transform the feature from the previous frame for reuse
in the current frame. This is done by computing the cosine similarity between
the outputs of the feature extractor from the previous frame and current frame
denoted as $\bf t$ and $\bf e$ as follows:

$\textit{m}_{sp,i}({\bf t},{\bf e})=-0.5\times cos(t_{i},e_{i})+0.5,$ (8)

where $\textit{m}_{sp,i}$ denotes a spatial mask of the $i$-th pixel and we
scale the value to be in the range of $[0,1]$.

#### Knowledge Distillation

However, the cosine similarity between ${\bf t}$ and ${\bf e}$ from adjacent
frames alone is not sufficient for encoding meaningful spatial-temporal
information. To further improve the quality of the features, we add an
auxiliary loss to match the generated spatial mask with the ground-truth
distortion map. Since not all frames in the dataset are labeled, we utilize
knowledge distillation [16] to generate ground-truth distortion map as shown
in Figure 6 (b). Specifically, we generate the ground-truth distortion map by
subtracting the segmentation maps obtained by applying a strong image
segmentation model on two adjacent frames. Following Deng et al. [7], the loss
function for spatial mask consists of the binary cross-entropy loss and dice
loss, as follows:

$L_{recon}({\bf m_{sp}},{\bf n})=L_{bce}({\bf m_{sp}},{\bf n})+L_{dice}({\bf
m_{sp}},{\bf n}),$ (9)

where $\bf n$ denotes the ground-truth distortion map obtained by knowledge
distillation. The dice loss alleviates the class-imbalance problem, as only
partial spatial areas are distorted between adjacent frames. The dice loss,
$L_{dice}$, is computed as follows:

$L_{dice}=1-\frac{2\sum{m_{sp,i}n_{i}}+\kappa}{\sum{m_{sp,i}^{2}}+\sum{n_{i}^{2}}+\kappa},$
(10)

where $\kappa$ is a smoothing value set as 1 to avoid zero division.
$L_{dice}$ is the cornerstone for generating crisp edges while optimizing only
the cross-entropy loss leads to relatively coarse results.

#### Feature Aggregation

Using the generated spatial mask, we aggregate features of the segmentation
model from the previous frame and current frame as follows:

$\pi^{\prime}({{\bf I}_{cur}})={\bf m_{sp}}\cdot\pi({{\bf
I}_{cur}})+(\mathbf{I}-{\bf m_{sp}})\cdot\pi({{\bf I}_{pre}}),$ (11)

where $\pi(\bf{\cdot})$ is the output of the backbone segmentation network. In
particular, we intend the network to focus more on features for distorted
regions, while reusing features from previous frames for static regions.
Specifically, we pass the aggregated feature $\pi^{\prime}({{\bf I}_{cur}})$
to the segmentation head module to obtain the semantic segmentation map for
the current frame.

Algorithm 1 Distortion-aware scheduling

Input: Frames
$\textbf{V}=\\{\textbf{I}_{0},\textbf{I}_{1},\dots,\textbf{I}_{n}\\}$.
Output: Video segmentation maps
$\textbf{S}=\\{\textbf{S}_{0},\textbf{S}_{1},\dots,\textbf{S}_{n}\\}$.

$\textbf{f}_{0}\leftarrow$ Key Frame

$\sigma\leftarrow|\bf{m_{sp;1}}|=\text{distortion}(\textbf{I}_{1},\textbf{I}_{0})$

for i in range(1, n) do

if $\left|\bf m_{sp;i}\right|>\sigma$ then

$\textbf{f}_{i}\leftarrow$ Key Frame

$\sigma\leftarrow\gamma_{1;i}*|\bf m_{sp;i}|$

else

$\textbf{f}_{i}\leftarrow$ Non-Key Frame

$\sigma\leftarrow\gamma_{2;i}*|\bf m_{sp;i}|$ $(\gamma_{2;1}=1)$

end if

end for

Figure 5: (Left): Illustration of distortion-aware key frame selection. At the
arrival of a new frame, we first calculate the distortion of adjacent frames
to determine key and non-key frames. (Right): Pseudo-code for distortion-aware
scheduling policy. For good balance between key and non-key frames, we update
the threshold value scaled by a factor $\gamma$.

###  3.3 Distortion-Aware Scheduling Policy

While we compensate for information loss by reusing the features from previous
frames, error in the features propagate as time progresses. Hence, we propose
a scheduling policy that applies the proposed module only to specific non-key
frames. We do this by capturing the amount of distortion using the spatial
mask for each frame. Specifically, if the distortion for a given frame is
greater than some set threshold, we consider it as a key frame and compute the
full network on the frame. However, if the distortion is lower than the
threshold, we consider the frame a non-key frame and transform the features
from the previous frame for reuse as well as perform partial computation of
the residual blocks using the generated input-dependent mask.

#### Scaling Factor

After each computation, we update the threshold value with the distortion
amount of the current frame scaled by a factor $\gamma$. To prevent the full
network from being applied to more than one frame in a row, we assign a value
of 2 to $\gamma_{1}$ and update the threshold value with a multiplicative
factor of $\gamma_{1}$ right after a key frame so that the next frame can be
determined as a non-key frame. Additionally, to avoid situations where partial
computations are repeated over sequential frames, we assign a value of 0.95 to
$\gamma_{2}$ right after a non-key frame and update the threshold value with
the current distortion factor multiplied by $\gamma_{2}$ so that the next
frame can be slightly induced as a key frame. Most video semantic segmentation
frameworks adopt a strategy of propagating features across multiple frames
sequentially, however this require a heavy optical flow computation since
errors propagate as time progresses. Since our method targets real-time speed-
up without optical flow computation, we adopt a strategy of frequently
switching between key and non-key frames with proposed scaling rules.

These scaling rules provide a good balance between key and non-key frames
while simultaneously capturing video dynamics and leveraging spatial-temporal
locality. An illustration of this scheduling policy together with pseudo-code
is provided in Figure 5.

Table 1: Performance comparison with real-time (> 30 FPS) semantic
segmentation models on Cityscapes [6].

Model | mIoU | FPS | FLOPs | Params
---|---|---|---|---
PP-LiteSeg-T1 [36] | 73.1 | 273.6 | - | -
SwiftNetRN-18 [35] | 75.4 | 39.9 | 104.0G | 11.8M
SFNet-DF2 [26] | 75.8 | 61.0 | 48.5G | 10.5M
BiSeNetV2-L [49] | 75.8 | 47.3 | 118.5G | -
HyperSeg [33] | 76.2 | 36.9 | 7.5G | 10.1M
CABiNet [24] | 76.6 | 76.5 | 12.0G | 2.6M
STDC2-Seg75 [8] | 77.0 | 97.0 | - | -
RegSeg [10] | 78.1 | 30.0 | 39.1G | 3.3M
DDRNet-39 [17] | 80.3 | 22.1 | 261.8G | 32.4M
\- STMG-DDR39 | 79.1 | 32.6 | 175.2G | 22.9M
DDRNet-23-Slim [17] | 77.6 | 108.2 | 33.8G | 5.7M
\- STMG-DDR23-Slim | 76.2 | 116.0 | 27.2G | 4.9M

Table 2: Performance comparison with video semantic segmentation models on
Cityscapes [6].

Model | mIoU | FPS | Max Latency
---|---|---|---
DVSNet [47] | 63.2 | 30.3 | -
CLK [39] | 64.4 | 6.3 | 198 (ms)
DFF [54] | 69.2 | 6.4 | 575 (ms)
GRFP(5) [32] | 73.6 | 3.9 | 255 (ms)
TD4-Bise18 [18] | 75.0 | 47.6 | 21 (ms)
FANet18 [19] | 75.5 | 72.0 | 14 (ms)
PEARL [22] | 76.5 | 1.3 | 800 (ms)
LVS [28] | 76.8 | 5.8 | 380 (ms)
TD2-PSP50 [18] | 79.9 | 5.6 | 178 (ms)
TMANet50 [42] | 80.3 | 2.0 | 500 (ms)
NetWarp [9] | 80.6 | 0.3 | 3004 (ms)
STMG-DDR39 | 79.1 | 32.6 | 46 (ms)
STMG-DDR23-Slim | 76.2 | 116.0 | 10 (ms)

##  4 Experiments

###  4.1 Setup and Implementation

#### Dataset

We evaluate our method on Cityscapes [6]. Cityscapes is a dataset for urban-
scene parsing consisting of 5,000 images of urban street scenes with 19
classes. The dataset contains 2,975 finely annotated images for training, 500
images for validation, and 1,525 images for testing. We evaluate our method
based on the standard semantic segmentation evaluation metrics: mean class-
wise intersection over union (mIoU), and Frames Per Second (FPS). Cityscapes
is made up of snippets each consisting of 30 frames with the 20th frame of
each snippet annotated. Hence, we train and evaluate our model on the 20th
frame of each snippet with the image from the previous frame given. We also
evaluate our method on CamVid [3] which is a road-scene parsing dataset. The
results for the CamVid experiments are reported in Section A of the
supplementary file.

#### Models and Baselines

We select two state-of-the-art real-time semantic segmentation models for our
experiments: DDRNet-39 and DDRNet-23-Slim. Since DDRNet models have residual
blocks both in the high and low resolution branches, we apply our dynamic
block pruning mechanism in both branches. To preserve the output dimension of
each residual layer, we do not drop the first residual block in each layer.
This is because dropping the first block in each layer will not allow forward
propagation of the features in residual networks. Specifically, there are a
total of 6 and 17 prunable residual blocks in DDRNet-23-Slim and DDRNet-39
respectively.

#### Implementation

To train our models, we use mini-batch stochastic gradient descent (SGD) with
weight decay of 0.0005 and the momentum of 0.9. We use a batch size of 16 for
training on the Cityscapes dataset. We set the initial learning rate as 0.01
and train the model for 1,000 epochs for DDRNet-23-Slim and 484 epochs for
DDRNet-39. We also apply random cropping for data augmentation with crop size
of 1024 × 1024. We implement all models and conduct all experiments using
PyTorch. We train each model with two GeForce RTX 3090 GPUs for DDRNet-39 base
models, and two GeForce RTX 2080 Ti GPUs for DDRNet-23-Slim base models to
reproduce the results.

#### Inference Speed Measurement

We conduct inference speed measurement on a single RTX 2080 Ti. Following the
evaluation procedure of Si et al. [40], Orsic et al. [35], and Hong et al.
[17], we remove the batch normalization layers after each convolutional layer
during the speed measurement. Specifically, we use the protocols established
by Chen et al. [4] for a fair comparison. Since not all frames are annotated
in Cityscapes, we evaluate the speed of our model by sampling a portion of the
entire video sequence to include both key and non-key frames in the annotated
frames. Further, we report the maximum latency of our model along with the
speed of our model as video semantic segmentation models have varying latency
between key and non-key frames. We compare the speed of our model on two
semantic segmentation backbone architectures: DDR-39 and DDR-23-Slim.

Figure 6: Qualitative results of (a) our method and (b) visualization of the
spatial mask. Black and white represent static and distorted regions,
respectively. During the training phase, a distortion map is provided to STMG
to better capture the spatial-temporal locality between adjacent frames. The
degree of sparsity refers to the pruning ratio among prunable residual blocks
in the backbone.

Table 3: Ablation study on spatial and block pruning mask. $m_{sp}$ denotes a
blending ratio of previous and current features while $m_{avg}$ is the average
value of $\bf m_{sp}$ in the corresponding STMG.

Pruning and Masking Methodology | mIoU | FLOPs | Params
---|---|---|---
Random Block Pruning (w/o ft) | 71.40 $\pm$ 0.14 | 175.6 $\pm$ 0.4G | 22.8M
Random Spatial Masking | 78.06 $\pm$ 0.02 | 175.2G | 22.9M
Uniform Blending ($m_{sp}=m_{avg}$) | 78.69 | 175.2G | 22.9M
Uniform Blending ($m_{sp}=0.5$) | 78.25 | 175.2G | 22.9M
Ours (Spatial-Temporal Masking) | 79.09 | 175.2G | 22.9M

Table 4: Ablation study on distortion-aware scheduling policy. $R$ denotes a
key frame duration length for the fixed policy and $\gamma$ denotes a
threshold scaling factor for the distortion-aware policy.

Scheduling Policy | mIoU | FLOPs | Params
---|---|---|---
Distortion-Aware Scheduling ($\gamma=0.95$) | 79.09 | 175.2G | 22.9M
Distortion-Aware Scheduling ($\gamma=0.9$) | 79.43 | 187.6G | 24.2M
Fixed Scheduling ($R=2$) |  79.73 $\pm$ 0.03 | 189.9G | 24.5M

Table 5: FPS gain results. For a fair comparison, we select and compare the
model and framework whose FPS of the base network is closest to the base
network of ours.

Framework (Model) | FPS gain | mIoU drop
---|---|---
TD4-Bise18 (Bise34) [18] | 28.57% | 1.32%
STMG-DDR39 (DDR39) |  47.51% | 1.47%

Table 6: Ablation study on FAM. For a fair comparison, we use the same
sparsity inducing prior $\beta_{p}$ and the same KL scale factor for per-frame
block-wise pruning inspired by dependent beta-Bernoulli dropout [25].

Model | mIoU | Drop | FPS | Gain
---|---|---|---|---
DDRNet-39 | 80.27 | - | 22.1 | -
\- Per-Frame Pruning | 76.31 | 4.93% | 41.7 | 88.7%
\- Ours (STMG-DDR39) | 79.09 | 1.47% | 32.6 | 47.5%

###  4.2 Results

In Table 3.3, we report the experimental results on Cityscapes for the
DDRNet-39 backbone network. The results show that our method reaches 32.6 FPS
which is a 47.5% gain over the base network which runs at 22.1 FPS on high
resolution images. The segmentation outputs are shown in Figure 6 (a) and in
Section C of the supplementary file. We report similar results for the
DDRNet-23-Slim architecture in Table 3.3. As shown in Table 3.3, our method
reaches 116.0 FPS with 7.2% FPS gain compared to the base network. In both
architectures, we increase the FPS with less than 2% drop in accuracy. That
is, STMG provides a better trade-off between accuracy and speed compared to
state-of-the-art models. We further visualized this speed-accuracy trade-off
in Figure 2.

#### Degree of Distortion and Sparsity

We visualize the generated spatial masks in Figure 6 (b). We observe that the
mask generator successfully encodes meaningful spatial-temporal information
and captures the distorted regions such as the edge movement of people or
bicycles. As shown in Figure 6 (b), each example shows different degrees of
sparsity and the results show a strong correlation between sparsity and degree
of distortion. To further improve the encoding of spatial-temporal
information, we add a distortion bias $\eta$ to DBB [25]-inspired block
pruning mechanism as follows:

$\varphi_{k}(\mathbf{x}_{n},\beta_{k})=\mathrm{clamp}\bigg{(}\gamma_{k}\frac{g_{k}(\mathbf{x}_{n})+\eta-\mu_{k}}{\sigma_{k}}+\beta_{k},\tau\bigg{)},\;\;\eta=-\overline{m_{sp,i}}+0.5,$
(12)

where $\varphi_{k}(\cdot)$ denotes a block pruning probability of the $k$-th
residual block. The resulting pruning mechanism, distrotion bias-based DBB
(dbb-DBB) achieves a stronger correlation between sparsity and degree of
distortion as shown in Figure 8. The comparison results are reported in Table
4.2.

#### Distortion-Aware Scheduling Policy

As shown in Figure 7, the adaptive scheduling policy successfully determines
key and non-key frames by utilizing the distortion between adjacent frames.
Diverse dynamic key frame paths are generated which shows that the scheduling
policy does not generate a fixed configuration of key and non-key frames. For
example, the 3rd row shows a repeated non-key frame as the distortion over
consecutive frames are small, and the 5th row shows repeated switching between
key and non-key frame when the distortion across frames becomes large.

Figure 7: Qualitative results of distortion-aware scheduling policy. Frames
highlighted in red represent key frames and other frames represent non-key
frames. The third row shows that for static video streams, the scheduling
policy continues to perform partial computations on non-key frames.

Table 7: Performance on Cityscapes over the DDRNet-39. "+dbb-DBB" represents a
pruning mechanism using the proposed distortion bias-based DBB (dbb-DBB). STMG
uses a default DBB [25]-inspired pruning.

Model | mIoU | FLOPs | Params
---|---|---|---
STMG | 79.1 | 175.2G | 22.9M
STMG + dbb-DBB | 79.3 | 188.7G | 24.8M

Figure 8: Correlation between the degree of distortion estimated by the
spatial mask generator and pruning sparsity in dbb-DBB.

#### Ablation Studies

We conduct ablation studies to examine the effectiveness of each component in
our framework. As described in Table 4.1, we conduct an ablation study on
spatial masking and block pruning. The results show that our proposed spatial
masking trained with knowledge distillation outperforms the model that uses
random or uniform blending with features of previous frame. The masking method
differs during the inference time and the identical trained weights are used
during the ablation study. Further, our DBB [25]-inspired pruning mechanism
outperforms the random pruning at the same computational cost. As described in
Table 4.1, the results show that our distortion-aware scheduling policy
achieves lower FLOPs and parameters compared to the fixed scheduling policy.
In Table 4.1, we conduct an ablation study on our proposed feature aggregation
module (FAM), and we observe that the information from the previous frame can
compensate for the information loss due to the dropped blocks in the backbone
for the current frame.

We discuss the limitations and societal impacts of our work in the
supplementary file, in Section E.

##  5 Conclusion

In this work, we proposed an efficient framework for real-time video semantic
segmentation that exploits the spatial-temporal locality of videos. Our
framework reuses the features from previous frames, and perform partial
evaluation of the backbone network by taking into account the differences
between two consecutive frames. This partial network evaluation is done using
an input-dependent gating mechanism that decides which features and blocks to
prune. Our framework is model-agnostic and can be applied to any semantic
segmentation models with residual backbone networks to speed up their
inference. We validated our method on two benchmark datasets with different
semantic segmentation models and showed that it significantly improves
inference-time speed at the expense of marginal drop in accuracy compared to
baselines approaches that achieve lower inference speed-ups with similar
decreases in model accuracy.

Acknowledgements
This research was supported by Institute of Information & communications
Technology Planning & Evaluation (IITP) grant funded by the Korea
government(MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School
Program(KAIST)), and Service Development and Demonstration on DNA Drone
Technologies Program through the National Research Foundation of Korea(NRF)
funded by the Ministry of Science and ICT(1711119697).

## References

  * Azimi et al. [2019] Seyed Majid Azimi, Corentin Henry, Lars Sommer, Arne Schumann, and Eleonora Vig.  Skyscapes fine-grained semantic understanding of aerial scenes.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7393–7403, 2019.
  * Balloch et al. [2018] Jonathan C Balloch, Varun Agrawal, Irfan Essa, and Sonia Chernova.  Unbiasing semantic segmentation for robot perception using synthetic data feature transfer.  _arXiv preprint arXiv:1809.03676_ , 2018.
  * Brostow et al. [2009] Gabriel J Brostow, Julien Fauqueur, and Roberto Cipolla.  Semantic object classes in video: A high-definition ground truth database.  _Pattern Recognition Letters_ , 30(2):88–97, 2009\.
  * Chen et al. [2019] Wuyang Chen, Xinyu Gong, Xianming Liu, Qian Zhang, Yuan Li, and Zhangyang Wang.  Fasterseg: Searching for faster real-time semantic segmentation.  _arXiv preprint arXiv:1912.10917_ , 2019.
  * Choi et al. [2020] Sungha Choi, Joanne T Kim, and Jaegul Choo.  Cars can’t fly up in the sky: Improving urban-scene segmentation via height-driven attention networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9373–9383, 2020.
  * Cordts et al. [2016] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3213–3223, 2016.
  * Deng et al. [2018] Ruoxi Deng, Chunhua Shen, Shengjun Liu, Huibing Wang, and Xinru Liu.  Learning to predict crisp boundaries.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 562–578, 2018.
  * Fan et al. [2021] Mingyuan Fan, Shenqi Lai, Junshi Huang, Xiaoming Wei, Zhenhua Chai, Junfeng Luo, and Xiaolin Wei.  Rethinking bisenet for real-time semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9716–9725, 2021.
  * Gadde et al. [2017] Raghudeep Gadde, Varun Jampani, and Peter V Gehler.  Semantic video cnns through representation warping.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 4453–4462, 2017.
  * Gao [2021] Roland Gao.  Rethink dilated convolution for real-time semantic segmentation.  _arXiv preprint arXiv:2111.09957_ , 2021.
  * Ghahramani and Griffiths [2005] Zoubin Ghahramani and Thomas Griffiths.  Infinite latent feature models and the indian buffet process.  _Advances in neural information processing systems_ , 18, 2005.
  * He et al. [2020] Fei He, Naiyu Gao, Qiaozhe Li, Senyao Du, Xin Zhao, and Kaiqi Huang.  Temporal context enhanced feature aggregation for video object detection.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 10941–10948, 2020.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * He et al. [2021] Wei He, Meiqing Wu, Mingfu Liang, and Siew-Kei Lam.  Cap: Context-aware pruning for semantic segmentation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 960–969, 2021.
  * He et al. [2017] Yihui He, Xiangyu Zhang, and Jian Sun.  Channel pruning for accelerating very deep neural networks.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1389–1397, 2017.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2(7), 2015.
  * Hong et al. [2021] Yuanduo Hong, Huihui Pan, Weichao Sun, Yisong Jia, et al.  Deep dual-resolution networks for real-time and accurate semantic segmentation of road scenes.  _arXiv preprint arXiv:2101.06085_ , 2021.
  * Hu et al. [2020a] Ping Hu, Fabian Caba, Oliver Wang, Zhe Lin, Stan Sclaroff, and Federico Perazzi.  Temporally distributed networks for fast video semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8818–8827, 2020a.
  * Hu et al. [2020b] Ping Hu, Federico Perazzi, Fabian Caba Heilbron, Oliver Wang, Zhe Lin, Kate Saenko, and Stan Sclaroff.  Real-time semantic segmentation with fast attention.  _IEEE Robotics and Automation Letters_ , 6(1):263–270, 2020b.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International conference on machine learning_ , pages 448–456. PMLR, 2015.
  * Jain et al. [2019] Samvit Jain, Xin Wang, and Joseph E Gonzalez.  Accel: A corrective fusion network for efficient semantic segmentation on video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8866–8875, 2019.
  * Jin et al. [2017] Xiaojie Jin, Xin Li, Huaxin Xiao, Xiaohui Shen, Zhe Lin, Jimei Yang, Yunpeng Chen, Jian Dong, Luoqi Liu, Zequn Jie, et al.  Video scene parsing with predictive feature learning.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 5580–5588, 2017.
  * Jung et al. [2021] Sanghun Jung, Jungsoo Lee, Daehoon Gwak, Sungha Choi, and Jaegul Choo.  Standardized max logits: A simple yet effective approach for identifying unexpected road obstacles in urban-scene segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15425–15434, 2021.
  * Kumaar et al. [2021] Saumya Kumaar, Ye Lyu, Francesco Nex, and Michael Ying Yang.  Cabinet: Efficient context aggregation network for low-latency semantic segmentation.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 13517–13524. IEEE, 2021.
  * Lee et al. [2018] Juho Lee, Saehoon Kim, Jaehong Yoon, Hae Beom Lee, Eunho Yang, and Sung Ju Hwang.  Adaptive network sparsification with dependent variational beta-bernoulli dropout.  _arXiv preprint arXiv:1805.10896_ , 2018.
  * Li et al. [2020a] Xiangtai Li, Ansheng You, Zhen Zhu, Houlong Zhao, Maoke Yang, Kuiyuan Yang, Shaohua Tan, and Yunhai Tong.  Semantic flow for fast and accurate scene parsing.  In _European Conference on Computer Vision_ , pages 775–793. Springer, 2020a.
  * Li et al. [2020b] Yanwei Li, Lin Song, Yukang Chen, Zeming Li, Xiangyu Zhang, Xingang Wang, and Jian Sun.  Learning dynamic routing for semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8553–8562, 2020b.
  * Li et al. [2018] Yule Li, Jianping Shi, and Dahua Lin.  Low-latency video semantic segmentation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 5997–6005, 2018.
  * Lin et al. [2020a] Mingbao Lin, Rongrong Ji, Yuxin Zhang, Baochang Zhang, Yongjian Wu, and Yonghong Tian.  Channel pruning via automatic structure search.  _arXiv preprint arXiv:2001.08565_ , 2020a.
  * Lin et al. [2020b] Peiwen Lin, Peng Sun, Guangliang Cheng, Sirui Xie, Xi Li, and Jianping Shi.  Graph-guided architecture search for real-time semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4203–4212, 2020b.
  * Liu et al. [2021] Mingyuan Liu, Dan Schonfeld, and Wei Tang.  Exploit visual dependency relations for semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9726–9735, 2021.
  * Nilsson and Sminchisescu [2018] David Nilsson and Cristian Sminchisescu.  Semantic video segmentation by gated recurrent flow propagation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6819–6828, 2018.
  * Nirkin et al. [2021] Yuval Nirkin, Lior Wolf, and Tal Hassner.  Hyperseg: Patch-wise hypernetwork for real-time semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4061–4070, 2021.
  * Oh et al. [2020] Dokwan Oh, Daehyun Ji, Cheolhun Jang, Yoonsuk Hyun, Hong S Bae, and Sungju Hwang.  Segmenting 2k-videos at 36.5 fps with 24.3 gflops: Accurate and lightweight realtime semantic segmentation network.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 3153–3160. IEEE, 2020.
  * Orsic et al. [2019] Marin Orsic, Ivan Kreso, Petra Bevandic, and Sinisa Segvic.  In defense of pre-trained imagenet architectures for real-time semantic segmentation of road-driving images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12607–12616, 2019.
  * Peng et al. [2022] Juncai Peng, Yi Liu, Shiyu Tang, Yuying Hao, Lutao Chu, Guowei Chen, Zewu Wu, Zeyu Chen, Zhiliang Yu, Yuning Du, et al.  Pp-liteseg: A superior real-time semantic segmentation model.  _arXiv preprint arXiv:2204.02681_ , 2022.
  * Ren et al. [2011] Lu Ren, Yingjian Wang, Lawrence Carin, and David Dunson.  The kernel beta process.  _Advances in Neural Information Processing Systems_ , 24:963–971, 2011.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical image computing and computer-assisted intervention_ , pages 234–241. Springer, 2015.
  * Shelhamer et al. [2016] Evan Shelhamer, Kate Rakelly, Judy Hoffman, and Trevor Darrell.  Clockwork convnets for video semantic segmentation.  In _European Conference on Computer Vision_ , pages 852–868. Springer, 2016.
  * Si et al. [2019] Haiyang Si, Zhiqiang Zhang, Feifan Lv, Gang Yu, and Feng Lu.  Real-time semantic segmentation via multiply spatial fusion network.  _arXiv preprint arXiv:1911.07217_ , 2019.
  * Veit et al. [2016] Andreas Veit, Michael J Wilber, and Serge Belongie.  Residual networks behave like ensembles of relatively shallow networks.  _Advances in neural information processing systems_ , 29:550–558, 2016.
  * Wang et al. [2021] Hao Wang, Weining Wang, and Jing Liu.  Temporal memory attention for video semantic segmentation.  _arXiv preprint arXiv:2102.08643_ , 2021.
  * Wang et al. [2018] Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez.  Skipnet: Learning dynamic routing in convolutional networks.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 409–424, 2018.
  * Wang et al. [2020] Yujiang Wang, Mingzhi Dong, Jie Shen, Yang Wu, Shiyang Cheng, and Maja Pantic.  Dynamic face video segmentation via reinforcement learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6959–6969, 2020.
  * Williamson et al. [2010] Sinead Williamson, Peter Orbanz, and Zoubin Ghahramani.  Dependent indian buffet processes.  In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , pages 924–931. JMLR Workshop and Conference Proceedings, 2010.
  * Wu et al. [2018] Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris.  Blockdrop: Dynamic inference paths in residual networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 8817–8826, 2018.
  * Xu et al. [2018] Yu-Syuan Xu, Tsu-Jui Fu, Hsuan-Kung Yang, and Chun-Yi Lee.  Dynamic video segmentation network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6556–6565, 2018.
  * Yu et al. [2018] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang.  Bisenet: Bilateral segmentation network for real-time semantic segmentation.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 325–341, 2018.
  * Yu et al. [2021] Changqian Yu, Changxin Gao, Jingbo Wang, Gang Yu, Chunhua Shen, and Nong Sang.  Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation.  _International Journal of Computer Vision_ , 129(11):3051–3068, 2021.
  * Zhao et al. [2017] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2881–2890, 2017.
  * Zhao et al. [2018] Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia.  Icnet for real-time semantic segmentation on high-resolution images.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 405–420, 2018.
  * Zhou et al. [2011] Mingyuan Zhou, Hongxia Yang, Guillermo Sapiro, David Dunson, and Lawrence Carin.  Dependent hierarchical beta process for image interpolation and denoising.  In _Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics_ , pages 883–891. JMLR Workshop and Conference Proceedings, 2011.
  * Zhou et al. [2018] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang.  Unet++: A nested u-net architecture for medical image segmentation.  In _Deep learning in medical image analysis and multimodal learning for clinical decision support_ , pages 3–11. Springer, 2018.
  * Zhu et al. [2017] Xizhou Zhu, Yuwen Xiong, Jifeng Dai, Lu Yuan, and Yichen Wei.  Deep feature flow for video recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2349–2358, 2017.
  * Zhuang et al. [2020] Jiafan Zhuang, Zilei Wang, and Bingke Wang.  Video semantic segmentation with distortion-aware feature correction.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 31(8):3128–3139, 2020.

####  Supplementary File

#### Organization

The supplementary file is organized as follows. In Section A, we first
describe the experimental results on CamVid [3] of the proposed STMG framework
and describe the implementation detail and spatial masking method on CamVid.
Also, we provide the code and dataset in Section B. Then, we provide
additional experimental results on Cityscapes [6] with visualization of
examples in Section C. Finally, we provide the additional experimental results
on CamVid in Section D and potential negative societal impacts and limitation
of our work in Section E.

##  Appendix A Experiments on CamVid

#### CamVid Dataset

CamVid [3] is a road-scene parsing dataset which is taken from the point of
view of a driving automobile. The dataset consists of 701 annotated images
with 11 classes at 1 Hz and in part 15Hz, in which 367, 101, and 233 images
for training, validation, and testing, respectively.

Figure 9: Qualitative results on CamVid.

Table 8: Performance comparison with real-time (> 30 FPS) semantic
segmentation models on CamVid [3] without pretraining on Cityscapes [6].

Model | mIoU | FPS | Resolution
---|---|---|---
BiSeNetV2 [49] | 72.4 | 124.5 | 720x960
GAS [30] | 72.8 | 153.1 | 720x960
PP-LiteSeg-T [36] | 73.3 | 222.3 | 720x960
SFNet-RN-18 [26] | 73.8 | 35.5 | 720x960
STDC2-Seg [8] | 73.9 | 152.2 | 720x960
MSFNet [40] | 75.4 | 91.0 | 768x1024
HyperSeg [33] | 78.4 | 38.0 | 576x768
DDRNet-39 [17] | 76.7 | 59.1 | 720x960
\- STMG-DDR39 | 75.5 | 70.2 | 720x960

Table 9: Performance comparison with video semantic segmentation models on
CamVid [3].

Model | mIoU | FPS
---|---|---
DFF [54] | 66.0 | 9.8
GRFP(5) [32] | 66.1 | 4.3
ACCEL-18 [21] | 66.7 | 5.9
NetWarp [9] | 67.1 | 2.8
ACCEL-50 [21] | 67.7 | 4.2
FANet-18 [19] | 69.0 | 154
FANet-34 [19] | 70.1 | 121
TD4-PSP18 [18] | 72.6 | 25.0
TD2-PSP50 [18] | 76.0 | 11.1
STMG-DDR39 | 75.5 | 70.2

#### Spatial Masking on CamVid

We first learn the spatial masks using knowledge distillation as described in
the main paper. However, we found that it was difficult to learn spatial masks
since the amount of distortion between adjacent frames is too large in CamVid
due to the low frame rate. Thus, we utilize a fixed spatial mask value to
aggregate the features from previous and current frames. Specifically, the
feature aggregation is computed as follows:

$\pi^{\prime}({{\bf I}_{cur}})=m_{sp}\cdot\pi({{\bf
I}_{cur}})+(1-m_{sp})\cdot\pi({{\bf I}_{pre}}),$ (13)

where, $m_{sp}$ is fixed to $0.8$. We found that setting a higher $m_{sp}$ is
crucial to compensate for information loss due to the low frame rate by using
more features from current frame than in the previous frame.

#### Implementation Detail

We set the initial learning rate to 0.001 and train the model for 968 epochs
for DDRNet-39 and corresponding STMG module. We use a batch size of 4 with a
cropping resolution of 720 × 960 for CamVid and conduct all experiments using
PyTorch. For a fair comparison, we apply the same training setting to the
corresponding STMG module. We train each model with two GeForce RTX 2080 Ti
GPUs for DDRNet-39 and corresponding STMG module in the same setting.

#### Experimental Results

We further report the experimental results on CamVid [3] in Table 9, using
DDRNet-39 backbone network [17]. The results show that our method can achieve
70.2 FPS, achieving 18.8% gain over the base model. We increase the FPS with
less than 2% drop in accuracy. That is, STMG provides a good trade-off
comparable to state-of-the-art models on CamVid.

##  Appendix B Code and Dataset

Code and dataset are available at
https://anonymous.4open.science/r/neurips2022stmg.

##  Appendix C Additional Results on Cityscapes

Figure 10: Qualitative results on Cityscapes. Visualization examples of our
STMG. First row: Target Frame. Second row: STMG-DDR39. Third row: Ground
Truth. Zoom in for best view.

Figure 11: Qualitative results of distortion-aware scheduling policy. Frames
highlighted in red represent key frames and other frames represent non-key
frames. The fourth row shows that for static video streams, the scheduling
policy continues to perform partial computations on non-key frames.

Figure 12: Qualitative results of generated spatial mask visualization on
Cityscapes. First row: Target Frame. Second row: Generated Mask. Third row:
Distortion Map. Zoom in for best view.

##  Appendix D Additional Results on CamVid

Figure 13: Qualitative results of distortion-aware scheduling policy on
CamVid. Frames highlighted in red represent key frames and other frames
represent non-key frames. The fifth row shows that the scheduling policy
continues to perform partial computations for static video streams.

##  Appendix E Discussion

In this section, we discuss the limitation and potential negative societal
impacts of our work.

#### Potential Societal Impacts

A potential ethical threat is that our proposed method could be exploited with
malicious purpose to harm a society such as terrorism with limited computing
resources (e.g. drones for terrorism). Further, our method could be exploited
in mobile applications that can generate antisocial content, including real-
time portrait matting for unethical deepfake videos. We hope that our proposed
method would not be utilized for such malicious purposes to harm the society.

#### Limitation

In this work, we propose a novel framework for real-time speed up of video
semantic segmentation models by exploiting the spatial-temporal locality of
video. However, since our method targets real-time speed up and state-of-the-
art real-time segmentation models are much faster than optical flow
estimation, our method does not utilize flow-based feature propagation. Due to
these properties, our method adopts a scheduling policy with a relatively
short duration between key and non-key frames compared to previous video
segmentation frameworks, as errors propagate when non-key frames are repeated.
Further, since DBB [25]-inspired pruning method optimizes the segmentation
network while learning the pruning mask, our method uses different weights for
key and non-key frames. This results in memory inefficiency both in inference
time and training time, as two networks have to be loaded on the GPU. Because
training a segmentation network requires a large memory, training is performed
only on two adjacent frames: a fixed key frame for the previous frame and a
trainable non-key frame for the current frame. We leave this limitation as
future work.