11institutetext: Xi’an Jiaotong University, Xi’an, Shaanxi, China
22institutetext: University of Illinois at Chicago, Chicago, IL, USA
33institutetext: HERE Technologies, Chicago, IL, USA 44institutetext: State
University of New York at Buffalo, Buffalo, NY, USA 55institutetext: Wormpex
AI Research, Bellevue, WA, USA

# Two-Stream Consensus Network for Weakly-Supervised Temporal Action
Localization

Yuanhao Zhai 11 Le Wang Corresponding author.11 Wei Tang 22 Qilin Zhang 33
Junsong Yuan 44 Gang Hua 55

###### Abstract

Weakly-supervised Temporal Action Localization (W-TAL) aims to classify and
localize all action instances in an untrimmed video under only video-level
supervision. However, without frame-level annotations, it is challenging for
W-TAL methods to identify false positive action proposals and generate action
proposals with precise temporal boundaries. In this paper, we present a Two-
Stream Consensus Network (TSCN) to simultaneously address these challenges.
The proposed TSCN features an iterative refinement training method, where a
frame-level pseudo ground truth is iteratively updated, and used to provide
frame-level supervision for improved model training and false positive action
proposal elimination. Furthermore, we propose a new attention normalization
loss to encourage the predicted attention to act like a binary selection, and
promote the precise localization of action instance boundaries. Experiments
conducted on the THUMOS14 and ActivityNet datasets show that the proposed TSCN
outperforms current state-of-the-art methods, and even achieves comparable
results with some recent fully-supervised methods.

###### Keywords:

Temporal Action Localization; Weakly-Supervised Learning

##  1 Introduction

The task of Weakly-supervised Temporal Action Localization (W-TAL) aims at
simultaneously localizing and classifying all action instances in a long
untrimmed video given only video-level categorical labels in the learning
phase. Compared to its fully-supervised counterpart, which requires frame-
level annotations of all action instances during training, W-TAL greatly
simplifies the procedure of data collection and avoids annotation bias of
human annotators, therefore has been widely studied [18, 41, 34, 27, 30, 1,
23, 46, 24, 28, 26, 43, 20] in recent years.

Several W-TAL methods [41, 30, 27, 23, 28, 26, 20] adopt a Multiple Instance
Learning (MIL) framework, where a video is treated as a bag of frames/snippets
to perform the video-level action classification. During testing, the trained
model slides over time and generates a Temporal-Class Activation Map (T-CAM)
[49, 27] (i.e., a sequence of probability distributions over action classes at
each time step) and an attention sequence that measures the relative
importance of each snippet. The action proposals are generated by thresholding
the attention value and/or the T-CAM. This MIL framework is usually built on
two feature modalities, i.e., RGB frames and optical flow, which are fused in
two possible ways. Early fusion methods [30, 34, 1, 23, 24, 20] concatenate
the RGB and optical flow features before they are fed into the network, and
late fusion methods [27, 23, 28, 26] compute a weighted sum of their
respective outputs before generating action proposals. An example of late
fusion is shown in Fig. 1.

Figure 1: Visualization of two-stream outputs and their late fusion result.
The first two rows are an input video and the ground truth action instances,
respectively. The last three rows are attention sequences (scaled from $0$ to
$1$) predicted by the RGB stream, the flow stream and their weighted sum
(i.e., the fusion result), respectively, and the horizontal and vertical axes
denote the time and the intensity of attention values, respectively. The green
boxes denote the localization results generated by thresholding the attention
at the value of $0.5$. By properly combining the two different attention
distributions predicted by the RGB and flow streams, the late fusion result
achieves a higher true positive rate and a lower false positive rate, and thus
has better localization performance

Despite these recent development, two major challenges still persist. One of
the most critical problems that prior W-TAL methods suffer from is the lack of
ability to rule out false positive action proposals. Without frame-level
annotations, they localize action instances that do not necessarily correspond
to the video-level labels. For example, a model may falsely localize the
action “swimming” by only checking the existence of water in the scene.
Therefore, it is necessary to exploit more fine-grained supervision to guide
the learning process. Another problem lies in the generation of action
proposals. In previous methods, action proposals are generated by thresholding
the activation sequence with a fixed threshold, which is preset empirically.
It has a significant impact on the quality of action proposals: a high
threshold may result in incomplete action proposals while a low threshold can
bring more false positives. But how to get out of this dilemma was rarely
studied.

In this paper, we introduce a Two-Stream Consensus Network (TSCN) to address
the two aforementioned problems. To eliminate false positive action proposals,
we design an iterative refinement training scheme, where a frame-level pseudo
ground truth is generated from late fusion attention sequence, and serves as a
more precise frame-level supervision to iteratively update two-stream models.
Our intuition is simple: late fusion is essentially a voting ensemble of the
RGB and flow streams, and if a proper fusion parameter (i.e., the
hyperparameter to control the relative importance of two streams) is selected,
late fusion can provide more accurate result compared with each individual
stream. The advantage of combining these two streams has been demonstrated by
the Two-Stream Convolutional Networks [37] for action recognition. As shown in
Fig. 1, the two streams produce different activation distributions, which lead
to different false positives and false negatives. However, when they are
combined, the false positive action proposals that only exist in one stream
can be largely eliminated, and a high activation value occurs only when both
streams are confident that an action instance exists. Since the late fusion
result is of higher quality than single stream result, it can in turn serve as
a frame-level pseudo ground truth to supervise and refine both streams. To
generate high-quality action proposals, we introduce a new attention
normalization loss. It pushes the predicted attention to approach extreme
values, _i.e._ , 0 and 1, so as to avoid ambiguity. As a result, simply
setting the threshold to 0.5 yields high-quality action proposals.

Formally, given an input video, RGB and optical flow features are first
extracted from pre-trained deep networks. Then two-stream base models are
trained with video-level labels on RGB and optical flow features,
respectively, where the attention normalization loss is used to learn the
attention distribution. After obtaining two-stream attention sequences, a
frame-level pseudo ground truth is generated based on their weighted sum
(i.e., the late fusion attention sequence), and in turn provides frame-level
supervision to improve the two-stream models. We iteratively update the pseudo
ground truth and refine the two-stream base models, and the normalization term
at the same time forces the predicted attention to approach a binary
selection. The final localization result is obtained by thresholding the late
fusion attention sequence.

To summarize, our contribution is threefold:

  * •

We introduce a Two-Stream Consensus Network (TSCN) for W-TAL. The proposed
TSCN uses an iterative refinement training method, where a pseudo ground truth
generated from late fusion attention sequence at previous iteration can
provide more precise frame-level supervision to current iteration.

  * •

We propose an attention normalization loss function, which forces the
attention to act like a binary selection, and thus improves the quality of
action proposals generated by the thresholding method.

  * •

Extensive experiments are conducted on two standard benchmarks (i.e., THUMOS14
and ActivityNet) to demonstrate the effectiveness of the proposed method. Our
TSCN significantly outperforms previous state-of-the-art W-TAL methods, and
even achieves comparable results to some recent fully-supervised TAL methods.

##  2 Related Work

Action Recognition. Traditional methods [19, 7, 8, 39] aim to model spatio-
temporal information via hand-crafted features. Two-Stream Convolutional
Networks [37] use two separate Convolutional Neural Networks (CNNs) to exploit
appearance and motion clues from RGB frames and optical flow, respectively,
and use a late fusion method to reconcile the two-stream outputs. [10] focuses
on studying different ways to fuse the two streams. The Inflated 3D ConvNet
(I3D) [3] expands the 2D CNNs in two-stream networks to 3D CNNs. Several
recent methods [47, 5, 35, 40, 31] focus on directly learning motion clues
from RGB frames instead of calculating optical flow.

Fully-supervised Temporal Action Localization. Fully-supervised TAL requires
frame-level annotations of all action instances during training. Several
large-scale datasets have been created for this task, such as THUMOS [15, 13],
ActivityNet [2], and Charades [36]. Many methods [33, 48, 12, 14, 6, 42, 22,
4] adopt a two-stage pipeline, _i.e._ , action proposal generation followed by
action classification. Several methods [42, 6, 11, 4] adopt the Faster R-CNN
[32] framework to TAL. Most recently, some methods [22, 25, 21] try to
generate action proposals with more flexible durations. Zeng et al. [45] apply
the Graph Convolutional Networks (GCN) [17, 38] to TAL to exploit proposal-
proposal relations.

Weakly-supervised Temporal Action Localization. W-TAL, which only requires
video-level supervision during training, greatly relieves the data annotation
efforts, and draws more and more attention from the community recently. Hide-
and-Seek [18] randomly hides part of the input video to guide the network to
discover other relevant parts. UntrimmedNet [41] consists of a selection
module to select the important snippets and a classification module to perform
per snippet classification. Sparse Temporal Pooling Network (STPN) [27]
improves UntrimmedNet by adding a sparse loss to enforce the sparsity of
selected segments. W-TALC [30] jointly optimizes a co-activity similarity loss
and a multiple instance learning loss to train the network. AutoLoc [34] is
one of the first two-stage methods in W-TAL, and it first generates initial
action proposals and then regresses the boundaries of the action proposals
with an Outer-Inner-Contrastive loss. CleanNet [24] improves AutoLoc by
leveraging the temporal contrast in snippet-level action classification
predictions. Liu et al. [23] propose a multi-branch network to model different
stages of action. Besides, several methods [28, 20] focus on modeling the
background and achieve state-of-the-art performances.

Recently, RefineLoc [1] uses an iterative refinement method to help the model
capture a complete action instance. And our method is distinct from RefineLoc
in three main aspects. (1) We adopt a late fusion framework, while RefineLoc
adopts an early fusion framework. (2) Our pseudo ground truth is generated
from two-stream late fusion attention sequences, which provides better
localization performance than each single stream, while RefineLoc generates
the pseudo ground truth by expanding previous localization results, which
might result in coarser and over-complete action proposals. (3) We introduce a
new attention normalization loss to explicitly avoid the ambiguity of
attention, while RefineLoc has no explicit constraints on attention values.

Figure 2: An overview of the proposed Two-Stream Consensus Network, which
consists of three parts: (1) RGB and optical flow snippet-level features are
extracted with pre-trained models; (2) two-stream base models are separately
trained using these RGB and optical flow features; (3) frame-level pseudo
ground truth is generated from the two-stream late fusion attention sequence,
and in turn provides frame-level supervision to two-stream base models

##  3 Two-Stream Consensus Network

In this section, we first formulate the task of Weakly-supervised Temporal
Action Localization (W-TAL), and then describe the proposed Two-Stream
Consensus Network (TSCN) in detail. The overall architecture is shown in Fig.
2.

###  3.1 Problem Formulation

Assume we are given a set of training videos. For each video $v$, we only have
its video-level categorical label $\mathbf{y}$, where
$\mathbf{y}\in\mathbb{R}^{C}$ is a normalized multi-hot vector, and $C$ is the
number of action categories. The goal of temporal action localization is to
generate a set of action proposals $\\{(t_{s},t_{e},c,\psi)\\}$ for each
testing video, where $t_{s},t_{e},c,\psi$ denote the start time, the end time,
the predicted action category and the confidence score of the action proposal,
respectively.

###  3.2 Feature Extraction

Following recent W-TAL methods [34, 27, 30, 23, 24, 28, 26, 43, 20], we
construct TSCN upon snippet-level feature sequences extracted from the raw
video volume. The RGB and optical flow features are extracted with pre-trained
deep networks (e.g., I3D [3]) from non-overlapping fixed-length RGB frame
snippets and optical flow snippets, respectively. They provide high-level
appearance and motion information of the corresponding snippets. Formally,
given a video with $T$ non-overlapping snippets, we denote the RGB features
and optical flow features as $\\{\mathbf{f}_{\text{RGB},i}\\}_{i=1}^{T}$ and
$\\{\mathbf{f}_{\text{flow},i}\\}_{i=1}^{T}$, respectively, where
$\mathbf{f}_{\text{RGB},i},\mathbf{f}_{\text{flow},i}\in\mathbb{R}^{D}$ are
the feature representations of the $i$-th RGB frame and optical flow snippet,
respectively, and $D$ denotes the channel dimension.

###  3.3 Two-Stream Base Models

After obtaining the RGB and optical flow features, we first use two-stream
base models to perform the video-level action classification, and then
iteratively refine the base models with a frame-level pseudo ground truth. The
features of two modalities are fed into two separate base models,
respectively, and the two base models use the same architecture but do not
share parameters. Therefore, in this subsection, we omit the subscript RGB and
flow for conciseness.

Since the features are not originally trained for the W-TAL task, we
concatenate the $T$ input features $\\{\textbf{f}_{i}\\}_{i=1}^{T}$, and use a
set of temporal convolutional layers to generate a set of new features
$\\{\textbf{x}_{i}\\}_{i=1}^{T}$, where
$\textbf{x}_{i}\in\mathbb{R}^{D^{\prime}}$, and $D^{\prime}$ denotes the
output feature dimension.

As a video may contain background snippets, to perform video-level
classification, we need to select snippets that are likely to contain action
instances and meanwhile filter out snippets that are likely to contain
background. To this end, an attention value $A_{i}\in(0,1)$ to measure the
likelihood of the $i$-th snippet containing an action is given by a fully-
connected (FC) layer:

$A_{i}=\sigma\left(\mathbf{w}_{A}\cdot\mathbf{x}_{i}+b_{A}\right),$ (1)

where $\sigma(\cdot)$, $\mathbf{w}_{A}$, and $b_{A}$ are the sigmoid function,
weight vector and bias of the attention layer. We then perform attention-
weighted pooling over the feature sequence to generate a single foreground
feature $\mathbf{x}_{\text{fg}}$, and feed it to an FC softmax layer to get
the video-level prediction:

$\mathbf{x}_{\text{fg}}=\frac{1}{\sum_{i=1}^{T}A_{i}}\sum_{i=1}^{T}A_{i}\mathbf{x}_{i},$
(2)
$\hat{y}_{c}=\frac{e^{\mathbf{w}_{c}\cdot\mathbf{x}_{\text{fg}}+b_{c}}}{\sum_{i=1}^{C}e^{\mathbf{w}_{i}\cdot\mathbf{x}_{\text{fg}}+b_{i}}},$
(3)

where $\hat{y}_{c}$ is the probability that the video contains the $c$-th
action, and $\mathbf{w}_{c}$ and $b_{c}$ are the weight and bias of the FC
layer for category $c$. The classification loss function
$\mathcal{L}_{\text{cls}}$ is defined as the standard cross entropy loss:

$\mathcal{L}_{\text{cls}}=-\sum_{c=1}^{C}y_{c}\log(\hat{y}_{c}),$ (4)

where $y_{c}$ denotes the value of label vector y at index $c$.

Ideally, an attention value is expected to be binary, where $1$ indicates the
presence of action while $0$ indicates background. Recently, several methods
[28, 20] introduce a background category, and use the background
classification to guide the learning of attention. In this work, instead of
using background classification, we introduce an attention normalization term
to force the attention to approach extreme values:

$\mathcal{L}_{\text{att}}=\frac{1}{l}\min_{\begin{subarray}{c}A\subset\\{A_{i}\\}\\\
|A|=l\end{subarray}}\sum_{a\in
A}a-\frac{1}{l}\max_{\begin{subarray}{c}A\subset\\{A_{i}\\}\\\
|A|=l\end{subarray}}\sum_{a\in A}a,$ (5)

where $l=\max\left(1,\lfloor\frac{T}{s}\rfloor\right)$ and $s$ is a
hyperparameter to control the selected snippets. This normalization loss aims
to maximize the difference between the average top-$l$ attention values and
the average bottom-$l$ attention values, and force the foreground attention to
be $1$ and background attention to be $0$.

Therefore, the overall loss for the base model training is the weighted sum of
the classification loss and the attention normalization term:

$\mathcal{L}_{\text{base}}=\mathcal{L}_{\text{cls}}+\alpha\mathcal{L}_{\text{att}},$
(6)

where $\alpha$ is a hyperparameter to control the weight of the normalization
loss.

In addition, the temporal-class activation map (T-CAM)
$\\{\textbf{s}_{i}\\}_{i=1}^{T}$, $\textbf{s}_{i}\in\mathbb{R}^{C}$ is also
generated by sliding the classification FC softmax layer over all snippets:

$s_{i,c}=\frac{e^{\mathbf{w}_{c}\cdot\mathbf{x}_{i}+b_{c}}}{\sum_{j=1}^{C}e^{\mathbf{w}_{j}\cdot\mathbf{x}_{i}+b_{j}}},$
(7)

where $s_{i,c}$ is the T-CAM value of $i$-th snippet for category $c$.

###  3.4 Pseudo Ground Truth Generation

We iteratively refine the two-stream base models with a frame-level pseudo
ground truth. Specifically, we divide the whole training process into several
refinement iterations. At refinement iteration $0$, only video-level labels
are used for training. And at refinement iteration $n+1$, a frame-level pseudo
ground truth is generated at refinement iteration $n$, and provides frame-
level supervision for the current refinement iteration. However, without true
ground truth annotations, we can neither measure the quality of the pseudo
ground truth, nor guarantee the pseudo ground truth can help the base models
achieve higher performance.

Inspired by two-stream late fusion, we introduce a simple yet effective method
to generate the pseudo ground truth. Intuitively, locations at which both
streams have high activations are likely to contain ground truth action
instances; locations at which only one stream has high activations are likely
to be either false positive action proposals or true action instances that
only one stream can detect; locations at which both streams both have low
activations are likely to be the background.

Following this intuition, we use the fusion attention sequence
$\\{A_{\text{fuse},i}^{(n)}\\}_{i=1}^{T}$ at refinement iteration $n$ to
generate pseudo ground truth $\\{\mathcal{G}_{i}^{(n+1)}\\}_{i=1}^{T}$ for
refinement iteration $n+1$, where $A_{\text{fuse},i}^{(n)}=\beta
A_{\text{RGB},i}^{(n)}+(1-\beta)A_{\text{flow},i}^{(n)}$, and $\beta\in[0,1]$
is a hyperparameter to control the relative importance of RGB and flow
attentions. We introduce two pseudo ground truth generation methods.

Soft pseudo ground truth means to directly use the fusion attention values as
pseudo labels: $\mathcal{G}_{i}^{(n+1)}=A_{\text{fuse},i}^{(n)}$. The soft
pseudo labels contain the probability of a snippet being the foreground
action, but also add uncertainty to the model.

Hard pseudo ground truth thresholds the attention sequence to generate a
binary sequence:

$\mathcal{G}_{i}^{(n+1)}=\left\\{\begin{array}[]{lr}1,\quad
A_{\text{fuse},i}^{(n)}>\theta;&\\\ 0,\quad
A_{\text{fuse},i}^{(n)}\leq\theta,&\\\ \end{array}\right.$ (8)

where $\theta$ is the threshold value. Setting a large value of $\theta$ will
eliminate the action proposals that only one stream has high activations, and
therefore reduces the false positive rate. In contrast, setting a small value
of $\theta$ will help models to generate more action proposals and achieve a
higher recall. Hard pseudo labels remove the uncertainty and provide stronger
supervision, but introduce a hyperparameter.

After generating the frame-level pseudo ground truth, we force the attention
sequence generated by each stream to be similar to the pseudo ground truth
with a mean square error (MSE) loss111Although it is straightforward to use a
cross entropy loss for hard pseudo ground truth, we found in practice that the
cross entropy loss and the MSE loss achieve similar performance. To simplify
training, we use the MSE loss for both kinds of pseudo ground truth.:

$\mathcal{L}_{\mathcal{G}}^{(n+1)}=\frac{1}{T}\sum_{i=1}^{T}\left(A_{i}^{(n+1)}-\mathcal{G}_{i}^{(n+1)}\right)^{2}.$
(9)

At refinement iteration $n+1$, the total loss for each stream is

$\mathcal{L}_{\text{total}}^{(n+1)}=\mathcal{L}_{\text{base}}+\gamma\mathcal{L}_{\mathcal{G}}^{(n+1)},$
(10)

where $\gamma$ is a hyperparameter to control the relative importance of two
losses.

###  3.5 Action Localization

During testing, following BaS-Net [20], we first temporally upsample the
attention sequence and T-CAM by a factor of $8$ via linear interpolation.
Then, we select top-$k$ action categories from the fusion video-level
prediction $\hat{\textbf{y}}_{\text{fuse}}$ to perform action localization,
where
$\hat{\textbf{y}}_{\text{fuse}}=\beta\hat{\textbf{y}}_{\text{RGB}}+(1-\beta)\hat{\textbf{y}}_{\text{flow}}$.
For each of these categories, following our intention that the attention
performs a binary selection, we generate action proposals by directly
thresholding the attention value at $0.5$ and concatenating consecutive
snippets. The action proposals are scored via a variant of the Outer-Inner-
Constrastive score [34]: instead of using average T-CAM, we use attention
weighted T-CAM to measure the outer and inner temporal contrast. Formally,
given action proposal $(t_{s},t_{e},c)$, fusion attention
$\\{A_{\text{fuse},i}\\}_{i=1}^{T}$ and T-CAM
$\\{\textbf{s}_{\text{fuse},i}\\}_{i=1}^{T}$, where
$\textbf{s}_{\text{fuse},i}=\beta\textbf{s}_{\text{RGB},i}+(1-\beta)\textbf{s}_{\text{flow},i}$,
the score $\psi$ is computed as

$\psi=\frac{\sum_{i=t_{s}}^{t_{e}}A_{\text{fuse},i}s_{\text{fuse},i,c}}{t_{e}-t_{s}}-\frac{\sum_{i=T_{s}}^{T_{e}}A_{\text{fuse},i}s_{\text{fuse},i,c}-\sum_{i=t_{s}}^{t_{e}}A_{\text{fuse},i}s_{\text{fuse},i,c}}{T_{e}-T_{s}-(t_{e}-t_{s})},$
(11)

where $T_{s}=t_{s}-\frac{L}{4}$, $T_{e}=t_{e}+\frac{L}{4}$, and
$L=t_{e}-t_{s}$. We discard action proposals with confidence scores lower than
$0$.

##  4 Experiments

###  4.1 Dataset and Evaluation

THUMOS14 dataset [15] contains $200$ validation videos and $213$ testing
videos within $20$ categories for the TAL task. We use the $200$ validation
videos for training, and use the $213$ testing videos for evaluation.

ActivityNet dataset [2] has two release versions, i.e., ActivityNet v1.3 and
ActivityNet v1.2. ActivityNet v1.3 covers $200$ action categories, with a
training set of $10,024$ videos and a validation set of $4,926$ videos.
ActivityNet v1.2 is a subset of ActivityNet v1.3, and covers $100$ action
categories, with $4,819$ and $2,383$ videos in the training and validation
set, respectively.222In our experiments, there are $9,937$ and $4,575$ videos
in training and validation set of ActivityNet v1.3, respectively, and $4,471$
and $2,211$ videos in training and validation set of ActivityNet v1.2,
respectively, because the rest of the videos are unaccessible from YouTube. We
use the training set and the validation set for training and testing,
respectively.

Evaluation Metrics. Following the standard protocol on temporal action
localization, we evaluate our method with mean Average Precision (mAP) under
different Intersection-over-Union (IoU) thresholds. We use the evaluation code
provided by
ActivityNet333https://github.com/activitynet/ActivityNet/tree/master/Evaluation
to perform the experiments.

###  4.2 Implementation Details

Two off-the-shelf feature extraction backbones are used in our experiments,
i.e., UntrimmedNet [41] and I3D [3], with snippet lengths of $15$ frames and
$16$ frames, respectively. The two backbones are pre-trained on ImageNet [9]
and Kinetics [3], respectively, and are not fine-tuned for fair comparison.
The RGB and flow snippet-level features are extracted at the global_pool layer
as $1024$-D vectors.

The networks are implemented in PyTorch [29]. We use the Adam [16] optimizer
with a fixed learning rate $0.0001$. We train the base models $200$ and $80$
epochs at refinement iteration $0$, and $100$ and $40$ epochs for later
refinement iterations for ActivityNet and THUMOS14, respectively. We set the
maximal number of refinement iterations to $4$ for the THUMOS14 dataset, and
$24$ for the ActivityNet datasets, and choose base models that achieve the
lowest loss at the previous refinement iteration to generate the pseudo ground
truth. To eliminate fragmentary action proposals, temporal max pooling of
kernel size $5$ and stride $1$ is used on the fusion attention sequence before
pseudo ground truth generation on ActivityNet dataset. We use a whole video as
a batch. All hyperparameters are determined via grid search: $s=8$,
$\alpha=0.1$, $\beta=0.4$, $\gamma=2$. We set $\theta$ to $0.55$ and $0.5$ for
THUMOS14 and ActivityNet, respectively. We choose top-$2$ action categories
and also reject categories whose fusion classification prediction scores are
lower than $0.1$ to perform action localization.

Table 1: Comparison of our method with state-of-the-art TAL methods on the THUMOS14 testing set. UNT and I3D are abbreviations for UntrimmedNet feature and I3D feature, respectively | Method | mAP@IoU (%)
---|---|---
0.1 |  0.2 |  0.3 |  0.4 |  0.5 |  0.6 |  0.7 |  0.8 |  0.9
Fully-supervised | Yuan et al. [44] |  51.0 |  45.2 |  36.5 |  27.8 |  17.8 |  - |  - |  - |  -
CDC [33] |  - |  - |  40.1 |  29.4 |  23.3 |  13.1 |  7.9 |  - |  -
R-C3D [42] |  54.5 |  51.5 |  44.8 |  35.6 |  28.9 |  - |  - |  - |  -
SSN [48] |  66.0 |  59.4 |  51.9 |  41.0 |  29.8 |  - |  - |  - |  -
BSN [22] |  - |  - |  53.5 |  45.0 |  36.9 |  28.4 |  20.0 |  - |  -
TAL-Net [4] |  59.8 |  57.1 |  53.2 |  48.5 |  42.8 |  33.8 |  20.8 |  - |  -
GTAN [25] |  69.1 |  63.7 |  57.8 |  47.2 |  38.8 |  - |  - |  - |  -
BMN [21] |  - |  - |  56.0 |  47.4 |  38.8 |  29.7 |  20.5 |  - |  -
Weakly-supervised | UntrimmedNet [41] |  44.4 |  37.7 |  28.2 |  21.1 |  13.7 |  - |  - |  - |  -
STPN (UNT) [27] |  45.3 |  38.8 |  31.1 |  23.5 |  16.2 |  9.8 |  5.1 |  2.0 |  0.3
AutoLoc (UNT) [34] |  - |  - |  35.8 |  29.0 |  21.2 |  13.4 |  5.8 |  - |  -
W-TALC (UNT) [30] |  49.0 |  42.8 |  32.0 |  26.0 |  18.8 |  - |  6.2 |  - |  -
Liu et al. (UNT) [23] |  53.5 |  46.8 |  37.5 |  29.1 |  19.9 |  12.3 |  6.0 |  - |  -
RefineLoc (UNT) [1] |  - |  - |  36.1 |  - |  22.6 |  - |  5.8 |  - |  -
CleanNet (UNT) [24] |  - |  - |  37.0 |  30.9 |  23.9 |  13.9 |  7.1 |  - |  -
BaS-Net (UNT) [20] |  56.2 |  50.3 |  42.8 |  34.7 |  25.1 |  17.1 |  9.3 |  3.7 |  0.5
Ours (UNT) |  58.9 |  52.9 |  45.0 |  36.6 |  27.6 |  18.8 |  10.2 |  4.0 |  0.5
STPN (I3D) [27] |  52.0 |  44.7 |  35.5 |  25.8 |  16.9 |  9.9 |  4.3 |  1.2 |  0.1
W-TALC (I3D) [30] |  55.2 |  49.6 |  40.1 |  31.1 |  22.8 |  - |  7.6 |  - |  -
Liu et al. (I3D) [23] |  57.4 |  50.8 |  41.2 |  32.1 |  23.1 |  15.0 |  7.0 |  - |  -
RefineLoc (I3D) [1] |  - |  - |  40.8 |  - |  23.1 |  - |  5.3 |  - |  -
Nguyen et al. (I3D) [28] |  60.4 |  56.0 |  46.6 |  37.5 |  26.8 |  17.6 |  9.0 |  3.3 |  0.4
BaS-Net (I3D) [20] |  58.2 |  52.3 |  44.6 |  36.0 |  27.0 |  18.6 |  10.4 |  3.9 |  0.5
Ours (I3D) |  63.4 |  57.6 |  47.8 |  37.7 |  28.7 |  19.4 |  10.2 |  3.9 |  0.7

###  4.3 Comparison with the State-of-the-art

Experiments on THUMOS14. Table 1 summarizes the performance comparison between
the proposed TSCN and state-of-the-art fully-supervised and weakly-supervised
TAL methods on the THUMOS14 testing set. With UntrimmedNet features, TSCN
outperforms other W-TAL methods by a large margin, and even achieves
comparable results to some recent W-TAL methods with I3D features (e.g.,
Nguyen et al. [28] and BaS-Net [20]) at high IoU thresholds.

With I3D features, our performance boosts significantly, and outperforms
previous W-TAL methods at most IoU thresholds. We note the proposed TSCN can
achieve a comparable performance to some recent fully-supervised methods
(e.g., R-C3D [42]). TSCN even outperforms TAL-net [4] at IoU thresholds $0.1$
and $0.2$. However, as the IoU threshold increases, the performance of TSCN
drops significantly, because generating more precise action boundaries need
true frame-level ground truth supervision.

Experiments on ActivityNet. The performance comparisons on ActivityNet v1.2
and v1.3 are shown in Table 3 and Table 3, respectively, where our models are
trained with I3D features. The proposed TSCN outperforms previous W-TAL
methods at the average mAP at IoU threshold $0.5:0.05:0.95$ on both release
versions of ActivityNet, verifying the efficacy of our design intuition.

Table 2: Comparison of our method with state-of-the-art W-TAL methods on the ActivityNet v1.2 validation set. The Avg column indicates the average mAP at IoU thresholds 0.5:0.05:0.95 Method | mAP@IoU (%) |  Avg
---|---|---
0.5 |  0.75 |  0.95 |
UntrimmedNet [41] |  7.4 |  3.2 |  0.7 |  3.6
AutoLoc [34] |  27.3 |  15.1 |  3.3 |  16.0
W-TALC [30] |  37.0 |  - |  - |  18.0
Liu et al. [23] |  36.8 |  22.0 |  5.6 |  22.4
Ours |  37.6 |  23.7 |  5.7 |  23.6

Table 3: Comparison of our method with state-of-the-art W-TAL methods on the ActivityNet v1.3 validation set. The Avg column indicates the average mAP at IoU thresholds 0.5:0.05:0.95 Method | mAP@IoU (%) |  Avg
---|---|---
0.5 |  0.75 |  0.95 |
STPN [27] |  29.3 |  16.9 |  2.7 |  -
Liu et al. [23] |  34.0 |  20.9 |  5.7 |  21.2
Nguyen et al. [28] |  36.4 |  19.2 |  2.9 |  -
Ours |  35.3 |  21.4 |  5.3 |  21.7

###  4.4 Ablation Study

In this subsection, a set of ablation studies is conducted on the THUMOS14
testing set with UntrimmedNet feature to analyze the efficacy of each
component in the proposed TSCN.

Ablation study on $\mathcal{L}_{\text{att}}$. The goal of
$\mathcal{L}_{\text{att}}$ in Equation (5) is to force the attention values to
approach extreme values, and therefore generate a clean foreground feature
$\textbf{x}_{\text{fg}}$ and improve action proposal quality. Some recent
methods [28, 20] introduce background classification to W-TAL. Particularly,
background classification loss $\mathcal{L}_{bg}$ [28] is introduced to
classify the background, where a background attention is defined as $1-A_{i}$,
and a background feature is generated via background attention-weighted
pooling over all snippets to perform the background classification. Therefore,
$\mathcal{L}_{bg}$ is in essence an implicit attention normalization loss.
However, one drawback of such background loss is that assigning background
labels to all videos will make the value of the background category in the
T-CAM increase. We reproduce $\mathcal{L}_{bg}$ in our model, compare it with
our proposed $\mathcal{L}_{\text{att}}$, and list the results in Table 4. The
results reveal that both $\mathcal{L}_{bg}$ and $\mathcal{L}_{\text{att}}$
help improve the performance. And the proposed $\mathcal{L}_{\text{att}}$
achieves higher attention variance and better localization performance than
$\mathcal{L}_{bg}$, demonstrating that the our attention normalization term
$\mathcal{L}_{\text{att}}$ can better avoid the ambiguity of attention.
Surprisingly, with both $\mathcal{L}_{bg}$ and $\mathcal{L}_{\text{att}}$, the
localization performance is still lower than that with only
$\mathcal{L}_{\text{att}}$, and we think this is because the noise of
background classification reduces the accuracy of action proposal scores.

Table 4: Comparison of our method with different attention normalization
functions on the THUMOS14 testing set. $\mathcal{L}_{bg}$ is the background
classification loss introduced in [28], and $\mathcal{L}_{att}$ is defined in
Equation (5). The var column denotes the average attention variance over the
whole testing set

$\mathcal{L}_{\text{cls}}$ | $\mathcal{L}_{bg}$ | $\mathcal{L}_{att}$ | mAP@IoU (%) | Var
---|---|---|---|---
0.3 |  0.5 |  0.7
✓ | - | - |  29.6 |  16.1 |  4.1 | 0.0440
✓ | ✓ | - |  34.3 |  19.3 |  6.7 | 0.0599
✓ | - | ✓ |  40.9 |  24.0 |  8.2 | 0.0937
✓ | ✓ | ✓ |  40.6 |  23.6 |  7.8 | 0.0886

Figure 3: Comparison between models trained with different pseudo ground truth
on the THUMOS14 testing set. The upper bounds denote models trained with
ground truth actionness sequence

Ablation study on Pseudo Ground Truth. Fig. 3 plots performance comparison
between different pseudo ground truth methods at different refinement
iterations. Both soft and hard pseudo ground truth help improve the
localization performance. The hard pseudo ground truth removes uncertainty to
the model, and thus achieves higher performance improvement. However, with the
same frame-level supervision, the flow stream outperforms the RGB stream by a
large margin. We think this is because of the nature of two modalities: the
RGB modality is less sensitive to actions than the optical flow modality. To
demonstrate this, we generate a true frame-level ground truth actionness
sequence (action categories are not used), train our model in the same way as
the pseudo ground truth. The results are plotted in Fig. 3 as an upper bound.
The results verify our hypothesis and demonstrate that the optical flow
modality is more suitable for the action localization task than the RGB
modality.

Table 5: Comparison between the model trained with only video-level labels and the model trained with hard pseudo ground truth on the THUMOS14 testing set. The label column denotes the supervision used in training, where “video” indicates only video-level labels are leveraged, and “frame” indicates the hard pseudo ground truth is also leveraged during training. Precision, recall and F-measure are calculated under IoU threshold $0.5$ Modality |  Label | mAP@IoU (%) | Precision (%) | Recall (%) | F-measure
---|---|---|---|---|---
|  0.3 |  0.4 |  0.5 |  0.6 |  0.7
RGB |  video |  19.8 |  13.2 |  8.2 |  4.5 |  1.9 | 10.2 | 20.9 | 0.1371
RGB |  frame |  31.4 |  22.1 |  14.4 |  8.9 |  5.2 | 20.9 | 30.8 | 0.2489
Flow |  video |  40.2 |  32.0 |  23.2 |  15.4 |  7.2 | 25.5 | 43.3 | 0.3207
Flow |  frame |  40.8 |  32.7 |  24.1 |  16.8 |  8.7 | 30.9 | 42.4 | 0.3573
Fusion |  video |  40.9 |  32.4 |  24.0 |  15.9 |  8.2 | 23.6 | 44.4 | 0.3078
Fusion |  frame |  45.0 |  36.5 |  27.6 |  18.8 |  10.2 | 31.3 | 44.6 | 0.3680

Table 5 lists the detailed performance comparison between the model trained
with only video-level labels and that trained with the hard pseudo ground
truth. The results show that pseudo ground truth improves the localization
performance for both modalities at all IoU thresholds, and thus improves the
performance of the fusion result. Also, the pseudo ground truth greatly
improves the precision and recall for the RGB stream and the fusion result,
and improves the precision for the flow stream with a minor loss of recall
(the overall F-measure improves significantly), which demonstrates that the
pseudo ground truth can help eliminate false positive action proposals.

Qualitative Analysis. Three representative examples of TAL results are plotted
in Fig. 4 to illustrate the efficacy of the proposed pseudo supervision. In
the first example of diving and cliff diving, with only video-level labels,
the RGB stream provides worse localization result than the flow stream, and
thus leads to a noisy fusion attention sequence. The pseudo ground truth
guides the RGB stream to identify false positive action proposals and discover
true action instances, and further leads to a cleaner fusion attention
sequence, where high activations correspond better to the ground truth. In the
second example of cricket shot, with only video-level supervision, the RGB
stream can only distinguish certain scenes, and fails to separate proximate
action instances. In contrast, the flow stream can precisely detect action
instances. Therefore, the pseudo ground truth helps the RGB stream to separate
consecutive action instances. In the last example of soccer penalty, both
streams have high activations on certain false positive temporal locations.
Under this circumstance, the false positive action proposals will have higher
activations under frame-level pseudo supervision. To eliminate such false
positive action proposals, however, need true ground truth supervision. To
summarize, the two modalities have their own strengths and limitations: the
RGB stream is sensitive to appearance, thus it fails in scenes shot from
unusual angles or separating proximate action instances in the same scene; the
flow stream is sensitive to motion, and provides more accurate results, but it
fails in slow or occluded motion. Qualitative results reveal that the pseudo
ground truth helps two streams reach a consensus at most temporal locations.
Therefore, the fusion attention sequence becomes cleaner and helps generate
more precise action proposals and more reliable confidence scores.

Figure 4: Qualitative results on the THUMOS14 testing set. The eight rows in
each example are input video, ground truth action instance, RGB stream, flow
stream, and fusion attention sequences from the model trained with only video-
level labels and frame-level pseudo ground truth, respectively. Action
proposals are represented by green boxes. The horizontal and vertical axes are
time and intensity of attention, respectively

##  5 Conclusions

In this paper, we propose a Two-Stream Consensus Network (TSCN) for W-TAL,
which benefits from an iterative refinement training method and a new
attention normalization loss. The iterative refinement training uses a novel
frame-level pseudo ground truth as fine-grained supervision, and iteratively
improves the two-stream base models. The attention normalization loss function
reduces the ambiguity of attention values, and thus leads to more precise
action proposals. Experiments on two benchmarks demonstrate the proposed TSCN
outperforms current state-of-the-art methods, and verify our design intuition.

## Acknowledgement

This work was supported partly by National Key R&D Program of China Grant
2018AAA0101400, NSFC Grants 61629301, 61773312, and 61976171, China
Postdoctoral Science Foundation Grant 2019M653642, Young Elite Scientists
Sponsorship Program by CAST Grant 2018QNRC001, and Natural Science Foundation
of Shaanxi Grant 2020JQ-069.

## References

  * [1] Alwassel, H., Pardo, A., Heilbron, F.C., Thabet, A., Ghanem, B.: Refineloc: Iterative refinement for weakly-supervised action localization. arXiv preprint arXiv:1904.00227 (2019)
  * [2] Caba Heilbron, F., Escorcia, V., Ghanem, B., Carlos Niebles, J.: Activitynet: A large-scale video benchmark for human activity understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 961–970 (2015)
  * [3] Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6299–6308 (2017)
  * [4] Chao, Y.W., Vijayanarasimhan, S., Seybold, B., Ross, D.A., Deng, J., Sukthankar, R.: Rethinking the faster r-cnn architecture for temporal action localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1130–1139 (2018)
  * [5] Crasto, N., Weinzaepfel, P., Alahari, K., Schmid, C.: Mars: Motion-augmented rgb stream for action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7882–7891 (2019)
  * [6] Dai, X., Singh, B., Zhang, G., Davis, L.S., Qiu Chen, Y.: Temporal context network for activity localization in videos. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 5793–5802 (2017)
  * [7] Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 886–893 (2005)
  * [8] Dalal, N., Triggs, B., Schmid, C.: Human detection using oriented histograms of flow and appearance. In: Proceedings of the European Conference on Computer Vision. pp. 428–441 (2006)
  * [9] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Li, F.F.: Imagenet: A large-scale hierarchical image database. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition pp. 248–255 (2009)
  * [10] Feichtenhofer, C., Pinz, A., Zisserman, A.: Convolutional two-stream network fusion for video action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1933–1941 (2016)
  * [11] Gao, J., Yang, Z., Chen, K., Sun, C., Nevatia, R.: Turn tap: Temporal unit regression network for temporal action proposals. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 3628–3636 (2017)
  * [12] Gao, J., Yang, Z., Nevatia, R.: Cascaded boundary regression for temporal action detection. arXiv preprint arXiv:1705.01180 (2017)
  * [13] Gorban, A., Idrees, H., Jiang, Y.G., Zamir, A.R., Laptev, I., Shah, M., Sukthankar, R.: Thumos challenge: Action recognition with a large number of classes (2015)
  * [14] Heilbron, F.C., Barrios, W., Escorcia, V., Ghanem, B.: Scc: Semantic context cascade for efficient action detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3175–3184 (2017)
  * [15] Jiang, Y.G., Liu, J., Zamir, A.R., Toderici, G., Laptev, I., Shah, M., Sukthankar, R.: Thumos challenge: Action recognition with a large number of classes (2014)
  * [16] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [17] Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
  * [18] Kumar Singh, K., Jae Lee, Y.: Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 3524–3533 (2017)
  * [19] Laptev, I.: On space-time interest points. International Journal of Computer Vision pp. 107–123 (2005)
  * [20] Lee, P., Uh, Y., Byun, H.: Background suppression network for weakly-supervised temporal action localization. In: Proceedings of the AAAI Conference on Artificial Intelligence (2020)
  * [21] Lin, T., Liu, X., Li, X., Ding, E., Wen, S.: Bmn: Boundary-matching network for temporal action proposal generation. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 3889–3898 (2019)
  * [22] Lin, T., Zhao, X., Su, H., Wang, C., Yang, M.: Bsn: Boundary sensitive network for temporal action proposal generation. In: Proceedings of the European Conference on Computer Vision. pp. 3–19 (2018)
  * [23] Liu, D., Jiang, T., Wang, Y.: Completeness modeling and context separation for weakly supervised temporal action localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1298–1307 (2019)
  * [24] Liu, Z., Wang, L., Zhang, Q., Gao, Z., Niu, Z., Zheng, N., Hua, G.: Weakly supervised temporal action localization through contrast based evaluation networks. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 3899–3908 (2019)
  * [25] Long, F., Yao, T., Qiu, Z., Tian, X., Luo, J., Mei, T.: Gaussian temporal awareness networks for action localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 344–353 (2019)
  * [26] Narayan, S., Cholakkal, H., Khan, F.S., Shao, L.: 3c-net: Category count and center loss for weakly-supervised action localization. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 8679–8687 (2019)
  * [27] Nguyen, P., Liu, T., Prasad, G., Han, B.: Weakly supervised action localization by sparse temporal pooling network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6752–6761 (2018)
  * [28] Nguyen, P.X., Ramanan, D., Fowlkes, C.C.: Weakly-supervised action localization with background modeling. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 5502–5511 (2019)
  * [29] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Advances in neural information processing systems, pp. 8024–8035 (2019)
  * [30] Paul, S., Roy, S., Roy-Chowdhury, A.K.: W-talc: Weakly-supervised temporal activity localization and classification. In: Proceedings of the European Conference on Computer Vision. pp. 563–579 (2018)
  * [31] Piergiovanni, A., Ryoo, M.S.: Representation flow for action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019)
  * [32] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Proceedings of Neural Information Processing Systems. pp. 91–99 (2015)
  * [33] Shou, Z., Chan, J., Zareian, A., Miyazawa, K., Chang, S.F.: Cdc: Convolutional-de-convolutional networks for precise temporal action localization in untrimmed videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5734–5743 (2017)
  * [34] Shou, Z., Gao, H., Zhang, L., Miyazawa, K., Chang, S.F.: Autoloc: Weakly-supervised temporal action localization in untrimmed videos. In: Proceedings of the European Conference on Computer Vision. pp. 154–171 (2018)
  * [35] Shou, Z., Lin, X., Kalantidis, Y., Sevilla-Lara, L., Rohrbach, M., Chang, S.F., Yan, Z.: Dmc-net: Generating discriminative motion cues for fast compressed video action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1268–1277 (2019)
  * [36] Sigurdsson, G.A., Varol, G., Wang, X., Farhadi, A., Laptev, I., Gupta, A.: Hollywood in homes: Crowdsourcing data collection for activity understanding. In: Proceedings of the European Conference on Computer Vision. pp. 510–526 (2016)
  * [37] Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Proceedings of Neural Information Processing Systems. pp. 568–576 (2014)
  * [38] Tan, M., Shi, Q., van den Hengel, A., Shen, C., Gao, J., Hu, F., Zhang, Z.: Learning graph structure for multi-label image classification via clique generation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4100–4109 (2015)
  * [39] Wang, H., Kläser, A., Schmid, C., Liu, C.L.: Action recognition by dense trajectories. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3169–3176 (2011)
  * [40] Wang, L., Koniusz, P., Huynh, D.Q.: Hallucinating idt descriptors and i3d optical flow features for action recognition with cnns. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 8698–8708 (2019)
  * [41] Wang, L., Xiong, Y., Lin, D., Van Gool, L.: Untrimmednets for weakly supervised action recognition and detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4325–4334 (2017)
  * [42] Xu, H., Das, A., Saenko, K.: R-c3d: Region convolutional 3d network for temporal activity detection. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 5783–5792 (2017)
  * [43] Yu, T., Ren, Z., Li, Y., Yan, E., Xu, N., Yuan, J.: Temporal structure mining for weakly supervised action detection. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 5522–5531 (2019)
  * [44] Yuan, Z., Stroud, J.C., Lu, T., Deng, J.: Temporal action localization by structured maximal sums. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3684–3692 (2017)
  * [45] Zeng, R., Huang, W., Tan, M., Rong, Y., Zhao, P., Huang, J., Gan, C.: Graph convolutional networks for temporal action localization. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 7094–7103 (2019)
  * [46] Zhai, Y., Wang, L., Liu, Z., Zhang, Q., Hua, G., Zheng, N.: Action coherence network for weakly supervised temporal action localization. In: Proceedings of the IEEE International Conference on Image Processing. pp. 3696–3700 (2019)
  * [47] Zhao, Y., Xiong, Y., Lin, D.: Recognize actions by disentangling components of dynamics. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6566–6575 (2018)
  * [48] Zhao, Y., Xiong, Y., Wang, L., Wu, Z., Tang, X., Lin, D.: Temporal action detection with structured segment networks. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2914–2923 (2017)
  * [49] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2921–2929 (2016)


# Performance Evaluation of Video Streaming Applications with Target Wake Time
in Wi-Fi 6

Govind Rajendran1, Rishabh Roy2, Preyas Hathi3, Nadeem Akhtar3 and Samar
Agnihotri1<EMAIL_ADDRESS>1 Indian Institute of Technology Mandi,
India
3 Arista Networks
2 Indian Institute of Technology Dharwad, India
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>{3preyas.hathi<EMAIL_ADDRESS>
###### Abstract

The Target Wake Time (TWT) feature, introduced in Wi-Fi 6, was primarily meant
as an advanced power save mechanism. However, it has some interesting
applications in scheduling and resource allocation. TWT-based resource
allocation can be used to improve the user experience for certain
applications, e.g., VoIP, IoT, video streaming, etc. In this work, we analyze
the packet arrival pattern for streaming traffic and develop a synthetic video
streaming traffic generator that mimics real-world streaming traffic. We
propose a two-stage approach where we calculate the TWT duty cycle in the
first step. In the subsequent step, we determine the Multiplication Factor
(MF), which jointly dictates the required TWT schedule for the synthetic
traffic model. Initial testing shows that key QoS metrics can be met for
sustained performance of synthetic traffic upon enabling TWT, even in the
presence of peak background congestion in the network.

##  I Introduction

Wireless local area network (WLAN) technology, popularly referred to as Wi-Fi,
has a major impact in our day to day lives for high speed wireless access
because of its low cost and ease of deployment. In homes, corporate offices
and public places such as stadiums, railway stations, airports etc, Wi-Fi
provides high-speed connectivity to the Internet. However, due to the use of
shared, unlicensed spectrum for Wi-Fi, congestion is a major problem, leading
to network performance deterioration both in terms of throughput and latency.
WLAN client devices are typically portable, equipped with power limited
batteries. As a consequence, power saving mechanisms play an important role in
battery life longevity and device performance. Wi-Fi 6 was introduced, based
on the IEEE 802.11ax version of the WLAN standard, to improve the end user
Quality of Service (QoS) by efficient use of the frequency spectrum in dense
WLAN deployments and reduced power consumption for power limited devices. Wi-
Fi 6 introduces 1024-QAM, orthogonal frequency division multiple access
(OFDMA), 8-stream multi user-multiple input multiple output (MU-MIMO), spatial
reuse to improve end user performance and increase spectral efficiency even in
dense scenarios[1, 2]. In addition to that, to improve existing power
management methods, a new power save mechanism called Target Wake Time (TWT)
is introduced in Wi-Fi 6. The TWT feature is primarily meant as an advanced
power save mechanism, which may not be used in mission critical scenarios
where the client needs to stay awake for the entire duration of operation.
However, it has some interesting applications in scheduling and resource
allocation[3]. Wi-Fi 6 supports two types of TWT mechanisms: (i) Individual,
and (ii) Broadcast. In the Individual TWT mode, AP creates service schedules
on a per-client basis. A TWT schedule consists of wake-up and sleep intervals
where clients can transmit/receive frames in the wake-up intervals and turn
off their transceivers and go to a power-saving mode during the sleep
intervals. It is also possible to group clients and create group-level service
schedules using Broadcast TWT mode. With this, APs can create a quasi-TDMA
type of resource scheduling, on top of OFDMA [4, 5] and MU-MIMO[6]. With pre-
defined wake-up times for each client, TWT can help reduce channel contention
and improve airtime utilization since only a subset of the associated clients
are expected to be active at a given time. In a managed, multi-AP Wi-Fi
network, an additional level of coordination can be applied for co-channel
neighbor APs to reduce contention across APs. The challenge is to choose the
appropriate TWT mode and the schedules to address the needs of clients and
applications. One important point to remember is that the opportunistic
channel access method, CSMA/CA rule still applies during the wake-up intervals
and the AP has to contend for the TWT opportunities in an environment where
clients that do not understand TWT or are not assigned TWT schedules are also
present.

TWT-based resource allocation can be used to improve the user experience for
certain applications, e.g. VoIP. Note that VoIP follows the ON-OFF model,
corresponding to active talking and silent periods [7]. Also note that the
VoIP payload size during the active and silent periods is fixed (depending on
the codec used). For such an application, a TWT schedule can be created
considering the traffic model features. If there is a dedicated Wi-Fi network
for VoIP, then clients connecting to this Wi-Fi network can be grouped
together and assigned appropriate TWT schedules. Same approach can be applied
to IoT applications, for example sensors which generate telemetry at periodic
intervals. TWT can also be used for real-time applications like video
streaming. In this case, TWT is used to provide guaranteed airtime to users.
The process requires characterizing the traffic model first and then
generating the schedule to fit the packet arrival process. While there are
well-documented statistical models for video streaming[7], most of the popular
applications like YouTube [8], Netflix, etc. have proprietary mechanisms in
place to adapt the video stream to the link quality, which necessitates
characterizing the traffic flows and optimizing the TWT schedules separately
for each of these applications.

In this work we present the performance of streaming applications running on a
single TWT enabled client operating in a congested network and the impact of
enabling TWT on the network’s overall performance. The main contributions of
this paper can be summarized as follows:

  * •

We create a synthetic video streaming traffic generator model that
approximates real-world video streaming and also define some relevant QoS
metrics for performance evaluation, like ‘average throughput’ and ‘buffer
health status’.

  * •

We propose a two-stage approach to find a TWT schedule satisfying the QoS
requirements for streaming traffic, thereby improving end-user experience.

##  II Application Traffic Model and QoS Metrics

We pick video streaming application because the bulk of today’s network
traffic consists of streaming and video-on-demand (VOD) services[9]. We
analyze the packet arrival pattern for video streaming traffic and develop a
synthetic streaming traffic generator that in essence mimics real-world
streaming traffic. To analyze the performance of TWT on streaming services, we
need to define appropriate Quality of Service (QoS) metrics that will
quantitatively measure the system performance.

###  II-A Synthetic traffic model

Most streaming services use DASH (Dynamic Adaptive Streaming over HTTP)[10] as
an application layer protocol to stream videos. In DASH, the content server
stores a video by breaking it down into bursts, usually worth a few seconds of
video time. Initially, when the video starts playing, the server sends these
bursts at a high rate in order to fill up the playback buffer with $45$-$60$
seconds worth of video time (known as Startup phase). The server then tops up
the buffer (known as Bursty phase) periodically. In the bursty phase, bursts
generally carry $2$-$10$ seconds worth of video data [11]. This buffered
approach allows the streaming service to handle intermittent network issues
without compromising video quality. By not buffering the whole video at once,
DASH also minimizes the data loss in case the user decides to switch to a
different video. DASH also stores the video at different picture resolutions
(different size bursts) and dynamically switches among them based on the
client’s bandwidth. In DASH, it is the client who measures the available
bandwidth as:

$\text{Available bandwidth}=\frac{\text{Size of the last burst}}{\text{Time
taken to serve the last burst}}$

If the network cannot accommodate the current bitrate, the client requests for
lower bit rate video from the server.[12]

Parameter |  Description
---|---
Frame Size |  Weibull distribution with parameters, $k=0.8099$ and
$\lambda=(\frac{6950\times\text{\rm Video bit rate}}{2})$ [13]
Inter-burst time |  Truncated normal distribution with mean of $6$ seconds and variance of $1.8$ seconds and lies within $2$-$10$s [10]
Frame Rate |  $30$ frames per second
TABLE I: VBR Traffic Model

We consider two synthetic traffic models, Constant Bit Rate (CBR) and the
Variable Bit Rate (VBR). Both models use TCP as the transport layer protocol.
Traffic for both models is sent over a multi-hop global network to incorporate
real network delays. We define the following parameters to characterize the
streaming traffic models in their bursty phase:

  * •

Inter-burst time:  It is the time duration, in seconds, between two
consecutive bursts.

  * •

Burst Size:  The size of each individual burst, expressed in Megabytes (MB).

####  II-A1 CBR Model

In the CBR model, a burst of fixed size, corresponding to the video bit rate,
is generated every set amount of inter-burst time. To approximate the bursty
nature of DASH protocol, the inter-burst time is set at $6$ seconds, which is
the mean value of the inter-burst time distribution with parameters mentioned
in the Table I. It acts as our baseline traffic model whose QoS requirements
need to be satisfied, before we test with the more dynamic VBR model.

####  II-A2 VBR Model

In order to closely mimic the bursty state of real streaming, we consider the
VBR model where its traffic parameters are mentioned in Table I. Inter-burst
time is sampled and multiplied with the required frame rate (i.e., the number
of frames per second) to get the expected number of frames for that duration.
We then sample the frame size for each frame to be transmitted and get the
total size of the burst.

####  II-A3 The need for a synthetic streaming model

Though the traffic of real streaming follows the bursty trend, the exact
pattern with reference to inter-burst time and burst size is dependent on
multiple factors including picture quality, genre of video playing, rate and
magnitude of change in content across frames, compression algorithm used by
the streaming service, transport layer protocol used (TCP, UDP or QUIC) etc.
The synthetic streaming model described incorporates video frame generation
from [13] and inter-burst time generated from the statistics provided in [10],
[11]. We use this synthetic model rather than emulating traffic patterns of a
real video as the patterns of each video are different and by using the
synthetic streaming model we are able to incorporate this variability while
also keeping the trend of bursty traffic. It also gives us a greater control
over the experimental parameters like bit-rate of video and duration and
allows us to quantitatively compare performance while also capturing sender
side statistics.

###  II-B QoS parameters

The perceived experience of the end user while using an application is the
true measure of the Quality of Service provided by the network. Metrics such
as the Mean Opinion Score are subjective and often take a lot of human time
and effort to rate the application’s user experience. Instead, objective
measurements of certain quantifiable factors which are known to contribute to
the experience of an application is easier, involves less human effort, and
allows us to quickly measure the performance of the network. The following QoS
metrics are relevant for evaluating the performance of streaming:

  * •

Throughput: Application layer throughput is one of the most basic yet
important metrics that determine performance. If a streaming service’s
throughput requirement is not met it will lead to buffering or quality change
in the video.

  * •

Buffer Health Status: As described above, DASH uses a buffer to maintain a
threshold of video playback time. The buffer at the client stores the incoming
burst to facilitate smooth playback of the video. When the traffic falls below
this threshold i.e. a buffer underrun occurs where the rate of consumption is
higher than the rate at which traffic enters the buffer, DASH adapts and
changes the video to a lower resolution to maintain the playback time. Change
in video rate severely affects the experience of watching the video, hence
maintaining the buffer health is one of the most important QoS metrics for
streaming [14].

  * •

Throughput Variation: While we may meet the average throughput requirements,
there might be large fluctuations in the instantaneous throughput. Variation
in throughput is not an absolute metric of performance, but it can severely
affect the buffer health due to the bursty nature of streaming traffic and
potentially be one of the contributing factors of QoS failure.

##  III TWT Background

In this work, we consider the Individual TWT operation mode where the Wi-Fi 6
AP negotiates TWT agreements with Wi-Fi 6 capable devices at a per-client
basis. Individual Target Wake Time comprises of the following parameters
(refer to Fig. 1):

Figure 1: An illustration of Individual TWT.

  * •

Target Wake Duration:  It is the minimum amount of time for which the TWT-
enabled client is awake and transmits/receives data. We denote Target Wake
Duration as TWT-SP. Each TWT-SP consists of multiple Wi-Fi 6 frame
transmissions.

  * •

Target Wake Interval:  It is the time difference between two consecutive TWT-
SPs where the client can go to sleep. We denote Target Wake Interval as TWT-
WI. TWT-WI can be periodic (known as ‘implicit’ TWT) or aperiodic (known as
‘explicit’ TWT) depending on the type of agreement AP negotiates with the
client.

  * •

Target Wake Time:  It is the time offset from the TWT negotiation till the
starting of the first TWT-SP.

In this work, we use the ‘Unannounced’ setting inside the Individual TWT
agreement, where the AP expects the clients to be awake throughout a service
period and the clients need not send any PS-POLL or APSD trigger frames to AP.
We also consider the negotiated TWT to be an ‘implicit’ type of agreement
where a TWT-enabled client wakes up and goes back to its sleep state
periodically.

###  III-A The TWT Duty Cycle and Multiplication Factor

We define TWT _Duty Cycle_ to be the percentage of time a TWT-enabled client
is awake throughout the TWT schedule. It is calculated as: $\frac{(TWT-
SP)}{(TWT-SP+TWT-WI)}\times 100\%$
It essentially dictates guaranteed wake time for the TWT-enabled client to
transmit/receive frames. How often the TWT-enabled client wakes up in a
particular TWT Duty Cycle is governed by another parameter, referred to as
_Multiplication Factor_ (MF) i.e. a TWT-enabled client can wake up more
(respectively, less) frequently if it uses a higher (respectively, lower) MF,
while maintaining the same TWT Duty cycle.

###  III-B Calculation of TWT Schedules

The IEEE 802.11ax standard defines the method to calculate the TWT schedules
which consist of the parameters mentioned in [1]. The implementation of TWT
puts an upper limit on the duration of TWT-SP, precisely $65535$ $\mu$S. As a
direct consequence, it impacts the TWT schedules, Duty Cycles and MF
calculations. The TWT schedule calculated with TWT-SP = $65535$ $\mu$S
corresponds to $\text{MF}=1$ for a particular TWT duty cycle. We derive TWT
schedules corresponding to a higher MF value by dividing both the TWT-SP and
the TWT-WI obtained from MF = 1, with the desired MF (approximately).

##  IV Experimental Setup and results

Figure 2: Experimental setup

The test bed used to perform our experiments is shown in the Fig. 2. We use an
enterprise-grade Wi-Fi 6 capable AP which supports individual TWT. The AP is
configured to use a 20MHz channel. AP is the central node that receives and
sends all data, provides a centralized way to solicit the TWT commands and
keep track of the performance of the network. The AP also doubles up as an
iPerf server to generate downlink TCP multistream traffic to load the network.
We use multiple non-TWT Wi-Fi 6 clients namely Client 1, Client 2, Client 3
that act as iPerf clients; to mimic real deployments, the clients have
different RSSI values. The device under test (DUT) is a TWT enabled client to
which we send synthetic traffic from the synthetic traffic server through a
multi-hop network and evaluate QoS. The Table II briefly describes the
physical parameters that are relevant to our experimentation.

Client Index |  RSSI (dBm) |  Peak Standalone Throughput |  Multi-client Scenario Throughput
---|---|---|---
1 |  -46 |  63.5 Mbps |  14.3 Mbps
2 |  -45 |  75.4 Mbps |  23 Mbps
3 |  -37 |  163 Mbps |  17.6 Mbps
4 (DUT) |  -36 |  95 Mbps |  21.3 Mbps
TABLE II: Setup Parameters.

###  IV-A Methodology

In this subsection, we discuss the methodology to obtain the appropriate TWT
schedule for the synthetic traffic model which guarantees the required QoS
requirements mentioned in Section II-B. We follow a two-stage approach where
at each stage, we satisfy one of the two important QoS requirements for this
work, i.e., average throughput and buffer health status, for a specified video
bit-rate. The procedure works as follows:

####  IV-A1 Phase 1

In this phase, we obtain the smallest TWT duty cycle required for a specified
bit-rate video stream. The metric used for this phase is average throughput;
the goal is to find the TWT duty cycle that satisfies the QoS requirement.
Towards this, we increase the duty cycle in steps of $5$% and observe the
corresponding average throughput values for a single TCP stream iPerf session
from the AP to the DUT. The rationale behind using iPerf here is that it is
one of the most common, open-source, tools for measuring network performance.
This gives us an upper bound on the average throughput that can be achieved by
the system for the given TWT duty cycle in an unloaded network.

Figure 3: Single TCP stream throughput variation with different TWT duty
cycles

From Fig. 3 we can look-up the required smallest TWT duty cycle value to
achieve the average throughput requirement for the DUT.

####  IV-A2 Phase 2

In this phase, the QoS metric to meet is buffer health status. In our
experimentation the buffer health status though being a client side metric, is
actually measured at the server for the following reasons:

  * •

The client is oblivious to the inter-burst time as it is sampled from Table I
by the server.

  * •

Since TCP is used (end to end transmission is guaranteed) the server knows
exactly when the burst has been served out. Coupled with the fact that the
server already knows the inter-burst time it makes it convenient to measure
the buffer health status from the server side.

This metric places a stricter demand on throughput; in order to maintain the
buffer health, i.e., avoid buffer underruns, we need the average throughput
during the inter-burst time to be larger than the video bit rate. Towards
this, we start with the smallest TWT duty cycle obtained from Phase $1$ and
observe the instantaneous throughput variation for different MF. From Fig. 4,
we observe that when MF is low, there is a large number of intermittent
deviations of the instantaneous throughput from the average throughput. These
fluctuations coupled with the bursty nature of streaming video results in a
situation where a burst is not serviced within the inter-burst time, causing a
buffer underrun event. In order to minimize the fluctuation in instantaneous
throughput, we increase MF in multiples of two until we observe stable
performance.

Figure 4: Instantaneous throughput variation for Different MF’s.

####  IV-A3 Phase 3

In this phase, we test the performance of the synthetic traffic model and
calculate the corresponding QoS metrics. From Phase $1$ and $2$, we obtain the
minimum required TWT duty cycle and MF required to maintain the QoS, and test
the performance of the CBR synthetic traffic model. If the TWT schedule does
not meet the QoS requirements, we increment the duty cycle by $5\%$ and
reevaluate the performance. Once we obtain the TWT schedule for the CBR model,
we test the performance of the VBR model.

###  IV-B Results

We generate video traffic (both CBR and VBR) at $15.6$ Mbps bit-rate, as
mentioned in Section II, and analyze the performance in terms of the required
QoS metrics, i.e., average throughput and buffer health status. Throughout
this section, QoS-1 refers to the average throughput and QoS-2 refers to the
number of buffer underrun events (i.e., the number of times the buffer health
status metric is not met) in each session. The set-up used for all the
experiments is according to Fig. 2 where the radio parameters are mentioned in
Table II. The performance of the system and the impact of adding the DUT in
the network are also analyzed.

Traffic model | Performance without TWT | Performance with TWT
---|---|---
Iteration 1 |  Iteration 2 |  Iteration 3 |  Iteration 4 |  Iteration 5
CBR | 61.2 |  66.3 |  56.6 |  62.4 |  59.4 |  61.4
VBR | 56.8 |  62.8 |  61.0 |  65.6 |  64.2 |  58.1
TABLE III: System performance with and without TWT.

####  IV-B1 Impact of TWT on the system performance

In this section we observe the impact of adding the DUT (which is receiving a
15.6 Mbps CBR stream) to the network, in the presence of three iPerf sessions
from the AP to Client 1, Client 2 and Client 3. Each iPerf session contains 8
parallel TCP streams (referred to as peak background congestion). The
following table shows the performance of the system and the impact of a $30$%
TWT duty cycle on the net throughput of the system. Table III shows the sum of
the throughput (in Mbps) of all iPerf sessions, not including the throughput
of the synthetic traffic stream.

From Table III, we observe that even with a $30$% TWT duty cycle, the net
performance of the system does not deteriorate. Essentially, the AP uses the
extra airtime to satisfy the demands of non-TWT clients.

####  IV-B2 Impact of MF on synthetic traffic performance

We evaluate the impact of varying MF on the performance of the CBR stream
while keeping the TWT duty cycle constant. We run the experiment with peak
background congestion. We define time spent in buffer underrun as time elapsed
after the inter-burst time to serve out the burst. In Fig. 5 we plot the total
time spent in buffer underrun state over the duration of a simulation for
different MF’s. From Fig. 5, we observe an increasing trend in performance,
with the best performance for MF 8 followed by a sharp decrease thereafter.
From Wireshark packet capture analysis, we note that when the value of MF is
low, the corresponding TWT schedule has a large sleep time (TWT-WI) which
delays TCP window size progression and leads to long round trip times at the
tail end of the service periods. On the other hand, for higher MF values, the
corresponding TWT schedule has a small wake duration (TWT-SP) which
essentially hinders MAC layer frame aggregation, causing poor performance.

Figure 5: Total time spent in buffer underrun state over the duration of a simulation for different MF’s. Iteration No. | Duty cycle parameters
---|---
25% MF 8 | 30% MF 8
| QoS 1 | QoS 2 | QoS 1 | QoS 2
1 | 15 Mbps | 5 | 16 Mbps | 3
2 | 15 Mbps | 5 | 16 Mbps | 2
3 | 16 Mbps | 2 | 16 Mbps | 1
4 | 16 Mbps | 4 | 16 Mbps | 0
5 | 15 Mbps | 4 | 16 Mbps | 1
TABLE IV: Performance comparison between $25$% and $30$% TWT duty cycle with MF $8$ on CBR stream Iteration No. | Traffic Model
---|---
CBR | VBR
| QoS 1 | QoS 2 | QoS 1 | QoS 2
1 | 16 Mbps | 3 | 14 Mbps | 2
2 | 16 Mbps | 2 | 15 Mbps | 0
3 | 16 Mbps | 1 | 14 Mbps | 2
4 | 16 Mbps | 0 | 15 Mbps | 3
5 | 16 Mbps | 1 | 14 Mbps | 1
TABLE V: Performance comparison of VBR and CBR traffic model for $30$% TWT
duty cycle with MF $8$

####  IV-B3 Performance variation for different TWT duty cycles

From the Section 3 we obtain a $30$% TWT duty cycle with MF $8$ as the ideal
choice to start testing the performance of the CBR stream. From Table IV we
observe that it satisfies both the QoS requirements. These tests were
performed with peak background congestion. Table IV summarizes the performance
of CBR traffic for $25$% and $30$% TWT duty cycle with MF $8$. Each session of
synthetic traffic lasted for about $2$ minutes.

####  IV-B4 Performance Comparison of CBR and VBR traffic model

As described in Section II-A2 the requirements of VBR traffic are more
stringent than CBR traffic. In this experiment, both traffic sources emulate a
$15.6$ Mbps bit-rate video with each streaming session running for about $2$
minutes. Table V summarizes the performance of both CBR and VBR models for
$30$% TWT duty cycle with MF $8$. There is no discernible difference in the
performance of both the synthetic traffic models with the TWT parameters
obtained from our approach.

##  V Conclusion and Future Works

In this work, we consider a network which comprises of a Wi-Fi $6$ AP, a TWT-
enabled client, and multiple background non-TWT clients running at peak
congestion.

Figure 6: Planned future work.

We analyze the performance evaluation metrics of synthetic traffic models that
emulate streaming and show the effect of TWT on the network performance.
Initial testing shows that key QoS metrics can be met for streaming traffic
upon enabling TWT even in the presence of peak background congestion in the
network. The unused airtime (i.e. the time remaining after servicing the TWT-
enabled client) obtained from enabling TWT is utilized by the AP in servicing
other non-TWT clients while satisfying the necessary QoS requirements of the
TWT-enabled client. Since the objective of this work is to primarily schedule
video streaming traffic in a congested network scenario, we do not measure the
power consumption. Consequently by providing a TWT schedule, we indeed save
power when compared to staying awake all the time while satisfying the QoS
metrics. As an extension of this work, we plan to analyze different classes of
applications and their respective QoS requirements. We also plan to evaluate
the performance of an application and provide a suitable TWT schedule in order
to satisfy the QoS needs of the application.

## References

  * [1] “IEEE Standard for Information Technology–Telecommunications and Information Exchange between Systems Local and Metropolitan Area Networks–Specific Requirements Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications Amendment 1: Enhancements for High-Efficiency WLAN,” _IEEE Std 802.11ax-2021 (Amendment to IEEE Std 802.11-2020)_ , pp. 1–767, 2021.
  * [2] E. de Carvalho Rodrigues, A. García-Rodríguez, L. G. Giordano, and G. Geraci, “On the latency of IEEE 802.11ax WLANs with parameterized spatial reuse,” 2020. [Online]. Available: https://arxiv.org/abs/2008.07482
  * [3] M. Nurchis and B. Bellalta, “Target wake time: Scheduled access in IEEE 802.11ax WLANs,” _IEEE Wirel. Commun._ , vol. 26, no. 2, pp. 142–150, 2019\.
  * [4] K. Dovelos and B. Bellalta, “Optimal resource allocation for uplink OFDMA in 802.11ax networks,” _CoRR_ , vol. abs/1811.00957, 2018. [Online]. Available: http://arxiv.org/abs/1811.00957
  * [5] S. Bhattarai, G. Naik, and J.-M. J. Park, “Uplink resource allocation in ieee 802.11ax,” in _ICC 2019 - 2019 IEEE International Conference on Communications (ICC)_ , 2019, pp. 1–6.
  * [6] “Multi-User MIMO in Wi-Fi 6 - Arista Networks”,. [Online]. Available: https://www.arista.com/assets/data/pdf/Whitepapers/MU-MIMO-Whitepaper.pdf
  * [7] “11ax Evaluation Methodology”, IEEE 802.11-14/0571r12. [Online]. Available: https://mentor.ieee.org/802.11/dcn/14/11-14-0571-12-00ax-evaluation-methodology.docx
  * [8] A. Mondal, S. Sengupta, B. R. Reddy, M. Koundinya, C. Govindarajan, P. De, N. Ganguly, and S. Chakraborty, “Candid with youtube: Adaptive streaming behavior and implications on data consumption,” in _Proceedings of the 27th Workshop on Network and Operating Systems Support for Digital Audio and Video_ , 2017, pp. 19–24.
  * [9] “Streaming video - mbs to gbs - mobility report - ericsson.” [Online]. Available: https://www.ericsson.com/en/reports-and-papers/mobility-report/articles/streaming-video
  * [10] “What is MPEG-DASH? HLS vs. DASH — cloudflare.com.” [Online]. Available: https://www.cloudflare.com/learning/video/what-is-mpeg-dash/
  * [11] T. Zhang, F. Ren, W. Cheng, X. Luo, R. Shu, and X. Liu, “Modeling and analyzing the influence of chunk size variation on bitrate adaptation in DASH,” in _IEEE INFOCOM 2017-IEEE Conference on Computer Communications_. IEEE, 2017, pp. 1–9.
  * [12] J.-M. Jeong and J.-D. Kim, “Effective bandwidth measurement for dynamic adaptive streaming over http,” in _2015 International Conference on Information Networking (ICOIN)_ , 2015, pp. 375–378.
  * [13] “11ax Evaluation Methodology”, IEEE 802.11-14/0571r12. [Online]. Available: https://mentor.ieee.org/802.11/dcn/14/11-14-0571-12-00ax-evaluation-methodology.docx
  * [14] M. Ghobadi, Y. Cheng, A. Jain, and M. Mathis, “Trickle: Rate limiting YouTube Video Streaming,” in _2012 USENIX Annual Technical Conference (USENIX ATC 12)_ , 2012, pp. 191–196.