# How Much Temporal Long-Term Context is Needed for Action Segmentation?

Emad Bahrami1 Gianpiero Francesca2 Juergen Gall1,3
1University of Bonn, Germany 2Toyota Motor Europe, Belgium
3Lamarr Institute for Machine Learning and Artificial Intelligence, Germany

###### Abstract

Modeling long-term context in videos is crucial for many fine-grained tasks
including temporal action segmentation. An interesting question that is still
open is how much long-term temporal context is needed for optimal performance.
While transformers can model the long-term context of a video, this becomes
computationally prohibitive for long videos. Recent works on temporal action
segmentation thus combine temporal convolutional networks with self-attentions
that are computed only for a local temporal window. While these approaches
show good results, their performance is limited by their inability to capture
the full context of a video. In this work, we try to answer how much long-term
temporal context is required for temporal action segmentation by introducing a
transformer-based model that leverages sparse attention to capture the full
context of a video. We compare our model with the current state of the art on
three datasets for temporal action segmentation, namely 50Salads, Breakfast,
and Assembly101. Our experiments show that modeling the full context of a
video is necessary to obtain the best performance for temporal action
segmentation.

##  1 Introduction

Temporal action segmentation can be used in many real-world applications such
as monitoring production lines or studying animal behavior. In these settings,
the videos can be very long and it is required to recognize the start and end
of all actions that occur in a video as illustrated in Fig. 1.

Figure 1: Datasets like Assembly101 contain long videos of assembly tasks and
an action label needs to be predicted for each frame. The first row shows some
frames of a video. The second row shows the ground-truth labels for all frames
of the video where different colors correspond to different action labels.
Rows 3-6 show the predictions of the proposed model for different amounts of
long-term context where 100% means the temporal context of the full video.

Recently, combinations of temporal convolutional networks [1, 30] with self-
and cross-attention from transformers [66, 4] have shown impressive results
for temporal action segmentation. These works are in line with other hybrid
models [16, 9, 66, 63] that combine the attention modules with convolutions to
compensate for the lack of strong inductive bias of pure transformers.
However, the emergence of datasets like Assembly101 [46], where subjects
perform assembly tasks, poses a new challenge in the area of temporal action
segmentation due to the existence of long videos that can last up to 25
minutes. Since modeling the long-term context of such a video is very
expensive, Yi _et al_. [66] proposed to compute the attention for a local
temporal window. In order to understand the impact of the window size on the
temporal action segmentation accuracy, we analyze the impact of the window
size on two datasets with long videos in Section 4.1. Fig. 1 shows some
qualitative results of this study on Assembly101. Indeed, the results show
that modeling the long-term context of an entire video is very important for
temporal action segmentation.

Based on this finding, we revisit how temporal attention is modeled in
transformer architectures for temporal action segmentation. [66, 4] use a
hierarchy of temporal windows, making training on long video sequences as they
occur in Assembly101 [46] very expensive. Inspired by works that decompose
attention over the spatial and temporal domain for short video clips [6, 68],
we propose to iterate between computing windowed local attention and sparse
long-term context attention such that both short and long-term context is
modeled. This approach is particularly suitable for temporal action
segmentation since the local attentions focus on the similarity or
dissimilarity of features within an action segment or between neighboring
action segments whereas the long-term context attention focuses on the
relations between actions within the entire video. The source code is
available at https://github.com/LTContext/LTContext.

##  2 Related Work

The traditional sliding window with non-maxima suppression has been among the
early approaches for action segmentation [20, 45]. [24, 22, 51] adopted hidden
Markov Models (HMMs) for temporal modeling. [44] incorporates length and
temporal context and uses dynamic programming for inference. Temporal
convolutional networks (TCN) with temporal pooling are used by [25] to
classify each video frame. Later, [1] introduced a multi-stage TCN capable of
maintaining high temporal resolution, which is necessary for fine-grained
recognition. [19] reduces the over-segmentation error by adding an action
boundary regression network to refine the frame-wise results. [18] proposed
another refinement method based on a graph convolutional network.

Transformers [57] were originally used for natural language processing and
only rely on attention for sequence modeling. Recently, transformers have been
widely adopted in vision [12, 62, 34, 59], speech recognition [16, 41], and
action recognition [6, 2]. The original transformers suffered from two issues.
First, the cost of the self-attention operation is quadratic with respect to
the sequence length. This has been addressed by many methods that improve the
memory efficiency of transformers [5, 38, 10]. We refer to [52] for a
comprehensive survey of efficient transformers. For instance, restricting the
attention to a fixed window size [5] is one approach. Multi-axis self-
attention [56] combines block-attention with grid-attention, which is based on
a spatial grid overlaid on the entire 2D space. Adapting sparse attention for
capturing global information has shown to be an effective solution in vision
tasks such as high-resolution image generation [69, 55], object detection [56,
65], and instance segmentation [56, 65]. MViT [13] uses a pooling operation to
reduce the space-time resolution before attention and MViTv2 [31] improves
MViT by adding the relative position along (x,y,t) and residual connections.
The second issue of transformers is the poor generalization due to a
relatively weak inductive bias [9, 54] compared to convolutional neural
networks (CNN). Hybrid models that combine self-attention and convolution
layers have thus been proposed for vision tasks [63, 9, 56, 62] and speech
recognition [16, 41].

In action recognition, works such as [6, 68] adopt the idea of using attention
for video understanding. In TimesFormer [6], the self-attention is first
applied along the temporal dimension for each single patch (time attention),
i.e., over all frames of a short video clip of a few seconds. In a second
step, the attention is applied over the patches of each frame (space
attention). VIDTR [68] also decomposes self-attention into spatial and
temporal attention, but additionally down-samples the temporal dimension. In
our work, we do not apply attention spatially and temporally, but we address
the question of how temporal attention can be computed for very long sequences
that last 25 minutes as it is required for temporal action segmentation.

For temporal action segmentation, [66] proposed an architecture, called
ASFormer, which is based on a multi-stage TCN [1] and equips the temporal
convolutions with a local window attention [5]. This is done in a hierarchy
where the size of the local window grows with each layer.

Recently, [4] proposed a decoder on top of the encoder that generates action
segments in an autoregressive manner. It uses two heads where the head on top
of the encoder predicts frame-wise probabilities and the head of the decoder
predicts the sequence of actions. Finally, an alignment decoder fuses the
output of the two heads and aligns the predicted sequence of actions to the
frames. [3] introduced a hybrid Temporal Convolution Transformer (TCTr) where
they adapt an action boundary detector to adaptively estimate attention from
local neighboring frames. [64] proposed to use additional constraints during
training, but the approach assumes that action sequences can be modelled by a
directed acyclic graph, which does not allow that actions occur more than once
in a video. Recently, multi-modal approaches that combine language models with
vision transformers have been proposed. For instance, [29] uses prompt
engineering to extract features from pre-trained vision-language models such
as ActionCLIP [60].

##  3 Long-Term Context for Action Segmentation

Recently, transformers combined with temporal convolutional neural networks
[66, 4, 3] have shown very good performance for temporal action segmentation.
For this task, the frame-wise labels $c_{1},\ldots,c_{T}$, where
$c_{t}\in\mathcal{C}$ and $C{=}|\mathcal{C}|$ denotes the number of action
classes, need to be predicted for a given a video $X=(x_{1},\ldots,x_{T})$
with $T$ frames, where $x_{t}$ represents a feature map of size $D$ at frame
$t$. Since $T$ can be very large, [66, 4, 3] limit the self-attention to a
local temporal window.

In order to understand how much temporal long-term context is needed for
temporal action segmentation, we limited the temporal input window and
evaluated the quality of the temporal action segmentation in Section 4.1. The
results in Fig. 4 show that temporal long-term context has a strong impact on
the performance. Based on our analysis, we thus revisit the windowed attention
of previous works for temporal action segmentation and propose to model the
temporal long-term context of a video using sparse attentions [5, 42, 40].
Additionally, we equip our method with windowed attention to capture the
locality between neighboring frames. In this way, we obtain a flexible design
that is capable of providing long-term and local temporal context. While we
describe first the Long-Term Context (LTContext) block in Section 3.1, the
entire network is described in Section 3.2.

Figure 2: Illustration of windowed and long-term context attentions with a
window of size 2. For the windowed attention, the sequence is partitioned into
small windows and the attentions are computed for each window. For long-term
context attention, the sequence is reordered such that the attentions are
computed over the whole, but sparsely sampled sequence. After the attention,
the output is reordered again to preserve the original order. Best viewed in
color.

###  3.1 Temporal Context Attention

The self-attention blocks of transformers are advantageous over convolutions
in the aggregation of global information. However, applying attention to long
sequences such as untrimmed videos is impractical due to the quadratic
complexity of the self-attention blocks. To address this issue, we adopt an
attention mechanism where we leverage sparse and windowed attention to model
long-term and local temporal context.

The attention function transforms the input into a query, key, and value and
computes the output as a weighted sum of the values. For example, given a
sequence of features, $X\in\mathbb{R}^{T\times D}$, the attention can be
written as follows:

$Attention(Q,K,V)=\operatorname{softmax}_{row}\left(\frac{QK^{T}}{\sqrt{D}}\right)V$
(1)

where $Q,K,V\in\mathbb{R}^{T\times D}$ are linearly transformed from $X$.
Since $T$ is very large for long sequences, we need to modify $Q,K,V$ to
enable the modeling of long-term and local temporal context as illustrated in
Fig. 2.

#### Temporal Windowed Attention

For the temporal Windowed Attention (WA), we partition the sequence into non-
overlapping windows of size $W$. Fig. 2 illustrates the case for $W=2$, but we
use $W=64$ in practice. The impact of $W$ is evaluated in Section 4. Instead
of computing the attention over the entire sequence of length $T$, we compute
the attention $\frac{T}{W}$ times, where each query $Q\in\mathbb{R}^{W\times
D}$ corresponds to each window. For the keys and values, we use an overlap
where we use the next window in addition, _i.e_., for each query $Q$ we have
$K,V\in\mathbb{R}^{2W\times D}$. We perform masking when $K$ and $V$ exceed
the input sequence. We evaluate the impact of the overlap in Section 4.

#### Temporal Long-Term Context Attention

For the temporal Long-Term Context (LTC) attention, the input is also
partitioned into non-overlapping windows of size $G$. However, instead of
computing the attention over each window, the attention is computed over all
windows where from each window only one element is taken. In the illustration
of Fig. 2 with $G=2$, we compute the attention over the first feature of all
windows and the attention over the second feature of all windows. In general,
we compute the attentions for ${G}$ queries $Q\in\mathbb{R}^{\frac{T}{G}\times
D}$ where the keys and values are the same, _i.e_.,
$K,V\in\mathbb{R}^{\frac{T}{G}\times D}$. The parameter $G$ provides the
flexibility to adjust the sparseness based on the available memory budget,
_e.g_., $G=1$ corresponds to the case where the attention is applied over the
full sequence. In practice, we use $G=64$ and we evaluate the impact of $G$ in
Section 4.

#### LTContext block

The top of Fig. 3 illustrates the entire LTContext block. As in previous works
[1, 66, 4], we use a 1D dilated temporal convolution with kernel size 3, where
the dilation factor increases by factor $2$ for each layer. The dilated
temporal convolution is followed by a Gaussian Error Linear Unit (GELU). In
the LTContext block, we first use the windowed and then the long-term context
attention, which are shown in Fig. 2. We evaluate the impact of the order in
Section 4. Finally, we use a linear layer with a residual connection to output
the features for each frame, $F\in\mathbb{R}^{T\times D}$.

Figure 3: The network architecture of LTContext with LTContext blocks (top).

###  3.2 LTContext Architecture

The entire LTContext network is depicted in Fig. 3. For a fair comparison, we
will use the features that are provided for the corresponding datasets as
input. In all cases, the dimensionality of the features is 2048. As in
previous works, we use a linear layer to reduce the dimensionality of the
features to 64. The output of each LTContext block is the feature map
$F\in\mathbb{R}^{T\times D}$. We repeat each LTContext block $N$ times where
the dilation factor of the temporal convolution increases in each layer. In
practice, we use $N=9$ and we evaluate the impact of $N$ in Section 4. After
the first $N$ layers of LTContext blocks, we use an additional linear layer to
reduce the dimensionality $D$ further to 32. The dimensionality reduction
reduces the number of parameters from $1.42$ million to $0.72$ million without
reducing the accuracy. We also use an additional linear layer followed by a
softmax layer to generate the frame-wise class probabilities
$P\in\mathbb{R}^{T\times C}$.

We continue with three additional stages where each stage consists of $N$
layers of LTContext blocks. Note that we reset the dilation factor to 1 for
the temporal convolution at the beginning of each stage and we compute the
frame-wise class probabilities $P\in\mathbb{R}^{T\times C}$ after each stage,
which contributes to the multi-stage loss. We use the cross-entropy loss
combined with the mean squared error smoothing loss as introduced by [1] and
used in [30, 66] for a fair comparison. Inspired by [66], we use the cross-
attention for the LTContext blocks in stages 2 to 4. Instead of using the
features $F$ for the queries and keys for windowed and long-term context
attention, the predictions $P$ are used. We thus have $Q\in\mathbb{R}^{W\times
C}$, $K\in\mathbb{R}^{2W\times C}$, and $V\in\mathbb{R}^{2W\times D}$ for the
windowed attention and $Q\in\mathbb{R}^{\frac{T}{G}\times C}$,
$K\in\mathbb{R}^{\frac{T}{G}\times C}$, and $V\in\mathbb{R}^{\frac{T}{G}\times
D}$ for the long-term context attention. While the cross-attention is not
shown in Fig. 3, it only means that $P$ is an additional input for the
windowed and long-term context attention in stages 2-4. We evaluate the impact
of the number of stages in Section 4.

##  4 Experiments

Datasets. We evaluate the performance of our proposed model on three
challenging action segmentation datasets: 50Salads [50], Breakfast [23], and
Assembly101 [46].

50Salads [50] contains 50 videos annotated with 17 action classes. On average,
each video is 6.4 minutes long and has 18 action segments. Following previous
works [1, 66, 4], we use five-fold cross-validation and report the average.

Breakfast contains 1,712 videos of breakfast preparation activities with an
average length of 2.3 minutes. There are 48 action classes and each video has
on average 6.6 action segments. For evaluation, we report the average of the 4
splits for cross-validation as in [1].

Assembly101 [46] is the largest dataset among the three datasets with 4,321
videos and 202 coarse action classes composed of 11 verbs and 61 objects.
Assembly101 is a procedural activity dataset containing videos of people
assembling and disassembling 101 toy vehicles. On average, each video includes
24 action segments and is 7.1 minutes long. Compared to Breakfast, Assembly101
has 2.5 times more videos, 6.7 times more hours of video footage, 9.3 times
more action segments, and 4.2 more action classes. For our evaluation, we
follow the setting for temporal action segmentation [46] and report the
results on the validation set.

For a fair comparison, we use the features that are provided for the datasets
and that have been used in previous works. We use the I3D [7] features for the
50Salads and Breakfast datasets and TSM [32] features for the Assembly101
dataset [46]. Both features are 2048 dimensional vectors. Following [1], we
also used the temporally downsampled features for 50Salads to compensate for
the different frame-rates of the datasets.

Evaluation Metrics. We report the frame-wise accuracy (Acc), segmental Edit
distance, and segmental F1 score at the overlapping thresholds of $10\%$,
$25\%$, and $50\%$ denoted by $F1@\\{10,25,50\\}$. The intersection over the
union (IoU) ratio is used as the overlapping threshold. The edit distance
measures only the order of the actions but not the duration. The frame-wise
accuracy measures the accuracy per frame. It is dominated by actions that have
a long duration and it is not very sensitive to over-segmentation errors that
occur when a few frames within a ground-truth action segment are wrongly
classified. The F1 score is the most reliable measure.

###  4.1 How much temporal long-term context is needed?

We first present the results of our analysis on the impact of using the full
sequence as input compared to using a temporal window. The goal of this
experiment is to shed light on how much temporal long-term context is needed
for the task of temporal action segmentation. For the analysis, we use only
50Salads and Assembly101 since the videos in Breakfast are too short. For the
experiments, we train our approach (LTContext) and ASFormer [66] either on the
full video sequences or we divide the videos into shorter sequences where the
full context is lacking.

Fig. 4 shows the result of this experiment on the 50Salads and Assembly101
dataset. We report the window size as a percentage of the average length of a
video in the corresponding dataset and 100% means that the full sequence has
been used. The results clearly show that the full context of an entire
sequence is advantageous over allowing the model to see only a window of the
input sequence even if the window is large (50%). We can also observe that our
approach benefits more from the full sequence than ASFormer.

We furthermore evaluated whether the impact of the window size is stronger for
longer videos. To this end, we sorted all test videos into four quarters by
their length. Fig. 5 (left) shows that the difference between the window size
50% and the full video (100%) is larger for long videos. This shows that long-
term context is in particular for long videos important. We also evaluated
whether choosing a window for each video instead of choosing a window based on
the average video length (fixed) performs better. For the video-specific
window size, the window size is set to the percentage of each video. Fig. 5
(right) shows that a video-specific window size performs much worse than a
fixed window. Varying the amount of context for each video is thus not
beneficial. Fig. 1 shows qualitative results of our approach for different
amounts of temporal context with a fixed window for a video from the
Assembly101 dataset.

Figure 4: Impact of different sizes of the input window on the 50Salads
dataset (left) and the Assembly101 dataset (right). The window size is given
in percentage of the average length of a video in the corresponding dataset.
100% denotes the entire video.

Figure 5: Results for different video lengths (rights). Comparison of a fixed
with a video-specific input window size (left). Both plots are for 50Salads.

Method | Note | Breakfast | 50Salads
---|---|---|---
F1@{10, 25, 50} | Edit | Acc | F1@{10, 25, 50} | Edit | Acc
IDT+LM [44] | DF1 | - | - | - | - | - | 44.4 | 38.9 | 27.8 | 45.8 | 48.7
ST-CNN [26] | DF2 | - | - | - | - | - | 55.9 | 49.6 | 37.1 | 45.9 | 59.4
ED-TCN [25] | DF2 | - | - | - | - | - | 68.0 | 63.9 | 52.6 | 59.8 | 64.7
TDRN [28] | DF2 | - | - | - | - | - | 72.9 | 68.5 | 57.2 | 66.0 | 68.1
SSA-GAN [14] | DF3 | - | - | - | - | 43.3 | 74.9 | 71.7 | 67.0 | 69.8 | 73.3
Bridge-Prompt [29] | DF4 | - | - | - | - | - | 89.2 | 87.8 | 81.3 | 83.8 | 88.1
C2F-TCN [47] | TA | 72.2 | 68.7 | 57.6 | 69.6 | 76.0 | 84.3 | 81.8 | 72.6 | 76.4 | 84.9
UVAST [4]+Viterbi | P | 75.9 | 70.0 | 57.2 | 76.5 | 66.0 | 89.1 | 87.6 | 81.7 | 83.9 | 87.4
UVAST [4]+FIFA [48] | P | 76.9 | 71.5 | 58.0 | 77.1 | 69.7 | 88.9 | 87.0 | 78.5 | 83.9 | 84.5
Liu [35] \+ ASRF [19] | P | 77.5 | 72.3 | 59.5 | 76.7 | 73.7 | 87.9 | 86.6 | 80.5 | 82.7 | 86.6
DTL [64] | C | 78.8 | 74.5 | 62.9 | 77.7 | 75.8 | 87.1 | 85.7 | 78.5 | 80.5 | 86.9
MS-TCN [1] | - | 52.6 | 48.1 | 37.9 | 61.7 | 66.3 | 76.3 | 74.0 | 64.5 | 67.9 | 80.7
MS-TCN++ [30] | - | 64.1 | 58.6 | 45.9 | 65.6 | 67.6 | 80.7 | 78.5 | 70.1 | 74.3 | 83.7
DTGRM [58] | - | 68.7 | 61.9 | 46.6 | 68.9 | 68.3 | 79.1 | 75.9 | 66.1 | 72.0 | 80.0
MuCon [49] | - | 73.2 | 66.1 | 48.4 | 76.3 | 62.8 | - | - | - | - | -
Gao  _et al_. [15] | - | 74.9 | 69.0 | 55.2 | 73.3 | 70.7 | 80.3 | 78.0 | 69.8 | 73.4 | 82.2
BCN [61] | - | 68.7 | 65.5 | 55.0 | 66.2 | 70.4 | 82.3 | 81.3 | 74.0 | 74.3 | 84.4
SSTDA [8] | - | 75.0 | 69.1 | 55.2 | 73.7 | 70.2 | 83.0 | 81.5 | 73.8 | 75.8 | 82.2
C2F-TCN [47] | - | 70.1 | 66.6 | 56.2 | 68.2 | 73.5 | 76.6 | 73.0 | 62.5 | 69.2 | 80.1
ASRF [19] | - | 74.3 | 68.9 | 56.1 | 72.4 | 67.6 | 84.9 | 83.5 | 77.3 | 79.3 | 84.5
UVAST [4] | - | 76.7 | 70.0 | 56.6 | 77.2 | 68.2 | 86.2 | 81.2 | 70.4 | 83.9 | 79.5
DPRN [39] | - | 75.6 | 70.5 | 57.6 | 75.1 | 71.7 | 87.8 | 86.3 | 79.4 | 82.0 | 87.2
LGTNN [53] | - | 76.2 | 71.5 | 57.5 | 75.2 | 72.5 | 87.5 | 86.2 | 79.8 | 82.0 | 86.1
ASFormer [66] | - | 76.0 | 70.6 | 57.4 | 75.0 | 73.5 | 85.1 | 83.4 | 76.0 | 79.6 | 85.6
TCTr [3] | - | 76.6 | 71.1 | 58.5 | 76.1 | 77.5 | 87.5 | 86.1 | 80.2 | 83.4 | 86.6
LTContext (Ours) | - | 77.6 | 72.6 | 60.1 | 77.0 | 74.2 | 89.4 | 87.7 | 82.0 | 83.2 | 87.7

Table 1:  Results on the Breakfast and 50Salads datasets. The best and second
best results for the methods in the bottom half are shown in bold and
underlined since only methods without additional notes are directly
comparable. P: additional post-processing; C: additional constraints; DF:
different features (DF1: Improved Dense Trajectories (IDT), DF2: Spatio-
temporal VGG-style CNN, DF3: Generative Adversarial Network (GAN), DF4:
ActionCLIP); TA: test augmentation. In the top half, we highlight results that
are better in italic bold formatting.

Method | Assembly101
---|---
F1@{10, 25, 50} | Edit | Acc | Params (M) | Inference (sec)
MS-TCN++ [30] | 31.6 | 27.8 | 20.6 | 30.7 | 37.1 | 1.08 | 0.1
UVAST [4]* | 32.1 | 28.3 | 20.8 | 31.5 | 37.4 | 1.22 | 3.2
C2F-TCN [47] | 33.3 | 29.0 | 21.3 | 32.4 | 39.2 | 6.89 | 0.1
ASFormer [66]* | 33.4 | 29.2 | 21.4 | 30.5 | 38.8 | 1.13 | 2.1
LTContext (Ours) | 33.9 | 30.0 | 22.6 | 30.4 | 41.2 | 0.72 | 0.6

Table 2:  Comparison with state-of-the-art methods on the Assembly101 dataset.
The best and second best results are shown in bold and underlined. *We trained
UVAST [4] and ASFormer [66] using the code of the authors.

###  4.2 Comparison with State of the Art

We present the performance comparison of our method with state-of-the-art
methods on the Breakfast and 50Salads datasets in Table 1 and on the
Assembly101 dataset [46] in Table 2.

On Breakfast and 50Salads, our method outperforms all comparable methods in
terms of F1 score at all thresholds, which is the most important measure. Our
method also achieves a better Edit score than all methods except for UVAST [4]
and TCTr [3]. UVAST uses ASFormer [66] as encoder and an additional alignment
decoder. The higher Edit score of UVAST is expected since the approach has an
additional head that predicts the sequence of actions and thus maximizes the
Edit score by an additional loss. This, however, comes at the cost of a much
lower frame accuracy. TCTr [3] achieves a higher frame-wise accuracy on
Breakfast and a higher Edit score on 50 Salads, but it performs worse for the
other metrics. The approach uses a boundary detection module and compresses
the temporal features, which is a complementary approach. It needs to be noted
that our approach outperforms TCTr for all metrics on 50Salads if we use 10
instead of 9 layers as shown in Table 9. However, 9 layers perform better than
10 layers for the other datasets. We therefore report in Tables 1 and 2 only
the results with 9 layers.

We also compare to approaches that use different input features, perform
additional test augmentation, or add additional constraints. On Breakfast,
only DTL [64] reports better results. DTL [64] uses additional logic-based
constraints for training ASFormer and assumes that action sequences can be
modeled by a directed acyclic graph. This is very efficient on the Breakfast
dataset, which has relatively few actions per video compared to the other
datasets. However, our approach outperforms DTL for all metrics on 50Salads
where the structure of the action sequences is more complex. Furthermore, DTL
cannot be applied to datasets like Assembly101 where actions occur more than
once in a video. On 50Salads, only Bridge-Prompt [29] performs slightly better
for some metrics. Bridge-Prompt proposes an approach for feature learning
using a vision-language model such as ActionCLIP [60] in combination with
ASFormer. The approach is thus complementary and not comparable to our
approach. Nevertheless, our approach achieves higher $F1@10$ and $F1@50$
scores on 50Salads. If we use 10 instead of 9 layers as shown in Table 9, we
even outperform Bridge-Prompt for all measures except for frame-wise accuracy.
In summary, our approach outperforms all methods in terms of F1 score, which
is the most precise measure of the segmentation quality, at all thresholds on
50Salads.

Assembly101 is the largest dataset both in terms of the number of videos and
their length, and it is the most challenging dataset. Since ASFormer [66] and
UVAST [4] have not been evaluated on this dataset, we trained ASFormer and
UVAST on Assembly101 using the publicly available source code and report the
results in Table 2 as well. We outperform all methods on Assembly101 in F1
scores. C2F-TCN [47] achieves the best edit score. Since we trained ASFormer
[66] and UVAST [4] by ourselves, we can compare the training time on
Assembly101. While our approach requires 1 day and 18 hours, ASFormer [66] and
UVAST [4] needed 4 weeks and 2 weeks, respectively.

Figure 6: Qualitative results on Assembly101. The three rows show the ground-
truth labels, the predictions by ASFormer, and the predictions by the proposed
approach LTContext. It can be best viewed by using the zoom function of a PDF
viewer.  Figure 7: Qualitative results on Breakfast. The three rows show the
ground-truth labels, the predictions by ASFormer, and the predictions by the
proposed approach LTContext. Figure 8: Qualitative results on 50Salads. The
three rows show the ground-truth labels, the predictions by ASFormer, and the
predictions by the proposed approach LTContext.

###  4.3 Qualitative Evaluation

In Figs. 6-8, we present some qualitative results for the Assembly101,
Breakfast, and 50Salads dataset. The first row of each figure shows the middle
frame of each ground-truth action segment. In the second, third, and fourth
rows, the ground truth segmentation, the predictions of ASFormer [66], and the
prediction of our model (LTContext) are shown, respectively. In Fig. 6, our
approach shows much fewer errors than ASFormer. Although our approach
recognizes all action classes that occur in the video, there are several
errors where some instances are missed like the last action segment of the
bottom row which corresponds to the action ‘detach base’. This indicates how
challenging the Assembly101 dataset is. In Fig. 7, the predictions of our
model are very close to the ground-truth. ASFormer overestimates the duration
of the green segment, which corresponds to the action ‘spoon powder’, and
hallucinates purple segments at the beginning and end of the video. In Fig. 8,
both methods estimate the segments well, but ASFormer predicts wrongly two
orange segments, which correspond to the action ‘mix dressing’, and the olive
segment is too short, which corresponds to the action ‘add salt’.

###  4.4 Ablation Studies

We finally evaluate the impact of each component of our architecture. For the
ablation studies, we report the results averaged over the 5 splits of the
50Salads dataset.

Model Architecture | F1@{10, 25, 50} | Edit | Acc
---|---|---|---
LTContext | 89.4 | 87.7 | 82.0 | 83.2 | 87.7
\- Windowed Attention | 87.7 | 85.1 | 77.4 | 82.1 | 85.5
\- LTContext Attention | 84.1 | 82.6 | 74.1 | 78.2 | 85.3
WA (S1) + LTContext (S2-S4) | 87.8 | 85.0 | 79.4 | 81.4 | 85.2

Table 3:  Impact of using LTContext and Windowed Attention (WA) on 50Salads.
In the case of Windowed Attention, we use two windowed attention blocks
instead of a combination of windowed and LTContext attention. In the case of
LTContext Attention, we use two LTContext attention blocks. The last row
corresponds to using only windowed attention in stage 1 and only LTContext
attention in stages 2 to 4.

#### Impact of attention types

In Table 3, we show the impact of using the combination of windowed and long-
term context (LTContext) attention in the LTContext block illustrated in Fig.
3. We first compare it to two variants where we use only windowed or only
long-term context attention. In order to keep the number of parameters the
same, we still use in these cases two attention blocks within LTContext. The
results show that combining both types of attention leads to better results.
In particular, the F1@50 score is substantially higher.

As shown in Fig. 3, we use four LTContext blocks. We also evaluate what
happens if we vary the attention not within an LTContext block but between the
four LTContext blocks. For this, we use only windowed attention for the first
LTContext block and only long-term context attention for the other three
LTContext blocks. The last row in Table 3 shows that this does not perform
better than using only windowed attention and it is worse than combining
windowed and long-term context attention within an LTContext block.

#### Impact of different values of $W$ and $G$

The parameter $W$ controls the size of the local window for the local
attention and the parameter $G$ controls the sparseness of the global
attention. If not otherwise specified, we use $W=G=64$ in our experiments. The
results in Table 4 show that the F1 score drops when the size of the local
window $W$ becomes smaller. Note that larger values of $W$ increase the memory
and computational cost. When we decrease $G$, the F1 score also drops but not
as drastic as for $W$.

$W$ | $G$ | F1@{10, 25, 50} | Edit | Acc
---|---|---|---|---
8 | 64 | 83.8 | 82.1 | 74.5 | 77.0 | 87.0
16 | 64 | 85.4 | 82.9 | 77.1 | 77.1 | 87.0
32 | 64 | 87.6 | 85.8 | 79.8 | 81.5 | 86.6
64 | 64 | 89.4 | 87.7 | 82.0 | 83.2 | 87.7
64 | 32 | 88.8 | 87.2 | 81.3 | 83.2 | 87.7
64 | 16 | 88.6 | 87.0 | 80.3 | 82.6 | 86.2
64 | 8 | 88.6 | 87.1 | 79.9 | 82.9 | 86.6

Table 4:  Impact of the parameters $W$ and $G$ on 50 Salads.

#### Impact of overlaps for windowed attention

As described in Section 3.1, we use an overlap for the keys and values for the
windowed attention. In Table 5, we also report the result when the keys and
values do not overlap, _i.e_., they are the same as the queries. It is
interesting to note that our attention can be interpreted as a combination of
a reshaping operation with axial attention [17] in the special case without
overlap. The results, however, show that an overlap improves the results.
Using a larger overlap where the keys and values consist of three consecutive
windows does not improve the results further.

| F1@{10, 25, 50} | Edit | Acc
---|---|---|---
Non-Overlapping | 87.9 | 85.5 | 79.1 | 81.4 | 85.1
Overlap (2 Windows) | 89.4 | 87.7 | 82.0 | 83.2 | 87.7
Overlap (3 Windows) | 89.2 | 88.0 | 80.8 | 83.2 | 86.6

Table 5:  Impact of using an overlap for the keys and values for the local
attention on 50Salads.

#### Impact of the order of the attention

The LTContext block shown in Fig. 3 uses first windowed attention and then
long-term context attention. Table 6 shows the results when we change the
order of windowed and long-term context attention within the LTContext block.
If the order is changed, the performance decreases. Since the LTContext blocks
are repeated, the drop in performance is moderate.

| F1@{10, 25, 50} | Edit | Acc
---|---|---|---
WA-LTContext | 89.4 | 87.7 | 82.0 | 83.2 | 87.7
LTContext-WA | 88.3 | 86.4 | 80.4 | 81.6 | 86.1

Table 6:  Impact of the attention order within the LTContext block on
50Salads. Our model computes first windowed attention (WA) and then long-term
context attention (row 1).

#### Impact of the cross-attention

As described in Section 3.2, we use cross-attention in stages 2 to 4. Table 7
shows that the performance drastically decreases without cross-attention.

| F1@{10, 25, 50} | Edit | Acc
---|---|---|---
Without Cross-Attention | 82.2 | 78.5 | 70.4 | 73.6 | 80.2
LTContext | 89.4 | 87.7 | 82.0 | 83.2 | 87.7

Table 7:  Impact of the cross-attention on 50Salads.

#### Impact of using convolutions

The LTContext block shown in Fig. 3 starts with a dilated 1D convolution with
kernel size 3. In Table 8, we evaluate the impact of the dilated convolution
by comparing it to a LTContext block that uses a convolutional kernel of the
same size but without dilation factor, and a LTContext block without 1D
convolution. The results show that dilated convolutions have a very high
impact on performance.

| F1@{10, 25, 50} | Edit | Acc
---|---|---|---
Conv with Dilation | 89.4 | 87.7 | 82.0 | 83.2 | 87.7
Conv without Dilation | 80.1 | 78.0 | 67.7 | 72.6 | 79.3
Without Conv | 81.8 | 79.2 | 67.1 | 72.7 | 78.9

Table 8:  Impact of using LTContext blocks with 1D convolution but without
dilation (row 2) and using LTContext blocks without 1D convolution (row 3).

#### Impact of the number of layers

As shown in Fig. 3, we repeat the LTContext blocks at each stage $N$ times. We
used $N{=}9$ layers in all experiments. Table 9 shows the impact of varying
the number of layers. On 50Salads, all measures improve by increasing $N$. It
needs to be noted that we use $N{=}9$ for the results reported in Table 1
although we can get even better results with 10 layers on 50 Salads. For the
Breakfast and Assembly101 dataset, the best performance is achieved with 9
layers.

number of layers (N) | 50Salads
---|---
F1@{10, 25, 50} | Edit | Acc
8 | 87.2 | 85.1 | 77.6 | 80.8 | 85.7
9 | 89.4 | 87.7 | 82.0 | 83.2 | 87.7
10 | 89.5 | 88.1 | 82.4 | 84.1 | 87.7
| Breakfast
8 | 75.7 | 70.5 | 57.7 | 74.5 | 73.1
9 | 77.6 | 72.6 | 60.1 | 77.0 | 74.2
10 | 77.3 | 72.4 | 59.7 | 76.4 | 73.5
| Assembly101
8 | 31.9 | 28.4 | 21.3 | 27.8 | 41.0
9 | 33.9 | 30.0 | 22.6 | 30.4 | 41.2
10 | 32.6 | 29.3 | 21.9 | 28.7 | 41.5

Table 9:  Impact of the number of layers on 50Salads, Breakfast, and
Assembly101.

#### Impact of the number of attention heads

In our implementation, we do not use multiple attention heads. Nevertheless,
we evaluated the impact of using multiple heads in Table 10 since most
transformers use multiple heads. The results, however, show that there is no
benefit in using multiple heads. The results are consistent with the
observations in [37, 33]. For example, [37] shows that many attention heads
can often be reduced to a single head without impacting the performance. They
also argue that some tasks are more reliant on multiple heads than others.
Temporal action segmentation seems to be a task where one head is sufficient.

number of heads | F1@{10, 25, 50} | Edit | Acc
---|---|---|---
1 | 89.4 | 87.7 | 82.0 | 83.2 | 87.7
2 | 88.8 | 87.0 | 80.5 | 82.9 | 87.0
4 | 89.2 | 87.1 | 80.8 | 83.1 | 86.9
8 | 88.2 | 86.4 | 80.5 | 82.1 | 86.1

Table 10:  Impact of the number of attention heads.

#### Impact of the number of stages

As shown in Fig. 3, we use four stages of LTContext blocks, each of them with
9 layers. We evaluate the impact of the number of stages in Table 11. As can
be seen, using multiple stages helps to reduce the over-segmentation error and
improves the F1 score and Edit score significantly compared to using only one
stage. Increasing the number of stages up to 4 improves all metrics, but using
5 stages decreases the performance and the network starts to overfit.

number of stages | F1@{10, 25, 50} | Edit | Acc
---|---|---|---
1 | 56.3 | 54.1 | 49.9 | 45.6 | 84.7
2 | 86.2 | 84.5 | 78.2 | 80.5 | 86.7
3 | 87.6 | 85.7 | 78.3 | 81.2 | 85.5
4 | 89.4 | 87.7 | 82.0 | 83.2 | 87.7
5 | 89.0 | 86.8 | 80.0 | 83.0 | 85.7

Table 11:  Impact of the number of stages on 50Salads.

##  5 Conclusion

In this work, we addressed the question of how much temporal long-term context
is needed for action segmentation. Our analysis indicates that allowing
networks to operate on the full input sequence is more beneficial compared to
the case where the model has only access to a subset of the input. Based on
our analysis, we presented LTContext, an approach for temporal action
segmentation, where we leverage sparse attention to capture the long-term
context of a video and windowed attention to model the local information in
the neighboring frames. Our approach achieves state-of-the-art segmental F1
scores on the 50Salads and Assembly101 datasets, which contain long videos.

## Acknowledgement

Juergen Gall has been supported by the Deutsche Forschungsgemeinschaft (DFG,
German Research Foundation) GA 1927/4-2 (FOR 2535 Anticipating Human
Behavior), the project iBehave (receiving funding from the programme
“Netzwerke 2021”, an initiative of the Ministry of Culture and Science of the
State of Northrhine Westphalia), and the ERC Consolidator Grant FORHUE
(101044724). The sole responsibility for the content of this publication lies
with the authors.

## References

  * [1] Yazan Abu Farha and Juergen Gall.  MS-TCN: Multi-stage temporal convolutional network for action segmentation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [2] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  ViViT: A video vision transformer.  In IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [3] Nicolas Aziere and Sinisa Todorovic.  Multistage temporal convolution transformer for action segmentation.  Image and Vision Computing, 2022.
  * [4] Nadine Behrmann, S. Alireza Golestaneh, Zico Kolter, Juergen Gall, and Mehdi Noroozi.  Unified fully and timestamp supervised temporal action segmentation via sequence to sequence translation.  In European Conference on Computer Vision (ECCV), 2022.
  * [5] Iz Beltagy, Matthew E Peters, and Arman Cohan.  Longformer: The long-document.  arXiv preprint arXiv:2004.05150, 2020.
  * [6] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  In International Conference on Machine Learning (ICML), 2021.
  * [7] João Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [8] Min-Hung Chen, Baopu Li, Yingze Bao, Ghassan AlRegib, and Zsolt Kira.  Action segmentation with joint self-supervised temporal domain adaptation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [9] Zihang Dai, Hanxiao Liu, Quoc Le, and Mingxing Tan.  Coatnet: Marrying convolution and attention for all data sizes.  Advances in Neural Information Processing Systems (NeurIPS), 2021\.
  * [10] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context.  In Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
  * [11] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.  FlashAttention: Fast and memory-efficient exact attention with IO-awareness.  In Advances in Neural Information Processing Systems (NeurIPS), 2022\.
  * [12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations (ICLR), 2021\.
  * [13] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer.  Multiscale vision transformers.  In IEEE/CVF International Conference on Computer Vision (ICCV), 2021\.
  * [14] Harshala Gammulle, Simon Denman, Sridha Sridharan, and Clinton Fookes.  Fine-grained action segmentation using the semi-supervised action gan.  Pattern Recognition, 2020.
  * [15] Shang-Hua Gao, Qi Han, Zhong-Yu Li, Pai Peng, Liang Wang, and Ming-Ming Cheng.  Global2local: Efficient structure search for video action segmentation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [16] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang.  Conformer: Convolution-augmented transformer for speech recognition.  In INTERSPEECH, 2020.
  * [17] Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans.  Axial attention in multidimensional transformers.  arXiv, 2019.
  * [18] Yifei Huang, Yusuke Sugano, and Yoichi Sato.  Improving action segmentation via graph-based temporal reasoning.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [19] Yuchi Ishikawa, Seito Kasai, Yoshimitsu Aoki, and Hirokatsu Kataoka.  Alleviating over-segmentation errors by detecting action boundaries.  In IEEE Winter Conference on Applications of Computer Vision (WACV), 2021.
  * [20] Svebor Karaman, Lorenzo Seidenari, and Alberto Del Bimbo.  Fast saliency based pooling of fisher encoded dense trajectories.  In European Conference on Computer Vision (ECCV) Workshops, 2014\.
  * [21] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In International Conference on Learning Representations (ICLR), 2015\.
  * [22] Hilde Kuehne, Ali Arslan, and Thomas Serre.  The language of actions: Recovering the syntax and semantics of goal-directed human activities.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
  * [23] Hilde Kuehne, Ali Arslan, and Thomas Serre.  The language of actions: Recovering the syntax and semantics of goal-directed human activities.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 780–787, 2014.
  * [24] Hilde Kuehne, Juergen Gall, and Thomas Serre.  An end-to-end generative framework for video segmentation and recognition.  In IEEE Winter Conference on Applications of Computer Vision (WACV), 2016.
  * [25] Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager.  Temporal convolutional networks for action segmentation and detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [26] Colin Lea, Austin Reiter, René Vidal, and Gregory D Hager.  Segmental spatiotemporal cnns for fine-grained action segmentation.  In European Conference on Computer Vision (ECCV), 2016.
  * [27] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza.  xformers: A modular and hackable transformer modelling library.  https://github.com/facebookresearch/xformers, 2022.
  * [28] Peng Lei and Sinisa Todorovic.  Temporal deformable residual networks for action segmentation in videos.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [29] Muheng Li, Lei Chen, Yueqi Duan, Zhilan Hu, Jianjiang Feng, Jie Zhou, and Jiwen Lu.  Bridge-prompt: Towards ordinal action understanding in instructional videos.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [30] Shi-Jie Li, Yazan Abu Farha, Yun Liu, Ming-Ming Cheng, and Juergen Gall.  MS-TCN++: Multi-stage temporal convolutional network for action segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020.
  * [31] Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer.  Mvitv2: Improved multiscale vision transformers for classification and detection.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [32] Ji Lin, Chuang Gan, and Song Han.  Tsm: Temporal shift module for efficient video understanding.  In IEEE International Conference on Computer Vision (ICCV), 2019\.
  * [33] Liyuan Liu, Jialu Liu, and Jiawei Han.  Multi-head or single-head? an empirical comparison for transformer training.  arXiv, 2021.
  * [34] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In IEEE/CVF International Conference on Computer Vision (ICCV), 2021\.
  * [35] Zhichao Liu, Leshan Wang, Desen Zhou, Jian Wang, Songyang Zhang, Yang Bai, Errui Ding, and Rui Fan.  Temporal segment transformer for action segmentation.  arXiv preprint arXiv:2302.13074, 2023.
  * [36] Ilya Loshchilov and Frank Hutter.  SGDR: stochastic gradient descent with warm restarts.  In International Conference on Learning Representations (ICLR), 2017\.
  * [37] Paul Michel, Omer Levy, and Graham Neubig.  Are sixteen heads really better than one?  In Advances in Neural Information Processing Systems (NeurIPS), 2019\.
  * [38] Piotr Nawrot, Szymon Tworkowski, Michał Tyrolski, Łukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski.  Hierarchical transformers are more efficient language models.  arXiv preprint arXiv:2110.13711, 2021.
  * [39] Junyong Park, Daekyum Kim, Sejoon Huh, and Sungho Jo.  Maximization and restoration: Action segmentation through dilation passing and temporal reconstruction.  Pattern Recognition, 2022.
  * [40] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran.  Image transformer.  In International Conference on Machine Learning (ICML), 2018.
  * [41] Yifan Peng, Siddharth Dalmia, Ian Lane, and Shinji Watanabe.  Branchformer: Parallel MLP-attention architectures to capture local and global context for speech recognition and understanding.  In International Conference on Machine Learning (ICML), 2022.
  * [42] Jiezhong Qiu, Hao Ma, Omer Levy, Wen-tau Yih, Sinong Wang, and Jie Tang.  Blockwise self-attention for long document understanding.  Findings of Empirical Methods in Natural Language Processing (EMNLP), 2020\.
  * [43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning (ICML), 2021.
  * [44] Alexander Richard and Juergen Gall.  Temporal action detection using a statistical language model.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [45] Marcus Rohrbach, Sikandar Amin, Mykhaylo Andriluka, and Bernt Schiele.  A database for fine grained activity detection of cooking activities.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
  * [46] F. Sener, D. Chatterjee, D. Shelepov, K. He, D. Singhania, R. Wang, and A. Yao.  Assembly101: A large-scale multi-view video dataset for understanding procedural activities.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [47] Dipika Singhania, Rahul Rahaman, and Angela Yao.  Coarse to fine multi-resolution temporal convolutional network.  arXiv preprint, 2021.
  * [48] Yaser Souri, Yazan Abu Farha, Fabien Despinoy, Gianpiero Francesca, and Juergen Gall.  Fifa: Fast inference approximation for action segmentation.  In DAGM German Conference on Pattern Recognition (GCPR), 2021.
  * [49] Yaser Souri, Mohsen Fayyaz, Luca Minciullo, Gianpiero Francesca, and Juergen Gall.  Fast weakly supervised action segmentation using mutual consistency.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021.
  * [50] Sebastian Stein and Stephen J. McKenna.  Combining embedded accelerometers with computer vision for recognizing food preparation activities.  In ACM International Joint Conference on Pervasive and Ubiquitous Computing, page 729–738, 2013.
  * [51] Kevin Tang, Li Fei-Fei, and Daphne Koller.  Learning latent temporal structure for complex event detection.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
  * [52] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler.  Efficient transformers: A survey.  ACM Comput. Surv., 2022.
  * [53] Xiaoyan Tian, Ye Jin, and Xianglong Tang.  Local–global transformer neural network for temporal action segmentation.  Multimedia Systems, 2022.
  * [54] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In International Conference on Machine Learning (ICML), 2021.
  * [55] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li.  Maxim: Multi-axis mlp for image processing.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [56] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li.  Maxvit: Multi-axis vision transformer.  In European Conference on Computer Vision (ECCV), 2022.
  * [57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems (NeurIPS), 2017\.
  * [58] Dong Wang, Di Hu, Xingjian Li, and Dejing Dou.  Temporal relational modeling with self-supervision for action segmentation.  AAAI Conference on Artificial Intelligence, 2020.
  * [59] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen.  MaX-DeepLab: End-to-end panoptic segmentation with mask transformers.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [60] Mengmeng Wang, Jiazheng Xing, and Yong Liu.  Actionclip: A new paradigm for video action recognition.  arXiv preprint arXiv:2109.08472, 2021.
  * [61] Zhenzhi Wang, Ziteng Gao, Limin Wang, Zhifeng Li, and Gangshan Wu.  Boundary-aware cascade networks for temporal action segmentation.  In European Conference on Computer Vision (ECCV), 2020.
  * [62] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang.  Cvt: Introducing convolutions to vision transformers.  In IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [63] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollar, and Ross Girshick.  Early convolutions help transformers see better.  In Advances in Neural Information Processing Systems (NeurIPS), 2021\.
  * [64] Ziwei Xu, Yogesh S Rawat, Yongkang Wong, Mohan Kankanhalli, and Mubarak Shah.  Don’t pour cereal into coffee: Differentiable temporal logic for temporal action segmentation.  In Advances in Neural Information Processing Systems (NeurIPS), 2022\.
  * [65] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao.  Focal attention for long-range interactions in vision transformers.  In Advances in Neural Information Processing Systems (NeurIPS), 2021\.
  * [66] Fangqiu Yi, Hongyu Wen, and Tingting Jiang.  ASFormer: Transformer for action segmentation.  In British Machine Vision Conference (BMVC), 2021.
  * [67] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al.  Big bird: Transformers for longer sequences.  Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
  * [68] Yanyi Zhang, Xinyu Li, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe.  Vidtr: Video transformer without convolutions.  In IEEE/CVF International Conference on Computer Vision (ICCV), 2021\.
  * [69] Long Zhao, Zizhao Zhang, Ting Chen, and Dimitris Metaxas abd Han Zhang.  Improved transformer for high-resolution GANs.  In Advances in Neural Information Processing Systems (NeurIPS), 2021\.

## Appendix

We provide additional experiments and implementation details.

##  Appendix A Implementation Details

As mentioned in the paper, we use 9 layers and 4 stages for all datasets. We
use $W=G=64$ for Assembly101 and 50Salads, and $W=64$ and $G=8$ for the
Breakfast dataset since the videos are shorter than Assembly101 and 50Salads.
We use Adam [21] optimizer and cosine learning rate decay [36]. The starting
learning rate for Breakfast and Assembly101 is 0.00025 and the decay to
0.00005 starts after 15 epochs. We train Breakfast for 150 epochs and
Assembly101 for 120 epochs. The model for 50Salads is trained for 200 epochs
with a fixed learning rate of 0.00065.

##  Appendix B Impact of Temporal Downsampling

Fig. 9 shows the impact of temporally downsampling the input. In this
experiment, the model has access to the full context of a video but in a lower
temporal resolution since the input is temporally downsampled. The performance
of the model degrades compared to no downsampling.

Figure 9: Impact of different downsampling rates on the 50Salads dataset
(left) and the Assembly101 dataset (right).

##  Appendix C Other Features

In order to evaluate the impact of using vision-language models, we extract
features using CLIP [43] from 50Salads and report the result of action
segmentation in Table 12. Without additional fine-tuning, the features do not
perform well.

Features | F1@{10, 25, 50} | Edit | Acc
---|---|---|---
CLIP | 65.8 | 57.6 | 44.2 | 64.2 | 62.4
I3D | 89.4 | 87.7 | 82.0 | 83.2 | 87.7

Table 12:  Results are on 50Salads.

##  Appendix D Alternative Efficient Attentions

We compare in Table 13 our approach with RandomAttention [67] from XFormer
[27] and FlashAttention [11]. These types of attention focus on sparseness and
result in fragmented segments, which is indicated by high accuracy, but very
low F1 and Edit scores.

Attention | F1@{10, 25, 50} | Edit | Acc
---|---|---|---
FlashAttention | 55.2 | 53.0 | 48.7 | 42.6 | 84.6
RandomAttention | 49.0 | 45.7 | 41.8 | 37.2 | 85.6
Ours | 89.4 | 87.7 | 82.0 | 83.2 | 87.7

Table 13:  Results are on 50Salads.


# MetaAID: A Flexible Framework for Developing Metaverse Applications via AI
Technology and Human Editing

Hongyin Zhu
Department of Computer Science and Technology, Tsinghua University, Beijing,
China
<EMAIL_ADDRESS>

###### Abstract

Achieving the expansion of domestic demand and the economic internal
circulation requires balanced and coordinated support from multiple industries
(domains) such as consumption, education, entertainment, engineering
infrastructure, etc., which is indispensable for maintaining economic
development. Metaverse applications may help with this task and can make many
industries more interesting, more efficient, and provide a better user
experience. The first challenge is that metaverse application development
inevitably requires the support of various artificial intelligence (AI)
technologies such as natural language processing (NLP), knowledge graph (KG),
computer vision (CV), and machine learning (ML), etc. However, existing
metaverse application development lacks a lightweight AI technology framework.
This paper proposes a flexible metaverse AI technology framework MetaAID that
aims to support language and semantic technologies in the development of
digital twins and virtual humans. The second challenge is that the development
process of metaverse applications involves both technical development tasks
and manual editing work, and often becomes a heavyweight multi-team
collaboration project, not to mention the development of metaverse
applications in multiple industries. Our framework summarizes common AI
technologies and application development templates with common functional
modules and interfaces. Based on this framework, we have designed 5
applications for 3 industries around the expansion of domestic demand and
economic internal circulation. Experimental results show that our framework
can support AI technologies when developing metaverse applications in
different industries.

## Introduction

Metaverse applications are designed to map real-world entities or sensations
into the virtual world, thereby generating parallel virtual avatars. The
blueprint of metaverse (?; ?; ?) is to build a digital twin of the real world.
Metaverse applications may make various industries more interesting, more
efficient, and provide a better user experience, which will help the expansion
of domestic demand and the economic internal circulation. Metaverse
application needs the support of AI technologies to realize smarter and more
interesting functions. Besides, AI technologies may reduce the cost of system
development and operation and maintenance (O&M). AI has a silent but important
role in the foundation and development of the metaverse (?). However, in terms
of technical support, there is a lack of lightweight frameworks for the AI
tech stack. Due to the unclear connection between applications in different
industries, applications tend to be developed separately, resulting in a lack
of collaboration and information interconnection. The expansion of domestic
demand and the economic internal circulation is not limited to one industry,
but also requires the balanced and coordinated development of multiple
industries (domains) such as consumption, entertainment, education,
engineering infrastructure, technological innovation, etc. The coordinated
development of different industries is inseparable from the interconnection of
information. It is well known that there is a lot of duplication of code
between applications.

Most AI frameworks focus on a single task, such as text classification,
sequence labeling, information extraction, knowledge graph reasoning, etc.
When it comes to developing an application, it simply requires multiple AI
frameworks, resources, front-end and back-end development, system operation
and maintenance, etc. When multiple industries are involved, different
development teams are required to complete the applications separately. This
also leads to less collaboration between different industries. Different from
prior works, we propose MetaAID, a flexible metaverse framework that uses AI
technology and human editing to support the collaborative development of
applications in multiple industries. This framework is lightweight and
flexible and can provide rich language and semantic technical support for
developing digital twins and virtual humans. This framework can also be seen
as the integration, consolidation, reorganization, and extension of existing
frameworks. A digital twin (?) is a virtual representation of an object or
system that spans its lifecycle and uses simulation, machine learning, and
reasoning to help decision-making. A virtual human (or digital human) (?) is a
simulation of a person on a computer that can be used for interaction between
human and machine intelligence.

There are 3 main challenges in implementing this framework. (1) Accumulation
and organization of the tech stack. Summarizing and implementing the AI tech
stack required for application development is a lengthy process that may take
years to complete. This includes the process of collecting and comparing
technology options (solutions) and finally, we adopt the most agile and
extensible approach. (2) The repository of human-machine collaboration
templates. Technical templates or code snippets that can be widely used by
developers are essential, such as templates for application development and
templates for different programming languages. (3) Framework evaluation.
Evaluating the framework by developing online applications can generate direct
results. For the first challenge, we collect various technical frameworks and
solutions over 8 years. For the second challenge, we organize resources to
facilitate high-quality human editing work, including summarizing reusable
templates and solutions to common problems. For the third challenge, we have
multiple applications in different industries developed by one developer using
this framework.

Digital twin technology can be used to map the real-world entities into the
virtual world, such as stores, merchandise, teachers, etc. Virtual reality
(VR) and augmented reality (AR) aim at the first step in mapping entities into
the virtual world, such as dealing with the use of 3D scanners to create human
shapes in 3D graphics. Our framework focuses on the second step, which is to
provide the virtual avatar with technical support (intelligence) in terms of
language and semantics.

Achieving the expansion of domestic demand and economic internal circulation
requires balanced and coordinated support from multiple industries.
Applications in different industries are interconnected in technical support
and data analysis. As an illustrative example, we apply the MetaAID framework
to 3 industries that are indispensable for the expansion of domestic demand
and economic internal circulation, i.e., entertainment (NLP-based
entertainment), online education (AI subjects and English learning), daily
consumption (food and beverages). The metaverse platform requires not only
machine programs but also human editing of high-quality and personalized
content. The advantage of machine programs lies in their computational
efficiency, while the advantage of humans lies in their wealth of creativity
and knowledge. Our framework aims to integrate technology with human editing
work, emphasizing not only technology but also the win-win and mutual progress
of humans and technology.

To evaluate this framework, we develop 5 applications on different platforms,
including iOS apps, online websites, and WeChat mini-programs. The innovations
of this paper are as follows.

1\. We propose MetaAID, a flexible metaverse framework that uses AI technology
and human editing to support the collaborative development of metaverse
applications in multiple industries.

2\. We build a repository of human-machine collaboration templates to support
agile human-machine content creation.

3\. To evaluate the features of this framework, we develop 5 applications in 3
industries around the topic of the expansion of domestic demand and economic
internal circulation.

## Related Work

### AI Technology

Artificial intelligence (?) is the simulation of human intelligence processes
by machines, especially computer systems. The concept of artificial
intelligence consists of multiple disciplines, e.g., knowledge representation
and reasoning, commonsense, machine learning, natural language understanding
and generation, machine vision, psychology, philosophy, etc.

Machine learning is the process of using mathematical models of data to help a
computer learn without direct instruction. Machine learning algorithms have
great potential in metaverse applications. These methods can be divided into
supervised learning (?), semi-supervised learning (?), unsupervised learning
(?), self-supervised learning (?) and reinforcement learning (?). Models can
also be divided into generative models and discriminative models (?). These
methods can also be divided into traditional machine learning methods and deep
learning (?) methods. In recent years, neural network models have been widely
studied and applied. With the development of computational power and
optimization algorithms, researchers can train deep and large neural network
models. These models achieve state-of-the-art (SOTA) performance on perceptual
and cognitive tasks.

Due to space limitations, considering the scope of this paper, we mainly
introduce some frameworks in natural language processing, knowledge graph,
computer vision, and deep learning. NLP frameworks are designed to process
textual data efficiently, enabling machines to directly understand the
unstructured text. There are several representative toolkits that play
different roles in text processing, understanding, generation, etc., for
example, transformers111https://github.com/huggingface/transformers, Stanford
CoreNLP (?), Spacy222https://spacy.io/, NLTK (?), Gensim (?). KG frameworks
aim to build machine-readable knowledge graphs from heterogeneous data and to
construct linked open data (LOD) through the semantic web technology
standards. There are several representative toolkits under this tech stack
that deal with knowledge graph construction, representation, reasoning,
question answering, etc., for example, RDFLib (?), Jena (?), BLINK (?), OpenKE
(?), etc.

Computer vision frameworks aim to enable computers and systems to derive high-
level understanding from digital images, videos, and other visual inputs.
There are several representative toolkits that deal with image classification,
object detection, face recognition, semantic segmentation, etc., for example,
OpenCV (?), YOLO (?), SimpleCV333http://simplecv.org/, Caffe (?), etc. Deep
learning frameworks are used to build deep neural networks and support
parallel training of large-scale data and can train large models with tens of
billions of parameters. Representative frameworks are Tensorflow (?), Pytorch
(?), Keras444https://keras.io/, etc. In terms of metaverse frameworks, some
companies, such as Meta Platforms (Facebook), Microsoft, and Roblox, have
gradually built metaverse platforms, but there are few lightweight metaverse
frameworks. Zhu (?) proposes the MetaOnce framework for adding more
constraints and game theory to entities, relations, and events in metaverses.
They introduce multi-scene graphs and propose the entity-relation-event game
rule controller.

### Metaverse Applications

The Metaverse system (?) is mainly used in 3 scenarios. In terms of content
distribution platforms, applications mainly cover the fields of games, social
networks, sports, etc. In terms of display platforms and devices, it mainly
includes smart wearable devices, VR, and AR. In terms of software and hardware
technical support, it mainly includes blockchain, 5G networks, artificial
intelligence (AI), cloud computing, chip design, etc.

There are some well-known metaverse systems in the world. There are different
services in Facebook’s metaverse platform (?; ?), covering social networks,
games, work scene, collaboration, productivity, etc. Microsoft’s metaverse
system (?) includes digital twin-based games, augmented reality glasses, a
mixed reality collaboration platform, cloud computing services, etc. Roblox is
committed to providing users with the tools and technologies to create virtual
worlds (?). The content covers educational innovation, games, etc. Epic Games
(?) is working on building the metaverse into game systems. They build a high-
fidelity digital human platform based on Unreal Engine in the cloud. Different
from the above platforms, our metaverse framework is designed to provide AI
technologies when developing virtual humans and digital twins.

## Approach

In this section, we first provide an overview of the MetaAID framework and
then describe our approach to developing applications in 3 industries.

### Framework Overview

Figure 1: Framework overview

Our framework is divided into 3 layers, i.e., the human editing layer, AI
technology layer, and app development layer, as shown in Figure 1. These
layers are divided according to technical characteristics, including their
tasks, required background knowledge, and roles in the system development
process. Efficient collaboration can be achieved between different layers, and
modules at different layers are responsible for different tasks. It can also
be simply seen as a tech stack similar to some technology companies. A tech
stack is a combination of technologies that a company uses to build and run an
application or project. Different from a company, our framework is portable,
highly open, and customizable, with greater growth potential.

The bottom layer is the app development layer, which aims to efficiently
develop applications, including website development, mobile application
development, data storage management, server operation and maintenance.
Currently, the website development module only supports the development of
monolithic architecture and will be upgraded to support distributed monoliths
and microservices applications in the future. This module mainly includes
front-end and back-end technical templates, technical usage instructions, bug
solving instructions, etc. Mobile application development aims to develop iOS,
Android, and WeChat mini-program. Due to the different development languages
and operating mechanisms of different platforms, app interface development is
different from website front-end development. We summarize some templates for
mobile app development. Mobile apps and websites can share the same backend
services. Data storage management is designed to manage the data center for
private and federated analysis. In our framework, application data of all
industries are stored separately in the same data center, which not only
ensures the independence and privacy of databases of different industries but
also facilitates the collaboration and joint analysis of multi-industry data.
The databases we use mainly include relational databases, graph databases, and
simple JSON-based databases. The operation and maintenance module includes
instructions for using common tools such as Docker, Tomcat, hardware
management, network management, and shell scripts.

The middle layer is the AI technology layer, including NLP models, KG
technology, algorithms, and data processing pipelines. Our NLP models include
text classification, text generation, sequence labeling, information
extraction, question answering, machine reading comprehension, dialogue, etc.
KG technology module contains ontology construction, entity linking, entity
expansion, KG representation, KG reasoning, KG question answering, etc. The
algorithm module contains APIs for hundreds of algorithms for solving some
complex problems, such as graph search problems, dynamic programming, etc. The
data processing pipeline includes not only some data cleaning algorithms but
also some traditional machine learning algorithms including supervised and
unsupervised models, such as the SVM, K-means, decision tree, random forest,
etc.

The upper layer is the human editing layer, which allows people to edit more
personalized and high-quality content. This layer consists of knowledge
editing tools, standard specifications, web crawlers, and software. Knowledge
editing tools include data annotation tools, domain dictionary construction
tools, protege, tools for updating knowledge graphs, etc. The standard
specification aims to help editors produce high-quality, positive, and user-
satisfying material, and includes principles and presentation techniques,
e.g., video tutorial recording tips, PPT presentation tips, HTML editing tips,
etc. Web crawlers are designed to fetch customized content from the Internet
to improve work efficiency. Software is a necessary part to help with the
manual editing process, such as optical character recognition (OCR) software,
UI design software, screen recording software, etc. Based on this framework,
we build 5 applications in 3 industries. Next, we will describe the problems
encountered during the application building process and our solutions.

### Entertainment Industry Application

NLP technologies are widely used in public opinion monitoring, scientific
literature mining, news mining, biomedicine, etc. NLP techniques have achieved
good results on tasks such as dialogue, question answering, and machine
reading comprehension, and have surpassed human-level performance on some
datasets. People look forward to interacting happily with intelligent machines
and enjoying the convenience brought by technological progress.

While NLP techniques are widely used, especially for text classification,
sequence labeling, and text generation tasks, few mobile applications use NLP
techniques to entertain users. We develop a Chinese analysis app “What you
say” (你说啥子), designed to automatically analyze human language and generate
responses on mobile phones. This technology can be used to develop virtual
humans in metaverse applications. This app contains machine reading
comprehension and emotion recognition functions. The machine reading
comprehension task is defined as follows. Given a document $t$ and a question
$q$, the model answers the question by extracting the text spans in the
document, which is also known as extractive question answering. The emotion
recognition task is defined as follows. Given a text $t$, the model aims to
predict the most likely emotion type.

$\displaystyle c_{i}=argmax(p(c_{1}),p(c_{2}),...,p(c_{n}))$ (1)

where $p(c_{i})$ is the probability of emotion $c_{i}$.

The challenge lies in NLP models and mobile app development. For the first
challenge, we use the NLP module in the AI technology layer. We adopt BERT
(?), which is further fine-tuned by (?) to deal with these two tasks. , As
shown in Figure 2, BERT is composed of transformer blocks (?), and the formula
for the transformer block is as follows.

Figure 2: Transformer block architecture

$\displaystyle h=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V$ (2)

The above equation shows the scaled dot-product attention where Q,K,V are the
query, key, value matrices. The input consists of queries and keys of
dimension $d_{k}$, and values of dimension $d_{v}$.

$\displaystyle H_{mul}=(h_{1}\oplus h_{2}...\oplus h_{n})W^{O}$ (3)

where $W^{O}$ is the parameter matrix.

$\displaystyle h_{norm}=LayerNorm(V+SubLayer_{1}(V))$ (4)

The above equation is a residual connection (?) around the sub-layer, followed
by layer normalization. $SubLayer_{1}(\cdot)$ is the multi-head self attention
function.

$\displaystyle h_{ffn}=max(0,h_{norm}W_{1}+b1)W_{2}+b_{2}$ (5)

$W_{1}$ and $W_{2}$ are liner transformations. $b_{2}$ is the bias. The linear
transformations are the same across different positions, and they use
different parameters from layer to layer.

$\displaystyle h_{out}=LayerNorm(h_{norm}+SubLayer_{2}(h_{norm}))$ (6)

The above equation is another layer normalization and residual connection.
$SubLayer_{2}(\cdot)$ is the position-wise feed-forward network.

For the second challenge, we develop an iOS app using the mobile app
development module to facilitate user interaction. In terms of cloud services,
since the training and inference of deep learning models require GPUs, we
built cloud services using server operation and maintenance modules.

### Education Industry Application

With the development of science and technology, the problem of uneven
distribution of educational resources is being alleviated. However, the
dissemination of scientific and technological knowledge on frontier research
topics needs to be further resolved. There needs to be more sharing of
learning experiences from novice to expert.

We build 3 types of educational applications, corresponding to knowledge
integration, knowledge dissemination, and knowledge testing. This technique
can be used to develop educational virtual humans. For knowledge integration,
we develop an online knowledge resource aggregation website “Top bros online”
(小牛人在线). Existing online education websites rarely consider searching other
educational resource websites, and return all the results to users after
aggregating, because different educational websites are separated. The
difficulty of this problem lies in the integration of multi-source knowledge
on the Internet. We build a website to aggregate massive information
technology videos and construct a knowledge graph to support comprehensive
resource SPARQL. The data comes from multiple Chinese online education
websites, e.g., chinahadoop.cn, study.163.com, xue.taobao.com, etc.

For knowledge dissemination, we manually record hundreds of video tutorials.
The challenge lies in the production of scarce video resources. We record the
videos on AI disciplines, such as pattern recognition, natural language
processing, algorithm and data structure, matrix theory, probability theory
and mathematical statistics, etc. We develop a WeChat mini-program “Theory
bros online” (师兄在线) where users can select a video playlist of a course to
watch the entire video list. The app uses manually recorded videos to address
the spread of AI theoretical knowledge that is difficult for beginners to
understand. In terms of structured knowledge, our knowledge card module
extracts structured knowledge points to make the books more interesting and
easier to learn. First, we scan paper books and use OCR to recognize images,
and then use multi-strategy information extraction to extract structured data.

For knowledge testing, we develop an online educational website “English
doctor” (英语医师) which aims to test users’ knowledge of English. A major feature
of this website is that we organize English knowledge from keywords, knowledge
points, question types, etc. Our website has the function of reciting words
and knowledge testing. This site uses questions from common exams such as
CET-4, CET-6, and the graduate entrance examination to test users’ knowledge.
On this English testing system, users only need to answer the questions, and
the system will automatically verify the correctness of the answers. The
system will record each user’s wrong answers and unremembered words,
increasing the probability of checking these knowledge points, and allowing
users to deepen their memory. These features are supported by knowledge graph
technology, and we construct an English test knowledge graph.

### Consumption Industry Application

While the e-commerce and food delivery industries are booming, the market
share of physical stores declines. However, physical store consumption has its
irreplaceable advantages, such as allowing people to better integrate into
society, experience different environments, and buy or learn unexpected new
products. Due to the lack of attractiveness and interest in offline
consumption scenarios, people are more inclined to choose convenient online
consumption, especially when they can stay at home and within reach.

We propose “Talk over drinks” (谁来煮酒) app, which aims to increase the fun of
food and beverages consumption. This app promotes offline consumption, drives
the market growth of related industries, and helps to enhance interpersonal
relationships and social interaction. This technology can be used to model
digital twins for physical stores or products in the metaverse application. By
integrating multi-source and multi-modal data of search engines, as well as
store review websites such as dianping.com and meituan.com, etc, it
facilitates users to make search queries and comprehensive comparisons to
enhance their leisure experience. Meanwhile, this app can recommend
interesting content.

The difficulty of KG construction lies in several aspects, such as data
acquisition, ontology construction, and information extraction. We use the KG
technology module in the AI technology layer. We first obtain raw data from
search engines, then use the information extraction algorithms to extract
structured information and build a knowledge graph. The content of food and
beverages is not monotonous but consists of many aspects. We propose the
multi-perspective consumption (MC) KG and divide the application into the
following aspects, i.e., classification of alcoholic beverages, celebrities
and alcoholic drinks, late-night snack, physical stores, hotels,
transportation, and blind boxes. For the ontology, as shown in Figure 3, the
top-level concepts contain Person, Consumption level, Hotel, Physical store,
Blind box, Taxi.

Figure 3: Ontology of the MCKG

Classification of alcoholic beverages contains a detailed description, such as
liquor, imported wine, red wine, champagne, cocktail, beer, etc. For the above
alcoholic beverages, we curate content from the Internet or books. Celebrities
and alcoholic drinks describe the relation of (celebrity, likeToDrink,
alcoholic drinks ). We first created a list of celebrities and a list of
common alcoholic beverages, which were then automatically linked to the Baidu
encyclopedia through a program. We extract the relations (celebrity,
likeToDrink, alcoholic drinks ) from search engines and keep the evidence of
these facts. We first find webpages that mention both celebrities and
alcoholic drinks, then use an information extraction algorithm to extract the
facts where celebrities and alcoholic beverages co-occurrence in the same
passage, and select the frequently occurring facts. Based on the output
candidate facts, we further perform a manual check to verify the correctness
of the facts.

Physical stores are divided into bars and restaurants. We manually set seed
entities for automatic data acquisition and entity linking. We used
dianping.com to find highly rated stores and then organized them by geographic
location. In terms of hotels, we select hotels with high ratings on ctrip.com
and organize them by geographic location. Users can tap the phone screen to
view detailed information and navigate to other well-known apps.

## Experiment

### Setup

We developed 5 applications in 3 industries based on this framework and
invited users to test them. These applications run on 3 platforms, i.e., the
website, iOS, and WeChat mini-program. These cloud services run on an AMD
Ryzen 5 1500X Quad-Core Processor @ 3.5GHz (Mem: 16G) and 1 Tesla T4 GPU
(16G).

### Results of the Entertainment Industry Application

Figure 4 shows the results from the machine reading comprehension (MRC)
function and the emotion recognition function. For the MRC function, the user
inputs a document about “the Eiffel Tower”, then asks different questions, and
the system returns corresponding answers based on the document. For example,
the input question is “what is the meaning of the Eiffel Tower?” The answer is
“Celebrating the 100th anniversary of the victory of the French Revolution”.
If the input question changes to “Will the Eiffel Tower fall down in the
future?”, the system will return “By the Seine”. While the answer to the
latter question is incorrect, it can be seen as an entertainment scenario.
Users can ask any question and get an interesting response based on the input
document.

Figure 4: Examples of MRC function

Figure 5 shows the results of emotion recognition. Emotions are divided into 5
levels, with the lowest level-1 representing the worst emotion and the highest
level-5 representing the happiest emotion. When the user inputs “The weather
is fine today, but I have a cold.”, the system responds with a sentiment score
of 2. When the user inputs “you are the sea king”, the system responds with a
sentiment score of 5. Although the connotation of the latter sentence is
derogatory, the system still judges it as a positive emotion.

Figure 5: Examples of emotion recognition function

### Results of the Education Industry Application

For the knowledge integration website, we collect the educational resources of
technologies such as big data, AI, machine learning, data mining, etc. on the
Internet. We build a knowledge graph where users can search for resources
through our websites. In the user center, users can upload new videos,
download resources, play videos, manage resources, etc. As shown in Figure 6,
in the upper part of the diagram, the keywords and resource type entered are
“big data” and “video” respectively, and then the site returns video tutorials
related to big data. In the lower part of the diagram, the returned search
results are displayed in a paginated manner, and users can bookmark these
videos and watch them online.

Figure 6: The interface of our knowledge integration website

In the user center, users can view their favorite videos and playlists, as
shown in the upper part of Figure 7. When the user clicks on a video playback
link, videos uploaded by other users will be played on this site. For
resources crawled on the Internet, users will be navigated to the target
website after clicking, rather than playing on our website. As shown in the
lower part of Figure 7, the user is watching a video on pattern recognition -
unsupervised pattern recognition.

Figure 7: The user center of our knowledge integration website

For the scarce resources on the Internet, we recorded video tutorials and
wrote lectures. As shown in Figure 8, the user chooses to watch the playlist
of pattern recognition video tutorials through the WeChat mini-program. Users
can watch playlists on several AI disciplines, such as pattern recognition,
natural language processing, algorithm and data structure, matrix theory,
probability theory and mathematical statistics, etc.

Figure 8: The interface of our WeChat mini-program

For knowledge testing, as shown in the upper part of Figure 9, administrators
can build their exam systems by generating RDF (Resource Description Framework
in Semantic Web Standards) files and uploading them to the site. The lower
part of the diagram shows that this website supports organizing questions by
keyword, knowledge points, and question types. As the user fills in a slot,
the system validates each answer in real-time and displays it below the
question.

Figure 9: The interface of our knowledge test website

### Results of the Consumption Industry Application

Figure 10 shows the results of the “Talk over drinks” app. The left part of
the diagram is the classification of alcoholic beverages, including liquor,
red wine, imported wine, cocktails, etc. Users can click “Details” to view
more information and recommended reading content. The right part of the
diagram is the alcoholic beverages that celebrities like to drink. Users can
click on a celebrity to see information about that celebrity and click on
“Details” to see the story that the celebrity likes to drink.

Figure 10: The classification of alcoholic beverages (left) and celebrities
and alcoholic drinks (right)

As shown in Figure 11, the left part of the diagram shows an overview of the
venue, i.e., pubs, and the right part of the diagram shows the restaurants
serving late-night snacks. We reorganize the information about physical stores
based on subway stations, bus stops, and consumption levels. Users can easily
search for these physical stores by location, name, and consumption level, and
can also view more information and reviews of these stores by clicking on the
“Details”. There are buttons in the detailed descriptions to guide users to
apps such as dianping.com, meituan.com, and WeChat official accounts.

Figure 11: The list of physical stores (left) and late night snacks (right)

## Conclusion and Future Work

This paper proposes a flexible Metaverse AI technology framework MetaAID,
which aims to support AI technology in the development of metaverse
applications. This framework uses AI technology and human editing to support
the collaborative development of applications in multiple industries. We apply
the MetaAID framework to 3 industries that are indispensable for the expansion
of domestic demand and economic internal circulation, i.e., entertainment
(NLP-based entertainment), online education (AI subjects and English
learning), daily consumption (food and beverages). The experimental results
demonstrate that our framework can be used to support the development of
metaverse applications.

The innovation of this paper lies in the flexibility of the framework and the
diversity of techniques within this tech stack, including app development
technologies, AI technologies, and human editing technologies, which can be
used to build applications in different industries. Applications built on this
framework can facilitate information exchange and federated data analysis. In
the future, we hope to expand this framework to support scalable web
architecture and distributed systems.

## References

  * [Abadi et al.2016] Abadi, M.; Barham, P.; Chen, J.; Chen, Z.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Irving, G.; Isard, M.; et al.  2016\.  $\\{$TensorFlow$\\}$: A system for $\\{$Large-Scale$\\}$ machine learning.  In 12th USENIX symposium on operating systems design and implementation (OSDI 16), 265–283.
  * [Bird and Loper2004] Bird, S., and Loper, E.  2004\.  Nltk: the natural language toolkit.  Association for Computational Linguistics.
  * [Bishop2006] Bishop, C. M.  2006\.  Pattern Recognition and Machine Learning.  Springer.
  * [Bradski and Kaehler2000] Bradski, G., and Kaehler, A.  2000\.  Opencv.  Dr. Dobb’s journal of software tools 3:2.
  * [Devlin et al.2019] Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K.  2019\.  BERT: pre-training of deep bidirectional transformers for language understanding.  In Proceedings of NAACL-HLT, 4171–4186.  Association for Computational Linguistics.
  * [François-Lavet et al.2018] François-Lavet, V.; Henderson, P.; Islam, R.; Bellemare, M. G.; and Pineau, J.  2018\.  An introduction to deep reinforcement learning.  arXiv preprint arXiv:1811.12560.
  * [Goodfellow, Bengio, and Courville2016] Goodfellow, I.; Bengio, Y.; and Courville, A.  2016\.  Deep learning.  MIT press.
  * [Han et al.2018] Han, X.; Cao, S.; Lv, X.; Lin, Y.; Liu, Z.; Sun, M.; and Li, J.  2018\.  Openke: An open toolkit for knowledge embedding.  In Proceedings of EMNLP, 139–144.  Association for Computational Linguistics.
  * [He et al.2016] He, K.; Zhang, X.; Ren, S.; and Sun, J.  2016\.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778.
  * [Huynh-The et al.2022] Huynh-The, T.; Pham, Q.-V.; Pham, X.-Q.; Nguyen, T. T.; Han, Z.; and Kim, D.-S.  2022\.  Artificial intelligence for the metaverse: A survey.  arXiv preprint arXiv:2202.10336.
  * [Jia et al.2014] Jia, Y.; Shelhamer, E.; Donahue, J.; Karayev, S.; Long, J.; Girshick, R.; Guadarrama, S.; and Darrell, T.  2014\.  Caffe: Convolutional architecture for fast feature embedding.  In Proceedings of the 22nd ACM international conference on Multimedia, 675–678.
  * [Kaur and Gupta2021] Kaur, M., and Gupta, B.  2021\.  Metaverse technology and the current market.
  * [Kim2021] Kim, J.  2021\.  Advertising in the metaverse: Research agenda.
  * [Krech2006] Krech, D.  2006\.  Rdflib: A python library for working with rdf.  Online https://github. com/RDFLib/rdflib.
  * [Ledell Wu2020] Ledell Wu, Fabio Petroni, M. J. S. R. L. Z.  2020\.  Zero-shot entity linking with dense entity retrieval.  In EMNLP.
  * [Lee2021] Lee, B.-K.  2021\.  The metaverse world and our future.  Review of Korea Contents Association 19(1):13–17.
  * [Liu et al.2021] Liu, X.; Zhang, F.; Hou, Z.; Mian, L.; Wang, Z.; Zhang, J.; and Tang, J.  2021\.  Self-supervised learning: Generative or contrastive.  IEEE Transactions on Knowledge and Data Engineering.
  * [Magnenat-Thalmann and Thalmann2005] Magnenat-Thalmann, N., and Thalmann, D.  2005\.  Virtual humans: thirty years of research, what next?  The Visual Computer 21(12):997–1015.
  * [Manning et al.2014] Manning, C. D.; Surdeanu, M.; Bauer, J.; Finkel, J. R.; Bethard, S.; and McClosky, D.  2014\.  The stanford corenlp natural language processing toolkit.  In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, 55–60.
  * [McBride2002] McBride, B.  2002\.  Jena: A semantic web toolkit.  IEEE Internet computing 6(6):55–59.
  * [Mitchell1997] Mitchell, T. M.  1997\.  Machine learning, International Edition.  McGraw-Hill Series in Computer Science. McGraw-Hill.
  * [Park and Kim2022] Park, S.-M., and Kim, Y.-G.  2022\.  A metaverse: taxonomy, components, applications, and open challenges.  IEEE Access.
  * [Paszke et al.2019] Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al.  2019\.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems 32\.
  * [Rasheed, San, and Kvamsdal2020] Rasheed, A.; San, O.; and Kvamsdal, T.  2020\.  Digital twin: Values, challenges and enablers from a modeling perspective.  Ieee Access 8:21980–22012.
  * [Redmon et al.2016] Redmon, J.; Divvala, S.; Girshick, R.; and Farhadi, A.  2016\.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, 779–788.
  * [Řehůřek and Sojka2010] Řehůřek, R., and Sojka, P.  2010\.  Software Framework for Topic Modelling with Large Corpora.  In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, 45–50.  Valletta, Malta: ELRA.  http://is.muni.cz/publication/884893/en.
  * [Russell and Norvig2002] Russell, S., and Norvig, P.  2002\.  Artificial intelligence: a modern approach.
  * [Sparkes2021] Sparkes, M.  2021\.  What is a metaverse.
  * [Van Engelen and Hoos2020] Van Engelen, J. E., and Hoos, H. H.  2020\.  A survey on semi-supervised learning.  Machine Learning 109(2):373–440.
  * [Vaswani et al.2017] Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I.  2017\.  Attention is all you need.  Advances in neural information processing systems 30\.
  * [Wang et al.2022] Wang, F.-Y.; Qin, R.; Wang, X.; and Hu, B.  2022\.  Metasocieties in metaverse: Metaeconomics and metamanagement for metaenterprises and metacities.  IEEE Transactions on Computational Social Systems 9(1):2–7.
  * [Wilson, Gosling, and Graham2012] Wilson, R. E.; Gosling, S. D.; and Graham, L. T.  2012\.  A review of facebook research in the social sciences.  Perspectives on psychological science 7(3):203–220.
  * [Zhao et al.2019] Zhao, Z.; Chen, H.; Zhang, J.; Zhao, X.; Liu, T.; Lu, W.; Chen, X.; Deng, H.; Ju, Q.; and Du, X.  2019\.  Uer: An open-source toolkit for pre-training models.  EMNLP-IJCNLP 2019 241\.
  * [Zhu2022] Zhu, H.  2022\.  Metaonce: A metaverse framework based on multi-scene relations and entity-relation-event game.  arXiv preprint arXiv:2203.10424.