11institutetext: Excellence Cluster Science of Intelligence, Technische
Universitat Berlin 11email<EMAIL_ADDRESS>
# Action-based Contrastive Learning for Trajectory Prediction

Marah Halawa  Olaf Hellwich  Pia Bideau

###### Abstract

Trajectory prediction is an essential task for successful human-robot
interaction, such as in autonomous driving. In this work, we address the
problem of predicting future pedestrian trajectories in a first-person view
setting with a moving camera. To that end, we propose a novel action-based
contrastive learning loss, that utilizes pedestrian action information to
improve the learned trajectory embeddings. The fundamental idea behind this
new loss is that trajectories of pedestrians performing the same action should
be closer to each other in the feature space than the trajectories of
pedestrians with significantly different actions. In other words, we argue
that behavioral information about pedestrian action influences their future
trajectory. Furthermore, we introduce a novel sampling strategy for
trajectories that is able to effectively increase negative and positive
contrastive samples. Additional synthetic trajectory samples are generated
using a trained Conditional Variational Autoencoder (CVAE), which is at the
core of several models developed for trajectory prediction. Results show that
our proposed contrastive framework employs contextual information about
pedestrian behavior, i.e. action, effectively, and it learns a better
trajectory representation. Thus, integrating the proposed contrastive
framework within a trajectory prediction model improves its results and
outperforms state-of-the-art methods on three trajectory prediction benchmarks
[31, 32, 26].

##  1 Introduction

Predicting the future trajectories of pedestrians is an important task in many
applications, such as in social robot interaction and autonomous driving.
Typically, the future trajectory of an agent/pedestrian is predicted based on
its own past movement history [33]. Nonetheless, integrating additional
information is possible, such as the trajectories of surrounding agents [1,
9], or visual scene data [34]. When the surrounding agents in the scene are
cars or robots, modeling the motion information based on past trajectories
only is a reasonable way to solve the task. However, in this work, we argue
that when other agents in the scene are pedestrians, then limiting the
information used for prediction to past trajectories is not sufficient. In
those cases additional information about pedestrian behavior (e.g. action)
plays an important role for predicting their future trajectory. For example,
the future trajectories of a pedestrian who is walking while texting on a
phone could be different from a pedestrian carrying an object or pushing a
baby stroller even if they have the same previous observed trajectories, and
the same end goal.

In this work, we study the influence of observed pedestrians’ actions on their
predicted trajectories. We propose a novel contrastive learning loss called
Action-based Contrastive Loss. This novel loss is employed as a regularizer to
the main trajectory prediction loss. The action-based contrastive loss
encourages the trajectory embeddings of agents performing the same action
(called positive samples) to come closer to each other in the feature space,
and the embeddings of trajectories observed while performing different actions
(called negative samples) far away from each other. For instance, the
representations of trajectories of walking pedestrians are encouraged to
become closer in the feature space, but farther from the representations of
trajectories of pedestrians riding bikes or standing, as illustrated in Fig.
1.

Contrastive learning losses, including ours (action-based contrastive loss),
utilize a mechanism called negative sampling/mining, which aims to choose the
samples that are deemed different and therefore their corresponding features
are driven farther in the embedding space. In our case, the negatives are
trajectories of pedestrians that have different actions. Commonly used
negative sampling techniques include choosing all other samples from the same
mini-batch [5] or from a fixed-size memory bank [14]. Nevertheless, while
these mechanisms prove effective on natural imaging datasets, we find they do
not provide similarly high gains on trajectory datasets. We conjecture that
this is due to the higher variation in visual data compared to trajectory
data, and most importantly, to the larger sizes of imaging datasets, e.g.
Imagenet [8] contains 1.6M images compared to PIE [31] that contains 738,970
trajectory samples. This results in limited numbers of negative samples, an
issue that becomes more evident when conditioning samples by class
information, e.g. action or behavior. Few works attempt to address this issue
via designing special heuristics for negative mining [23, 38, 13].
Alternatively, in this work, we propose to utilize the data distribution
learned by a Conditional Variational Auto-Encoder (CVAE) [35]. This avoids
designing special heuristics for negative mining. While this form of sampling
may be utilized to create negative samples only, we employ it to create both
positive and negative samples. This is possible due to the different
definition of our contrastive loss compared to the traditional Noise
Contrastive Estimation loss (NCE loss); the notion of positive/negative in our
case is tied to the different classes of action in the data. As explained
above, the samples that belong to the same action class are positives from the
point of view of this class, and other samples are negative.

Contributions. Our main contributions in this paper are as follows:

  * •

A novel contrastive loss, called action-based contrastive loss, which provides
the model with additional information about the action of an agent by guiding
the development of the embedding space for trajectories during learning.

  * •

A novel sampling/mining technique that utilizes the latent trajectory
distributions learned by CVAEs, circumventing the need to design special
mechanisms based on heuristics.

Our proposed contrastive learning framework improves the performance results
on three first-person view trajectory prediction benchmarks. It also provides
evidence that utilizing agent behavior information, in the form of action type
in this case, is beneficial for trajectory prediction, aligning with [26].
However, our proposed learning framework requires action information only
during training.

Figure 1: Overview of our action-based contrastive learning framework during
training phase. The contrastive loss $\mathcal{L}_{\text{Act-Con}}$ gets as
input both positive (green) and negative (red) embeddings $h$ for an anchor
(blue). The positive and negative samples are the samples other than the
anchor in the batch, as well as the synthetic samples from the CVAE. The parts
shown in yellow refer to our novel action-based contrastive learning
framework. It is worth mentioning that the action-based contrastive loss
illustrated in this figure updates only the weights of encoder $f$, and it is
jointly optimized with $\mathcal{L}_{traj}$ that updates both encoder $f$ and
decoder $g$. $\mathcal{L}_{traj}$ is not shown in the figure.

##  2 Related Work

Multi-modal trajectory prediction: A human can reach a desired location
following many possible trajectories. Therefore, multiple works utilize multi-
modal trajectory models, instead of predicting a single-path solution. Lee et
al. [21] proposed multi-modal trajectory model by incorporating samples from
the Gaussian distribution of a trained conditional variational autoencoder
(CVAE) into a long short-term memory encoder-decoder (LSTMED) model. Mangalam
et al. [27] predict the multi-modal trajectory of an agent by modeling three
factors: the desired endpoint goal, the social interaction with other agents
in the scene, and the planned trajectories with respect to the environmental
constraints in the scene. Similarly, their model is based on CVAE, which takes
as input both the encodings of the past trajectory and of the endpoint goal.
Sadeghian et al. [34] additionally include the past/observed trajectories of
all agents for future trajectory prediction. To provide additional context
information, top view images are incorporated. The distribution over feasible
future paths is modeled for each agent using LSTM-based GAN module. Similarly,
Yao et al. [39] predict trajectories conditioned on an estimated goal using a
bi-directional RNN decoder. While our method has the potential of being added
to any trajectory prediction method, we base our contrastive framework on
BiTraP [39], in the first-person view setting.

Using human actions to improve trajectory prediction: In literature, many
works employed video data to predict human activities [30]. Montes et al. [28]
used a 3D-CNN as a feature extraction network then pass the learned
representation to an RNN to exploit the time component in video data
effectively. Ma et al. [24] improved the performance of LSTMs in human
activity prediction by implementing ranking losses that penalize the
prediction model on inconsistency in prediction scores from the sequence
frames. Liang et al. [22] predicted a pedestrian’s future trajectory
simultaneously with future activities in a multi-task learning scheme. Rasouli
et al. [31], studied the influence of an estimated pedestrian intention on the
predicted trajectory by combining the intention representation with the
observed trajectory coordinates, then used this representation as input to the
decoder. Malla et al. [26] incorporates pedestrian action information with a
trajectory prediction model. They require this information as prior
information and learn a joint representation for both observed trajectory and
pedestrian action. In this work, we also highlight the importance of analyzing
the pedestrian’s behavior and action in the prediction of their future
trajectory. However, we propose to incorporate action information only during
training using a novel action-based contrastive loss.

Contrastive trajectory prediction: Contrastive learning is a representation
learning approach, first proposed by [29]. This approach encourages similar
high-dimensional input vectors to be mapped closely to each other in a lower-
dimensional embedding manifold, and the dissimilar ones are mapped far away
from each other. Contrastive learning has been applied in several unsupervised
[29, 15, 5, 11, 41, 4, 14, 6, 10, 17] and supervised [18] representation
learning methods. Recently, only few works applied contrastive learning to
trajectory prediction in a multi-agent setting. The flexibility of defining a
contrastive loss by using positive and negative samples addresses the shortage
problem in critical and challenging scenarios in training datasets. Such rare
scenarios are necessary for the model, as the agent could face these in the
real-world. Makansi et al. [25] utilize this idea by separating the hard and
critical samples in the feature space that do not satisfy some certain
favorable criterion far away from the positive easy samples. Liu et al. [23]
proposed a social sampling strategy that relies on augmenting negative samples
with prior knowledge about undesired scenarios in the multi-agent setting.
Both methods use the contrastive loss as a weighted combination to the future
trajectory forecasting loss, which may be the mean squared error (MSE) or
negative log-likelihood (NLL). Our method follows this family of algorithms,
and uses a novel action-based contrastive loss to add context information
about pedestrian actions to the trajectory prediction model.

Supervised contrastive loss: Khosla et al. [18] proposed a supervised
contrastive loss that is a generalization of the Triplet loss [16]. In this
supervised contrastive loss for each anchor there are more than one positive
sample, in addition to many negative samples. There are two major differences
between our proposed action-based contrastive loss and the supervised
contrastive loss used in [18]. First, they employ the supervised contrastive
loss to replace the cross-entropy loss for training the image classifier using
image labels. However, we utilize the contrastive loss to regularize the
trajectory prediction loss, which may be MSE or NLL. Second, due to the
differences between the nature of datasets we use in this paper and the image
data used in [18], it is simpler to extract many positive and negative samples
from a large dataset, such as ImageNet [8]. However, in first-person view
trajectory prediction datasets, the number of pedestrians with same actions is
limited, therefore we address this with a novel sampling process from a CVAE
trained to predict trajectories based on observing a short past trajectory.
This CVAE predictive model ensures consistency between observed and predicted
trajectories. Thus, it allows sampling additional positive and negative
samples that belong to specific actions. Using this novel sampling technique
avoids designing hard negative mining techniques, which use heuristics, as in
[36, 19] for domain adaptation.

##  3 Methodology

In this section, we present our method for the task of pedestrian trajectory
prediction, that focuses on integrating contextual information such as actions
for more reliable future predictions. We address this by employing an action-
based contrastive loss that enhances the trajectory prediction model with
action information.

###  3.1 Problem Formulation

For each pedestrian we have an observed past trajectory
$S_{t}=[s_{1},...,s_{t-1},s_{t}]$ at time $t$, and we predict a future
trajectory $Y_{t}=[y_{t+1},y_{t+2},...,y_{T}]$, where $s$ and $y$ are bounding
box coordinates for the observed and predicted trajectories, respectively. $T$
is the maximum predicted trajectory time length in the future. In addition, we
also have for each trajectory the action class information $a$, where the set
of available actions $a\in\\{a_{1},a_{2},..\\}$ may vary across different
datasets. Then in the training data, we assume there are $N$ different
training samples, where for each sample $i\in[1,..,N]$, we know $S^{i}$,
$Y^{i}$, and $a^{i}$. Finally, we process the dataset samples in mini-batches,
where each batch contains $B$ samples.

###  3.2 Multi-modal Trajectory Prediction

We follow the commonly used approach of an encoder-decoder prediction model,
where an encoder $f$ learns the representation $h$ given an observed
trajectory $S_{t}$ as an input, then a decoder $g$ uses the representation $h$
together with a sampled latent variable $z$ to predict the future trajectory
$Y_{t}$. We employ a standard long-short term encoder-decoder model (LSTMED)
[21]. In fact, we extend on the bi-directional version of LSTMED, proposed in
Yao et al.’s BiTraP [39]. The possibility to draw multiple future trajectories
for each observed trajectory is achieved with a CVAE, which is a non-
parametric model, that learns the distribution of target trajectory through a
stochastic latent variable. The distribution learned by the CVAE is essential
for our proposed contrastive framework, which we explain below. As a
trajectory prediction loss function $\mathcal{L}_{traj}$, the Best-of-Many
(BoM) L2-loss [3] between predicted and target trajectory is used. It is
noteworthy that we do not restrict our proposed framework, explained below, to
these choices of model architectures or loss functions; we adopt standard and
effective techniques to study its influence on predicted trajectories. The
essential factor for our learning framework is that the predicting future
trajectory model is based on CVAE, similar to trajectory prediction models in
[39, 27].

###  3.3 Action-based Contrastive Learning Framework

In order to enhance the model with contextual information about the pedestrian
actions, we propose a novel loss that is called action-based contrastive loss,
which acts as a regularizer for the trajectory prediction loss, and they
jointly train the trajectory prediction model. The proposed action-based
contrastive loss is based on a novel action-based sampling strategy shown in
Fig. 1. We first describe the proposed contrastive loss in the simple case,
without including additional samples from the CVAE distribution, and we
generalize it later.

####  3.3.1 Action-based Contrastive Loss

Let $B$ be the number of samples within a batch. For each observed past
trajectory $S^{i}$ where $i\in\\{1,..,B\\}$, called the anchor, there exists
multiple positive and negative samples. The positive samples $S^{i+}$ are the
trajectories that have the same action class as the anchor, which are denoted
by ${S^{i}}^{\prime\prime}$. Moreover, we also add an augmented version
${S^{i}}^{\prime}$ of the anchor trajectory as a positive sample, following
[23], which is created by adding small white noise $\epsilon$ to the bounding
box coordinates of the anchor trajectory.

Formally:

${S^{i}}^{\prime}=\left\\{S^{i}+\epsilon\right\\}$

${S^{i}}^{\prime\prime}=\left\\{S^{j}\right\\};\text{where}\;0<j<B,a^{j}=a^{i},i\neq
j$

$S^{i+}={S^{i}}^{\prime}\cup{S^{i}}^{\prime\prime}$

Negative samples $S^{i-}$ are trajectories belonging to a different action
class than the anchor.

$S^{i-}=\left\\{S^{k}\right\\};\text{where}\;0<k<B,a^{k}\neq a^{i},i\neq k$

Afterwards, all batch samples $\left\\{S^{i}\right\\}_{i=1}^{B}$ are processed
by the model encoder $f$ to produce their hidden representations
$\left\\{\mathbf{h}^{i}\right\\}_{i=1}^{B}$. Assuming $M$ positive samples and
$K$ negative samples in the batch, with $B=M+K$. The proposed loss is
calculated as follows:

$\begin{split}\ell_{\text{Act-
Con}}&=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\sum^{M}_{j=1,j\neq
i,a^{j}=a^{i}}\exp(sim(\mathbf{h}^{i},\mathbf{h}^{j})/\tau)}{\sum^{K}_{k=1,k\neq
i}\exp(sim(\mathbf{h}^{i},\mathbf{h}^{k})/\tau)}\\\ \mathcal{L}_{\text{Act-
Con}}&=\frac{1}{N/B}\sum^{N/B}\ell_{\text{Act-Con}}\end{split}$ (1)

where $sim$ is the similarity between the vector representations of the
samples, for which we use the dot-product. $\tau$ is the temperature
hyperparameter. The above loss function encourages the embeddings
$\mathbf{h}^{i}$ of positive sample trajectories to be closer to each other in
the embedding space, and far away from the embeddings of negative samples. The
complete loss function sums both the trajectory prediction loss
$\mathcal{L}_{traj}$ and the action-based contrastive loss
$\mathcal{L}_{\text{Act-Con}}$:

$\mathcal{L}_{final}=\mathcal{L}_{traj}+\beta\mathcal{L}_{\text{Act-Con}}$ (2)

where $\beta$ is a hyper-parameter that controls the contribution of action-
based contrastive loss. It is worth mentioning that additional behavioral
information such as pedestrian’s action class is only needed during training.
However, during inference, the model only takes the observed trajectory as
input to predict the future trajectory.

####  3.3.2 Action-based Synthetic Trajectory Sampling

The above loss formulation assumes no additional synthetic samples, i.e. it
considers observed trajectories in the batch only. However, due to the
relatively limited sizes of trajectory datasets, and the shortage of diversity
in action classes in captured scenes, commonly used negative sampling
techniques may not be sufficient. Those include sampling from the same mini-
batch [5] or from a fixed-size memory bank [14]. More comprehensive negative
and positive samples, from various behavior scenarios are rather needed.
Therefore, we extend training samples by drawing trajectories from the
distribution learned by the generative Conditional Variational Autoencoder
(CVAE) model. CVAE is a generative model that introduces a stochastic latent
variable $Z$ in order to learn the distribution of target future trajectory
$P\left(Y^{i}|S^{i},Z\right)$. This distribution is conditioned on the input
observed trajectories $S^{i}$, and the stochastic latent variable $Z$. Thus,
the model is able to predict multiple feasible trajectories $Y^{i}$ given the
input $S^{i}$. We assume the latent variable following a Gaussian distribution
$Z\sim N\left(\mu_{Z},\sigma_{Z}^{2}\right)$, and we train the CVAE to capture
this distribution. Afterwards, the training dataset is extended by sampling
from the Gaussian latent space multiple times, and passing samples through the
decoder $g$ to effectively predict different feasible future trajectories
conditioned on an observed trajectory. The conditioning on the observed
trajectory ensures a consistent behavior in the predicted future trajectory.
This behavior is captured in both the continuity of the trajectory as well as
the identical action class in both observed and future trajectories. We employ
the same encoder-decoder trajectory prediction model explained above in Sec.
3.2, which is a CVAE that predicts multiple feasible future trajectories, as
the example in Fig. 2 shows. This sampling strategy is illustrated in Fig. 1,
and it has the advantage that it avoids designing heuristics for negative
sample mining techniques, as mentioned before. The intuition behind this
sampling strategy is that the encoder-decoder CVAE model is capable of
generating future trajectories with the same behavior of the observed
trajectory. Since it is trained to predict the future trajectory of an
observed trajectory, then it captures the characteristics of the observed
trajectory.

Let $\left\\{Y^{i,l}\right\\}_{l=1}^{L}$ be the multiple predicted
trajectories for an observed trajectory $S^{i}$. Here, $Y^{i,l}$ is sampled
from $P\left(Y^{i,l}|S^{i},Z\right)$, and $L$ is the number of times we sample
a different $Z$ from the normal distribution
$N\left(\mu_{Z},\sigma_{Z}^{2}\right)$. Given these synthetic trajectory
samples, the set of positive samples for trajectory $S^{i}$ are then
reformulated as follows:

$S^{i+}_{1:t}=\left\\{{S^{i}}^{\prime}_{1:t}\right\\}\cup\left\\{{S^{i}}^{\prime\prime}_{1:t}\right\\}\cup\left\\{Y^{j,l}_{t+1:T}\right\\}_{l=0}^{L}$

where $i,j\in{1,..,B}$ and $a^{l}=a^{i}$ and $a^{j}=a^{i}$. And the negative
samples are reformulated as follows:

$S^{i-}_{1:t}=\left\\{S^{k}_{1:t}\right\\}\cup\left\\{Y^{k,l}_{t+1:T}\right\\}_{l=0}^{L}$

where $i,k\in{1,..,B}$ and $i\neq k$ and $a^{i}\neq a^{k}$ and $a^{i}\neq
a^{l}$ and $a^{k}=a^{l}$. In words, the synthetic samples created for sample
$S^{k}$, which we denote $Y^{k,l}$, are considered negative from the point of
view of sample $S^{i}$. These synthetic samples have the same action class of
sample $S^{k}$, hence denoted $a^{k}=a^{l}$.

The described action-based synthetic trajectory sampling strategy changes the
sets of positive and negative samples used in creating training batches.
However, the proposed contrastive loss equation Eq. 1 remains the same, only
$M$ and $K$ are affected.

Figure 2: Examples from TITAN dataset showing the multi-modality in the
trajectory prediction space. CVAE is able to predict multiple feasible future
trajectories (Green bounding boxes), conditioned on previously observed
trajectories (Blue bounding boxes). The red bounding boxes refer to the ground
truth future trajectories.

##  4 Experiments

In this section, we present the evaluation results of our method on three
first-person-view trajectory prediction datasets [31, 32, 26]. First, we
describe the used datasets. Then, we provide an overview of the experimental
setup and used evaluation metrics. Finally, we discuss our results and
findings.

###  4.1 Datasets

We evaluate our method on first-person view datasets. In this domain, the
Pedestrian Intention Estimation (PIE) [31] and the Joint Attention for
Autonomous Driving (JAAD) [32] datasets are the most commonly used benchmarks
in literature. The PIE dataset provides 293,437 annotated frames, containing
1,842 pedestrians with behavior annotations such as walking, standing,
crossing, looking, etc. Since a pedestrian could be “walking” and “looking” at
the same time, for example, then a pedestrian could have multiple behavior
labels in a single frame. Therefore, we only use two classes “walking” and
“standing”, which are exclusive. We use the same train and test splits in
[31]. On the other hand, the JAAD dataset provides 82,032 annotated frames,
containing 2,786 pedestrians, 686 of them have behavior annotations. Similar
to the PIE dataset, we only use for JAAD dataset two classes “walking” and
“standing”, which are exclusive. We use the same train and test splits in
[32].

We also use a third dataset named TITAN [26], which contains more action
classes compared to PIE and JAAD. TITAN provides 75,262 frames with 395,770
pedestrians with multiple action labels organized in five hierarchical
contextual activities, such as individual atomic actions, simple scene
contextual actions, complex contextual actions, transportive actions, and
communicative actions. For the same reason of not having multiple labels for
each pedestrian, we use individual atomic actions labels for the TITAN
dataset. The atomic action labels describe the primitive action, and are
categorized into 9 labels (sitting, standing, walking, running, bending,
kneeling, squatting, jumping, laying down).

Table 1: The quantitative results on PIE and JAAD datasets. The evaluation metrics are reported for different prediction lengths in squared pixels. ABC+ is our proposed action-based contrastive framework with sampling from a learned CVAE. BiTraP is a baseline trajectory prediction model without adding any contrastive loss. The other baseline results are obtained from [39]. Lower is better. Method | PIE | JAAD
---|---|---
ADE | C-ADE | C-FDE | ADE | C-ADE | C-FDE
0.5 |  1.0 |  1.5 |  1.5 |  |  0.5 |  1.0 |  1.5 |  1.5 |
Linear [31] |  123 |  477 |  1365 |  950 |  3983 |  233 |  857 |  2303 |  1565 |  6111
LSTM [31] |  172 |  330 |  911 |  837 |  3352 |  289 |  569 |  1558 |  1473 |  5766
B-LSTM [2] |  101 |  296 |  855 |  811 |  3259 |  159 |  539 |  1535 |  1447 |  5615
FOL-X [40] |  47 |  183 |  584 |  546 |  2303 |  147 |  484 |  1374 |  1290 |  4924
$PIE_{traj}$ [31] |  58 |  200 |  636 |  596 |  2477 |  110 |  399 |  1280 |  1183 |  4780
BiTraP [39] |  23 |  48 |  102 |  81 |  261 |  38 |  94 |  222 |  177 |  565
ABC+ |  16 |  38 |  87 |  65 |  191 |  40 |  89 |  189 |  145 |  409

###  4.2 Experimental Setup

We use the same setup for all datasets, where we observe 0.5 seconds and
predict 0.5, 1.0, and 1.5 seconds, following [31, 39]. The predicted
trajectories have two forms: bounding boxes coordinates and centers, that are
evaluated separately. PIE and JAAD datasets are both annotated at 30Hz
frequency, therefore we observe 15 frames and predict 45 frames. However,
TITAN dataset is annotated at 10HZ sampling frequency. Thus, we observe 5
frames and predict 15 frames.

####  4.2.1 Implementation details.

We use 256 as the size for all hidden layers in the encoder-decoder model that
is detailed in Sec. 3.2. It is noteworthy that we implement the loss in Eq. 1
using the efficient matrix-form (especially on GPU machines), instead of
performing expensive pairwise computations. We train the model on all datasets
with Adam optimizer [20] using a batch size of 128 and a learning rate of
0.001. On training datasets, we perform hyper-parameter tuning for $\beta$
(Eq. 2). We achieve our best results using $\beta=0.75$ for all datasets.

####  4.2.2 Evaluation metrics.

Following the commonly used evaluation protocols in literature [31, 39, 37],
we use the following evaluation metrics: i) Bounding box Average Displacement
Error (ADE), ii) Bounding box Center ADE (C-ADE), iii) Bounding box Final
Displacement Error (FDE), and iv) Bounding box Center FDE (C-FDE). All are
computed in squared pixels. The bounding box ADE is the mean square error
(MSE) for all predicted trajectories and ground-truth future trajectories.
This error is calculated using the bounding box upper-left and lower-right
coordinates. However, in C-ADE, otherwise called C-MSE, the error is
calculated using the centers of the bounding boxes. Bounding box FDE,
otherwise called FMSE, is the distance between the destination point of the
predicted trajectory and of the ground truth at the last time step. FDE is
also calculated using the bounding boxes coordinates. Finally, C-FDE or C-FMSE
is the mean squared error between the centers of final destination bounding
boxes.

Table 2: The quantitative results on TITAN dataset. The evaluation metrics are reported for observing 15 time steps and predicting 45 time steps of trajectories in squared pixels. ABC+ is our proposed action-based contrastive framework with sampling from a learned CVAE. BiTraP is a baseline trajectory prediction model without adding any contrastive loss. Lower is better. Method | ADE | C-ADE | C-FDE
---|---|---|---
0.5 |  1.0 |  1.5 |  1.5 |
BiTraP [39] |  194 |  352 |  658 |  498 |  989
ABC+ |  165 |  302 |  575 |  434 |  843
Table 3: The quantitative results on TITAN dataset. The evaluation metrics are reported for observing 10 time steps and predicting 20 time steps of trajectories in pixels. ABC+ is our proposed action-based contrastive framework with sampling from a learned CVAE. The other baseline results are obtained from [26]. Lower is better. Method |  ADE |  FDE
---|---|---
Social-LSTM [1] |  37.01 |  66.78
Social-GAN [12] |  35.41 |  69.41
Titan-vanilla [26] |  38.56 |  72.42
Titan-AP [26] |  33.54 |  55.80
ABC+ |  30.52 |  46.84

####  4.2.3 Baselines.

The trajectory prediction model trained with our proposed Action-Based
Contrastive framework (loss and sampling strategy) is indicated by (ABC+).
First, we evaluate the performance of our action-based contrastive framework
by comparing its results to the original BiTraP trajectory prediction model
[39] on all datasets, i.e. without adding our contrastive loss. This baseline
aims to highlight the gains obtained by our proposed contrastive framework.
BiTraP had previously achieved state-of-the-art on PIE and JAAD datasets.
Additionally, for PIE and JAAD datasets, we compare our results with PIEtraj
[31], FOL-X [40], B-LSTM [2], LSTM [31], and Linear [31] trajectory prediction
models. On the TITAN dataset, we first report the evaluation results compared
to BiTraP using observed and predicted lengths equal to PIE and JAAD. However,
to fairly compare our results on the TITAN dataset with prior work of Malla et
al. [26], Social-LSTM [1], and Social-GAN [12], we follow the same
experimental setup used in [26]. To that end, we retrain both the BiTraP
baseline model, and our proposed model (ABC+) to predict 20 frames, after
observing 10 frames, and we report our results using ADE and FDE in pixels,
not in squared pixels.

###  4.3 Trajectory Prediction Results

The evaluation results are shown in Tab. 1 for PIE and JAAD. For TITAN, in
Tab. 2 we show the evaluation results when observing 10 frames and predicting
20 frames, similar to [26], and in Tab. 3 we compare to BiTraP when observing
5 frames and predicting 10 frames. As the tables show, our method (ABC+)
achieves superior performance compared to the baseline BiTraP, which does not
use our action-based contrastive loss. This result highlights the
effectiveness of adding the proposed contrastive objective and sampling
strategy. Our proposed method also outperforms other baseline methods, with
significant margins.

These evaluation results confirm the gains obtained by using our proposed
action-based contrastive loss and sampling strategy. Utilizing action
information with our contrastive approach exhibits improved performance across
all evaluated benchmarks. In the TITAN dataset, particularly, the performance
benefits appear larger. We believe this is due to TITAN’s more comprehensive
action class structure, compared to PIE or JAAD. In other words, a more
diverse set of pedestrian action classes improves the learned embedding space
by our action-based contrastive loss. Nevertheless, our method improves the
obtained results even with a simpler binary action class structure in PIE and
JAAD.

Another significant result is that our method (ABC+) outperforms the baseline
Titan-AP [26] on TITAN, which incorporates the same action class information
with observed trajectory information, and produces a combined embedding to
predict the future trajectory. This indicates that our approach of supporting
the trajectory prediction model with behavioral context information by using
action-based contrastive loss is more effective than encoding the action
classes in the embedding space representation.

Table 4: Ablation results on PIE, JAAD, and TITAN datasets. ABC+ is our proposed action-based contrastive framework with sampling from a learned CVAE. ABC uses our action-based contrastive loss but without sampling from a learned CVAE. SimCLR uses a normal batch contrastive loss instead of our proposed action-based contrastive loss. Lower is better. |  Method | ADE | C-ADE | C-FDE
---|---|---|---|---
|  0.5 |  1.0 |  1.5 |  1.5 |
PIE |  SimCLR |  26 |  67 |  163 |  125 |  399
|  ABC |  16 |  40 |  93 |  69 |  213
|  ABC+ |  16 |  38 |  87 |  65 |  191
JAAD |  SimCLR |  50 |  124 |  273 |  211 |  608
|  ABC |  41 |  93 |  201 |  150 |  425
|  ABC+ |  40 |  89 |  189 |  145 |  409
TITAN |  SimCLR |  255 |  506 |  999 |  773 |  1805
|  ABC |  188 |  345 |  634 |  488 |  951
|  ABC+ |  165 |  302 |  575 |  434 |  843

Figure 3: C-FDE results (top) and C-ADE(1.5) (bottom) of trajectory prediction
model by applying different $\beta$ values 0.25,0.5,0.75 in Eq. 2. The results
are reported for TITAN, PIE, and JAAD datasets. Lower values are better.

###  4.4 Ablation study

In this section, we present the ablation studies to provide further insights
into our proposed action-based contrastive loss. Similar to the evaluation
results shown above, we refer to our proposed action-based contrastive
framework by ABC+, where we use the action-based contrastive loss as a
regularizer to the trajectory prediction loss, and we also increase negative
and positive samples during training by sampling synthetic trajectories from
CVAE.

Does action information improve the contrastive loss? The first ablation study
examines how the action-based contrastive loss Eq. 1 compares to the batch
contrastive loss, namely SimCLR [5], shown in Tab. 4. For this baseline, we
replace the action-based contrastive loss with SimCLR contrastive loss, and we
measure the trajectory prediction performance. The results demonstrate the
impact of utilizing contextual information in form of action on the future
trajectory prediction model.

Is the proposed action-based sampling strategy using a CVAE effective? The
second ablation study analyzes the impact of sampling synthetic trajectories
from CVAE on the trajectory prediction model performance. The baseline ABC in
Tab. 4 indicates the trajectory prediction model trained with action-based
contrastive loss without using the extra synthetic samples from the learned
CVAE. Comparing the quantitative results of ABC+ to the results of ABC
highlights the effectiveness of our novel sampling strategy on all datasets.

How does the weight of the contrastive loss affect the results? Finally, we
also study the influence of the hyper-parameter $\beta$ in Eq. 2, which
controls the impact of the action-based contrastive loss into the final
objective function. As shown in Fig. 3, we obtain the best results on all
benchmark datasets by setting $\beta$ to 0.75.

##  5 Discussion and Conclusions

We presented a contrastive framework for learning behavior-aware pedestrian
trajectory representations. Our proposed framework consists of an action-based
contrastive loss, and a novel trajectory sampling technique from a learned
distribution of a C-VAE model. The proposed framework significantly improves
the performance of trajectory prediction models on three different first-
person view benchmarks. Our evaluation results provide evidence that including
pedestrian behavior information, in the form of action or activity class in
this case, is beneficial for trajectory prediction. Moreover, our results also
confirm that our action-based contrastive loss, in conjunction with our
sampling strategy, is superior to alternative approaches that also utilize
action class information.

This work comes with a number of strengths. First, we ensure our proposed
contrastive loss can be easily integrated with commonly used trajectory
prediction models. Second, our proposed sampling strategy utilizes readily
learned distributions by generative models, such as CVAEs, and it avoids
designing data-specific heuristics. This allows for wider range of
applications, such as on animal trajectory data. Finally, contrastive learning
in general, and our proposed action-based framework, in particular, allow for
enhancing the quantities of underrepresented action classes in the data. This
line of work may help address the shortage of necessary edge-cases in training
datasets, which may be encountered in real-world scenarios.

This work also comes with a limitation. While effective, our proposed action-
based contrastive framework requires pedestrian action labels during the
training phase only. However, this requirement is mitigated by our new
trajectory sampling technique from CVAE, which does not require action labels
for generated samples. Making the training scheme semi-supervised in our
model. In addition, many modern trajectory datasets are increasingly providing
action information. Action prediction tasks from video data achieve high
performances [7], and hence can be performed efficiently and reliably as a
pre-processing step for our trajectory prediction framework. We deem
evaluating such idea as future work.

## Acknowledgment

This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German
Research Foundation) under Germany’s Excellence Strategy – EXC 2002/1 “Science
of Intelligence” – project number 390523135.

## References

  * [1] Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., Savarese, S.: Social lstm: Human trajectory prediction in crowded spaces. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 961–971 (2016). https://doi.org/10.1109/CVPR.2016.110
  * [2] Bhattacharyya, A., Fritz, M., Schiele, B.: Long-term on-board prediction of people in traffic scenes under uncertainty. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. 4194–4202 (2018)
  * [3] Bhattacharyya, A., Schiele, B., Fritz, M.: Accurate and diverse sampling of sequences based on a “best of many” sample objective. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8485–8493 (2018)
  * [4] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems 33, 9912–9924 (2020)
  * [5] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PMLR (2020)
  * [6] Chen, X., He, K.: Exploring simple siamese representation learning. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 15750–15758 (2021)
  * [7] Degardin, B., Proença, H.: Human behavior analysis: A survey on action recognition. Applied Sciences 11(18) (2021). https://doi.org/10.3390/app11188324, https://www.mdpi.com/2076-3417/11/18/8324
  * [8] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [9] Deo, N., Trivedi, M.M.: Convolutional social pooling for vehicle trajectory prediction. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp. 1549–15498 (2018). https://doi.org/10.1109/CVPRW.2018.00196
  * [10] Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., Zisserman, A.: With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In: International Conference on Computer Vision (ICCV). pp. 9588–9597 (October 2021)
  * [11] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P.H., Buchatskaya, E., Doersch, C., Pires, B.A., Guo, Z.D., Azar, M.G., Piot, B., Kavukcuoglu, K., Munos, R., Valko, M.: Bootstrap your own latent: A new approach to self-supervised learning (2020)
  * [12] Gupta, A., Johnson, J., Fei-Fei, L., Savarese, S., Alahi, A.: Social gan: Socially acceptable trajectories with generative adversarial networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. 2255–2264 (2018)
  * [13] Harwood, B., Kumar B G, V., Carneiro, G., Reid, I., Drummond, T.: Smart mining for deep metric learning. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (Oct 2017)
  * [14] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9729–9738 (2020)
  * [15] Henaff, O.: Data-efficient image recognition with contrastive predictive coding. In: International Conference on Machine Learning. pp. 4182–4192. PMLR (2020)
  * [16] Hoffer, E., Ailon, N.: Deep metric learning using triplet network. In: International workshop on similarity-based pattern recognition. pp. 84–92. Springer (2015)
  * [17] Jahanian, A., Puig, X., Tian, Y., Isola, P.: Generative models as a data source for multiview representation learning. In: International Conference on Learning Representations (2022), https://openreview.net/forum?id=qhAeZjs7dCL
  * [18] Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D.: Supervised contrastive learning. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 18661–18673. Curran Associates, Inc. (2020)
  * [19] Kim, D., Yoo, Y., Park, S., Kim, J., Lee, J.: Selfreg: Self-supervised contrastive regularization for domain generalization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9619–9628 (2021)
  * [20] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [21] Lee, N., Choi, W., Vernaza, P., Choy, C.B., Torr, P.H., Chandraker, M.: Desire: Distant future prediction in dynamic scenes with interacting agents. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 336–345 (2017)
  * [22] Liang, J., Jiang, L., Niebles, J.C., Hauptmann, A.G., Fei-Fei, L.: Peeking into the future: Predicting future person activities and locations in videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5725–5734 (2019)
  * [23] Liu, Y., Yan, Q., Alahi, A.: Social nce: Contrastive learning of socially-aware motion representations. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 15118–15129 (2021)
  * [24] Ma, S., Sigal, L., Sclaroff, S.: Learning activity progression in lstms for activity detection and early detection. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 1942–1950 (2016)
  * [25] Makansi, O., Çiçek, O., Marrakchi, Y., Brox, T.: On exposing the challenging long tail in future prediction of traffic actors. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 13147–13157 (October 2021)
  * [26] Malla, S., Dariush, B., Choi, C.: Titan: Future forecast using action priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11186–11196 (2020)
  * [27] Mangalam, K., Girase, H., Agarwal, S., Lee, K., Adeli, E., Malik, J., Gaidon, A.: It is not the journey but the destination: Endpoint conditioned trajectory prediction. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J. (eds.) Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II. Lecture Notes in Computer Science, vol. 12347, pp. 759–776. Springer (2020). https://doi.org/10.1007/978-3-030-58536-5_45, https://doi.org/10.1007/978-3-030-58536-5_45
  * [28] Montes, A., Salvador, A., Pascual, S., Giro-i Nieto, X.: Temporal activity detection in untrimmed videos with recurrent neural networks. In: 1st NIPS Workshop on Large Scale Computer Vision Systems (December 2016)
  * [29] Van den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv e-prints pp. arXiv–1807 (2018)
  * [30] Pareek, P., Thakkar, A.: A survey on video-based human action recognition: recent updates, datasets, challenges, and applications. Artificial Intelligence Review 54(3), 2259–2322 (2021)
  * [31] Rasouli, A., Kotseruba, I., Kunic, T., Tsotsos, J.K.: Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction. In: International Conference on Computer Vision (ICCV) (2019)
  * [32] Rasouli, A., Kotseruba, I., Tsotsos, J.K.: Are they going to cross? a benchmark dataset and baseline for pedestrian crosswalk behavior. In: Proceedings of the IEEE International Conference on Computer Vision Workshops. pp. 206–213 (2017)
  * [33] Rudenko, A., Palmieri, L., Herman, M., Kitani, K.M., Gavrila, D.M., Arras, K.O.: Human motion trajectory prediction: a survey. The International Journal of Robotics Research 39(8), 895–935 (2020). https://doi.org/10.1177/0278364920917446, https://doi.org/10.1177/0278364920917446
  * [34] Sadeghian, A., Kosaraju, V., Sadeghian, A., Hirose, N., Rezatofighi, H., Savarese, S.: Sophie: An attentive gan for predicting paths compliant to social and physical constraints. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)
  * [35] Sohn, K., Lee, H., Yan, X.: Learning structured output representation using deep conditional generative models. In: Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 28. Curran Associates, Inc. (2015)
  * [36] Sun, Y., Tzeng, E., Darrell, T., Efros, A.A.: Unsupervised domain adaptation through self-supervision. CoRR abs/1909.11825 (2019), http://arxiv.org/abs/1909.11825
  * [37] Wang, C., Wang, Y., Xu, M., Crandall, D.J.: Stepwise goal-driven networks for trajectory prediction. IEEE Robotics and Automation Letters 7(2), 2716–2723 (2022). https://doi.org/10.1109/LRA.2022.3145090
  * [38] Wu, C.Y., Manmatha, R., Smola, A.J., Krahenbuhl, P.: Sampling matters in deep embedding learning. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (Oct 2017)
  * [39] Yao, Y., Atkins, E., Johnson-Roberson, M., Vasudevan, R., Du, X.: Bitrap: Bi-directional pedestrian trajectory prediction with multi-modal goal estimation. IEEE Robotics and Automation Letters (RA-L) (2021)
  * [40] Yao, Y., Xu, M., Choi, C., Crandall, D.J., Atkins, E.M., Dariush, B.: Egocentric vision-based future vehicle localization for intelligent driving assistance systems. 2019 International Conference on Robotics and Automation (ICRA) pp. 9711–9717 (2019)
  * [41] Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S.: Barlow twins: Self-supervised learning via redundancy reduction. In: International Conference on Machine Learning. pp. 12310–12320. PMLR (2021)


We then set up a correspondence between a collection of simple surface defects $S_i(\Sigma)$ indexed by $i = 1,\ldots,n$ supporting fusion categories of topological lines $\mathsf{C}_{S_i} = \mathsf{C}_j$ and gapped boundary conditions for an auxiliary 3d TQFT $\cT_{\mathsf{B}}$ (by an abuse of notation, this depends only on the Morita equivalence class of $\mathsf{B}$).

In this spirit, it is again convenient to restrict attention to simple line defects and work with a formulation of $3\mathsf{Vec}$ where equivalent objects are identified:
\begin{itemize}
\item Objects are 3d TQFTs, determined by their 2-categories of boundary conditions $\mathsf{Mod}(\mathsf{B})$ for some multi-fusion category $\mathsf{B} = \mathsf{C}_1 \oplus ... \oplus \mathsf{C}_n$.
\item 1-morphisms are are topological surface interfaces between 3d TQFTs and correspond to (pseudo-)2-functors $A: \mathsf{Mod}(\mathsf{B}) \to \mathsf{Mod}(\mathsf{B}')$ between the corresponding 2-categories of boundary conditions.

\item 2-morphisms are topological line-like interfaces between topological surface interfaces and correspond to (pseudo-)natural transformation $\Phi: A \Rightarrow B$ between the corresponding 2-functors of 2-categories of boundary conditions.

\item Its 3-morphisms are topological pointlike junctions between line-like topological interfaces. As such, they correspond to modifications $\varphi: \Phi \Rrightarrow \Psi$ between the corresponding (pseudo-)natural transformations.
\end{itemize}
The fusion of objects corresponds to stacking the associated 3d TQFTs.

Using this picture, the action of the symmetry group $G$ on surface defects $S$ can now be translated into the $G$-equivariance of the attached TQFT $\mathcal{T}_{\mathsf{B}}$. Concretely, consider wedging $S$ between a small $(D-1)$-sandwich labelled by $g \in G$ as before. Due to the attached 3d TQFT, this now requires a choice of surface-like intersection $\mathcal{F}(g)$ between the defect $U_g$ and $\mathcal{T}_{\mathsf{B}}$, which we regard as a topological interface between $\mathcal{T}_{\mathsf{B}}$ and itself as illustrated in figure \ref{fig:3d-tqft-2}.
\begin{figure}[h]
\centering
\includegraphics[height=5.25cm]{3d-tqft-2.pdf}
\vspace{-5pt}
\caption{}
\label{fig:3d-tqft-2}
\end{figure}
As such, they can naturally be identified with 2-functors
\begin{equation}
\mathcal{F}(g)\, : \; \mathsf{Mod}(\mathsf{B}) \; \to \; \mathsf{Mod}(\mathsf{B})
\end{equation}
from the 2-category of boundary conditions of $\mathcal{T}_{\mathsf{B}}$ to itself, so that symmetry defects $g \in G$ can act on the simple boundary conditions $S \in \mathsf{Mod}(\mathsf{B})$ via
\begin{equation}
S \; \mapsto \; \mathcal{F}(g)(S) \, .
\end{equation}
Physically, this corresponds to shrinking down the $(D-1)$-sandwich formed by $g$ towards $S$ and thus implements the action of $g$ on the corresponding surface operator.

In general, the composition of these 2-functors may be controlled by a non-trivial associator
\begin{equation}
\alpha(g,h,k) \, : \;\; \big[ \mathcal{F}(g) \circ \mathcal{F}(h) \big] \circ \mathcal{F}(k) \;\; \Rightarrow \;\; \mathcal{F}(g) \circ \big[ \mathcal{F}(h) \circ \mathcal{F}(k) \big] \, .
\end{equation}
Physically, this means that when crossing the fusion of three surface interfaces $\mathcal{F}(g)$, $\mathcal{F}(h)$ and $\mathcal{F}(k)$, an invertible topological line defect $\alpha(g,h,k) \in \mathsf{Z(B)}^{\times}$ in $\mathcal{T}_{\mathsf{B}}$ emanates from their junction as illustrated in figure \ref{fig:3d-tqft-4}.
\begin{figure}[h]
\centering
\includegraphics[height=5.3cm]{3d-tqft-4.pdf}
\vspace{-5pt}
\caption{}
\label{fig:3d-tqft-4}
\end{figure}
Compatibility with the fusion of four surface interfaces then restrains the collection of these lines to form a 3-cocycle $\alpha \in Z^3(G,\mathsf{Z(B)}^{\times})$. The associated class $[\alpha] \in H^3(G,\mathsf{Z(B)}^{\times})$ represents the obstruction for the functors $\mathcal{F}(g)$ to define consistent intersections that are compatible with the fusion of symmetry defects in the bulk. It can be interpreted as the Postnikov class of a non-trivial 2-group formed by the intersections surfaces $\mathcal{F}(g)$ and the invertible lines of $\mathcal{T}_{\mathsf{B}}$.

If $[\alpha] = 1$, we can introduce natural equivalences
\begin{equation}\label{eq:3d-tqft-line-junctions}
F_{g,h} \, : \;\; \mathcal{F}(g) \, \circ \, \mathcal{F}(h) \;\, \Rightarrow \;\, \mathcal{F}(g \cdot h)
\end{equation}
that correspond to line-like junctions arising from the intersection of $\mathcal{T}_{\mathsf{B}}$ with the fusion of symmetry defects $g,h \in G$ in the bulk as illustrated in figure \ref{fig:3d-tqft-3}.
\begin{figure}[h]
\centering
\includegraphics[height=4.9cm]{3d-tqft-3.pdf}
\vspace{-5pt}
\caption{}
\label{fig:3d-tqft-3}
\end{figure}
Their compatibility with the fusion of three symmetry defects in the bulk is implemented by modifications
\begin{equation}\label{eq:3d-tqft-point-junctions}
f_{g,h,k} \, : \;\;\; F_{gh,k} \, \circ \, \big[ F_{g,h} \, \star \, \text{Id}_{\mathcal{F}(k)} \big]  \;\;\;
\Rrightarrow \;\;\; F_{g,hk} \, \circ \, \big[ \text{Id}_{\mathcal{F}(g)} \, \star \, F_{h,k} \big]  \, ,
\end{equation}
which correspond to point-like junctions arising from the intersection of $\mathcal{T}_{\mathsf{B}}$ with the fusion of $g,h,k \in G$ as illustrated in figure \ref{fig:3d-tqft-5}.
\begin{figure}[h]
\centering
\includegraphics[height=5.3cm]{3d-tqft-5.pdf}
\vspace{-5pt}
\caption{}
\label{fig:3d-tqft-5}
\end{figure}
In general, these junctions may respect the group law of $G$ only projectively\footnote{In this case, the 3d TQFT should itself be regarded as the boundary of a 4-dimensional theory.} in the sense that intersecting $\mathcal{T}_{\mathsf{B}}$ with the fusion of four symmetry defects $g,h,k,l \in G$ in two possible ways yields equivalent results up to a multiplicative phase $\pi(g,h,k,l) \in U(1)$ as illustrated in figure \ref{fig:3d-tqft-6}.
\begin{figure}[h]
\centering
\includegraphics[height=9cm]{3d-tqft-6.pdf}
\vspace{-5pt}
\caption{}
\label{fig:3d-tqft-6}
\end{figure}
The collection of these phases may be regarded as a map
\begin{equation}
\pi : \; G \times G \times G \times G \; \to \ U(1) \, ,
\end{equation}
which, in order to be compatible with the intersection of five symmetry defects, needs to form a 4-cocycle $\pi \in Z^4(G,U(1))$. The associated class $[\pi] \in H^4(G,U(1))$ represents the obstruction for the modifications $f_{g,h,k}$ to define consistent intersections that are compatible with the fusion of symmetry defects in the bulk. If $[\pi] = 1$, they can be renormalised to satisfy the appropriate compatibility conditions.

In summary, the action of the symmetry group $G$ on genuine surface operators in this framework can be described by the following data:
\begin{enumerate}
\item A 3d TQFT $\mathcal{T}_{\mathsf{B}}$ (an object in $\mathsf{3Vec}$),
\item a topological surface interface $\mathcal{F}(g)$ between $\mathcal{T}_{\mathsf{B}}$ and itself (a 1-endomorphism of $\mathcal{T}_{\mathsf{B}}$ in $\mathsf{3Vec}$) for ech $g \in G$,
\item a topological line-like junction $F_{g,h}$ (a 2-morphism in $\mathsf{3Vec}$) as in (\ref{eq:3d-tqft-line-junctions}) for each pair $g,h \in G$,
\item a topological point-like junction $f_{g,h,k}$ (a 3-morphism in $\mathsf{3Vec}$) as in (\ref{eq:3d-tqft-point-junctions}) for each triple $g,h,k \in G$ .
\end{enumerate}
The collection of this data together with the compatibility conditions they satisfy can be recognised as the data of a (pseudo-)3-functor
\begin{equation}
\mathcal{F} : \; \widehat{G} \; \to \; \mathsf{3Vec} \, ,
\end{equation}
where the group $G$ is regarded as a 3-category $\widehat{G}$ with a single object $\ast$, whose 1-endomorphisms are given by $\text{1-End}_{\widehat{G}}(\ast) = G$ and whose 2- and 3-morphisms are trivial. The collection of such 3-functors itself forms a 3-category whose
\begin{itemize}
\item objects are 3-functors $\mathcal{F}: \widehat{G} \to \mathsf{3Vec}$,
\item 1-morphisms are natural transformations $\eta: \mathcal{F} \Rightarrow \mathcal{F}'$,
\item 2-morphisms are modifications $\Xi: \eta \Rrightarrow \eta'$,
\item 3-morphisms are perturbations $\mathfrak{X}: \Xi \,\mathrel{\substack{\textstyle\Rightarrow\\[-0.5ex] \textstyle\Rightarrow}} \,\Xi'$.
\end{itemize}
We denote this category by $[\widehat{G},\mathsf{3Vec}]$ and recognise it as the fusion 3-category of finite 3-representations of $G$,
\begin{equation}
[\widehat{G},\mathsf{3Vec}] \; = \; \mathsf{3Rep}(G) \, .
\end{equation}
This reproduces the result from the previous subsection that surface operators transform in 3-representations of the symmetry group $G$. Equivalently, this may be regarded as the 3-category of $G$-equivariant 3d TQFTs.

\acknowledgments

The work of MB is supported by the EPSRC Early Career Fellowship EP/T004746/1 and the STFC Research Grant ST/T000708/1. The work of MB and AG is supported by the Simons Collaboration on Global Categorical Symmetry.

\appendix

\section{Higher representation theory}
\label{app:higher-representations}

In this appendix, we will review the higher representation theory of higher groups from an abstract categorical point of view. This should serve as a companion to the main body of the paper, where higher representations are constructed in a concrete physical context.

Generically, we will think of a linear $n$-representation of an $n$-group $\mathcal{G}$ as a functor
\begin{equation}
F : \; \widehat{\mathcal{G}} \, \to \, \mathsf{nVec}
\end{equation}
from the $n$-group thought of as an appropriate type of $n$-category $\widehat{\mathcal{G}}$ into the fusion $n$-category $\mathsf{nVec}$ of $n$-vector spaces. For $n>1$, there may be a multitude of models for $\mathsf{nVec}$ and different models may be natural for different applications\footnote{For a summary of models and relationships between them in the case $n=2$ see~[38].}. For our purposes, the most natural choice is the recursive definition
\begin{equation}
\mathsf{nVec} \, := \, \Sigma^n\mathbb{C}
\end{equation}
introduced in~[39], where $\Sigma\mathsf{C} := \text{Kar}(B\mathsf{C})$ denotes the Karoubi completion of the delooping of $\mathsf{C}$. Linear $n$-representations of $\mathcal{G}$ then form a fusion $n$-category which we denote by
\begin{equation}
\mathsf{nRep}(\mathcal{G}) \, := \, [\widehat{\mathcal{G}},\mathsf{nVec}] \, .
\end{equation}
We will uncover this definition concretely in the cases $n=1,2,3$ below.

\subsection{1-representations}
\label{app:1-representations}

Let us begin by reformulating the representation theory of ordinary groups in a categorical framework. To do this, we fix a finite group $G$ and note that we can associate to it a category $\widehat{G}$ defined as follows:
\begin{itemize}
\item Its set of objects contains a single element denoted by $\ast$.
\item The set of endomorphisms of $\ast$ is given by
\begin{equation}
\text{End}_{\widehat{G}}(\ast) \, = \, G
\end{equation}
with composition given by group multiplication in $G$.
\end{itemize}
This construction makes it clear that we can think of groups as special types of categories with a single object, all of whose morphisms are invertible.

We would like to linearly represent the group $G$ on finite dimensional complex vector spaces. The latter form a fusion category $\mathsf{Vec}$ defined as follows:

\begin{itemize}
\item Its objects are finite-dimensional complex vector spaces $V \cong \mathbb{C}^n$. Up to equivalence, there is a single simple object corresponding to the one-dimensional vector space $\mathbb{C}$.

\item Its morphisms between objects $V$ and $W$ are given by linear maps
\vspace{-1pt}
\begin{equation}
\begin{gathered}
\includegraphics[height=0.63cm]{eq-vec-morphism.pdf}
\end{gathered}
\vspace{-4pt}
\end{equation}
The composition of morphisms
\vspace{-1pt}
\begin{equation}
\begin{gathered}
\includegraphics[height=0.63cm]{eq-vec-composition.pdf}
\end{gathered}
\vspace{-4pt}
\end{equation}
is given by the composition of the corresponding linear maps.

\item The fusion of objects and morphisms
\vspace{-1pt}
\begin{equation}
\begin{gathered}
\includegraphics[height=1.28cm]{eq-vec-fusion.pdf}
\end{gathered}
\vspace{-4pt}
\end{equation}
is given by the tensor product of the corresponding vector spaces and linear maps.
\end{itemize}

Given the ingredients $\widehat{G}$ and $\mathsf{Vec}$, it is now straightforward to define the category $\mathsf{Rep}(G)$ of finite-dimensional representations of $G$:
\begin{itemize}
\item Its objects are functors $\mathcal{F}: \widehat{G} \to \mathsf{Vec}$. Concretely, this means that $\mathcal{F}$ assigns a finite dimensional complex vector space $V := \mathcal{F}(\ast) \in \mathsf{Vec}$ to the single object $\ast$ of $\widehat{G}$ and linear maps $\mathcal{F}(g) \in \text{End}(V)$ to the endomorphisms $g \in G$ of $\ast$ in $\widehat{G}$. Functoriality of $\mathcal{F}$ then ensures that the diagrams
\vspace{-1pt}
\begin{equation}\label{eq-rep-morphism}
\begin{gathered}
\includegraphics[height=2.3cm]{eq-rep-morphism.pdf}
\end{gathered}
\vspace{-4pt}
\end{equation}
commute for all $g,h \in G$, so that objects of $\mathsf{Rep}(G)$ correspond to ordinary representations of $G$ in the usual sense.

\item Its morphisms between objects $\mathcal{F}$ and $\mathcal{F}'$ are natural transformations $\eta: \mathcal{F} \Rightarrow \mathcal{F}'$. Concretely, this means that $\eta$ assigns a linear map $\varphi := \eta(\ast) \in \text{Hom}(V,V')$ to the single object $\ast$ of $\widehat{G}$ such that the diagrams
\vspace{-2pt}
\begin{equation}
\begin{gathered}
\includegraphics[height=2.1cm]{eq-rep-intertwiner.pdf}
\end{gathered}
\vspace{-4pt}
\end{equation}
commute for all $g \in G$. Morphisms in $\mathsf{Rep}(G)$ thus correspond to intertwiners between representations in the usual sense.
\end{itemize}

The fusion structure on $\mathsf{Vec}$ induces a fusion structure on $\mathsf{Rep}(G)$. In particular, this allows us to take direct sums and tensor products of representations in the usual sense.

Closely related is the category of projective representations of $G$, in which the diagram (\ref{eq-rep-morphism}) only commutes up to multiplicative phases $c(g,h) \in U(1)$. Pictorially, we denote this by adding a double arrow
\vspace{-1pt}
\begin{equation}
\begin{gathered}
\includegraphics[height=2.3cm]{eq-pr-rep-morphism.pdf}
\end{gathered}
\vspace{-4pt}
\end{equation}
to the diagram in (\ref{eq-rep-morphism}). These phases need to be compatible with the associativity of morphism composition in $\mathsf{Vec}$ in the sense that the diagram
\vspace{-1pt}
\begin{equation}
\begin{gathered}
\includegraphics[height=5.4cm]{eq-pr-rep-compatibility.pdf}
\end{gathered}
\vspace{-4pt}
\end{equation}
2-commutes, which we can rewrite as the 2-cocycle condition
\begin{equation}
(dc)(g,h,k) \; := \; \frac{c(h,k) \cdot c(g,hk)}{c(gh,k) \cdot c(g,h)} \; \stackrel{!}{=} \; 1 \, .
\end{equation}
The collection of phases can thus be thought of as a 2-cocycle $c \in Z^2(G,U(1))$. We denote the corresponding category of projective representations by $\mathsf{Rep}^c(G)$. The category of ordinary representations of $G$ is recovered by choosing $c=1$.

\subsection{2-representations}
\label{app:2-representations}

In the following, we would like to emulate the discussion of the previous section to define the fusion 2-category of 2-representations of a finite 2-group. We will review the notions of 2-groups and 2-vector spaces and provide a classification of 2-representations in terms of elementary group-theoretical data.

\subsubsection{2-groups}

In what follows, we will think of a 2-group $\mathcal{G}$ as a quadruple $\mathcal{G}=(G,A,\varphi,\alpha)$ consisting of
\begin{enumerate}
\item a finite group $G$,
\item a finite abelian group $A$,
\item a group action $\varphi: G \to \text{Aut}(A)$,
\item a Postnikov class representative $\alpha \in Z^3_{\varphi}(G,A)$.
\end{enumerate}

Similarly to the case of ordinary groups, we can associate to $\mathcal{G}$ a corresponding 2-category $\widehat{\mathcal{G}}$ that is defined as follows:
\begin{itemize}
\item Its set of objects contains a single element denoted by $\ast$.

\item The set of 1-endomorphisms of $\ast$ is given by
\begin{equation}
\text{1-End}_{\widehat{\mathcal{G}}}(\ast) \, = \, G
\end{equation}
with composition given by group multiplication in $G$.

\item The set of 2-morphisms between two 1-morphisms $g,h \in G$ is given by
\begin{equation}
\text{2-Hom}_{\widehat{\mathcal{G}}}(g,h) \; = \; \delta_{g,h} \cdot A
\end{equation}
with vertical composition given by group multiplication in $A$. The horizontal composition of two 2-morphisms $a \in \text{2-End}_{\widehat{\mathcal{G}}}(g)$ and $b \in \text{2-End}_{\widehat{\mathcal{G}}}(h)$ is given by
\begin{equation}
a \star b \; = \; a \cdot \varphi_g(b) \, \in \, \text{2-End}_{\widehat{\mathcal{G}}}(g \! \cdot \! h) \, .
\end{equation}

\item The 2-associator for the composition of 1-morphisms $g,h,k \in \text{1-End}_{\widehat{\mathcal{G}}}(\ast)$ is given by
\begin{equation}
\alpha(g,h,k) \, \in \, \text{2-End}_{\widehat{\mathcal{G}}}(g \! \cdot \! h \! \cdot \! k) \, .
\end{equation}
\end{itemize}
This construction makes it clear that we can alternatively think of 2-groups as special types of 2-categories with a single object, all of whose 1-morphisms and 2-morphisms are invertible.

\subsubsection{2-vector spaces}

Let us now consider the fusion 2-category $\mathsf{2Vec}$ of finite-dimensional 2-vector spaces, which can be described as follows:
\begin{itemize}
\item Its objects are finite-dimensional semi-simple associative algebras\footnote{Over a general field, this would be separable algebras. However, since we are working over $\mathbb{C}$, we can replace separable by semi-simple.}.
\item Its 1-morphisms are finite-dimensional bimodules.
\item Its 2-morphisms are bimodule maps.
\end{itemize}
As a consequence, equivalence between two objects $\mathcal{A}_1 \sim \mathcal{A}_2$ in $2\mathsf{Vec}$ corresponds to Morita equivalence of the corresponding finite-dimensional semi-simple algebras. Since any finite-dimensional semi-simple algebra $\mathcal{A}$ can be decomposed as a direct sum
\begin{equation}
\mathcal{A} \; \cong \; M_{r_1}(\mathbb{C}) \, \oplus \cdots \oplus \, M_{r_n}(\mathbb{C})
\end{equation}
of matrix algebras and any matrix algebra $M_r(\mathbb{C})$ is Morita equivalent to $\mathbb{C}$, any finite-dimensional semi-simple algebra is equivalent to a direct sum of $n$ copies of $\mathbb{C}$ in $\mathsf{2Vec}$. The equivalence classes of objects in $\mathsf{2Vec}$ are thus in 1-1 correspondence with natural numbers $n \in \mathbb{N}$, which reproduces the following description of the 2-category $\mathsf{2Vec}$ due to Kapranov and Voevodsky~[38]:
\begin{itemize}
\item Its objects are positive integers $n \in \mathbb{N}$. There is a single simple object corresponding to the positive integer $1$.

\item Its 1-morphisms between objects $n$ and $m$ are given by $(m \times n)$-matrices
\vspace{-1pt}
\begin{equation}
\begin{gathered}
\includegraphics[height=0.63cm]{eq-2vec-1-morphism.pdf}
\end{gathered}
\vspace{-4pt}
\end{equation}
whose entries are finite-dimensional vector spaces. The composition of 1-morphisms
\vspace{-1pt}
\begin{equation}
\begin{gathered}
\includegraphics[height=0.63cm]{eq-2vec-composition.pdf}
\end{gathered}
\vspace{-4pt}
\end{equation}
is given by the multiplication of the corresponding matrices using tensor products and direct sums of vector spaces.

\item Its 2-morphisms between 1-morphisms $A$ and $B$ are given by $(m \times n)$-matrices
\vspace{-2pt}
\begin{equation}
\begin{gathered}
\includegraphics[height=2cm]{eq-2vec-2-morphism.pdf}
\end{gathered}
\vspace{-4pt}
\end{equation}
whose entries are linear maps between the corresponding entries of $A$ and $B$. The vertical composition of 2-morphisms
\vspace{-4pt}
\begin{equation}
\begin{gathered}
\includegraphics[height=2.65cm]{eq-2vec-vertical-composition.pdf}
\end{gathered}
\vspace{-6pt}
\end{equation}
is given by entry-wise composition of linear maps. The horizontal composition of 2-morphisms
\vspace{-2pt}
\begin{equation}
\begin{gathered}
\includegraphics[height=2cm]{eq-2vec-horizontal-composition.pdf}
\end{gathered}
\vspace{-4pt}
\end{equation}
is given by the multiplication of the corresponding matrices using tensor products and direct sums of linear maps.

\item The fusion of objects, 1-morphisms and 2-morphisms
\begin{equation}
\begin{gathered}
\includegraphics[height=2.25cm]{eq-2vec-fusion.pdf}
\end{gathered}
\vspace{-2pt}
\end{equation}
is given by the tensor product of the corresponding matrices using tensor products and direct sums of vector spaces and linear maps, respectively.
\end{itemize}

\subsubsection{2-representations \& their classification}

In analogy to the case of ordinary groups, it is now straightforward to define the 2-category $\mathsf{2Rep}(\mathcal{G})$ of finite-dimensional 2-representations of the 2-group $\mathcal{G}$:
\begin{itemize}
\item Its objects are (pseudo-)-2-functors $\mathcal{F}: \widehat{\mathcal{G}} \to \mathsf{2Vec}$.
\item Its 1-morphisms are natural transformations $\eta: \mathcal{F} \Rightarrow \mathcal{F}'$.
\item Its 2-morphisms are modifications $\Xi: \eta \Rrightarrow \eta'$.
\end{itemize}

This rather abstract definition can be reformulated in terms of more elementary data by breaking down the definitions of (pseudo-)-2-functors, natural transformations and modifications. For instance, consider a fixed 2-representation $\mathcal{F}: \widehat{\mathcal{G}} \to \mathsf{2Vec}$. This then amounts to the following:
\begin{itemize}
\item To the single object $\ast$ of $\widehat{\mathcal{G}}$ the 2-functor $\mathcal{F}$ assigns a natural number $n := \mathcal{F}(\ast) \in \mathbb{N}$ which we call the dimension of the 2-representation.

\item To the 1-endomorphisms $g \in G$ of $\ast$ the 2-functor $\mathcal{F}$ assigns an invertible $(n \times n)$-matrix $\mathcal{F}(g)$ whose entries are finite dimensional complex vector spaces. Each row and column in $\mathcal{F}(g)$ can thus only contain one non-vanishing entry isomorphic to the one-dimensional vector space $\mathbb{C}$, so that up to isomorphism we can think of $\mathcal{F}(g)$ as a $(n \times n)$-permutation matrix. The 2-functor $\mathcal{F}$ thus induces a permutation action
\begin{equation}
\sigma: \; G \, \to \, S_n
\end{equation}
of the group $G$ on the set of $n$ elements.

\item To the 2-endomorphisms $a \in A$ of a 1-morphism $g \in G$ the 2-functor $\mathcal{F}$ assigns a $(n \times n)$-matrix whose entries are linear maps between the corresponding entries of $\mathcal{F}(g)$. Since the latter only contains one non-vanishing entry isomorphic to $\mathbb{C}$ in each row and column, we can view $\mathcal{F}(a)$ as a collection of $n$ phases $\chi_i(a) \in U(1)$. The 2-functor $\mathcal{F}$ thus induces a homomorphism
\begin{equation}
\chi: \; A \, \to \, U(1)^n \, ,
\end{equation}
which we can think of as a collection of $n$ characters $\chi \in (A^{\vee})^n$. 2-functoriality of $\mathcal{F}$ then implies that this collection satisfies
\begin{equation}
g \triangleright_{\sigma} \chi(a) \; = \; \chi(g \triangleright_{\varphi} a)
\end{equation}
for all $g \in G$ and $a \in A$, where we denoted by $\triangleright_{\sigma}$ and $\triangleright_{\varphi}$ the action of $G$ on $U(1)^n$ and $A$ induced by $\sigma$ and $\varphi$, respectively.

\item For each pair $g,h \in G$ we have a 2-isomorphism
\vspace{-1pt}
\begin{equation}
\begin{gathered}
\includegraphics[height=2.25cm]{eq-2rep-pseudo-morphism.pdf}
\end{gathered}
\vspace{-4pt}
\end{equation}
controlling the composition of $\mathcal{F}(g)$ and $\mathcal{F}(h)$. As such, they comprise linear maps between the $n$ one-dimensional entries of $F(gh)$ and $F(g) \circ F(h)$, so that we can view $F_{g,h}$ as a collection of $n$ phases $c(g,h) \in U(1)^n$. These phases need to be compatible with the associativity constraints on 1-morphism composition in $\mathsf{2Vec}$ in the sense that the diagram
\vspace{-1pt}
\begin{equation}
\begin{gathered}
\includegraphics[height=5.4cm]{eq-2rep-pseudo-compatibility.pdf}
\end{gathered}
\vspace{-4pt}
\end{equation}
2-commutes, which we can rewrite as the 2-cochain condition
\begin{equation}
(d_{\sigma}c)(g,h,k) \; = \; \frac{( g \triangleright_{\sigma} c(h,k) ) \cdot c(g,hk)}{c(gh,k) \cdot c(g,h)} \; \stackrel{!}{=} \; \braket{\chi, \alpha(g,h,k)} \, .
\end{equation}
The collection of phases can thus be thought of as a 2-cochain $c \in C^2(G,U(1)^n)$ that closes to the pairing $\braket{\chi,\alpha} \in Z^3(G,U(1)^n)$ of $\chi$ with the Postnikov representative $\alpha$.

\end{itemize}

In summary, we can think of 2-representations of the 2-group $\mathcal{G}$ as being labelled by quadruples $(n,\sigma,c,\chi)$ consisting of
\begin{enumerate}
\item a positive integer $n \in \mathbb{N}$,
\item a permutation action $\sigma: G \to S_n$,
\item a twisted 2-cochain $c \in C^2_{\sigma}(G,U(1)^n)$,
\item a collection of characters $\chi \in (A^{\vee})^n$,
\end{enumerate}
such that $g \triangleright_{\sigma}\chi(a) = \chi(g \triangleright_{\varphi} a)$ for all $g \in G$ and $a \in A$ and $d_{\sigma}c = \braket{\chi,\alpha}$. Two 2-representations labelled by quadruples $(n,\sigma,c,\chi)$ and $(n',\sigma',c',\chi')$ are considered equivalent if $n = n'$ and there exists a permutation $\tau \in S_n$ such that
\begin{equation}
\sigma' \, = \, \tau \circ \sigma \circ \tau^{-1} \, , \qquad [c'] \, = \, [\tau \triangleright c] \, , \qquad \chi' \, = \, \tau \triangleright \chi \, .
\end{equation}

\subsubsection{Direct sum and tensor product}

The fusion structure on $\mathsf{2Vec}$ induces a fusion structure on $\mathsf{2Rep}(G)$. In particular, this allows us to take direct sums and tensor products of 2-representations. In terms of the above labelling of 2-representations by elementary data, these can be described explicitly as follows: Consider two 2-representations $R=(n,\sigma,c,\chi)$ and $R'=(n',\sigma',c',\chi')$ of $\mathcal{G}$. Then, the following holds:
\begin{itemize}
\item Their direct sum is the 2-representation
\begin{equation}
R \oplus R' \; = \; \big(n + n', \, \sigma \oplus \sigma', \, c \oplus c', \, \chi \oplus \chi' \big) \, ,
\end{equation}
where the permutation action $\sigma \oplus \sigma': G \to S_{n + n'}$ is defined by
\begin{equation}
(\sigma \oplus \sigma')_g(i) \;\; := \;\;
\begin{cases}
\sigma_g(i) & \text{if} \;\; 1 \leq i \leq n \\
\sigma'_g(i-n) + n & \text{if} \;\; n+1 \leq i \leq n+n'
\end{cases} \, ,
\end{equation}
the twisted 2-cochain $c\oplus c' \in C^2_{\sigma \oplus \sigma'}(G,U(1)^{n+n'})$ is given by
\begin{equation}
(c \oplus c')_i(g,h) \; := \;
\begin{cases}
c_i(g,h) & \text{if} \;\; 1 \leq i \leq n \\
c'_{i-n}(g,h) & \text{if} \;\; n+1 \leq i \leq n+n'
\end{cases} \, ,
\end{equation}
and the collection of characters $\chi \oplus \chi' \in (A^{\vee})^{n+n'}$ is taken to be
\begin{equation}
(\chi \oplus \chi')_i \; := \;
\begin{cases}
\chi_i & \text{if} \;\; 1 \leq i \leq n \\
\chi'_{i-n} & \text{if} \;\; n+1 \leq i \leq n+n'
\end{cases} \, .
\end{equation}

\item Their tensor product is the 2-representation
\begin{equation}
R \otimes R' \; = \; \big(n \cdot n', \, \sigma \otimes \sigma', \, c \otimes c', \, \chi \otimes \chi' \big) \, ,
\end{equation}
where the permutation action $\sigma \otimes \sigma': G \to S_{n\cdot n'}$ is defined by
\begin{equation}
(\sigma \otimes \sigma')_g(i,j) \; := \; \big(\sigma_g(i),\sigma'_g(j)\big) \, ,
\end{equation}
the twisted 2-cochain $c\otimes c' \in C^2_{\sigma \otimes \sigma'}(G,U(1)^{n \cdot n'})$ is given by
\begin{equation}
(c \otimes c')_{(i,j)}(g,h) \; := \; c_i(g,h) \cdot c'_j(g,h) \, ,
\end{equation}
and the collection of characters $\chi \otimes \chi' \in (A^{\vee})^{n \cdot n'}$ is taken to be
\begin{equation}
(\chi \otimes \chi')_{(i,j)} \, := \, \chi_i \cdot \chi'_j \, .
\end{equation}
\end{itemize}

\subsubsection{Irreducibility}

The irreducible 2-representations are those which cannot be written as a direct sum of other 2-representations. In particular, this means that the permutation action $\sigma: G \to S_n$ of an irreducible 2-representation $S = (n,\sigma,c,\chi)$ is transitive, so that
\begin{equation}
\lbrace 1,...,n \rbrace \; \cong \; G / H
\end{equation}
as $G$-sets, where we denoted by $H := \text{Stab}_{\sigma}(1) \subset G$ the stabiliser of the permutation action. The twisted 2-cochain $c \in C^2_{\sigma}(G,U(1)^n)$ then induces an ordinary 2-cochain
\begin{equation}
u \; := \; c_1|_H \; \in \; C^2(H,U(1))
\end{equation}
on the subgroup $H$, which closes to $du = \braket{\lambda,\alpha|_H}$, where we denoted $\lambda := \chi_1 \in A^{\vee}$. In summary, we can label the irreducible 2-representations of $\mathcal{G}$ by triples $(H,u,\lambda)$ consisting of
\begin{enumerate}
\item a subgroup $H \subset G$,
\item a cochain $u \in C^2(H,U(1))$,
\item a $H$-invariant character $\lambda \in A^{\vee}$,
\end{enumerate}
such that $du = \braket{\lambda,\alpha|_H}$. Two irreducible 2-representations labelled by triples $(H,u,\lambda)$ and $(H',u',\lambda')$ are equivalent if there exists a group element $g \in G$ such that
\begin{equation}
H' \, = \, {}^{g\!}H \, , \qquad [u'] \, = \, [{}^gu] \, , \qquad \lambda' \, = \, {}^{g\hspace{-0.5pt}} \lambda \, ,
\end{equation}
where the conjugation action of $g$ on the entries of the triple $(H,u,\lambda)$ is defined by
\begin{equation}
{}^{g\!}H \; = \; g H g^{-1} \, , \quad\; ({}^gu)(h,h') \; = \; u(g^{-1}hg, g^{-1}h'g) \, , \quad\; ({}^g\lambda)(a) \, = \, \lambda(g^{-1} \triangleright_{\rho} a) \, .
\end{equation}

The irreducible 2-representations can be thought of as elementary building blocks in the sense that any generic 2-representation of $\mathcal{G}$ can be written as a direct sum of irreducible ones. In particular, the fusion of two irreducible 2-representations labelled by triples $(H,u,\lambda)$ and $(K,v,\eta)$ must again decompose as a direct sum of other irreducible 2-representations, which is reflected in Mackey's decomposition formula
\begin{equation}
(H,u,\lambda) \, \otimes \, (K,v,\eta) \;\; = \bigoplus_{[g] \, \in \, H \backslash G / K} \big( H \cap {}^{g\!}K, \; u \cdot {}^gv, \; \lambda \cdot {}^g\eta \big) \, .
\end{equation}
Similarly, the 1-morphism categories between irreducible 2-representations allow for a Mackey-type decomposition and are given by
\begin{equation}
\text{1-Hom}\big( (H,u,\lambda), \,(K,v,\eta) \big) \;\; = \bigoplus_{\substack{[g] \, \in \, H \backslash G / K \\[2pt] \overline{\lambda} \, \cdot \, {}^{g\hspace{-0.5pt}}\eta \; = \; 1}} \mathsf{Rep}^{\, \overline{u} \, \cdot \, {}^{g\hspace{-0.5pt}}v}(H \cap {}^{g\!}K) \, .
\end{equation}

\subsubsection{Example}

As an example, consider the 2-group $\mathcal{G} = (G,A,\varphi,\alpha)$ where
\begin{itemize}
\item $G= \mathbb{Z}_2$ is the cyclic group of order 2 with generator $x$,
\item $A= \mathbb{Z}_4$ is the cyclic group of order 4 with generator $y$,
\item $\varphi: G \to \text{Aut}(A)$ is defined on the generators by $\varphi_x(y) = y^3$,
\item $[\alpha] \in H^3_{\varphi}(G,A) = \mathbb{Z}_2$ is represented by a normalised 3-cocycle satisfying
\begin{equation}
\alpha(x,x,x) \, = \, 1 \qquad \text{or} \qquad \alpha(x,x,x) \, = \, y
\end{equation}
corresponding to a split or a non-split 2-group.
\end{itemize}
The irreducible 2-representations of $\mathcal{G}$ are then labelled by triples $(H,u,\lambda)$ as before, and can be summarised as follows:
\begin{itemize}
\item In the split case, there is no non-trivial choice of 2-cocycle $u$ since $H^2_{\varphi}(G,U(1)) = 1$. Up to equivalence, there are five irreducible 2-representations labelled by
\begin{equation}
\renewcommand{\arraystretch}{1.8}
\setlength{\tabcolsep}{10pt}
\begin{tabular}{c | c c c}
& $H$ & $u$ & $\lambda$  \\
\hline
$\mathbf{1}_+$ & $\mathbb{Z}_2$ & $1$ & 1\\
$\mathbf{1}_-$ & $\mathbb{Z}_2$ & $1$ & $\widehat{y}^2$ \\
$\mathbf{2}_+$ & $\lbrace 1 \rbrace$ & $1$ & $1$ \\
$\mathbf{2}_0$ & $\lbrace 1 \rbrace$ & $1$ & $\widehat{y}$ \\
$\mathbf{2}_-$ & $\lbrace 1 \rbrace$ & $1$ & $\widehat{y}^2$
\end{tabular}_{\strut^{\scalebox{1}{,}}}
\end{equation}
where we denoted by $\widehat{y}$ the generator of the Pontryagin dual $A^{\vee} = \mathbb{Z}_4$ of $A$. Their fusion rules are summarised by the fusion table
\begin{equation}
\renewcommand{\arraystretch}{1.8}
\setlength{\tabcolsep}{10pt}
\begin{tabular}{c | c  c  c  c  c}
$\otimes$ & $\mathbf{1}_+$ & $\mathbf{1}_-$ & $\mathbf{2}_+$ & $\mathbf{2}_0$ & $\mathbf{2}_-$ \\
\hline
$\mathbf{1}_+$ & $\mathbf{1}_+$ & $\mathbf{1}_-$ & $\mathbf{2}_+$ & $\mathbf{2}_0$ & $\mathbf{2}_-$ \\
$\mathbf{1}_-$ & $\mathbf{1}_-$ & $\mathbf{1}_+$ & $\mathbf{2}_-$ & $\mathbf{2}_0$ & $\mathbf{2}_+$ \\
$\mathbf{2}_+$ & $\mathbf{2}_+$ & $\mathbf{2}_-$ & $\mathbf{2}_+ \oplus \mathbf{2}_+$ & $\mathbf{2}_0 \oplus \mathbf{2}_0$ & $\mathbf{2}_- \oplus \mathbf{2}_-$ \\
$\mathbf{2}_0$ & $\mathbf{2}_0$ & $\mathbf{2}_0$ & $\mathbf{2}_0 \oplus \mathbf{2}_0$ & $\mathbf{2}_+ \oplus \mathbf{2}_-$ & $\mathbf{2}_0 \oplus \mathbf{2}_0$ \\
$\mathbf{2}_-$ & $\mathbf{2}_-$ & $\mathbf{2}_+$ & $\mathbf{2}_- \oplus \mathbf{2}_-$ & $\mathbf{2}_0 \oplus \mathbf{2}_0$ & $\mathbf{2}_+ \oplus \mathbf{2}_+$
\end{tabular}_{\strut^{\scalebox{1}{.}}}
\end{equation}
The 1-morphism spaces between simple objects are illustrated in figure \ref{fig:example-2-rep}.

\item In the non-split case, the condition $du = \braket{\lambda, \alpha|_H}$ is non-trivial only when $H = \mathbb{Z}_2$ and $\lambda = \widehat{y}^2$. However, since no such 2-cochain $u$ exists, the corresponding 2-representation $\mathbf{1}_-$ no longer exists. This is indicated by the red colouring of the $\mathbf{1}_-$ and its attached morphism spaces in figure \ref{fig:example-2-rep}.
\end{itemize}

\begin{figure}[h]
\centering
\includegraphics[height=5.7cm]{example-2-rep.pdf}
\caption{}
\label{fig:example-2-rep}
\end{figure}

\subsection{3-representations}
\label{app:3-representations}

In the following, we would like to define a notion of 3-representations that is analogous to the construction of 1- and 2-representations discussed in the previous subsections. Since the associated realm of fusion 3-categories is much richer and more intricate than before, we will not try to be systematic but only briefly highlight some of the most salient features. In general, one may define the notion of a 3-representations for a generic 3-group, but for simplicity we will restrict our attention to the study of 3-representations of ordinary finite groups $G$ (regarded as a special type of 3-group).

\subsubsection{3-vector spaces}

Let us first consider the fusion 3-category $\mathsf{3Vec}$ of finite-dimensional 3-vector spaces, which can be described as follows:
\begin{itemize}
\item Its objects are multi-fusion categories.
\item Its 1-morphisms are bimodule categories.
\item Its 2-morphisms are bimodule functors.
\item Its 3-morphisms are natural transformations.
\end{itemize}
As a consequence, equivalence of two objects $\mathsf{B}_1 \sim \mathsf{B}_2$ in $\mathsf{3Vec}$ corresponds to Morita equivalence of the corresponding multi-fusion categories. Since any multi-fusion category can be written as a direct sum of indecomposable ones and any indecomposable multi-fusion category is Morita equivalent to an ordinary fusion category, equivalence classes of objects in $\mathsf{3Vec}$ are in 1:1-correspondence with finite direct sums of fusion categories. In particular, there is certainly more than one simple object up to equivalence, which shows that objects in $\mathsf{3Vec}$ are now significantly richer than before.

\subsubsection{3-representations \& their classification}

Analogously to the cases of 1- and 2-representations, we can define the 3-category $\mathsf{3Rep}(G)$ of finite-dimensional 3-representations of an ordinary group $G$ as follows:
\begin{itemize}
\item Its objects are (pseudo-)3-functors $\mathcal{F}: \widehat{G} \to \mathsf{3Vec}$.
\item Its 1-morphisms are natural transformations $\eta: \mathcal{F} \Rightarrow \mathcal{F}'$.
\item Its 2-morphisms are modifications $\Xi: \eta \Rrightarrow \eta'$.
\item Its 3-morphisms are perturbations $\mathfrak{X}: \Xi \,\mathrel{\substack{\textstyle\Rightarrow\\[-0.5ex] \textstyle\Rightarrow}} \,\Xi'$.
\end{itemize}
Here, we regard the group $G$ as a special type of 3-category $\widehat{G}$ with a single object $\ast$, whose 1-endomorphisms are given by $\text{1-End}_{\widehat{G}}(\ast) = G$ and whose 2- and 3-morphisms are trivial.

This rather abstract construction of 3-representations can be reformulated in terms of more elementary data by breaking down the definitions of (pseudo-)3-functors, natural transformations, modification and perturbations. For instance, consider a fixed 2-representation $\mathcal{F}: \widehat{G} \to \mathsf{3Vec}$, which amounts to the following data:
\begin{itemize}
\item To the single object $\ast$ of $\widehat{G}$ the 3-functor $\mathcal{F}$ assigns a multifusion category $\mathsf{B}$. Up to equivalence, we can think of the latter as a direct sum $\mathsf{B} = \mathsf{C}_1 \oplus ... \oplus \mathsf{C}_n$ of ordinary fusion categories $\mathsf{C}_i$, whose number $n$ is called the dimension of the 3-representation. For simplicity, let us assume that the equivalence class of $\mathsf{B}$ is simply labelled by a single fusion category $\mathsf{C}$, corresponding to a one-dimensional 3-representation of $G$.

\item To the 1-endomorphisms $g \in G$ of $\ast$ the 3-functor assigns invertible $\mathsf{C}$-bimodule categories $\mathcal{F}(g)$, which induce a group homomorphism
\begin{equation}
\rho: \; G \, \to \, \text{BrPic}(\mathsf{C})
\end{equation}
from $G$ into the Brauer-Picard group $\text{BrPic}(\mathsf{C})$ of isomorphism classes of invertible $\mathsf{C}$-bimodules.

\item For each pair $g,h \in G$ we have a 2-isomorphism
\begin{equation}
\begin{gathered}
\includegraphics[height=2.25cm]{eq-3rep-pseudo-morphism.pdf}
\end{gathered}
\vspace{-4pt}
\end{equation}
controlling the composition of $\mathcal{F}(g)$ and $\mathcal{F}(h)$. As such, they correspond to invertible $\mathsf{C}$-bimodule functors which, up to equivalence, are given by the left action of a fixed invertible element $\kappa(g,h)$ of the Drinfeld centre $\mathsf{Z(C)}$ of $\mathsf{C}$. The collection of 2-isomorphisms $F_{g,h}$ then induces a 2-cocycle $\kappa \in Z^2(G,\mathsf{Z(C)}^{\times})$, where $\mathsf{Z(C)}^{\times}$ denotes the abelian group of isomorphism classes of invertible objects in $\mathsf{Z(C)}$.

\item For each triple $g,h,k \in G$ we have a 3-isomorphism
\vspace{-1pt}
\begin{equation}
\begin{gathered}
\includegraphics[height=5.4cm]{eq-3rep-pseudo-compatibility.pdf}
\end{gathered}
\vspace{-4pt}
\end{equation}
controlling the composition of the 2-isomorphisms $F_{g,h}$. As such, they correspond to natural isomorphisms of $\mathsf{C}$-bimodule functors given by multiplication by a phase $\mu(g,h,k) \in U(1)$. The collection of 3-isomorphisms $f_{g,h,k}$ then induces a 3-cocycle $\mu \in Z^2(G,U(1))$.
\end{itemize}

In summary, we can think of the one-dimensional 3-representations of $G$ as being labelled by quadruples $(\mathsf{C},\rho,\kappa,\mu)$ consisting of
\begin{enumerate}
\item a fusion category $\mathsf{C}$,
\item a group homomorphism $\rho: G \to \text{BrPic}(\mathsf{C})$,
\item a 2-cocycle $\kappa \in Z^2(G,\mathsf{Z(C)}^{\times})$,
\item a 3-cocycle $\mu \in Z^3(G,U(1))$.
\end{enumerate}
Analogously to the case of 2-representations, a general irreducible $n$-dimensional 3-representation of $G$ on a multifusion category $\mathsf{B} = \mathsf{C}_1 \oplus ... \oplus \mathsf{C}_n$ may be obtained as the induction of a one-dimensional 3-representation of a subgroup $H \subset G$.

\bibliographystyle{JHEP}
\providecommand{\href}[2]{#2}\begingroup\raggedright\begin{thebibliography}{10}

[1]
D.~Gaiotto, A.~Kapustin, N.~Seiberg, and B.~Willett, {\it {Generalized Global
Symmetries}},  {\em JHEP} {\bf 02} (2015) 172,
[\href{http://arxiv.org/abs/1412.5148}{{\tt arXiv:1412.5148}}].

[2]
E.~Sharpe, {\it {Notes on generalized global symmetries in QFT}},  {\em
Fortsch. Phys.} {\bf 63} (2015) 659--682,
[\href{http://arxiv.org/abs/1508.04770}{{\tt arXiv:1508.04770}}].

[3]
L.~Bhardwaj and Y.~Tachikawa, {\it {On finite symmetries and their gauging in
two dimensions}},  {\em JHEP} {\bf 03} (2018) 189,
[\href{http://arxiv.org/abs/1704.02330}{{\tt arXiv:1704.02330}}].

[4]
Y.~Tachikawa, {\it {On gauging finite subgroups}},  {\em SciPost Phys.} {\bf 8}
(2020), no.~1 015, [\href{http://arxiv.org/abs/1712.09542}{{\tt

[5]
F.~Benini, C.~C\'ordova, and P.-S. Hsin, {\it {On 2-Group Global Symmetries and
their Anomalies}},  {\em JHEP} {\bf 03} (2019) 118,
[\href{http://arxiv.org/abs/1803.09336}{{\tt arXiv:1803.09336}}].

[6]
P.-S. Hsin, H.~T. Lam, and N.~Seiberg, {\it Comments on one-form global
symmetries and their gauging in 3d and 4d},  {\em SciPost Phys.} {\bf 6}
(2019), no.~3 039, [\href{http://arxiv.org/abs/1812.04716}{{\tt

[7]
I.~n. Garc\'\i{}a~Etxebarria, B.~Heidenreich, and D.~Regalado, {\it {IIB flux
non-commutativity and the global structure of field theories}},  {\em JHEP}
{\bf 10} (2019) 169, [\href{http://arxiv.org/abs/1908.08027}{{\tt

[8]
J.~Eckhard, H.~Kim, S.~Schafer-Nameki, and B.~Willett, {\it {Higher-Form
Symmetries, Bethe Vacua, and the 3d-3d Correspondence}},  {\em JHEP} {\bf 01}
(2020) 101, [\href{http://arxiv.org/abs/1910.14086}{{\tt arXiv:1910.14086}}].

[9]
O.~Bergman, Y.~Tachikawa, and G.~Zafrir, {\it {Generalized symmetries and
holography in ABJM-type theories}},  {\em JHEP} {\bf 07} (2020) 077,
[\href{http://arxiv.org/abs/2004.05350}{{\tt arXiv:2004.05350}}].

[10]
C.~Cordova, T.~T. Dumitrescu, and K.~Intriligator, {\it {2-Group Global
Symmetries and Anomalies in Six-Dimensional Quantum Field Theories}},  {\em
JHEP} {\bf 04} (2021) 252, [\href{http://arxiv.org/abs/2009.00138}{{\tt

[11]
M.~Del~Zotto, I.~n. Garc\'\i{}a~Etxebarria, and S.~S. Hosseini, {\it {Higher
form symmetries of Argyres-Douglas theories}},  {\em JHEP} {\bf 10} (2020)
056, [\href{http://arxiv.org/abs/2007.15603}{{\tt arXiv:2007.15603}}].

[12]
L.~Bhardwaj and S.~Sch\"afer-Nameki, {\it {Higher-form symmetries of 6d and 5d
theories}},  {\em JHEP} {\bf 02} (2021) 159,
[\href{http://arxiv.org/abs/2008.09600}{{\tt arXiv:2008.09600}}].

[13]
P.-S. Hsin and H.~T. Lam, {\it {Discrete theta angles, symmetries and
anomalies}},  {\em SciPost Phys.} {\bf 10} (2021), no.~2 032,
[\href{http://arxiv.org/abs/2007.05915}{{\tt arXiv:2007.05915}}].

[14]
D.~R. Morrison, S.~Schafer-Nameki, and B.~Willett, {\it {Higher-Form Symmetries
in 5d}},  {\em JHEP} {\bf 09} (2020) 024,
[\href{http://arxiv.org/abs/2005.12296}{{\tt arXiv:2005.12296}}].

[15]
S.~Gukov, P.-S. Hsin, and D.~Pei, {\it {Generalized global symmetries of $T[M]$
theories. Part I}},  {\em JHEP} {\bf 04} (2021) 232,
[\href{http://arxiv.org/abs/2010.15890}{{\tt arXiv:2010.15890}}].

[16]
M.~Yu, {\it {Symmetries and anomalies of (1+1)d theories: 2-groups and symmetry
fractionalization}},  {\em JHEP} {\bf 08} (2021) 061,
[\href{http://arxiv.org/abs/2010.01136}{{\tt arXiv:2010.01136}}].

[17]
L.~Bhardwaj, {\it {2-Group symmetries in class S}},  {\em SciPost Phys.} {\bf
12} (2022), no.~5 152, [\href{http://arxiv.org/abs/2107.06816}{{\tt

[18]
Y.~Lee, K.~Ohmori, and Y.~Tachikawa, {\it {Matching higher symmetries across
Intriligator-Seiberg duality}},  {\em JHEP} {\bf 10} (2021) 114,
[\href{http://arxiv.org/abs/2108.05369}{{\tt arXiv:2108.05369}}].

[19]
F.~Apruzzi, L.~Bhardwaj, J.~Oh, and S.~Schafer-Nameki, {\it {The Global Form of
Flavor Symmetries and 2-Group Symmetries in 5d SCFTs}},
\href{http://arxiv.org/abs/2105.08724}{{\tt arXiv:2105.08724}}.

[20]
L.~Bhardwaj, M.~Hubner, and S.~Schafer-Nameki, {\it {1-form Symmetries of 4d
N=2 Class S Theories}},  {\em SciPost Phys.} {\bf 11} (2021) 096,
[\href{http://arxiv.org/abs/2102.01693}{{\tt arXiv:2102.01693}}].

[21]
N.~Ganter and M.~Kapranov, {\it Representation and character theory in
2-categories},  {\em Advances in Mathematics} {\bf 217} (2008), no.~5

[22]
J.~Elgueta, {\it Representation theory of 2-groups on kapranov and voevodsky's
2-vector spaces},  {\em Advances in Mathematics} {\bf 213} (2007), no.~1

[23]
A.~M. Osorno, {\it Explicit formulas for 2-characters},  {\em Topology and its
Applications} {\bf 157} (2010), no.~2 369--377.

[24]
L.~Bhardwaj, M.~Bullimore, A.~E.~V. Ferrari, and S.~Schafer-Nameki, {\it
{Anomalies of Generalized Symmetries from Solitonic Defects}},
\href{http://arxiv.org/abs/2205.15330}{{\tt arXiv:2205.15330}}.

[25]
D.~Delmastro, J.~Gomis, P.-S. Hsin, and Z.~Komargodski, {\it {Anomalies and
Symmetry Fractionalization}},  \href{http://arxiv.org/abs/2206.15118}{{\tt

[26]
T.~D. Brennan, C.~Cordova, and T.~T. Dumitrescu, {\it {Line Defect Quantum
Numbers \& Anomalies}},  \href{http://arxiv.org/abs/2206.15401}{{\tt

[27]
L.~Bhardwaj, M.~Bullimore, A.~E.~V. Ferrari, and S.~Schafer-Nameki, {\it
{Generalized Symmetries and Anomalies of 3d N=4 SCFTs}},
\href{http://arxiv.org/abs/2301.02249}{{\tt arXiv:2301.02249}}.

[28]
P.~Etingof, D.~Nikshych, and V.~Ostrik, {\it Fusion categories and homotopy
theory},  {\em Quantum topology} {\bf 1} (2010), no.~3 209--273.

[29]
T.~Bartsch, M.~Bullimore, A.~E.~V. Ferrari, and J.~Pearson, {\it
{Non-invertible Symmetries and Higher Representation Theory I}},
\href{http://arxiv.org/abs/2208.05993}{{\tt arXiv:2208.05993}}.

[30]
L.~Bhardwaj, S.~Schafer-Nameki, and A.~Tiwari, {\it {Unifying Constructions of
Non-Invertible Symmetries}},  \href{http://arxiv.org/abs/2212.06159}{{\tt

[31]
T.~Bartsch, M.~Bullimore, A.~E.~V. Ferrari, and J.~Pearson, {\it
{Non-invertible Symmetries and Higher Representation Theory II}},
\href{http://arxiv.org/abs/2212.07393}{{\tt arXiv:2212.07393}}.

[32]
L.~Bhardwaj, S.~Schafer-Nameki, and J.~Wu, {\it {Universal Non-Invertible
Symmetries}},  {\em Fortsch. Phys.} {\bf 70} (2022), no.~11 2200143,
[\href{http://arxiv.org/abs/2208.05973}{{\tt arXiv:2208.05973}}].

[33]
L.~Bhardwaj and S.~Schafer-Nameki, {\it {Generalized Charges, Part I:
Invertible Symmetries and Higher Representations}},
\href{http://arxiv.org/abs/2304.02660}{{\tt arXiv:2304.02660}}.

[34]
J.~Frohlich, J.~Fuchs, I.~Runkel, and C.~Schweigert, {\it {Duality and defects
in rational conformal field theory}},  {\em Nucl. Phys. B} {\bf 763} (2007)
354--430, [\href{http://arxiv.org/abs/hep-th/0607247}{{\tt hep-th/0607247}}].

[35]
A.~Davydov, L.~Kong, and I.~Runkel, {\it {Invertible Defects and Isomorphisms
of Rational CFTs}},  {\em Adv. Theor. Math. Phys.} {\bf 15} (2011), no.~1
43--69, [\href{http://arxiv.org/abs/1004.4725}{{\tt arXiv:1004.4725}}].

[36]
I.~Brunner, N.~Carqueville, and D.~Plencner, {\it {Discrete torsion defects}},
{\em Commun. Math. Phys.} {\bf 337} (2015), no.~1 429--453,
[\href{http://arxiv.org/abs/1404.7497}{{\tt arXiv:1404.7497}}].

[37]
M.~M. Kapranov and V.~A. Voevodsky, {\em 2-categories and Zamolodchikov
tetrahedra equations}, vol.~56 of {\em Proc. Sympos. Pure Math.},
\newblock Amer. Math. Soc., Providence, RI, 1994.

[38]
B.~Bartlett, C.~L. Douglas, C.~J. Schommer-Pries, and J.~Vicary, {\it Modular
categories as representations of the 3-dimensional bordism 2-category},

[39]
D.~Gaiotto and T.~Johnson-Freyd, {\it {Condensations in higher categories}},
\href{http://arxiv.org/abs/1905.09566}{{\tt arXiv:1905.09566}}.

[40]
J.~Gunarwardena, B.~Kahn, and C.~Thomas, {\it Stiefel-whitney classes of real
representations of finite groups},  {\em Journal of Algebra} {\bf 126}
(1989), no.~2 327--347.

[41]
A.~Davydov, M.~Mueger, D.~Nikshych, and V.~Ostrik, {\it The witt group of
non-degenerate braided fusion categories},  2011.

\end{thebibliography}\endgroup

\end{document}