# Social Context-aware GCN for Video Character Search via Scene-prior
Enhancement ††thanks:  * Tong Xu is the corresponding author. This work was
supported by the grants from National Natural Science Foundation of China
(No.U22B2059, 62222213, 62072423), the USTC Research Funds of the Double
First-Class Initiative (No.YD2150002009), and the Alibaba Group through
Alibaba Innovative Research (AIR) Program.

Wenjun Peng1,2, Weidong He1,2, Derong Xu1,2, Tong Xu1,2∗, Chen Zhu1,3, Enhong
Chen1,2 1University of Science and Technology of China,
2State Key Laboratory of Cognitive Intelligence,
3Alibaba.
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

With the increasing demand for intelligent services of online video platforms,
video character search task has attracted wide attention to support downstream
applications like fine-grained retrieval and summarization. However,
traditional solutions only focus on visual or coarse-grained social
information and thus cannot perform well when facing complex scenes, such as
changing camera view or character posture. Along this line, we leverage social
information and scene context as prior knowledge to solve the problem of
character search in complex scenes. Specifically, we propose a scene-prior-
enhanced framework, named SoCoSearch. We first integrate multimodal clues for
scene context to estimate the prior probability of social relationships, and
then capture characters’ co-occurrence to generate an enhanced social context
graph. Afterwards, we design a social context-aware GCN framework to achieve
feature passing between characters to obtain robust representation for the
character search task. Extensive experiments have validated the effectiveness
of SoCoSearch in various metrics.

###### Index Terms:

Person Search, Person Re-identification, Multimodal Learning

##  I INTRODUCTION

As an indispensable technology for supporting various video applications,
video character search plays an important role in downstream tasks such as
clip retrieval, video summarization, etc. For example, many video platforms,
such as Tencent and YouKu Video, provide an “only look at him” function by
searching the clips of specific characters in videos to meet fans’ needs,
which enhances user experience and brings significant business benefits.
However, the manual collection of clips for specific characters requires a lot
of human resources.

Figure 1: A film scene and its social relation graph.

To this end, automatic character search attaches much concentration from
academia and industry. Nevertheless, most mainstream video character search
methods mainly focus on visual information or coarse-grained social
information to identify characters. For example, classic video character
search methods, such as [1, 2], only focus on extracting visual clues and
ignoring other meta information. [3] tries to leverage social information, but
they only use heuristic rules based on co-occurrence and fail to fully use the
scene’s context to extract the prior probability of relationships, thus
limiting their effectiveness. Considering that, with the plot processing, the
characters may develop many social relationships. Estimating the impact of
these relationships on co-occurrence behavior without discrimination will
seriously weaken the differentiation of relationships. Let us take Figure 1 as
an example. There are two characters in this film scene, where the female is
easy to identify because of her clear visual clues. But as for the male one,
on the one hand, his back cannot provide enough visual clues to recognize him.
On the other hand, the numerous relationships of the female also make it
difficult to reveal its co-occurrence character. However, with carefully
checking the fine-grained information, such as the subtitles and the kinds of
social relationships between characters, we can reasonably infer the identity
of this male. Obviously, the subtitles and visual clues of the background
imply two characters are a “couple”. And the most likely person, dating with
the female, is her husband.

Based on the above observations, in this paper, we propose a scene prior
knowledge based Social Context-aware character Search framework named
SoCoSearch, which can integrate fine-grained social information and scene
context to search video characters effectively. Specifically, SoCoSearch first
generates a prior probability for different relationship types to obtain a
weighted social graph through integrated multimodal semantic clues. At the
same time, we use the characters’ visual features to select an anchor node and
mine characters’ co-occurrence. The co-occurrence serves as an extra clue to
enhance the weighted social graph into a social context graph, which helps to
learn the candidate set of potential matching characters. Furthermore, we
employ a social context-aware GCN to achieve feature passing between graph
nodes and obtain characters’ social features. Finally, we use anchor node
similarity to adjust the weight of visual and social features adaptively. The
technical contributions of this article can be summarized as follows:

  * •

We further discussed the assistance of social relationships on the video
character search task, and took the lead in using the scene context as prior
knowledge to enhance the model’s performance on the task.

  * •

We proposed a novel neural-based framework, which effectively integrates
multiple clues and automatically adjusts the weight of various features.

  * •

The experiments on real video datasets prove that our method outperforms
several state-of-the-art baselines in terms of mAP, mINP and R1.

##  II RELATED WORKS

###  II-A Person Search.

The main challenge of traditional methods is extracting robust and
discriminative visual features. [4, 5, 1] extract multiple level features from
earlier layers, and fuse them as the final representation to enhance the
performance of models. To guarantee that the model has better generalization
ability across domains, [6] introduces normalization modules by learning
domain styles to improve the model’s performance across different cameras. [7]
designs two novel modules named jigsaw patch module and side information
embeddings, which can improve the robustness and discrimination ability of the
model by rearranging patches and adding domain information.

Nevertheless, traditional datasets are often gathered from a few nearby
cameras in a short time, leading to the appearance of the same identity often
similar, with slight variance in posture, clothing or occlusion. To solve the
problem, [8] utilizes social context to make models adaptable to the video
character search task in unconstrained environments. [3] takes a step further
and uses high-level social relations to identify characters. Unfortunately,
this method relies on heuristic modeling and fails to reveal the prior
probability of the relationship in different scenes, thus limiting the
guidance capability of characters’ occurrence.

###  II-B Multimodal Learning.

Recently, some studies have applied multimodal learning techniques to video
applications. [9] proposes a novel framework for character-oriented video
summarization by combining visual and textual information for the first time.
[10] introduces a multi-stream architecture to recognize relations in videos.
[11] proposes a model that considers visual and dialogue clues to recognize
interactions and social relations jointly. Inspired by the above studies, we
design a dynamic relation weight module to predict the prior probability of
relationships in different scenes via multimodal clues.

##  III TECHNICAL FRAMEWORK

Figure 2: The overall structure of our framework.

###  III-A Framework Overview

The video character search task aims to retrieve all gallery RoIs belonging to
a specific query character in a social context-aware condition. In order to
better address this task, as illustrated in Figure 2, we propose a framework
that includes a dynamic relation weight module (DRWM), a co-occurrence miner
(COM), and a Social Context-aware Graph Convolutional Network (GCN). Firstly,
DRWM generates the prior probability for different social relationships to get
a weighted social relation graph. At the same time, COM uses visual features
to strengthen the social relation graph. Finally, the social features of
gallery characters are learned by social context-aware GCN, and integrated
with visual features to get the final representation.

###  III-B Dynamic Relation Weight Module

Given the visual and textual context as scene prior knowledge, the Dynamic
Relation Weight Module (DRWM) aims to predict the probabilities of different
relationships between the characters in the scene. These probabilities are
assigned to the corresponding edges of the predefined social relation graph to
construct a weighted social relation graph.

Specifically, we first leverage TSM[12] and Bert[13] to extract visual and
textual features, respectively. $h_{v}$ is denoted as the visual features, and
$h_{d}$ is denoted as the textual features. We concatenate $h_{v}$ with
$h_{d}$, and feed them into a linear layer to obtain the relation weight
$\hat{y_{r}}$. We assign the relation weight to the corresponding edge in the
predefined social graph to get the weighted social graph. For the example
shown in Figure 2 (DRWM Part), the relationship between $q_{1}$ and $q_{2}$ is
$``couple"$ and the corresponding relation weight output from DRWM is 0.6, so
the edge weight of $e_{ij}(``couple")$ is assigned to 0.6. We formulate the
relation recognition as a multi-label classification task, and the binary
cross-entropy loss $\mathcal{L_{R}}$ is expressed as:

$\mathcal{L_{R}}=-y_{r}\cdot log(\hat{y_{r}})-(1-y_{r})\cdot
log(1-\hat{y_{r}}),$ (1)

where $y_{r}$ is the true relationships between each pair of gallery
characters in the scene, and $\hat{y_{r}}$ is the prediction of the DRWM.

###  III-C Co-occurrence Miner

Given the RoIs of the query and gallery, Co-occurrence Miner (COM) aims to
reveal characters’ co-occurrence. In this module, we first calculate the
similarity between the RoIs of the query and gallery by euclidean distance.
Then, we convert the similarity to the possibility of characters’ co-
occurrence.

Specifically, we employ TransReID[7] as our visual backbone to learn the
visual features of characters. Afterwards, we calculate the euclidean distance
of each query-gallery features pair and then convert it to a co-occurrence
probability. The process can be expressed as follows:

$d(g_{i},q_{j})=||\boldsymbol{h_{g_{i}}}-\boldsymbol{h_{q_{j}}}||^{2}_{2},$
(2)

$s(g_{i},q_{j})=\frac{\bar{d}-d(g_{i},q_{j})}{\delta(d)},$ (3)

$p(g_{i},q_{j})=\frac{exp(s(g_{i},q_{j}))}{\sum_{j}{exp(s(g_{i},q_{j}))}},$
(4)

where $\boldsymbol{h_{g_{i}}}$ and $\boldsymbol{h_{q_{j}}}$ are visual
features of gallery $g_{i}$ and query $q_{j}$, respectively. $\bar{d}$ is the
average distance and $\delta(d)$ is the standard deviation of all distances.
$p(g_{i},q_{j})$ is the identity probability, representing the probability of
$g_{i}$ and $q_{j}$ belonging to the same character. Then, COM calculates the
co-occurrence probability of each pair. For example, in Figure 2 (COM Part),
given five query characters $\\{q_{j}|j=1,2,3,4,5\\}$, two gallery characters
$\\{g_{i}|i=1,2\\}$ and their respective probability distributions of
similarity, $g_{1}$ is regarded as the anchor character since the highest
confidence value 0.6. The distribution of $p(g_{1},q_{j})$ serves as the
possibility of co-occurrence of $p(g_{2},q_{j})$, which consists of the edge
weight of the social context graph. Specifically, $g_{1}$ and $g_{2}$ appear
simultaneously, and $g_{1}$ is probably $q_{j}$. Therefore, it can be
considered that $q_{j}$ is likely to co-occur with $g_{2}$, and the similarity
between $g_{1}$ and $q_{j}$ is the co-occurrence probability of $q_{j}$ and
$g_{2}$. Thus, we connect $g_{2}$ with all $q_{j}$ and assign co-occurrence
probability to the corresponding edge’s weight to represent characters’ co-
occurrence. For scenes with more than two characters, we choose the character
with the highest similarity as the anchor.

###  III-D Social Context-aware GCN

In order to make full use of characters’ relationships and co-occurrence
simultaneously, this module combines the weighted social relation graph
obtained from DRWM and the characters’ co-occurrence obtained from COM to
generate the social context graph. Then it performs message passing on the
graph to extract social features of the gallery, which are weighted and
combined with visual features to learn the final representation.

It is noting that, in this graph, the second-order neighbors of each gallery
node constitute a candidate character set, which makes the model pay more
attention to the characters who are likely to exist in the scene. As mentioned
before, DRWM assigns weights to different relationship edges to determine the
most likely social pairs in the scene. COM mines the characters’ co-occurrence
based on visual similarity to determine the most likely co-occurrence pairs in
this scene. Finally, as shown in Figure 2 (Social Context-aware GCN part), we
can get meta paths according to social and co-occurrence pairs and obtain the
gallery character candidates.

Therefore, in the social co-occurrence graph, we need to aggregate the feature
of the second-order neighbors into the corresponding gallery node to reduce
the difference between them. In addition, we remove the feature transformation
weight $W$ and activation function $f$ to obtain pure features of second-order
neighbors. The specific operations are as follows:

$\boldsymbol{h^{(l+1)}}=\boldsymbol{\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}}\boldsymbol{h^{(l)}},$
(5)

where $\tilde{A}\in\mathbb{R}^{N\times N}$ is the adjacency matrix of the
graph, and $\tilde{D}\in\mathbb{R}^{N\times 1}$ is the degree matrix of nodes,
$h^{(l)}\in\mathbb{R}^{N\times d}$ is the output from $l$ layer. Suppose
$h^{(2)}$ is the social feature output from 2-layer social content-aware GCN,
$h^{(0)}$ is the visual feature output from the visual backbone, the final
representation of the gallery node is:

$w_{s}=\frac{p_{a}}{\sum{\hat{y_{r}}}},$ (6)
$\boldsymbol{h_{f}}=w_{s}\boldsymbol{h^{(2)}}+(1-w_{s})\boldsymbol{h^{(0)}},$
(7)

where $p_{a}$ donates the highest anchor’s identity probability, $\hat{y_{r}}$
donates the relationships’ probability predicted by DRWM, which is used to
control the weight of social features. Because when the identity probability
of the anchor is very low or there are many relationships in a scene, the
possibility that the second-order neighbor of the gallery node is the
corresponding query character will also decrease. Therefore, we need to reduce
the weight of social feature $h^{(2)}$.

We formulate the cross entropy loss $\mathcal{L_{ID}}$ and triplet loss
$\mathcal{L_{T}}$ to optimize COM and Social Context-aware GCN, which is
expressed as:

$\hat{y}_{id}=\boldsymbol{W_{c}}*\boldsymbol{h_{f}}+\boldsymbol{b_{c}},$ (8)
$\mathcal{L_{ID}}=\sum{-y_{id}\cdot log(\hat{y}_{id})},$ (9)
$\mathcal{L_{T}}=log[1+exp(||\boldsymbol{h_{f}}-\boldsymbol{h_{p}}||^{2}_{2}-||\boldsymbol{h_{f}}-\boldsymbol{h_{n}}||^{2}_{2})],$
(10)

where $W_{c}$ and $b_{c}$ is weight and bias of classification layer.
$\hat{y}_{id}$, $y_{id}$ are the prediction and groundtruth of the character’s
ID respectively. $\boldsymbol{h_{f}}$ is the representation of a gallery
character $g_{i}$, $\boldsymbol{h_{p}}$ is the representation of the positive
sample that has same character ID with $g_{i}$, $\boldsymbol{h_{n}}$ is the
representation of the negative sample that has different character ID with
$g_{i}$. Finally, we combine three loss to get the overall loss function:

$\mathcal{L=L_{R}+L_{ID}+L_{T}}.$ (11)

##  IV EXPERIMENTS

###  IV-A Datasets and Pre-processing

To the best of our knowledge, there is no off-the-shelf dataset available for
the video character search task that contains RoIs of characters and their
corresponding video frames, textual information, and relation graph.
Therefore, following [3], we constructed two datasets named Social-Bilibili
and Social-MovieNet based on the Bilibili[9] and MovieNet[14] datasets,
respectively.

Social-Bilibili Dataset. The dataset contains 70 films with an average length
of 2 hours, 375 main characters, related subtitles and screen-bullet comments.
All characters’ relationships are grouped into five categories, which are
working, kinship, hostile, friend and couple respectively. Based on the
dataset, we sampled 325 characters, a total of 11478 images, for train, and
the remaining 50 characters, 1989 images, for test. Following [3], we selected
a clear image for each character as the query image, which constitutes our
query set. In addition, we collect the video frames and textual information
with time window $T_{t}=[t-12,t+12]$ as the context.

Social-MovieNet Dataset. We collected social relation graphs for 16 films from
MovieNet, with an average length of 1.7 hours, 79 main characters and related
subtitles. To be consistent with the Social-BiliBili dataset, we also group
all relationships into five categories, which are working, kinship, hostile,
friend and couple respectively. We sampled 59 characters, a total of 14462
images, for train, and the remaining 20 characters, 5991 images, for test.
Similarly, we selected a clear image for each character as a query image,
which constitutes our query set. Since MovieNet has split movies into shots,
we set the time window to $T_{t}=[t-2,t+2]$, where $t$ is the id of the shot.
All frames and subtitles in the time window $T_{t}$ will be collected as the
context.

###  IV-B Experiment Settings

Following the setting of TransReID[7], all training images are resized to
256$\times$128 and augmented with random horizontal flipping, padding, random
cropping and random erasing[15]. The batch size is set to 64 with 16 images
per character ID for triplet loss. Our model is optimized by SGD with a 0.9
momentum setting. During the first 120 epochs, we only train the visual
backbone with the initial learning rate of 0.08 and weight decay of 1e-4.
During the last 20 epochs, DRWM and social context-aware GCN are added to
train with visual search backbone parameters frozen. The initial learning rate
is set to 1e-4 and weight decay is set to 1e-5. All experiments are conducted
on two GeForce RTX 3090 with PyTorch and Torchvison toolboxes.

###  IV-C Baselines

We compare our model with the following state-of-the-art methods to evaluate
its performance:

  * •

HACNN[16], which contains two branches, one branch learns local features, and
the other learns global features. At the same time, a attention selection
module is adopted to learn a set of complementary attention maps to optimize
the representation.

  * •

MLFN[5], which contains multiple factor modules and factor selection modules
to catch the semantic information at different level, and dynamic select the
information extracted from multiple factor modules to explain content of each
image.

  * •

ResNet[17], which is widely used in computer vision tasks[12, 18, 19] and can
solve the learning degradation problem in traditional CNN-based networks.

  * •

ResNet-mid[4], which extracts the mid-level features and fuses them with the
output of the last layer as the final representation, making itself capable to
learn discriminative features at different semantic levels.

  * •

OSNet[1], which contains a unified aggregation gate that can fuse multi-scale
features caught by omni-scale residual blocks to learn homogeneous and
heterogeneous information to improve its performance.

  * •

OSNet-AIN[20], which introduces instance normalization layers to adapt the
styles across domains on the basis of OSNet. In addition, an architecture
search algorithm is performed to find the best locations for instance
normalization layers.

  * •

NFormer[21], which introduces a neighbor transformer network to eliminate
outlier features and generate more reliable representations by modeling the
relations between all the input images.

  * •

ViT[22], which splits the image into N patches, and then uses a linear
projection function to map them into 1-D tokens. In this way, the image has
converted into a sequence of tokens, which can be processed by transformer.

  * •

TransReID[7], which introduces jigsaw patch module and side information
embeddings to strengthen the connection between different patches and learn
domain style, enhancing the robustness of the model.

  * •

TP[3], which is proposed as baseline in [3]. Following their ideas, we use
LSTM to extract text features and calculate the similarity, then combine
textual and visual similarity to obtain the final measurement. TP-ResNet and
TP-ViT represent as their visual backbones.

  * •

SCPS[3], which introduces a relation-aware framework to identify characters by
using visual information and social relations revealed by multi-modal context.
SCPS-L and SCPS-P represent as linear and prudent label update strategies,
respectively.

###  IV-D Overall Performance

TABLE I: Overall performance on two datasets. TRM means Transformer. The best results are highlighted in bold. Method | Backbone | Social-Bilibili | Social-MoviNet
---|---|---|---
mAP | mINP | R1 | mAP | mINP | R1
HACNN[16] | CNN | 47.8 | 29.4 | 66.0 | 30.4 | 20.7 | 55.0
MLFN[5] | CNN | 55.7 | 34.6 | 72.0 | 30.6 | 20.6 | 40.0
ResNet[17] | CNN | 61.2 | 35.8 | 82.0 | 42.1 | 22.2 | 65.0
ResNet-mid[4] | CNN | 63.8 | 35.3 | 82.0 | 45.0 | 22.0 | 75.0
OSNet[1] | CNN | 64.6 | 38.2 | 80.0 | 49.1 | 21.2 | 75.0
OSNet-AIN[20] | CNN | 65.1 | 39.7 | 86.0 | 45.3 | 20.9 | 80.0
TP-ResNet[3] | CNN | 59.5 | 36.2 | 78.0 | 44.8 | 21.7 | 75.0
NFormer[21] | TRM | 63.8 | 36.3 | 80.0 | 37.8 | 20.7 | 65.0
ViT[22] | TRM | 76.5 | 49.1 | 90.0 | 59.4 | 23.0 | 90.0
TransReID[7] | TRM | 79.9 | 51.2 | 90.0 | 62.3 | 24.6 | 90.0
TP-ViT[3] | TRM | 68.1 | 40.3 | 86.0 | 59.2 | 24.1 | 95.0
SCPS-L[3] | TRM | 80.9 | 52.6 | 90.0 | 57.9 | 23.9 | 80.0
SCPS-P[3] | TRM | 81.0 | 53.1 | 90.0 | 58.9 | 24.1 | 80.0
Ours | TRM | 82.8 | 56.3 | 94.0 | 63.6 | 24.5 | 95.0

In Table I, we show the overall performance of our model and the baselines.
Following the re-ID community, we select mAP and CMC-Rank@1(R1) as our
evaluation metrics. In addition, because the video character search is used as
the upstream task of idols summarization and the enthusiastic fans usually
don’t want to miss any shots of their idols, we need to evaluate the cost of
searching for the last correct image. Therefore, we introduce mINP[23] to
measure the ability to retrieve the hardest match of models.

Overall, our social context-aware method performs better than traditional
methods. Taking TransReID as an example, on the Social-Bilibili dataset, our
method reaches 82.8%, 56.3% and 94.0%, surpassing TransReID 2.9%, 5.1% and
4.0% on the terms of mAP, mINP and R1 respectively. This is because TransReID
only uses the visual features and treats each character as an independent
instance, while our method can integrate characters’ relationships and co-
occurrence to assist the character search. However, on the Social-MovieNet
dataset, the mINP decreased by 0.1% compared with TransReID. Because as the
mAP of the visual search backbone decreases (the visual search task on the
Social-MovieNet dataset is more challenging), the anchor node selection
becomes less accurate, which makes it difficult to propagate features of the
corresponding query node to the gallery node.

In addition, we compared the different methods of using social clues. Take
SCPS-P as an example, on Social-Bilibili dataset, our model outperformed 1.8%,
3.2% and 4.0% on mAP, mINP and R1 respectively. Compared with our method, SCPS
doesn’t consider the role of the prior probability of the relationship between
the characters in different scenes, thus limiting the model performance. We
can observe similar results on the Social-MovieNet dataset.

###  IV-E Ablation Study

TABLE II: The performance of our method with different visual backbones on Social-Bilibili dataset Method | Backbone | mAP | mINP | R1
---|---|---|---|---
ResNet | ResNet50 | 61.2 | 35.8 | 82.0
ResNet-mid | ResNet50 | 63.8 | 35.3 | 82.0
TransReID | DeiT-S/16 | 65.4 | 42.3 | 88.0
TransReID | DeiT-B/16 | 75.6 | 48.7 | 90.0
TransReID | ViT-S/16 | 64.8 | 39.2 | 88.0
TransReID | ViT-B/16 | 79.9 | 51.2 | 90.0
SoCoSearch | ResNet50 | 63.2 | 38.1 | 84.0
SoCoSearch | ResNet-mid | 63.9 | 37.7 | 82.0
SoCoSearch | DeiT-S/16 | 67.9 | 45.1 | 88.0
SoCoSearch | DeiT-B/16 | 78.5 | 52.9 | 94.0
SoCoSearch | ViT-S/16 | 67.3 | 41.0 | 86.0
SoCoSearch | ViT-B/16 | 82.8 | 56.3 | 94.0

Different Backbones: In this section, we discuss the performance of our model
with different visual backbones. The experimental results are shown in Table
II. We can observe that both CNN-based methods and transformer-based methods
have improved after equipped social context-aware GCN on Social-Bilibili
Dataset. In addition, the improvement of transformer-based methods is more
significant than CNN-based methods’. Because SoCoSearch need to select anchors
before building the social context graph, and the mAP of transformer-based
methods are significantly higher than CNN-based methods’, which makes this
kind of method can select anchors more accurately. The correct anchors make
social-context aware GCN more likely to aggregate the features of
corresponding query characters to the gallery characters, thereby improving
the model performance.

Different Modalities: We also compare the performance of our model with
different modal information. The experimental results are shown in Table III.
For models that neither uses visual nor textual context, it can be regarded as
the TransReID, which only depends on the image features to match characters.
Hence, it can not be well applied to the video characters search task.
However, our model uses visual and textual information as clues to construct a
social co-occurrence graph, which improves the similarity between the
corresponding gallery and query nodes. It can be seen that when just using
visual or textual information, the model is still effective, but the
performance will be slightly decreased compared with the model with both
modalities. Because a single modality may introduce bias, which leads to the
DRWM being unable to allocate weights correctly, thus decreasing the model
performance. Thus, it is necessary to integrate multimodal clues for better
weights allocation.

Different Balance Strategies: Finally, to verify our adaptive strategy’s
effectiveness, we compared different methods of balancing social features.
Table IV shows three strategies to control social weight: mean, random, and
adaptive weight. According to the mean and random strategy results, we can
observe that inappropriate social weight will damage the original performance.
However, our adaptive strategy is well-adapted to different scenes. It can
adjust the social weight based on multimodal clues, making the model perform
better.

TABLE III: The performance of our method with different modal information on Social-Bilibili dataset Method | Visual | Textual | mAP | mINP | R1
---|---|---|---|---|---
w/o VT | - | - | 79.9 | 51.2 | 90.0
w/o V | - | Bert | 82.6 | 55.4 | 94.0
w/o T | TSM | - | 82.7 | 56.2 | 94.0
all | TSM | Bert | 82.8 | 56.3 | 94.0
TABLE IV: The performance of our method with different balance strategies on Social-Bilibili dataset Strategy | mAP | mINP | R1
---|---|---|---
Mean | 82.4 | 55.4 | 94.0
Random | 76.0 | 47.9 | 90.0
Adaptive(Ours) | 82.8(+0.4) | 56.3(+0.6) | 94.0(+0)

##  V CONCLUSION

In this paper, we proposed a social context-aware framework named SoCoSearch
to deal with the video character search task. SoCoSearch can integrate prior
knowledge and characters’ co-occurrence to construct a social context graph.
With the help of the graph, we can get a set of potential matching characters.
And it employed a social context-aware GCN to pass the candidate node’s
features to the corresponding gallery characters to improve the similarity
between the query-gallery pairs and complete the matching. Extensive
experiments on the two real-world datasets demonstrated an improvement over
several baselines, proving our method’s superiority in the video character
search task.

## References

  * [1] Kaiyang Zhou, Yongxin Yang, Andrea Cavallaro, and Tao Xiang,  “Omni-scale feature learning for person re-identification,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 3702–3712.
  * [2] Lin Wu, Yang Wang, Junbin Gao, and Xue Li,  “Where-and-when to look: Deep siamese attention networks for video-based person re-identification,”  IEEE Transactions on Multimedia, vol. 21, no. 6, pp. 1412–1424, 2019.
  * [3] Dan Li, Tong Xu, Peilun Zhou, Weidong He, Yanbin Hao, Yi Zheng, and Enhong Chen,  “Social context-aware person search in videos via multi-modal cues,”  ACM Transactions on Information Systems (TOIS), vol. 40, no. 3, pp. 1–25, 2021.
  * [4] Qian Yu, Xiaobin Chang, Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales,  “The devil is in the middle: Exploiting mid-level representations for cross-domain instance matching,”  arXiv preprint arXiv:1711.08106, 2017.
  * [5] Xiaobin Chang, Timothy M Hospedales, and Tao Xiang,  “Multi-level factorisation net for person re-identification,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 2109–2118.
  * [6] Zijie Zhuang, Longhui Wei, Lingxi Xie, Haizhou Ai, and Qi Tian,  “Camera-based batch normalization: an effective distribution alignment method for person re-identification,”  IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 1, pp. 374–387, 2021.
  * [7] Shuting He, Hao Luo, Pichao Wang, Fan Wang, Hao Li, and Wei Jiang,  “Transreid: Transformer-based object re-identification,”  in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 15013–15022.
  * [8] Yichao Yan, Qiang Zhang, Bingbing Ni, Wendong Zhang, Minghao Xu, and Xiaokang Yang,  “Learning context graph for person search,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 2158–2167.
  * [9] Peilun Zhou, Tong Xu, Zhizhuo Yin, Dong Liu, Enhong Chen, Guangyi Lv, and Changliang Li,  “Character-oriented video summarization with visual and textual cues,”  IEEE Transactions on Multimedia, vol. 22, no. 10, pp. 2684–2697, 2019.
  * [10] Tong Xu, Peilun Zhou, Linkang Hu, Xiangnan He, Yao Hu, and Enhong Chen,  “Socializing the videos: A multimodal approach for social relation recognition,”  ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), vol. 17, no. 1, pp. 1–23, 2021.
  * [11] Anna Kukleva, Makarand Tapaswi, and Ivan Laptev,  “Learning interactions and relationships between movie characters,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9849–9858.
  * [12] Ji Lin, Chuang Gan, and Song Han,  “Tsm: Temporal shift module for efficient video understanding,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 7083–7093.
  * [13] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova,  “Bert: Pre-training of deep bidirectional transformers for language understanding,”  arXiv preprint arXiv:1810.04805, 2018.
  * [14] Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin,  “Movienet: A holistic dataset for movie understanding,”  in European Conference on Computer Vision. Springer, 2020, pp. 709–727.
  * [15] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang,  “Random erasing data augmentation,”  in Proceedings of the AAAI conference on artificial intelligence, 2020, vol. 34, pp. 13001–13008.
  * [16] Wei Li, Xiatian Zhu, and Shaogang Gong,  “Harmonious attention network for person re-identification,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 2285–2294.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [18] Sirui Zhao, Huaying Tang, Shifeng Liu, Yangsong Zhang, Hao Wang, Tong Xu, Enhong Chen, and Cuntai Guan,  “Me-plan: A deep prototypical learning with local attention network for dynamic micro-expression recognition,”  Neural Networks, vol. 153, pp. 427–443, 2022.
  * [19] Sirui Zhao, Hongyu Jiang, Hanqing Tao, Rui Zha, Kun Zhang, Tong Xu, and Enhong Chen,  “Pedm: A multi-task learning model for persona-aware emoji-embedded dialogue generation,”  ACM Transactions on Multimedia Computing, Communications and Applications, 2022.
  * [20] Kaiyang Zhou, Yongxin Yang, Andrea Cavallaro, and Tao Xiang,  “Learning generalisable omni-scale representations for person re-identification,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [21] Haochen Wang, Jiayi Shen, Yongtuo Liu, Yan Gao, and Efstratios Gavves,  “Nformer: Robust person re-identification with neighbor transformer,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 7297–7307.
  * [22] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  arXiv preprint arXiv:2010.11929, 2020.
  * [23] Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven CH Hoi,  “Deep learning for person re-identification: A survey and outlook,”  IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 6, pp. 2872–2893, 2021.