∎

11institutetext: R. Koner 22institutetext: Ludwig Maximilian University of
Munich, Munich, Germany
22email<EMAIL_ADDRESS>33institutetext: S. Shit 44institutetext:
Technical University of Munich, Munich,Germany
44email<EMAIL_ADDRESS>55institutetext: V. Tresp 66institutetext:
Siemens AG, Munich, Germany
66email<EMAIL_ADDRESS>
# Relation Transformer Network

Rajat Koner  Suprosanna Shit  Volker Tresp

(Preprint : under review)

###### Abstract

The extraction of a scene graph with objects as nodes and mutual relationships
as edges is the basis for a deep understanding of image content. Despite
recent advances, such as message passing and joint classification, the
detection of visual relationships remains a challenging task due to sub-
optimal exploration of the mutual interaction among the visual objects. In
this work, we propose a novel transformer formulation for scene graph
generation and relation prediction. We leverage the encoder-decoder
architecture of the transformer for rich feature embedding of nodes and edges.
Specifically, we model the node-to-node interaction with the self-attention of
the transformer encoder and the edge-to-node interaction with the cross-
attention of the transformer decoder. Further, we introduce a novel positional
embedding suitable to handle edges in the decoder. Finally, our relation
prediction module classifies the directed relation from the learned node and
edge embedding. We name this architecture as Relation Transformer Network
(RTN). On the Visual Genome and GQA dataset, we have achieved an overall mean
of 4.85% and 3.1% point improvement in comparison with state-of-the-art
methods. Our experiments show that Relation Transformer can efficiently model
context across various datasets with small, medium, and large-scale relation
classification.

###### Keywords:

Scene Graph, Scene Understanding, Visual Relation Detection, Transformer

##  1 Introduction

A scene graph is a graphical representation of an image consisting of multiple
entities and their relationships, expressed in a triplet format like
$\langle$subject, predicate, object$\rangle$. Objects in the scene become
nodes, undirected interactions between nodes are represented by edges and a
directed edge is called a relationship or predicate. E.g. in Fig. 1,
‘Eye’,‘Hair’,‘Head’, and ‘Man’ are object or node labels and their mutual
relationships are described by the predicates ‘has’ and ‘on’. An extracted
scene graph can be used in many downstream applications like visual question
answering ghosh2019generating ; hildebrandt2020scene ; koner2021graphhopper ,
image retrieval Schroeder2020 , and image captioning xu2019scene .

A scene graph generation (SGG) task is executed in two steps: first, objects
present in the image are detected, and second, the most suitable predicates
are determined for selected object pairs. Current object detection approaches
have achieved outstanding performance in spatially locating objects in an
image. In contrast, performance on relation prediction is still quite limited.
Several recent works have tried to explore SGG from different perspectives.
Context information is exchanged either globally zellers2018neural  or across
neighborhoods Xu_2017_CVPR ; yang2018graph . newell2017pixels ;
zhang2019graphical  introduced a variant of contrastive loss for a better
representation of similar types of relations and some contemporary work
lin2020gps  improved message passing with direction and priority sensitive
loss. knyazev2020graph  addresses specifically the biases in different
datasets. Interactions among objects and their corresponding edges through
contextual attention for SGG, the topic of this paper, is still under-
explored. This paper proposes a novel transformer-based formulation for the
SGG task, namely Relation Transformer network (RTN). In the following, we
rationalize the self-attention and cross-attention of the proposed RTN in
context of SGG task.

First, it is crucial to understand the role of each object in an image and how
object labels are related and influenced by others in the context of the whole
image. For example, in Fig. 1, the presence of node labels like ‘Eye’, ‘Hair’,
‘Nose’, ‘Ear’ indicate that these together describe a face and indicate the
presence of a node with label ‘Face’ or ‘Head’ in their surroundings.
Additionally, the node label ‘Shirt’ implies that this is a face or head of a
‘Human’ and not an animal. The spatio-semantic co-occurrence of labels along
with the contextualization of nodes are both important for predicting pairwise
relations and node labels. To obtain “context aware nodes”, we have modeled
interactions among nodes using self-attention of the transformer
vaswani2017attention , which in this case is a node-to-node (N2N) attention
propagation.

Second, a subsequent challenge is to predict the exact relationship between
two objects. In this paper, we assume that node labels and edge labels are
mutually dependent and predicted by scene context. For example, in Figure 1,
the probabilities for edges between the node ‘Man’ and its body parts like
‘Head’ ,‘Eye’ and ‘Nose’ should be similar because of the spatio-semantic
similarity. Thus context propagation from nodes to edges directly influences
the classification of their directed relations ($\langle
Man,~{}has,~{}Head\rangle,\langle Head,~{}of,~{}Man\rangle$). Although a
directed edge label is mainly dependent on its two associated nodes, i.e.,
subject and object, we argue that attention from all other nodes to an edge
helps to leverage all mutually correlated relations and guides towards
identifying most consistent relationship in the global scene context. This
edge to nodes interaction is modeled using the cross-attention of the
transformer decoder, which we refer to as E2N attention.

(a) Scene consisting of a man’s face

(b) Corresponding scene graph

Figure 1: (a) is an example image of a face of a man. (b) describes the
corresponding scene graph, annotated with various object labels like head,
ear, shirt (color coded as the respective bounding box) and their mutual
relationships.

In addition, we introduce a novel positional encoding that preserves the local
context of the edge. Finally, a directed relation prediction module (RPM) 3.4
is used to classify the relation that takes advantage of the context-rich
nodes and edges. To summarize our contributions:

  1. 1.

We formulate the SGG task as a feature learning on a graph leveraging
transformer architecture to model object-to-object and object-to-relation
interaction. First, we extract object-to-object interaction via the self-
attention (N2N attention) of the transformer encoder.

  2. 2.

Next, we use this rich node representation to capture object-to-relation
interaction via the transformer decoder’s cross-attention (E2N attention). We
introduce a novel positional encoding for the edges in the transformer decoder
to accumulate a global scene environment while preserving local context.

  3. 3.

We employ an efficient directional relation prediction module to accumulate
the learned node and edge representation from the transformer and classify the
desired directed relation.

  4. 4.

We perform extensive experiments in multiple datasets to show our proposed
RTN’s generalizability and efficacy. We achieve an overall mean of 4.85% and
3.2% improvement on the challenging Visual Genome and GQA dataset over the
state-of-the-art models.

##  2 Related Work

Scene Understanding with Language Prior: Scene understanding evolved through
many phases throughout recent years. Initially, researchers tried to localize
objects or regions in an image, based on a given caption or text reference
mao2016generation ; nagaraja2016modeling ; hu2017modeling ;
plummer2015flickr30k . These approaches mostly matched the referenced text to
the matching part of the image. johnson2015image  introduced scene graphs for
image retrieval, and lu2016visual , proposed visual relationship detection
with language priors, and introduced an associated dataset named VRD.
baier2017improving  derived a knowledge graph model from the training data
labels and achieved generalization to new triples by knowledge graph
factorization approaches. Several works focused on combining the visual and
other semantic features of the subject, object, and predicates
zhang2019graphical ; newell2017pixels ; wan2018representation ; yin2018zoom ;
zhang2017visual ; lu2016visual  and enriched the features with a variant of
triplet loss, pooling, and multi-modal representations sah2019improving .

Context in Scene Graph: Contextual information has been shown to be helpful
for object detection liu2018structure , visual question answering
Anderson_2018_CVPR , and scene understanding nagaraja2016modeling . Recent
advancements in the attention approach provided an efficient way to model
complex interactions of entities in NLP networks vaswani2017attention , and
convolution networks wang2018non . Various recent relationship detection
networks have tried to incorporate context with attention or transformer
zareian2020learning ; woo2018linknet ; yang2018graph ; lin2020gps  or without
attention qi2019attentive ; zellers2018neural ; herzig2018mapping ;
Xu_2017_CVPR . Although our work is also based on attentional context, it
differs as it introduces the novel N2N and E2N attention. Furthermore, like
zellers2018neural , we do not only consider interactions between mutually co-
occurrence objects, but we also analyze how the presence of objects or
predicates jointly influences each other.

Transformers in Vision: After the release of the transformer
vaswani2017attention , it became one of the most popular approaches for
various vision koner2021oodformer ; carion2020end  or vision language tasks
from natural language processing li2019unicoder . In Vision-Language
pretraining tasks, BERT-style architectures devlin2018bert  became a default
choice, due to their ability to process sequential and also non-sequential
data and, in almost all cases, it improved upon the state of the art results.
In lu2019vilbert , a two-stream network for joint vision-language modalities
has been used to obtain an enhanced representation for tasks like visual
question answering and image captioning. li2019unicoder  uses a combination of
sentences and image patches jointly for pretraining and achieved a state of
the result on GQA hudson2019gqa  or tasks like Masked Object Classification
(MOC), Visual Linguistic Matching (VLM). Recently, carion2020end  proposed a
simple end-to-end object-detection framework; similar to our work, it uses a
transformer encoder-decoder architecture. This recent surge of interest shows
the importance and efficacy of the transformer and BERT-style architectures.
In comparison to similar approaches, the Relation Transformer in this paper
gives superior performance and interpretable results; at the same time, it has
a transparent modular architecture. An earlier study of our method can be
found at koner2021scenes .

##  3 Method

We frame the SGG task as a multi-hop attention-based context-propagation
problem among nodes, edges, and their joint classification in a directed
graph. This task is decomposed into four sub-tasks, (1) Object detection to
get nodes (object bounding boxes) and their edges (union of object bounding
boxes) [in Sec. 3.2]. (2) Modeling of interactions between the nodes [in Sec.
3.3.1] (3) Accumulation of necessary context from all nodes for an edge [in
Sec. 3.3.3] and, (4) Classification of directed relations between the objects
from the extracted context information. In the next sub-sections, we will
describe these sub-tasks along with a brief introduction of the attention
mechanism of the transformer. An overview of the proposed Relation Transformer
architecture is shown in Fig. 2 with nodes and edges derived from a
representative image.

Figure 2: An overview of the proposed Relation Transformer architecture. The
network consists of four stages: a) Extraction of initial node (colored
circle) and edge (dotted line) features using a region proposal network (RPN)
of an object detector, b) Creation of context-rich node embeddings using N2N
attention from initial node embeddings, c) Creation of edge embedding by
accumulating context from nodes through E2N attention and proposed edge
position embedding, d) direction aware classification of the relation using
$\langle subject,edge,objects\rangle$ triplet embedding. Best viewed in color.

###  3.1 Problem Decomposition

A scene graph $G$ consists of a set of nodes $N=\\{n_{i}\\}$, that represent
the objects in an image and a set of labeled directed edges $R\subset
N\times\mathcal{R}\times N$, where $\mathcal{R}$ is the set the relation
types. For each node $n_{i}$, $b_{i}\in\mathbb{R}^{4}$ denotes the bounding
box coordinates and $o_{i}$ denotes the class label. We denote $I$ as the
image and $B$ as the set of bounding boxes, and $O$ as the set of object class
labels. With the help of this notation, a generative model for the graph is
$Pr(B\rvert I)\>Pr(O\rvert B,I)\>Pr(R\rvert O,B,I).$ $Pr(B\rvert I)$ is
inferred by our object detection module. From the objects and bounding boxes
we construct an undirected object interaction graph [Fig. 2] with all the
detected objects as a nodes $\\{n_{i}\\}$ and possible node-pairs with edges
$\\{e_{ij}\\}$. Such an undirected edge is a candidate directed relation
without immediate knowledge of the direction (subject to object) and
interaction label (predicate). For the sake of this paper, we assume that, if
$e_{ij}$ exists, there is a unique relation type. We propose to estimate
$Pr(O\rvert B,I)$ by the N2N module, where other labels can influence the
presence of one object label. Subsequently, to model the relationships
$Pr(R\rvert O,B,I)$, we first process the candidate undirected edges
$\\{e_{ij}\\}$ between $n_{i}$ and $n_{j}$ using the E2N module and learn an
object interaction graph. Then the RPM predicts edge direction and relation
type $r_{{i}\rightarrow{j}}$.

###  3.2 Object Detection

We have used Faster-RCNN ren2015faster  with a VGG-16 simonyan2014very
backbone for object detection. For a node $n_{i}$, we obtain the spatial
embedding bounding box coordinates $b_{i}\in\mathbb{R}^{4}$, a visual feature
vector of a region of interest $v_{i}\in\mathbb{R}^{4096}$ from the feature
map $I_{feature}$ obtained from top layer of VGG-16. Also, we get initial
class probabilities $o^{\textit{init}}_{i}\in\mathbb{R}^{C}$ where $C$ is the
number of classes. To exploit the semantic information of the predicted class
label $(o^{\textit{init}}_{i})$, we multiply it with the GloVe embedding
pennington2014glove  of all classes to obtain the semantic features $s_{i}$.
This enforces hard attention of detected class probabilities across the word
embedding feature space.

###  3.3 Context Propagation via Transformer

The core concept of our approach is the efficient attention-based context
propagation across all nodes and edges using an encoder-decoder architecture
implemented as transformers vaswani2017attention . The transformer
architecture uses self-attention mechanisms for mapping of the global
dependencies. One defines attention as the matrix

$\displaystyle\text{Attention}(Q,K,V)=\text{softmax}(\dfrac{QK^{T}}{\sqrt{d_{k}}})V.$
(1)

where query (Q), keys (K), and value (V) are obtained through three learnable
layers, and $d_{k}$ is a scaling factor. The output is computed as a weighted
sum of the values, where the weight is computed by multiplying a query matrix
with its corresponding key.

In our transformer architecture, we reason two different attention schemes
based on our observation as discussed in the Sec. 1. We incorporate self-
attention module in the encoder of our transformer that serves as a N2N
attention. However, to model the optimal contextualization from all nodes to
edges, we employ E2N attention as the cross-attention in the decoder of our
transformer. To exploit both global and local context propagation in the E2N
attention, we introduce appropriate changes in the positional encoding of the
decoder.

####  3.3.1 Encoder N2N Attention

Contextualization of objects by exploring its surroundings not only enhances
object detection liu2018structure , but also encodes more discriminate
features for relation classification. For this purpose, we make a permutation
invariant sequential ordering of the nodes and pass this node sequence to the
transformer encoder. The initial node feature vector ($f_{i}^{\textit{in}}$)
for the $i^{th}$ node is obtained by applying a linear projection layer
($W_{\textit{node}}$) on its concatenated features as

$f^{\textit{in}}_{i}=W_{\textit{node}}([v_{i};s_{i};b_{i}])$ (2)

Additionally for $i^{th}$ node, we added a positional feature vector
($\textit{pos}(n_{i})$) with its initial feature $f^{\textit{in}}_{i}$. It
takes the categorical position of $i^{th}$ node in a linear ordering of all
nodes, and covert it into a continuous sinusoidal vector as described in
vaswani2017attention .

$\displaystyle f^{\textit{final}}_{i}$
$\displaystyle=\textit{encoder}(f^{\textit{in}}_{i}+\textit{pos}(n_{i}))$ (3)
$\displaystyle o^{\textit{final}}_{i}$
$\displaystyle=\text{argmax}(W_{\textit{classifier}}(f^{\textit{final}}_{i})).$
(4)

where encoder is a stack of multi-head attention layers as shown in Figure 2.
After the contextualization of the nodes by the encoder, we obtain final node
features $f^{\textit{final}}_{i}$. This semantically enriched node feature is
subsequently used for two purposes. First, it is passed through a linear
object classifier $W_{\textit{classifier}}$ to get accurate final object class
($o^{\textit{final}}_{i}\in C$) probability as described in Eq. 4 and, second,
the $f^{\textit{final}}_{i}$ is passed to the the decoder cross-attention (E2N
attention) for edge context propagation.

####  3.3.2 Decoder Edge Positional Encoding

We feed the edges of the undirected object interaction graph to the
transformer decoder along with its positional embedding, which we refer to as
Edge Queries. Since there is no explicit ordering among edges we proposed a
novel positional embedding for edge from both of its node position. The new
positional encoding vector
($\textit{pos}_{\textit{e}_{ij}}\in\mathbb{R}^{2048}$) for edges ($e_{ij}$),
encodes the position of both the source nodes in an interleaved manner. One of
these nodes will play the roles of either subject or object. Since our edge is
undirected, we hypotheses that our proposed edge positional embedding will be
helpful for the network to distinguish the source nodes (subject or object)
out of all distinct nodes. The goal is to accumulate the necessary global
context (all distinct object instances) without losing its focus on the local
context (subject or object nodes). We define,

$\displaystyle\begin{split}\textit{pos}_{e_{ij}}{(k,k+1)}&=\left[\sin\left(\frac{p_{i}}{m^{\frac{2k}{d}}}\right),\cos\left(\frac{p_{i}}{m^{\frac{2k}{d}}}\right)\right]\\\
\textit{pos}_{e_{ij}}{(k+2,k+3)}&=\left[\sin\left(\frac{p_{j}}{m^{\frac{2k}{d}}}\right),\cos\left(\frac{p_{j}}{m^{\frac{2k}{d}}}\right)\right].\end{split}$
(5)

Eq. 5 describes positional encoding for an edge, where $p_{i}$ and $p_{j}$ are
the positions of the nodes $n_{i}$ and $n_{j}$, $m$ is maximum number of
sequence of nodes, $d=2048$, and $k$ denotes the $k^{th}$ position in the
positional encoding features vector.

####  3.3.3 Decoder E2N Attention

For an edge $e_{ij}$ of the Edge Queries, (between nodes $n_{i}$ and $n_{j}$),
its bounding box location ($b_{ij}\in\mathbb{R}^{4}$) and initial visual
features $v_{ij}\in\mathbb{R}^{4096}$ are derived from the union of the
bounding boxes of both nodes as shown in Figure 2. We concatenate the GloVe
vector embedding from both of its node labels $s_{ij}$ with the previously
obtained box and visual features for the semantic enrichment of the edge.
Subsequently, a linear projection layer ($W_{\textit{edge}}$) is used to
obtain the initial edge feature vector ($f^{\textit{in}}_{ij}$) or the Edge
Queries as

$\displaystyle f^{in}_{ij}=W_{edge}([v_{ij};s_{ij};b_{ij}]).$ (6)

We argue that a well contextualized edge is needed for complex global scene
representation and this can only be achieved if the edge exploits an larger
scene context. In traditional transformer decoder, a masked attention is used,
limiting the edges attention only to a part of the sequence. The accumulation
of global context for an edge requires a unique mechanism so it can preserve
its local dependency while exploring global context.

Empirically we found that applying self-attention between edges does not help,
as necessary context can be accumulated using N2N and E2N attention. See also
Table 6. Hence, we have removed the edge-to-edge self-attention in our
decoder. At first, E2N cross-attention has been applied from an edge to all
the nodes. Finally, we get the contextual edge features (
$f^{\textit{final}}_{ij}\in\mathbb{R}^{2048}$) as,

$\displaystyle
f^{\textit{final}}_{ij}=\textit{decoder}(f^{\textit{in}}_{ij}+{pos}_{e_{ij}},f^{\textit{final}}_{i={1..N}}).$
(7)

where decoder is a stack of multi-head attention with our proposed E2N
attention, positional encoding.

###  3.4 Directed Relation Prediction Module (RPM)

Relation is a directional property, i.e., subject and object cannot be
exchanged. After obtaining the context rich node and edge embeddings, an
initial directed relational embedding
($rel^{\textit{in}}_{{i}\rightarrow{j}}\in\mathbb{R}^{2048\times 3+512}$) has
been created as

$\displaystyle\textit{rel}^{\textit{in}}_{{i}\rightarrow{j}}=[f^{\textit{final}}_{i};f^{\textit{final}}_{ij};f^{\textit{final}}_{j};GAP(I_{feature})].$
(8)

where $GAP(I_{feature})\in\mathbb{R}^{512}$ is the global average pool of the
image feature obtained from object detector. As a next step,
$rel^{\textit{in}}_{{i}\rightarrow{j}}$ is passed through a sequential block
of neural networks, which we call directed relation prediction module or
$RPM$. $RPM$ takes ($rel^{\textit{in}}_{{i}\rightarrow{j}}$) as input and the
result is normalized with layer norm ba2016layer  followed by two blocks of
linear layers and Leaky ReLU xu2015empirical  nonlinearity for the predicate
classification as described in Eq. 8. Details on $RPM$ can be found in the
supplementary material. We postulate that a larger embedding space ($RPM$),
with normalized embeddings from nodes and edges, will effectively combine the
necessary context, and

$\displaystyle\textit{rel}^{\textit{final}}_{{i}\rightarrow{j}}=RPM(rel^{\textit{in}}_{{i}\rightarrow{j}}).$
(9)

Finally, we get the softmax distribution over all $p$ predicate categories
from the final relation vector
$(\textit{rel}^{\textit{final}}_{{i}\rightarrow{j}}\in\mathbb{R}^{2048})$
through a linear layer ($W_{\textit{p}}$). Note that $p$ is the number of
relation present in the dataset. We have also added the frequency baseline
(fq) from zellers2018neural  to model the dataset bias and obtain as

$\displaystyle\begin{split}Pr(r_{{i}\rightarrow{j}}|I)=\text{softmax}(W_{\textit{p}}\textit{rel}^{\textit{final}}_{{i}\rightarrow{j}})+\textit{fq}(\textit{i},\textit{j})).\end{split}$
(10)

The $Pr(r_{{i}\rightarrow{j}}|I)$ denotes the final relationship distribution
among $n_{i}$ and $n_{j}$ for a given image $I$.

##  4 Experiments

This section will describe the dataset and explain implementation details of
our network pipeline and spatial embedding implementation. 111Code is
available at:https://github.com/rajatkoner08/rtn

###  4.1 Datasets

We have used three most commonly used scene graph dataset, i.e., Visual Genome
krishna2017visual , GQAhudson2019gqa  and VRD lu2016visual , for our
experimental evaluation.

Visual Genome (VG) is one of the most challenging datasets for scene graph
detection and generation for real world images. The original dataset consists
of 108,077 images with annotated object bounding boxes, class, and binary
relations among the objects. The annotations are quite noisy: e.g., multiple
bounding boxes are provided for a single object. To alleviate this problem, Lu
et al. Xu_2017_CVPR  proposed a refined version of the dataset, which consists
of the most frequently occurring 150 objects and 50 relationships. To have a
fair comparison with most of the present state of art model zellers2018neural
; newell2017pixels ; zhang2019graphical ; herzig2018mapping ; zhang2019large
we have used this refined dataset. Also, our train (55K), validation (5K), and
test (26K) split are the same as per the dataset.

GQA is one of the largest and diverse scene graph datasets consisting of 1704
classes and 311 relationship labels as proposed in hudson2019gqa . It uses the
same images from Visual Genomekrishna2017visual  with more clean (e.g., more
accurate spatial location) and normalizes class and relationship distribution.
GQA is more challenging than other datasets as each image is annotated with a
dense scene graph and a large number of relations. We have used K-fold data
for training and report our result on “val” set mentioned in GQA. We omitted
the classwise frequency distribution zellers2018neural  for GQA for two
reasons, first GQA is more normalized than VG, and second due to a large
number of classes and relationships present in GQA incurred a large memory
overhead.

VRD contains 4000 training and 1000 test images with 100 objects and 70
predicate categories. We evaluate our model with the same COCO pretrained
backbone as used in zhang2019graphical . The evaluation metric is the same as
lu2016visual  that report R@50 and R@100 metric for relationship, phrase, and
predicate detection.

Model | w/ Graph Constraint | w/o Graph Constraint | Mean
---|---|---|---
| SGDET | SGCLS | PRDCLS | SGCLS | PRDCLS
R@ | 20 | 50 | 100 | 20 | 50 | 100 | 20 | 50 | 100 | 50 | 100 | 50 | 100
IMP Xu_2017_CVPR  | 14.6 | 20.7 | 24.5 | 31.7 | 34.6 | 35.4 | 52.7 | 59.3 | 61.3 | 43.4 | 47.2 | 75.2 | 83.6 | 44.93
A. Emb. newell2017pixels  | 6.5 | 8.1 | 8.2 | 18.2 | 21.8 | 22.6 | 47.9 | 54.1 | 55.4 | 26.5 | 30.0 | 68.0 | 75.2 | 34.04
Freq. zellers2018neural  | 17.7 | 23.5 | 27.6 | 27.7 | 32.4 | 34.0 | 49.4 | 59.9 | 64.1 | 40.5 | 43.7 | 71.3 | 81.2 | 44.07
MotifNet zellers2018neural  | 21.4 | 27.2 | 30.3 | 32.9 | 35.8 | 36.5 | 58.5 | 65.2 | 67.1 | 44.5 | 47.7 | 81.1 | 88.3 | 49.50
CMAT chen2019counterfactual  | 22.1 | 27.9 | 31.2 | 35.9 | 39.0 | 39.8 | 60.2 | 66.4 | 68.1 | 48.6 | 52.0 | 82.2 | 90.1 | 50.97
VRU zhang2019large  | 20.7 | 27.9 | 32.5 | 36.0 | 36.7 | 36.7 | 66.8 | 68.4 | 68.4 | - | - | - | - | 52.16
KREN chen2019knowledge  | - | 27.1 | 29.8 | - | 36.7 | 37.4 | - | 54.2 | 59.1 | 45.9 | 49.0 | 81.9 | 88.9 | 51.00
ReIDN zhang2019graphical  | 21.1 | 28.3 | 32.7 | 36.1 | 36.8 | 36.8 | 66.9 | 68.4 | 68.4 | 48.9 | 50.8 | 93.8 | 97.8 | 52.83
GPS Net lin2020gps  | 22.3 | 28.9 | 33.2 | 41.8 | 42.3 | 42.3 | 67.6 | 69.7 | 69.7 | - | - | - | - | 46.42
RTN (Ours) | 22.5 | 29.0 | 33.1 | 43.8 | 44.0 | 44.0 | 68.3 | 68.7 | 68.7 | 61.3 | 62.3 | 97.2 | 99.1 | 57.1
Table 1: Comparison of our model with state-of-the-art methods tested on Visual Genome krishna2017visual . We have computed mean performance, as some of the works did not provide all scores. Evaluation with ‘Graph constraint’allows only one relation between an object pair, and “No Graph constraint” allows multiple relations between an object pair. Model | SGCLS | PREDCLS | Mean
---|---|---|---
R@ | 20 | 50 | 100 | 20 | 50 | 100. |
IMPXu_2017_CVPR  | 6.3 | 9.4 | 11.2 | 47.3 | 69.8 | 81.4 | 37.6
Neural Motifzellers2018neural  | 6.5 | 9.9 | 11.9 | 51.2 | 73.6 | 84.2 | 39.6
Unbiased TDEtang2020unbiased  | 5.8 | 8.8 | 10.6 | 51.6 | 74.0 | 84.6 | 39.2
RTN(ours) | 9.2 | 11.9 | 12.2 | 55.3 | 78.2 | 88.1 | 42.3
Table 2: Comparison of our model on the GQA dataset for predicate and scene graph classification. Mask-RCNNhe2017mask  is used as backbone object detector fine tuned in GQA.  Model | SGCLS | PREDCLS
---|---|---
mR@ | 100 | 100
VG |  |
IMP Xu_2017_CVPR  | 6.0 | 10.5
FREQ zellers2018neural  | 8.5 | 16.0
MotifNetzellers2018neural  | 8.2 | 15.3
KERNchen2019knowledge  | 10.0 | 19.2
VCTREE-HLtang2018learning  | 10.8 | 19.4
GPS-Nettang2018learning  | 12.6 | 22.8
RTN (Ours) | 12.6 | 20.3
GQA |  |
IMP Xu_2017_CVPR  | 0.5 | 2.2
MotifNetzellers2018neural  | 0.8 | 2.9
Unbiased TDEtang2020unbiased  | 0.7 | 2.8
RTN (Ours) | 1.4 | 4.5
Table 3: Comparison on the mean-Recall@100 metric between various methods across all the 50 and 311 relationship categories for VG and GQA correspondingly. abcdefModel | Relation | Phrase
---|---|---
Detection | Detection
R@ | 50 | 100 | 50 | 100
VTransE zhang2017visual  | 19.4 | 22.4 | 14.1 | 15.2
Vip-CNN li2017vip  | 17.3 | 20.0 | 22.8 | 27.9
KL distilationyu2017visual  | 19.2 | 21.3 | 23.1 | 24.0
Zoom-Netyin2018zoom  | 18.9 | 21.4 | 24.8 | 24.1
RelDN* zhang2019graphical  | 25.3 | 28.6 | 31.3 | 36.4
GPS-Net* lin2020gps  | 27.8 | 31.7 | 33.8 | 39.2
RTN (Ours)* | 28.1 | 32.0 | 33.5 | 38.7
Table 4: Comparison of our model on the VRD dataset for relation detection and
Phrase Detection. * uses the same detector pretrained on COCO.

###  4.2 Implementation Details

We have implemented our model in PyTorch and trained it in a single Nvidia RTX
3900 GPU. We have trained the network for 20 epochs; it took approximately two
days to train. The input to our model is an image with a size of $592\times
592$ pixels, as in zellers2018neural . As mentioned before, the encoder and
the decoder modules accept input features of size 2048. We have used 3 encoder
layers, 2 decoder layers and 12 attention heads for our network. Our model is
optimized by SGD with momentum. A learning rate of $10^{-3}$ and batch size of
16 has been used. We have used cross-entropy loss for both of our object and
relation classification loss.222List of all hyper-parameters are given in the
supplementary material. In training, we used one foreground edge (contain at
least one ground truth relation) for 4 background edges (without any
relation), and randomly flip some images as part of data augmentation. We have
followed the same evaluation as in current benchmarks zhang2019graphical  and
computed scene graph classification (SGCLS) and predicate classification
(PREDCLS).

For scene graph detection (SGDET), we have taken the top 64 object label
proposals from the object detector for each image after performing the non-
maximal suppression (NMS) with intersection over union (IoU) of 0.3, as
similar to zellers2018neural . To reduce computational load in relation
classification, we have only considered those pairs of nodes whose bounding
boxes are overlapping.

To have a fair comparison on evaluating on visual genome, we have used Faster-
RCNN ren2015faster  with VGG16 simonyan2014very  backbone pretrained on visual
genome dataset as per zellers2018neural ; zhang2019graphical . We have trained
Mask-RCNNhe2017mask  on GQA for the backbone object detector as per
tang2020unbiased .

##  5 Results and Discussion

###  5.1 Quantitative Results

For VG dataset, Table 1 shows the performance of our method in comparison with
other methods. It clearly demonstrates that our novel context propagation for
both objects and edges significantly improves most of the performance metrics.
Note that in training, we only used simple cross-entropy loss in contrast to
the recent literature, such as, contrastive or triplet loss (ReIDN
zhang2019graphical , VRU zhang2019large ) and Node-Priority-Sensitive (NPS)
loss (lin2020gps ). As shown under ‘No graph constraint’ accuracy of $99.1\%$
R@100 in PREDCLS indicates, our model is superior than the competing models in
learning the most likely relations. One of the key observations we made from
Table 1 is that our approach significantly improves on the R@20 metric,
compared to previous state of the art RelDN zhang2019graphical  with maximum
improvement of 7.7% on SGCLS. Our model achieves slightly lower score than our
contemporary lin2020gps  work on R@50 and R@100 for PREDCLS. We suspect that
the frequency softening method by lin2020gps  better captures the data
imbalance and long tailed distribution of relationship, than ours in those
cases. However, in this study we are not focusing on loss function instead of
looking deep into context propagation. And hence, we leave the class imbalance
study for future work.

(a) Image

(b) N2N Attention heatmap

(c) E2N Attention heatmap

(d) Generated scene graphs

Figure 3: A positive example outputs from our network with associated
attention map and scene graph.

Table 2, shows the performance of RTN on GQA dataset. Here RTN outperforms all
competitive models by mean 3.1%. Compare to VG, GQA contains six times more
relationship classes and three times more dense scene graph annotation. Thus
GQA is more challenging for efficient propagation of context that depicts the
full scene. As GQA has more normalized relationship distribution, class wise
frequency bias is nonessential. Our proposed transformer based multi-hop
context propagation is particularly well suited for extracting important
context of a given edge query out of large number of nodes and edges. Hence,
performance on GQA shows our RTN is superior over the state-of-the-art
methods. Especially performance on “PREDCLS” depicts the effectiveness of our
model on utilizing proper context information in the presence of large number
of nodes or edges or in a dense scene graph.

In order to gain deeper insight on class imbalance, we computed mean-recall
that takes the mean across all the relationship classes. Table 3 shows the
performance of our model with other competing methods for the mean recall. Our
RTN performs better than most of them in “SGCLS” and “PREDCLS”. However, the
performance on “PREDCLS” R@100 in VG dataset is marginally lower compared to
lin2020gps  because of the class imbalance. Since GQA is more normalized than
VG, thus it has less prominent class imbalance problem. We have achieved state
of the art result on mean recall across all metrics on GQA.

Table 4, shows the performance of our network on the VRD dataset; for a fair
comparison, we have used the same object detector as in zhang2019graphical .
Here also, we have performed better than or on par with other state-of-the-art
models on relations classification. Transformer based architectures are
reported to require a large number of training data, but despite of relatively
small sample size of VRD, our node and edge based context propagation helps
RTN for an efficient accumulation of scene context.

###  5.2 Qualitative Results

Figure. 3, shows a positive example with N2N and E2N attention. Here N2N
attention shows how the presence of an object influence other like boy-door,
child-umbrella etc. E2N attention heat map shows the importance of surrounding
nodes for a particular edge (e.g child, boy play an important role in most of
relation as they are same semantic object ). More positive and negative
examples can be found in the supplementary section.

Additionally, an analysis of the errors of the Relation Transformer network
provides insight into what the network has learned. For example, ‘on’ is the
most mispredicted relation in our evaluation settings for Visual Genome. The
relationship ‘on’ is falsely predicted as ‘of’ for 56.9% out of total false
predictions among all relationships. Interestingly, for most of the examples
like ‘Face, of, Woman ’ is more appropriate than ‘Face, on, Woman ’ ,
indicating that the network is not necessarily failing to predict correctly,
rather it is predicting more suitable or semantically similar relationships. A
probable reason for these false predictions is the significant bias in the
training dataset.

###  5.3 Ablation Study

In the following, we present the results of three ablation experiments in
order to demonstrate the efficacy of our proposed modules and the selection of
hyper-parameters on VG dataset.

####  5.3.1 Encoder-Decoder Layers and RPM

First, we compare the performance of the N2N and E2N modules for varying
number of layers in combination with the RPM. The results are presented in the
following Table 5. We make the following observations.

#N2N | #E2N | RPM |  abcdefgPREDCLS
---|---|---|---
R@20 | R@50 | R@100
1 | 1 | yes | 66.7 | 67.1 | 67.1
1 | 2 | yes | 68.0 | 68.4 | 68.4
2 | 1 | yes | 67.4 | 67.8 | 67.8
2 | 2 | yes | 68.1 | 68.5 | 68.5
3(ours) | 2(ours) | yes | 68.3 | 68.7 | 68.7
3 | 2 | no | 67.8 | 68.2 | 68.3
3 | 3 | yes | 68.2 | 68.5 | 68.5

Table 5: Ablation study of our proposed N2N, E2N and relation prediction
module on Visual Genome for PREDCLS.

  1. 1.

All nodes need to be contextualized well enough, in order to propagate context
for edges. This is demonstrated in rows 4 and 5 of Table 5. Adding more edge
context layers without adequate object context propagation limits the
performance. Thus, it shows the importance of contextualized objects.

  2. 2.

After objects are properly contextualized, adding the optimal number of edge-
context modules on top of object-context modules increases the performance.
Jointly it shows the importance of both modules (rows 4 and 5).

  3. 3.

If we use a large number ($\geqslant 3$) of E2N and N2N layers, then the
performance decreases. One of the possible reason is that, as the network size
grows, it becomes hard to optimize the network (row 7).

  4. 4.

Directed RPM facilitates the forming of the necessary embedding space after
completion of optimal contextualization of nodes and edges. If we replace the
RPM module with a simple linear layer without any normalization, it hurts the
performance (row 6).

From these observations, we can clearly infer that all three modules have
significantly contributed to relationship classification.

####  5.3.2 Decoder E2N Attention and Positional Encoding

Next, we present necessary ablation experiments in support of the benefits of
the proposed changes in the decoder architecture as discussed in Section 1. We
have conducted three experiments to understand the impact of these changes as
presented in Table 6. We make the following observations.

Model |  abcdPRDCLS |
---|---|---
R@ | 20 | 50 | 100
with vaswani2017attention ’s decoder | 66.5 | 67.1 | 67.1
with only E2N attention | 67.6 | 68.0 | 68.0
with only E2N attention |  |  |
\+ proposed PE (RTN) | 68.3 | 68.7 | 68.7
Table 6: Ablation study of our proposed changes in original transformer
decoder vaswani2017attention , like changes in positional encoding, removal of
decoder-decoder attention on Visual Genome dataset.

  1. 1.

With vaswani2017attention ’s decoder: shows the network performance without
any modification from vaswani2017attention ’s transformer decoder. In this
setting, at first, the decoder applies attention across all edges (decoder
self-attention), afterward, from edge to all nodes (decoder-encoder cross
attention or ours E2N attention). However, for SGG task, primary context of an
edge should come from the immediate and neighboring nodes (E2N attention).
Hence, self-attention before cross-attention, as in vaswani2017attention ’s
decoder, hinders node to edge context propagation and limits performance.

  2. 2.

With only E2N attention: shows our network performance with only E2N attention
(without self-attention in the decoder) and position-wise feed forward
network. Here we didn’t change the positional encoding for the decoder.
Removing the self-attention in decoder, improves the performance. One possible
reason for this could be easy accumulation of necessary context from the nodes
needed for inferring relation of the image.

  3. 3.

With only E2N attention + proposed PE (RTN): shows our Relation Transformer
model. Here we have applied our proposed positional encoding, and the results
show that the preservation of the source node identity of an edge helps the
network efficiently accumulate the global context without losing the local
context.

Our experiment shows that our design changes in the original transformer
architecture are effective and helpful for relation classification.

####  5.3.3 Transformer and RPM Feature Space

Finally, we report results on ablation studies to analyze the impact of
feature embeddings in transformer inputs i.e., semantics or GloVE and spatial
embedding. Additionally, we perform ablation on the RPM feature space i.e.,
global average pooling and frequency bias. Both the ablation studies are
presented in Table 7. We make the following observations.

Model |  abcdPRDCLS |
---|---|---
abcdR@ | 20 | 50 | 100
without word embedding | 67.7 | 68.1 | 68.1
without spatial embedding | 68.2 | 68.6 | 68.6
without Global Avg Pool | 68.0 | 68.4 | 68.4
without Freq Bias | 66.7 | 67.0 | 67.0
with every features | 68.3 | 68.7 | 68.7
Table 7: Ablation study of our proposed changes in original transformer
decoder vaswani2017attention , like changes in positional encoding, removal of
decoder-decoder attention on Visual Genome dataset.

  1. 1.

Without word embedding, we see a decrease in recall. We attribute this to the
fact that word embedding provides additional semantic cues based on the
correlated object classes, which is helpful for the SGG task.

  2. 2.

Without spatial embedding of the bounding box coordinates, the performance
sufferers, but the drop is marginal. For an accurately known global context of
an object, spatial embedding provides primarily redundant information.
However, when the object detector does not accurately localize the object,
spatial embedding can provide additional information to compensate for it.

  3. 3.

Without Global Avg Pool, we also observe a performance drop. We suspect a lack
of sufficient global image information behind the decrease in recall. With
Global Avg Pool, it is easier for the network to aggregate the context of
individual nodes and edges that are far apart in spatial location.

  4. 4.

Without frequency bias, we see significant performance drops. This is because
of the high-class imbalance in the VG dataset, which dominates the loss
function. Thus frequency bias is a crucial part of the RPM to mitigate the
issue of class imbalance.

##  6 Conclusion

In this paper, we propose a transformer view on the scene graph generation
task. Notably, we explore node-to-node and edge-to-node association through
the lens of self-attention and cross-attention, respectively. The resultant
node and edge embedding from the transformer encoder and decoder,
respectively, contributes to robust and discriminate relation classification
through our relation prediction module. We achieve a consistent improvement
over the state-of-the-art models in the evaluation metrics across small,
medium, and large-scale relational datasets. Our modular nature of RTN can
offer contextualized node and edge features for other various tasks, such as
visual question answering, text to image generation, etc. Although our work’s
current focus remains on attention-based supervised relational context
modeling, future work can be carried upon solving class imbalance issues and
end-to-end self-supervised relational feature learning.

## References

  * (1) Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering.  In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
  * (2) Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization.  arXiv preprint arXiv:1607.06450 (2016)
  * (3) Baier, S., Ma, Y., Tresp, V.: Improving visual relationship detection using semantic modeling of scene descriptions.  In: International Semantic Web Conference, pp. 53–68. Springer (2017)
  * (4) Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers.  arXiv preprint arXiv:2005.12872 (2020)
  * (5) Chen, L., Zhang, H., Xiao, J., He, X., Pu, S., Chang, S.F.: Counterfactual critic multi-agent training for scene graph generation.  In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4613–4623 (2019)
  * (6) Chen, T., Yu, W., Chen, R., Lin, L.: Knowledge-embedded routing network for scene graph generation.  In: Conference on Computer Vision and Pattern Recognition (2019)
  * (7) Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805 (2018)
  * (8) Ghosh, S., Burachas, G., Ray, A., Ziskind, A.: Generating natural language explanations for visual question answering using scene graphs and visual attention.  arXiv preprint arXiv:1902.05715 (2019)
  * (9) He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn.  In: Proceedings of the IEEE international conference on computer vision, pp. 2961–2969 (2017)
  * (10) Herzig, R., Raboh, M., Chechik, G., Berant, J., Globerson, A.: Mapping images to scene graphs with permutation-invariant structured prediction.  In: Advances in Neural Information Processing Systems, pp. 7211–7221 (2018)
  * (11) Hildebrandt, M., Li, H., Koner, R., Tresp, V., Günnemann, S.: Scene graph reasoning for visual question answering.  arXiv preprint arXiv:2007.01072 (2020)
  * (12) Hu, R., Rohrbach, M., Andreas, J., Darrell, T., Saenko, K.: Modeling relationships in referential expressions with compositional modular networks.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1115–1124 (2017)
  * (13) Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6700–6709 (2019)
  * (14) Johnson, J., Krishna, R., Stark, M., Li, L.J., Shamma, D., Bernstein, M., Fei-Fei, L.: Image retrieval using scene graphs.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3668–3678 (2015)
  * (15) Knyazev, B., de Vries, H., Cangea, C., Taylor, G.W., Courville, A., Belilovsky, E.: Graph density-aware losses for novel compositions in scene graph generation.  arXiv preprint arXiv:2005.08230 (2020)
  * (16) Koner, R., Li, H., Hildebrandt, M., Das, D., Tresp, V., Günnemann, S.: Graphhopper: Multi-hop scene graph reasoning for visual question answering (2021)
  * (17) Koner, R., Sinhamahapatra, P., Roscher, K., Günnemann, S., Tresp, V.: Oodformer: Out-of-distribution detection transformer (2021)
  * (18) Koner, R., Sinhamahapatra, P., Tresp, V.: Scenes and surroundings: Scene graph generation using relation transformer.  arXiv preprint arXiv:2107.05448 (2021)
  * (19) Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations.  International Journal of Computer Vision 123(1), 32–73 (2017)
  * (20) Li, G., Duan, N., Fang, Y., Jiang, D., Zhou, M.: Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training.  arXiv preprint arXiv:1908.06066 (2019)
  * (21) Li, Y., Ouyang, W., Wang, X., Tang, X.: Vip-cnn: Visual phrase guided convolutional neural network.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1347–1356 (2017)
  * (22) Lin, X., Ding, C., Zeng, J., Tao, D.: Gps-net: Graph property sensing network for scene graph generation.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3746–3753 (2020)
  * (23) Liu, Y., Wang, R., Shan, S., Chen, X.: Structure inference net: Object detection using scene-level context and instance-level relationships.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6985–6994 (2018)
  * (24) Lu, C., Krishna, R., Bernstein, M., Fei-Fei, L.: Visual relationship detection with language priors.  In: European conference on computer vision, pp. 852–869. Springer (2016)
  * (25) Lu, J., Batra, D., Parikh, D., Lee, S.: Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In: Advances in Neural Information Processing Systems, pp. 13–23 (2019)
  * (26) Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A.L., Murphy, K.: Generation and comprehension of unambiguous object descriptions.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 11–20 (2016)
  * (27) Nagaraja, V.K., Morariu, V.I., Davis, L.S.: Modeling context between objects for referring expression understanding.  In: European Conference on Computer Vision, pp. 792–807. Springer (2016)
  * (28) Newell, A., Deng, J.: Pixels to graphs by associative embedding.  In: Advances in neural information processing systems, pp. 2171–2180 (2017)
  * (29) Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation.  In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543 (2014)
  * (30) Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., Lazebnik, S.: Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In: Proceedings of the IEEE international conference on computer vision, pp. 2641–2649 (2015)
  * (31) Qi, M., Li, W., Yang, Z., Wang, Y., Luo, J.: Attentive relational networks for mapping images to scene graphs.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3957–3966 (2019)
  * (32) Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks.  In: Advances in neural information processing systems, pp. 91–99 (2015)
  * (33) Schroeder, B., Tripathi, S.: Structured query-based image retrieval using scene graphs.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 178–179 (2020)
  * (34) Sharifzadeh, S., Baharlou, S.M., Berrendorf, M., Koner, R., Tresp, V.: Improving visual relation detection using depth maps (2019)
  * (35) Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556 (2014)
  * (36) Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research 15(1), 1929–1958 (2014)
  * (37) Tang, K., Niu, Y., Huang, J., Shi, J., Zhang, H.: Unbiased scene graph generation from biased training.  In: Conference on Computer Vision and Pattern Recognition (2020)
  * (38) Tang, K., Zhang, H., Wu, B., Luo, W., Liu, W.: Learning to compose dynamic tree structures for visual contexts.  In: Conference on Computer Vision and Pattern Recognition (2019)
  * (39) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need.  In: Advances in neural information processing systems, pp. 5998–6008 (2017)
  * (40) Voita, E., Talbot, D., Moiseev, F., Sennrich, R., Titov, I.: Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.  arXiv preprint arXiv:1905.09418 (2019)
  * (41) Wan, H., Luo, Y., Peng, B., Zheng, W.S.: Representation learning for scene graph completion via jointly structural and visual embedding.  In: IJCAI, pp. 949–956 (2018)
  * (42) Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794–7803 (2018)
  * (43) Woo, S., Kim, D., Cho, D., Kweon, I.S.: Linknet: Relational embedding for scene graph.  In: Advances in Neural Information Processing Systems, pp. 560–570 (2018)
  * (44) Xu, B., Wang, N., Chen, T., Li, M.: Empirical evaluation of rectified activations in convolutional network.  arXiv preprint arXiv:1505.00853 (2015)
  * (45) Xu, D., Zhu, Y., Choy, C.B., Fei-Fei, L.: Scene graph generation by iterative message passing.  In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
  * (46) Xu, N., Liu, A.A., Liu, J., Nie, W., Su, Y.: Scene graph captioner: Image captioning based on structural visual representation.  Journal of Visual Communication and Image Representation 58, 477–485 (2019)
  * (47) Yang, J., Lu, J., Lee, S., Batra, D., Parikh, D.: Graph r-cnn for scene graph generation.  In: Proceedings of the European conference on computer vision (ECCV), pp. 670–685 (2018)
  * (48) Yin, G., Sheng, L., Liu, B., Yu, N., Wang, X., Shao, J., Change Loy, C.: Zoom-net: Mining deep feature interactions for visual relationship recognition.  In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 322–338 (2018)
  * (49) Yu, R., Li, A., Morariu, V.I., Davis, L.S.: Visual relationship detection with internal and external linguistic knowledge distillation.  In: Proceedings of the IEEE international conference on computer vision, pp. 1974–1982 (2017)
  * (50) Zareian, A., Wang, Z., You, H., Chang, S.F.: Learning visual commonsense for robust scene graph generation.  arXiv preprint arXiv:2006.09623 (2020)
  * (51) Zellers, R., Yatskar, M., Thomson, S., Choi, Y.: Neural motifs: Scene graph parsing with global context.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5831–5840 (2018)
  * (52) Zhang, H., Kyaw, Z., Chang, S.F., Chua, T.S.: Visual translation embedding network for visual relation detection.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5532–5540 (2017)
  * (53) Zhang, J., Kalantidis, Y., Rohrbach, M., Paluri, M., Elgammal, A., Elhoseiny, M.: Large-scale visual relationship understanding.  In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 9185–9194 (2019)
  * (54) Zhang, J., Shih, K.J., Elgammal, A., Tao, A., Catanzaro, B.: Graphical contrastive losses for scene graph parsing.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11535–11543 (2019)

##  Appendix A Implementation Details

In this section we will list out hyper-parameter used in final model.

  1. 1.

optimizer : Stochastic Gradient Descent(SGD)

  2. 2.

learning rate : $10^{-3}$ with reduce on plateau and patience $3$

  3. 3.

batch size : 16

  4. 4.

dropout srivastava2014dropout  : $0.25$

  5. 5.

Context Propagation of Objects : 3 E2N modules

  6. 6.

Context Propagation for Edges : 2 E2N modules

  7. 7.

weight initialization : Xavier normal

  8. 8.

attention head : 12 attention heads are used in both N2N and E2N.

  9. 9.

random seed : 42

  10. 10.

Directed Relation Prediction Module (RPM): As discussed in paper, a RPM module
leverages upon context rich nodes ($f_{i}^{final},f_{j}^{final}$) and
undirected edges ($f_{ij}^{final}$) to produce final directed relation
embedding between two nodes ($rel_{i\rightarrow j}^{final}$). The input to RPM
(($rel_{i\rightarrow j}^{in}$)) is normalized by LayerNorm ba2016layer then
followed by a linear layer ($W_{1}\in\mathbb{R}^{4096}$), dropout then another
linear layer ($W_{2}\in\mathbb{R}^{2048}$) and finally followed by Leaky ReLU
non-linearity.

##  Appendix B Qualitative Results

This section will provide a few more qualitative samples generated by our
network in both positive and negative scenarios. To improve visibility and
interpretability, we only consider the interaction among ground truth objects
and relations in these examples.

Fig. 4, shows the positive scenario, where our network is able to detect
correct relationships label despite the presence of repetitive bounding box
(boy and child) or similar objects (giraffe). Thus, it shows the robustness of
the method.

Fig. 5, shows the negative scenario, where network prediction is different
from ground truth labels. In most of these cases, it was found that predicted
labels are semantically closer to ground truth labels, and from a human
perspective, both could be right. For example man-at-beach and man-on-beach
both are grammatically correct.

While exploring various attention heads, we have found an interesting pattern
that few attention heads are focusing on the main object in the scene, few on
the combination of some objects while others focus on surroundings. Some
recent research workvoita2019analyzing  also explores the working patterns for
various attention heads.

(a) Image

(b) N2N Attention heatmap

(c) E2N Attention heatmap

(d) Generated scene graphs

Figure 4: Some positive example outputs from our network with associated
attention map and scene graph.

(a) Image

(b) N2N Attention heatmap

(c) E2N Attention heatmap

(d) Generated scene graphs

Figure 5: Some negative example outputs from our network with associated
attention map and scene graph.


# Radio observations of the tidal disruption event AT2020opy: a luminous non-
relativistic outflow encountering a dense circumnuclear medium

A. J. Goodwin,1, J. C. A. Miller-Jones1, S. van Velzen2, M. Bietenholz3,4, J.
Greenland1, B. Cenko5, S. Gezari,6, A. Horesh7, G. R. Sivakoff8, L. Yan9, W.
Yu10, and X. Zhang10,11
1International Centre for Radio Astronomy Research – Curtin University, GPO
Box U1987, Perth, WA 6845, Australia
2Leiden Observatory, Leiden University, PO Box 9513, 2300 RA Leiden, The
Netherlands
3 Department of Physics and Astronomy, York University, Toronto, M3J 1P3,
Ontario, Canada
4 SARAO/Hartebeesthoek Radio Observatory, PO Box 443, Krugersdorp, 1740, South
Africa
5 Astrophysics Science Division, NASA Goddard Space Flight Center, Mail Code
661, Greenbelt, MD 20771, USA
6 Space Telescope Science Institute, 3700 San Martin Drive, Baltimore, MD
21218, USA
7 Racah Institute of Physics, The Hebrew University of Jerusalem, Jerusalem
91904, Israel
8 CCIS 4-181 University of Alberta, Edmonton Alberta T6G 2E1, Canada
9 Caltech Optical Observatories, California Institute of Technology, Pasadena,
CA 91125, USA
10 Shanghai Astronomical Observatory, Chinese Academy of Sciences, 80 Nandan
Road, Shanghai 200030, China
11 University of Chinese Academy of Sciences, 19A Yuquanlu, Beijing 100049,
China
<EMAIL_ADDRESS>
(Accepted XXX. Received YYY; in original form ZZZ)

###### Abstract

Tidal disruption events (TDEs) occur when a star passes too close to a
supermassive black hole and is destroyed by tidal gravitational forces. Radio
observations of TDEs trace synchrotron emission from outflowing material that
may be ejected from the inner regions of the accretion flow around the SMBH or
by the tidal debris stream. Radio detections of tidal disruption events are
rare, but provide crucial information about the launching of jets and outflows
from supermassive black holes and the circumnuclear environment in galaxies.
Here we present the radio detection of the TDE AT2020opy, including three
epochs of radio observations taken with the Karl G. Jansky’s Very Large Array
(VLA), MeerKAT, and upgraded Giant Metrewave Radio telescope. AT2020opy is the
most distant thermal TDE with radio emission reported to date, and from
modelling the evolving synchrotron spectra we deduce that the host galaxy has
a more dense circumnuclear medium than other thermal TDEs detected in the
radio band. Based on an equipartition analysis of the synchrotron spectral
properties of the event, we conclude that the radio-emitting outflow was
likely launched approximately at the time of, or just after, the initial
optical flare. We find no evidence for relativistic motion of the outflow. The
high luminosity of this event supports that a dense circumnuclear medium of
the host galaxy produces brighter radio emission that rises to a peak more
quickly than in galaxies with lower central densities.

###### keywords:

transients: tidal disruption events – radio continuum: transients

††pubyear: 2022††pagerange: Radio observations of the tidal disruption event
AT2020opy: a luminous non-relativistic outflow encountering a dense
circumnuclear medium–Radio observations of the tidal disruption event
AT2020opy: a luminous non-relativistic outflow encountering a dense
circumnuclear medium

##  1 Introduction

When a star passes within the tidal radius of a black hole, the star can be
destroyed, producing a bright flare of electromagnetic radiation visible from
radio to X-ray wavelengths (e.g. Rees, 1988). The afterglow emission from such
tidal disruption events (TDEs) at different wavelengths gives insight into the
process by which the star was destroyed, the formation of accretion disks
around black holes, the magnetic and gravitational fields of the central black
hole, and the nuclear environment of the host galaxy (e.g. Lodato & Rossi,
2011).

TDEs show diverse optical, X-ray, and radio properties, thought to be
explained by the circumstances surrounding the stellar disruption and
subsequent behaviour of the debris such as the SMBH mass, viewing angle,
impact parameter as well as the circumnuclear environment of the host galaxy.
Simulations have shown that the bound stellar debris may circularise to form
an accretion disk (e.g. Bonnerot et al., 2016; Hayasaki et al., 2016; Liptai
et al., 2019; Bonnerot & Lu, 2020; Mummery & Balbus, 2020), emitting X-ray
radiation from accretion onto the SMBH, and optical radiation, from either re-
processing of the X-rays in the accretion disk or stream-stream collisions of
the tidal debris (e.g. see Roth et al., 2020, for a review). The time taken
for the stellar debris to circularise and accretion to begin onto the SMBH is
a matter of debate, with physical system properties such as the SMBH mass,
stellar orbit, and stellar properties thought to affect the organisation of
the debris (Hayasaki et al., 2016; Liptai et al., 2019; Lu & Bonnerot, 2020).
Observationally, X-ray properties of TDEs are extremely diverse (Auchettl et
al., 2017), with some events never detected in X-rays, others detected
immediately (e.g. Miller et al., 2015), and others showing delayed onset of
bright X-ray emission (e.g. Hinkle et al., 2021). The intersection of debris
streams and the circularisation process could be an important factor driving
the diversity in the observational properties of many TDEs (Lu & Bonnerot,
2020).

Radio emission from TDEs is rare; only $\sim 10\%$ of TDEs discovered have
reported radio detections. Radio observations of TDEs probe the outflowing
material ejected during the stellar destruction, including any jets or wind-
induced outflows, as well as their interactions with the circumnuclear medium
(CNM; see Alexander et al., 2020, for a review). Recent radio observations of
TDEs have identified two distinct populations: relativistic, non-thermal,
jetted events (e.g. Swift J1644+57; Bloom et al., 2011), and the more common
non-relativistic, thermal events (e.g. ASASSN-14li; Alexander et al., 2016;
van Velzen et al., 2016), as well as highlighting the diverse characteristics
of individual events within these populations.

Non-thermal TDEs are thought to produce a relativistic jet, giving rise to
bright radio emission with luminosities $>10^{40}$ erg s-1 (Levan et al.,
2011; Burrows et al., 2011; Zauderer et al., 2011; Bloom et al., 2011). In
contrast, thermal TDEs exhibit radio emission with spectral luminosities
$<10^{40}$ erg s-1 that often is observed within months after the initial
optical flare, and rises to a peak within a couple of years depending on the
frequency (e.g. Alexander et al., 2016; Anderson et al., 2020; Cendes et al.,
2021; Goodwin et al., 2022). Recently, it has been suggested that delayed
radio flares are common in TDEs (Horesh et al., 2021a; Horesh et al., 2021b;
Cendes et al., 2022; Perlman et al., 2022). However, without continuous radio
coverage of the TDE lightcurve, it cannot be determined if these are "flares"
or simply a slow rise to the radio peak with a structured radio lightcurve, as
was the case for AT2019azh (Goodwin et al., 2022; Sfaradi et al., 2022). There
are two strong cases in which there is evidence that a delayed mildly
relativistic jet was produced $>500$ d post initial disruption: ASASSN-15oi
and AT2018hyz (Horesh et al., 2021a; Cendes et al., 2022).

The radio emission from thermal TDEs is thought to arise from either a mildly-
collimated, sub-relativistic jet (e.g. van Velzen et al., 2016), a spherical
accretion-induced wind outflow (e.g. Alexander et al., 2016), the unbound
debris stream (e.g. Krolik et al., 2016), or a spherical outflow from stream-
stream collisions during the circularisation of the stellar debris (e.g. Lu &
Bonnerot, 2020). Existing radio observations of thermal TDEs have been unable
to convincingly discern the mechanism behind the non-relativistic outflows
that have been observed, and new observations are crucial in identifying if
there is a single mechanism behind all radio outflows from TDEs, or if it
differs from system to system.

In this work we present the radio detection of AT2020opy, including three
epochs of radio spectral observations of the event over 8 months. In Section 2
we describe the radio observations and the data processing. In Section 3 we
present the results and synchrotron modelling of the outflow. In Section 4 we
discuss the implications of the results and provide a comparison of AT2020opy
with other TDEs, and finally in Section 5 we summarise this work and provide
concluding remarks.

##  2 Observations

The TDE AT2020opy (ZTF20abjwvae) was first detected on 2020 July 08 by the
Zwicky Transient Facility (ZTF) as a transient coincident with the nucleus of
the galaxy SDSS J155625.72+232220.6 (Gezari et al., 2020). The source rose
slowly to a peak optical flux of $g=18.9$ mag on 2020 August 02 and showed a
featureless blue continuum in spectral observations taken with the Palomar
60in SED Machine. Swift follow-up observations on 2020 August 09 revealed
bright UV emission from the event but no associated X-ray source, motivating
Gezari et al. (2020) to classify the transient as a TDE. Based on the ZTF
observations and optical spectral properties, Hammerstein et al. (2022)
classified AT2020opy as an H+He TDE at a redshift of $z=0.159$ due to broad
H$\alpha$ and H$\beta$ emission lines, as well as a complex of He II emission
lines.

###  2.1 VLA

We observed the optical position of AT2020opy on four occasions with the Karl
G. Jansky Very Large Array (VLA; Proposal ID: 20A-392, PI: Van Velzen) between
2020 October 06 and 2021 June 03. Our initial observation on 2021 October 06
was taken at 8–12 GHz to search for radio emission from the event. We
discovered a point source consistent with the optical position of the galaxy
with a flux density of 65$\pm$7 $\mu$Jy at 10 GHz. We subsequently triggered 3
epochs of radio spectral observations of the source spanning 2–18 GHz over 8
months. The radio observations are summarised in Table 1.

Table 1: Dedicated radio observations of AT2020opy. $\nu$ is the central frequency of each sub-band (with bandwidth of 1 GHz at S-band and C-band, 2 GHz at X-band, 3 GHz at Ku-band, and 0.856 GHz for MeerKAT L-band), $F_{\nu}$ is the measured flux density of the source, and "Array" describes the VLA array configuration for the observations. Date (UTC) | Array | Band |  $\nu$ (GHz) |  $F_{\nu}$ (uJy)
---|---|---|---|---
06-Oct-2020 17:19:14 | VLA-B | X | 10 | 65$\pm$7
15-Oct-2020 23:15:51 | VLA-B | X | 11 | 61$\pm$6
|  | X | 9 | 68$\pm$10
|  | C | 4.55 | 40$\pm$13
|  | C | 5.04 | 47$\pm$16
|  | C | 6.13 | 51$\pm$63
|  | C | 7.6 | 63$\pm$12
17-Dec-2020 13:35:07 | VLA-A | Ku | 16.5 | 98.8$\pm$10
|  | Ku | 13.5 | 137$\pm$10
|  | X | 11 | 137.0$\pm$9.5
|  | X | 9 | 134$\pm$8
|  | C | 7.5 | 139$\pm$11
|  | C | 6.5 | 134$\pm$15
|  | C | 5.5 | 91$\pm$12
|  | C | 4.5 | 75$\pm$11
|  | S | 3.76 | 60$\pm$13
|  | S | 3.24 | 53$\pm$25
03-Jun-2021 00:52:34 | VLA-C->D | X | 11 | 139$\pm$11
|  | X | 9 | 198$\pm$9
|  | C | 7.5 | 175$\pm$14
|  | C | 6.5 | 241$\pm$16
|  | C | 5.5 | 256$\pm$17
|  | C | 4.5 | 252$\pm$17
|  | S | 3.5 | 228$\pm$27
|  | S | 2.5 | 251$\pm$100
14-Aug-2021 17:42:51 | MeerKAT | L | 1.28 | 94$\pm$15
11-May-2022 22:09:16 | MeerKAT | L | 1.28 | 141$\pm 18$
23-Jun-2022 14:20:56 | uGMRT | L | 1.26 | 141$\pm$29
| uGMRT | P | 0.65 | $<261$

All VLA data were reduced in the Common Astronomy Software package (CASA
5.6.3, McMullin et al., 2007) following standard procedures using the VLA
pipeline. For all observations, 3C 286 was used for flux density calibration,
J1609+2641 was used for phase calibration for frequency ranges 2–12 GHz (S, C,
and X-band), and J1619+2247 was used for phase calibration for 12–18 GHz (Ku-
band). To extract the source flux density, images of the target field were
made using the CASA task tclean and the flux density was measured in the image
plane by fitting an elliptical Gaussian point source fixed to the size of the
synthesised beam using the CASA task imfit. We split each frequency band into
2 or 4 sub-bands depending on the bandwidth available after radio frequency
interference (RFI) flagging.

###  2.2 MeerKAT

We observed AT2020opy with MeerKAT during an observing run on 2021 Aug. 14 and
2021 May 11. We used the “4K” (4096-channel) wideband continuum mode and
observed with bandwidth of 856 MHz around a central frequency of 1.28 GHz,
over a total time of about 3.7 hr of which $\sim$1 hr was spent on-source for
AT2020opy.

The data were reduced using the OxKAT scripts (Heywood, 2020). We used
observations of 3C 286 (ICRF J133108.2+303032) to set the flux density scale
and calibrate the bandpass, and PKS J1609+2641 (ICRF J160913.3+264129) as a
secondary calibrator. The final images were made using the WSClean
($w$-stacking CLEAN) imager (Offringa et al., 2014; Offringa & Smirnov, 2017),
and had a restoring beam of $12.9\arcsec\ \times 5.2\arcsec$ at $-20\deg$. We
obtained the final flux densities by fitting elliptical Gaussians to the
image. Since there is a relatively nearby confusing source with a flux density
similar to that of AT2020opy, about 8″ to the southwest, we simultaneously
fitted two elliptical Gaussians, one for AT2020opy and one for the confusing
source, along with a zero-level to account for any constant offsets in the
flux of the image. Our value for the flux density of AT2020opy is the flux
density of the fitted Gaussian, and the uncertainty includes both the
statistical uncertainty and a systematic one due to the uncertainty in the
flux-density bootstrapping, estimated at 5%.

###  2.3 uGMRT

We observed AT2020opy with the upgraded Giant Metrewave Telescope (uGMRT) on
2022 June 23 at band 4 (total bandwidth of 300 MHz with a central frequency of
0.65 GHz) and band 5 (total bandwidth of 460 MHz with a central frequency of
1.26 GHz) over a total time of 3 hr, with 51 min on target at band 4 and 34
min on target at band 5. Each frequency band was broken into 2048 spectral
channels. Data reduction was carried out in CASA using standard procedures
including flux and bandpass calibration with 3C286 and phase calibration with
ICRF J160913.3+264129. Images of the target field were created using the CASA
task tclean. Two phase only and three phase and amplitude rounds of self-
calibration were carried out on the band 4 data. As with the VLA images, the
target flux density was extracted in the image plane at both bands using the
CASA task imfit. Unfortunately no detection of the source was obtained in the
band 4 observation due to a nearby bright source on the edge of the primary
beam causing a high image rms. We instead report the 3$\sigma$ upper limit
that was obtained at 0.65 GHz. A 5$\sigma$ detection of the source was
obtained at 1.26 GHz (band 5) and reported in Table 1.

###  2.4 Archival radio observations

To explore the possibility of previous AGN activity in the host galaxy, we
searched the radio archives for observations covering the coordinates of
AT2020opy. The VLA Sky Survey (VLASS, Lacy et al., 2020) observed the
coordinates of AT2020opy at 3 GHz on 2020 July 16 and 2017 September 25 (35
months pre-optical flare). There was no detection of the host galaxy in either
observation, with 3$\sigma$ upper limits of 490$\,\mu$Jy and 340$\,\mu$Jy
respectively. The NRAO VLA Sky Survey (NVSS, Condon et al., 1998) also
observed the coordinates of AT2020opy at 1.4 GHz on 1995 February 28, but did
not detect the host galaxy with a 3$\sigma$ upper limit of 2.1 mJy. These
observations rule out the possibility of bright ($>300\,\mu$Jy) AGN activity
in the host galaxy in the past 20 yr, but we cannot eliminate the possibility
of the galaxy hosting a low luminosity AGN.

##  3 Results

The VLA radio lightcurve at 5.5 GHz for AT2020opy compared to other radio-
bright thermal TDEs is shown in Figure 1 and the broadband radio spectra for
each of our three epochs are plotted in Figure 2.

Figure 1: Radio spectral luminosity of AT2020opy at 5.5 GHz compared to a
selection of other radio-bright thermal TDEs. AT2020opy is the most radio
luminous TDE at early times found to date. TDE data are from AT2019azh Goodwin
et al. (2022); AT2019dsg Cendes et al. (2021); ASASSN-14li Alexander et al.
(2016); ASASSN-15oi Horesh et al. (2021a); AT2018hyz Cendes et al. (2022);
CNSS J0019+00 Anderson et al. (2020); XMMSL1 J0740-85 Alexander et al. (2017);
IGR J12850+0134 Perlman et al. (2022), Lei et al. (2016), and Nikołajuk &
Walter (2013); iPTF 16fnl Horesh et al. (2021b).

AT2020opy appears brighter than other thermal TDEs at early times relative to
the outflow launch date, with luminosity $\nu L_{\nu}\approx 5\times 10^{38}$
erg s-1, but is not as luminous as the relativistic event Swift J1644+57
($L_{\nu}\approx 2\times 10^{45}$ erg s-1, Zauderer et al., 2011). The radio
emission from AT2020opy is well-described by a peaked synchrotron spectrum
that evolves on timescales of months, consistent with an outflow travelling
through the circumnuclear medium surrounding the SMBH and accelerating
electrons along magnetic field lines in the resulting shock front.

###  3.1 Synchrotron spectral fitting

We fit the synchrotron spectra of AT2020opy using the same approach outlined
in Goodwin et al. (2022). We apply the Granot & Sari (2002) model assuming the
synchrotron self-absorption frequency is associated with the peak of the
spectrum and $\nu_{\rm{m}}<\nu_{\rm{a}}<\nu_{\rm{c}}$, where $\nu_{\rm{m}}$ is
the synchrotron minimum frequency, $\nu_{\rm{a}}$ is the synchrotron self-
absorption frequency, and $\nu_{\rm{c}}$ is the synchrotron cooling frequency.
This approach enables the total spectral flux density to be modelled as a
function of frequency in order to constrain the break frequencies and electron
energy index, $p$. We assume no contribution to the radio emission from the
host galaxy due to no previous radio detections of the host in archival
observations indicating that the dominant contribution to the radio emission
is the synchrotron transient component, and because earlier observations of
the transient were significantly fainter than later observations.

As in Goodwin et al. (2022), we use a Python implementation of Markov Chain
Monte Carlo (MCMC), emcee (Foreman-Mackey et al., 2013) to marginalise over
the synchrotron model parameters to determine the best fit parameters and
uncertainties. Due to the paucity of the data at high frequencies, we fix the
synchrotron energy index to $p=2.7$ (e.g. Cendes et al., 2021), but note that
the derived parameters do not deviate significantly from the 1$\sigma$
uncertainty ranges if we instead choose other reasonable values, such as
$p=2.5$ or $p=3$. Furthermore, $p=2.7$ is the best-fit spectral index when we
allow $p$ to be a free parameter while fitting the third epoch in which the
optically thin slope is best-constrained by the data.

The observed and modelled synchrotron spectra for AT2020opy are plotted in
Figure 2, and the best-fitting peak flux density and peak frequency for each
epoch are listed in Table 2. The synchrotron peak flux density rose
consistently between the three epochs and the peak frequency decreased between
the epochs.

Figure 2: Radio spectra of the TDE AT2020opy obtained with the VLA (circles)
and MeerKAT (star) telescopes and the median synchrotron spectra for the
corresponding epochs (solid lines). 50 random samples from the MCMC fitting
are plotted for each spectrum to demonstrate the uncertainty in the fits. The
radio emission from AT2020opy is well-fit by an evolving synchrotron spectrum
where the peak is associated with synchrotron self-absorption in each epoch.

###  3.2 Outflow modelling

We model the radio outflow based on the inferred synchrotron emission
properties using the same approach outlined in Goodwin et al. (2022), in which
following the model of Barniol Duran et al. (2013) we assume the ambient
electrons are accelerated into a power-law distribution by the blastwave from
the outflow, $N(\gamma)\propto\gamma^{-p}$, where $\gamma$ is the electron
Lorentz factor. In order to estimate the outflow radius, energy, magnetic
field strength, and velocity, we assume equipartition between the electron and
magnetic field energy densities, which enables the derivation of an
equipartition radius and energy. Once the equipartition radius and energy are
obtained, we then parameterise the deviation from equipartition to derive the
total energy and radius, from which other parameters can be derived. We refer
the reader to equations 4–13 of Goodwin et al. (2022) for the specific
equations also used in this work. To account for different outflow mechanisms,
we model two geometries of the outflow: a spherical outflow and a mildly
collimated conical outflow with a half opening angle of 30°. We note that we
model the outflow as non-relativistic (bulk Lorentz factor $\Gamma=1$), as a
relativistic outflow is only possible for very small ($\lesssim 0.1$ deg)
opening angles. The estimated physical outflow properties for AT2020opy are
plotted in Figure 3 and listed in Table 2 for each of these geometries.

Table 2: Synchrotron spectral fitting parameters and outflow model predictions
for AT2020opy. Fix values in table

|  $\delta t$ (d) |  $\nu_{\mathrm{peak}}$ (GHz) |  $F_{\rm peak}$ (mJy) | log10 $R$ (cm) | log10 $E$ (erg) | $\beta$ | log10 $B$ (G) | log10 $n_{e}$ (cm-3)
---|---|---|---|---|---|---|---|---
| 50 | $9.6\pm 1.2$ | $0.070\pm 0.007$ | $16.2\pm 0.1$ | $48.9\pm 0.3$ | $0.12\pm 0.04$ | $-0.4\pm 0.9$ | $4.7\pm 1.3$
Spherical | 116 | $9.4\pm 0.5$ | $0.141\pm 0.006$ | $16.3\pm 0.1$ | $49.3\pm 0.3$ | $0.08\pm 0.02$ | $-0.5\pm 0.7$ | $4.6\pm 1.2$
| 281 | $4.2\pm 0.4$ | $0.28\pm 0.02$ | $16.8\pm 0.1$ | $50.0\pm 0.3$ | $0.10\pm 0.03$ | $-0.9\pm 0.7$ | $3.8\pm 1.2$
| 50 | $9.6\pm 1.2$ | $0.070\pm 0.007$ | $16.6\pm 0.1$ | $49.4\pm 0.3$ | $0.24\pm 0.08$ | $-0.7\pm 0.9$ | $4.1\pm 1.3$
Conical | 116 | $9.4\pm 0.5$ | $0.141\pm 0.006$ | $16.7\pm 0.1$ | $49.8\pm 0.3$ | $0.16\pm 0.05$ | $-0.7\pm 0.7$ | $4.0\pm 1.2$
| 281 | $4.2\pm 0.4$ | $0.28\pm 0.02$ | $17.2\pm 0.1$ | $50.5\pm 0.3$ | $0.2\pm 0.06$ | $-1.1\pm 0.7$ | $3.3\pm 1.2$

Note: $\delta t$ is reported with reference to $t_{0}$, the estimated outflow
launch date of MJD 59087.

Figure 3: Inferred physical outflow properties from an equipartition analysis
of three epochs of the radio synchrotron emission from AT2020opy. Black
circles indicate parameters assuming a spherical homogeneous outflow, grey
circles indicate parameters assuming a collimated, conical outflow. The dashed
lines show a linear fit to the radius for both the spherical (black) and
conical (grey) geometries, which give an estimated outflow launch date of MJD
59087$\pm$41 d.

The radius increased approximately linearly with time, indicating
approximately constant velocity of the outflow. A simple linear fit to the
radius (Figure 3) gives an outflow launch date of MJD=59087$\pm$41 d or
MJD=59088$\pm$43 d for spherical and conical geometries, respectively; 50 d
after the optical flare was first observed. This predicted outflow launch date
is coincident with the optical peak, on MJD 59070, and is also coincident
within 2-$\sigma$ of the initial optical flare on MJD=59038.

The energy of the outflow increased approximately linearly with time, as is
expected for an increasing synchrotron peak flux density, which could be
indicative of constant energy injection into the outflow. The velocity and
magnetic field strength remained approximately constant at 0.1 $c$ (0.2 $c$)
and 0.28 G (0.15 G) for the spherical (conical) geometries, with no sign of
relativistic motion of the outflow. We note that the inferred radius, energy,
and ambient density are consistent with having remained constant between the
first and second epochs of observations at 50 and 116 d post outflow launch.
However, due to the paucity of data in the first epoch and the large
uncertainties in the resulting spectral fits, we deduce that it is more likely
that the outflow was evolving between these two epochs, as there is signficant
evolution between the first and last epochs.

##  4 Discussion

The radio properties of the TDE AT2020opy indicate a non-relativistic outflow
was launched at the time of or just after the initial optical flare. We deduce
that the outflow has an approximately constant velocity with $\beta\approx
0.1\,c$ and energy $\sim 10^{48}$ erg for radii $\sim 10^{16}$ cm. Between
2020 October and 2021 June the radio emission from AT2020opy was increasing in
peak flux density and the peak frequency of the synchrotron spectrum was
decreasing, consistent with a constant velocity outflow moving through the CNM
and sweeping up material.

Hammerstein et al. (2022) analysed the optical spectra of AT2020opy and
classified the event as a TDE with broad H$\alpha$ and H$\beta$ emission lines
as well as a complex of He II emission lines (H+He spectral class) and a
structured optical lightcurve with some flaring activity. In an analysis of 30
TDEs, Hammerstein et al. (2022) found some evidence that TDEs with structured
lightcurves tend to occur in galaxies with lower total mass, and thus could
occur around lower mass SMBHs. We note that this finding is strongly dependent
on an assumption of a relationship between total stellar mass in the host
galaxy and SMBH mass. Hammerstein et al. (2022) propose that the structured
flaring activity seen in the lightcurves of these TDEs could be due to longer
circularisation times of the lower mass SMBHs. A longer circularisation time
of the disk for AT2020opy could explain the lack of early X-ray emission from
the event (Gezari et al., 2020), as well as a radio outflow that was launched
after the initial optical flare.

###  4.1 The outflow mechanism

The radio measurements that enable determinations about the physical
properties of the outflow produced in AT2020opy enable some discrimination
between current models of non-relativistic outflows in TDEs. Firstly, the data
indicate that the outflow in this event is likely to have been launched
approximately 50 d after the initial optical flare, however we cannot rule out
a contemporaneous launch of the outflow at a high degree of confidence (we
infer it was launched at least 8 d after to 1$\sigma$). In comparison, the
radio outflows observed for the thermal TDEs AT2019azh and ASASSN-14li were
inferred to have been launched at the time of the initial optical flare
(Goodwin et al., 2022; Alexander et al., 2016; van Velzen et al., 2016).
Secondly, the observed velocity of the outflow is approximately constant
(under the assumption of ballistic motion) and the radio emission cannot be
explained by a relativistic outflow unless the jet has an unphysically small
opening angle. Hammerstein et al. (2022) found some evidence that TDEs with
structured lightcurves occur in lower mass host galaxies, which could
translate to lower mass central black holes with longer circularisation times,
making stream-stream collisions more important. A long circularisation time of
the stellar debris for AT2020opy could explain a delayed onset of the radio
outflow if the outflow was produced by either a disk wind or debris
collisions. Finally, the lack of X-ray emission from the event at early times
(Gezari et al., 2020) could either be due to intrinsically low X-ray activity
due to the accretion disk taking a long time to form, making the super-
Eddington accretion induced wind outflow scenario unlikely, or due to the
large distance to the source.

We thus conclude that the outflow from AT2020opy is more likely to be
explained by a spherical outflow from stream-stream collisions of the
circularising stellar debris (Lu & Bonnerot, 2020) than by an accretion
induced wind outflow from accretion onto the SMBH (e.g. Alexander et al.,
2016). We deduce that the unbound debris stream is unlikely to explain the
radio properties of this outflow due to the low mass predicted in the outflow
($\lesssim 10^{-2}M_{\odot}$) and predicted small opening angle of the unbound
debris stream (e.g. Guillochon et al., 2014).

Under the assumption that the outflow was produced by a collision induced
outflow (CIO) or accretion induced wind, the increasing energy with constant
velocity of the radio outflow is consistent with a single injection of energy
into the outflow that is sweeping up material from the CNM (Lu & Bonnerot,
2020). Under such an assumption, we can approximately infer the deceleration
time of the outflow using the model from Lu & Bonnerot (2020), where for
$k=2$, the deceleration radius is given by

$r_{\mathrm{dec}}=\frac{1}{\Omega}\frac{2E_{\rm k}}{Nm_{\rm p}v_{0}^{2}}$ (1)

where the outflow is assumed to be a thin shell covering solid angle
$\Omega\sim 2\pi$, $E_{\rm k}$ is the kinetic energy in the outflow, $m_{\rm
p}$ is the proton mass, $N$ is the electron density, and $v_{0}$ is the
average outflow velocity.

For $E_{\rm k}\sim 5\times 10^{51}$ ($10^{52})$ erg, $v_{0}=0.1\,c$, and
$N=10^{3}$ cm-3, we infer $r_{\mathrm{dec}}\approx 10^{17}$ cm ($2\times
10^{17}$ cm), and thus $t_{\mathrm{dec}}\approx$390 d (780 d). At 281 d we
observed the radio outflow to still be expanding and increasing in
luminosity/energy(Figure 3), and at 623–666 d we observed the radio emission
to still be increasing/constant in luminosity at 1.25 GHz (Figure 4). We thus
predict that the outflow could still be increasing in luminosity until 2022
November, depending on the kinetic energy available in the outflow. Further
radio observations of the event would help constrain the deceleration
radius/time of the outflow and enable further discrimination between outflow
models. Importantly, we note that for this kind of outflow, the radio emission
can continue to increase for up to years after the initial event, depending on
the energy available in the outflow and the density of the CNM. For AT2020opy,
the observed radio emission is more luminous likely due to a denser CNM, and
thus may peak earlier than other thermal TDEs.

Figure 4: Radio lightcurve of AT2020opy at 1.25, 3.5, 5.5, and 9 GHz. During
the first $\approx 300$ d post-disruption the radio flux density rose at all
frequencies. At 623–666 d post-disruption the radio flux density appears to
have plateaued at 1.25 GHz, indicating that the outflow may have begun to
decelerate.

###  4.2 Comparison with other TDEs

A comparison of the inferred outflow properties for AT2020opy with other
radio-bright TDEs is shown in Figure 5. AT2020opy clearly fits into the
population of non-relativistic events in terms of energy, velocity, and radius
from the central black hole.

Figure 5: Left: The scaled radius and ambient density for a selection of TDEs.
Right: The kinetic energy and outflow velocity for a selection of TDEs. TDE
data and assumed SMBH masses are from Cendes et al. (2021); Stein et al.
(2021) (AT2019dsg, $M_{\mathrm{BH}}=5\times 10^{6}$ $M_{\mathrm{\odot}}$),
Alexander et al. (2016) (ASASSN-14li, $M_{\mathrm{BH}}=1\times 10^{6}$
$M_{\mathrm{\odot}}$), Eftekhari et al. (2018) (Sw J1644+57,
$M_{\mathrm{BH}}=1\times 10^{6}$ $M_{\mathrm{\odot}}$), Anderson et al. (2020)
(CNSS J0019+00, $M_{\mathrm{BH}}=1\times 10^{7}$ $M_{\mathrm{\odot}}$),
Mattila et al. (2018) (Arp 299-B AT1, $M_{\mathrm{BH}}=2\times 10^{7}$
$M_{\mathrm{\odot}}$), Alexander et al. (2017) (XMMSL1 J0740-85,
$M_{\mathrm{BH}}=3.5\times 10^{6}$ $M_{\mathrm{\odot}}$), and Goodwin et al.
(2022) (AT2019azh, $M_{\mathrm{BH}}=3\times 10^{6}$ $M_{\mathrm{\odot}}$). For
AT2020opy we assume $M_{\mathrm{BH}}=1.12\times 10^{7}$ $M_{\mathrm{\odot}}$
(Hammerstein et al., 2022). $R_{s}$ is the Schwarzschild radius of the black
hole and Req is the predicted equipartition radius of the outflow. AT2020opy
fits well into the population of thermal TDEs, with a slightly higher density
at radii closer to the SMBH.

The ambient density is approximately proportional to $n\propto
R^{-1.5}$–$R^{-2.5}$ for AT2020opy, similar to other thermal TDEs. The CNM
density of AT2020opy appears to be $\sim 30\%$ denser than any of the other
TDEs observed to date, which could explain the higher luminosity (and higher
distance) of the radio emission from the event. Interestingly, AT2020opy is
also the most distant of the thermal TDEs with detected radio emission
reported, implying that for more energetic events radio emission may be
observed from further away. The inferred CNM density and radio luminosity of
AT2020opy further confirm that for galaxies with higher CNM densities, outflow
emission rises more quickly and is brighter than in galaxies with less-dense
CNMs (Lu & Bonnerot, 2020).

The delay relative to the optical flare in radio emission observed from
AT2020opy was not large, similar to the two other thermal TDEs that have been
observed in the radio-rise phase AT2019azh (Goodwin et al., 2022) and
AT2019dsg (Stein et al., 2021; Cendes et al., 2021), in contrast to the late-
time radio flare that was observed from ASASSN-15oi (Horesh et al., 2021a).
Our modelling constrains the onset of the radio outflow in AT2020opy to be
consistent with the time of or just after the optical flare was observed.

##  5 Conclusions

We followed the radio evolution of the tidal disruption event AT2020opy for 20
months with the VLA, MeerKAT, and uGMRT radio telescopes. Based on modelling
of the synchrotron emission observed, we find that the radio emission is
likely due to a non-relativistic outflow, which could take the form of a
spherical wind, collision induced outflow or a mildly collimated jet. We note
that based on modelling of the evolution of the radius, we find the outflow
was launched after or around the time that the initial optical flare was
observed. Through synchrotron spectral modelling of the radio emission, we
deduce that the circumnuclear medium of the host galaxy is denser than
inferred for other TDE hosts, which causes brighter, quickly rising radio
emission from the outflow.

Follow-up observations of this event are encouraged to continue to observe the
long-term decay of the radio emission, which we predict will reach a peak
luminosity at 390-780 d post-optical flare (up to 2022 November).

## Acknowledgements

The authors thank K. Alexander, N. Blagorodnova, P. Woudt, M. Bottcher, R.
Fender, J. Bright, and S. Kulkarni for their contributions to the observing
proposals that were instrumental to this work. This work was supported by the
Australian government through the Australian Research Council’s Discovery
Projects funding scheme (DP200102471). A.H. is grateful for the support by the
I-Core Program of the Planning and Budgeting Committee and the Israel Science
Foundation, and support by ISF grant 647/18. A.H. is grateful for support by
the Zelman Cowen Academic Initiatives. GRS is supported by NSERC Discovery
Grants RGPIN-2016-06569 and RGPIN-2021-0400. The National Radio Astronomy
Observatory is a facility of the National Science Foundation operated under
cooperative agreement by Associated Universities, Inc. The MeerKAT telescope
is operated by the South African Radio Astronomy Observatory, which is a
facility of the National Research Foundation, an agency of the Department of
Science and Innovation. We thank the staff of the GMRT that made these
observations possible. GMRT is run by the National Centre for Radio
Astrophysics of the Tata Institute of Fundamental Research.

## Data Availability

The spectral fitting and equipartition modelling software used in this work is
publicly available on Github at https://github.com/adellej/tde_spectra_fit.

## Software

This research made use of Matplotlib, a community-developed Python package
(Hunter, 2007), NASA’s Astrophysics Data System Bibliographic Services, the
Common Astronomy Software Application package CASA (McMullin et al., 2007),
The Cube Analysis and Rendering Tool for Astronomy (CARTA Comrie et al., 2021)
and the Python packages cmasher (van der Velden, 2020), and emcee (Foreman-
Mackey et al., 2013).

## References

  * Alexander et al. (2016) Alexander K. D., Berger E., Guillochon J., Zauderer B. A., Williams P. K. G., 2016, ApJ, 819, L25
  * Alexander et al. (2017) Alexander K. D., Wieringa M. H., Berger E., Saxton R. D., Komossa S., 2017, ApJ, 837, 153
  * Alexander et al. (2020) Alexander K. D., van Velzen S., Horesh A., Zauderer B. A., 2020, Space Sci. Rev., 216, 81
  * Anderson et al. (2020) Anderson M. M., et al., 2020, ApJ, 903, 116
  * Auchettl et al. (2017) Auchettl K., Guillochon J., Ramirez-Ruiz E., 2017, ApJ, 838, 149
  * Barniol Duran et al. (2013) Barniol Duran R., Nakar E., Piran T., 2013, ApJ, 772, 78
  * Bloom et al. (2011) Bloom J. S., et al., 2011, Science, 333, 203
  * Bonnerot & Lu (2020) Bonnerot C., Lu W., 2020, MNRAS, 495, 1374
  * Bonnerot et al. (2016) Bonnerot C., Rossi E. M., Lodato G., Price D. J., 2016, MNRAS, 455, 2253
  * Burrows et al. (2011) Burrows D. N., et al., 2011, Nature, 476, 421
  * Cendes et al. (2021) Cendes Y., Alexander K. D., Berger E., Eftekhari T., Williams P. K. G., Chornock R., 2021, ApJ, 919, 127
  * Cendes et al. (2022) Cendes Y., et al., 2022, arXiv e-prints, p. arXiv:2206.14297
  * Comrie et al. (2021) Comrie A., et al., 2021, CARTA: The Cube Analysis and Rendering Tool for Astronomy, Zenodo, doi:10.5281/zenodo.3377984
  * Condon et al. (1998) Condon J. J., Cotton W. D., Greisen E. W., Yin Q. F., Perley R. A., Taylor G. B., Broderick J. J., 1998, AJ, 115, 1693
  * Eftekhari et al. (2018) Eftekhari T., Berger E., Zauderer B. A., Margutti R., Alexander K. D., 2018, ApJ, 854, 86
  * Foreman-Mackey et al. (2013) Foreman-Mackey D., Hogg D. W., Lang D., Goodman J., 2013, PASP, 125, 306
  * Gezari et al. (2020) Gezari S., Perley D., Sollerman J., 2020, The Astronomer’s Telegram, 13944, 1
  * Goodwin et al. (2022) Goodwin A. J., et al., 2022, MNRAS, 511, 5328
  * Granot & Sari (2002) Granot J., Sari R., 2002, ApJ, 568, 820
  * Guillochon et al. (2014) Guillochon J., Manukian H., Ramirez-Ruiz E., 2014, ApJ, 783, 23
  * Hammerstein et al. (2022) Hammerstein E., et al., 2022, arXiv e-prints, p. arXiv:2203.01461
  * Hayasaki et al. (2016) Hayasaki K., Stone N., Loeb A., 2016, MNRAS, 461, 3760
  * Heywood (2020) Heywood I., 2020, oxkat: Semi-automated imaging of MeerKAT observations (ascl:2009.003)
  * Hinkle et al. (2021) Hinkle J. T., et al., 2021, MNRAS, 500, 1673
  * Horesh et al. (2021a) Horesh A., Cenko S. B., Arcavi I., 2021a, Nature Astronomy, 5, 491
  * Horesh et al. (2021b) Horesh A., Sfaradi I., Fender R., Green D. A., Williams D. R. A., Bright J. S., 2021b, ApJ, 920, L5
  * Hunter (2007) Hunter J. D., 2007, Computing in Science & Engineering, 9, 90
  * Krolik et al. (2016) Krolik J., Piran T., Svirski G., Cheng R. M., 2016, ApJ, 827, 127
  * Lacy et al. (2020) Lacy M., et al., 2020, PASP, 132, 035001
  * Lei et al. (2016) Lei W.-H., Yuan Q., Zhang B., Wang D., 2016, ApJ, 816, 20
  * Levan et al. (2011) Levan A. J., et al., 2011, Science, 333, 199
  * Liptai et al. (2019) Liptai D., Price D. J., Mandel I., Lodato G., 2019, arXiv e-prints, p. arXiv:1910.10154
  * Lodato & Rossi (2011) Lodato G., Rossi E. M., 2011, MNRAS, 410, 359
  * Lu & Bonnerot (2020) Lu W., Bonnerot C., 2020, MNRAS, 492, 686
  * Mattila et al. (2018) Mattila S., et al., 2018, Science, 361, 482
  * McMullin et al. (2007) McMullin J. P., Waters B., Schiebel D., Young W., Golap K., 2007, in Shaw R. A., Hill F., Bell D. J., eds, Astronomical Society of the Pacific Conference Series Vol. 376, Astronomical Data Analysis Software and Systems XVI. p. 127
  * Miller et al. (2015) Miller J. M., et al., 2015, Nature, 526, 542
  * Mummery & Balbus (2020) Mummery A., Balbus S. A., 2020, MNRAS, 492, 5655
  * Nikołajuk & Walter (2013) Nikołajuk M., Walter R., 2013, A&A, 552, A75
  * Offringa & Smirnov (2017) Offringa A. R., Smirnov O., 2017, MNRAS, 471, 301
  * Offringa et al. (2014) Offringa A. R., et al., 2014, MNRAS, 444, 606
  * Perlman et al. (2022) Perlman E. S., et al., 2022, ApJ, 925, 143
  * Rees (1988) Rees M. J., 1988, Nature, 333, 523
  * Roth et al. (2020) Roth N., Rossi E. M., Krolik J., Piran T., Mockler B., Kasen D., 2020, Space Sci. Rev., 216, 114
  * Sfaradi et al. (2022) Sfaradi I., Horesh A., Fender R., Green D. A., Williams D. R. A., Bright J., Schulze S., 2022, arXiv e-prints, p. arXiv:2202.00026
  * Stein et al. (2021) Stein R., et al., 2021, Nature Astronomy, 5, 510
  * Zauderer et al. (2011) Zauderer B. A., et al., 2011, Nature, 476, 425
  * van Velzen et al. (2016) van Velzen S., et al., 2016, Science, 351, 62
  * van der Velden (2020) van der Velden E., 2020, Journal of Open Source Software, 5, 2004