# OnlineRefer: A Simple Online Baseline for Referring Video Object
Segmentation

Dongming Wu1‡, Tiancai Wang2, Yuang Zhang3, Xiangyu Zhang2,4, Jianbing Shen5†
1 Beijing Institute of Technology, 2 MEGVII Technology, 3 Shanghai Jiao Tong
University,
4 Beijing Academy of Artificial Intelligence, 5 SKL-IOTSC, CIS, University of
Macau
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Referring video object segmentation (RVOS) aims at segmenting an object in a
video following human instruction. Current state-of-the-art methods fall into
an offline pattern, in which each clip independently interacts with text
embedding for cross-modal understanding. They usually present that the offline
pattern is necessary for RVOS, yet model limited temporal association within
each clip. In this work, we break up the previous offline belief and propose a
simple yet effective online model using explicit query propagation, named
OnlineRefer. Specifically, our approach leverages target cues that gather
semantic information and position prior to improve the accuracy and ease of
referring predictions for the current frame. Furthermore, we generalize our
online model into a semi-online framework to be compatible with video-based
backbones. To show the effectiveness of our method, we evaluate it on four
benchmarks, _i.e_., Refer-Youtube-VOS, Refer-DAVIS17, A2D-Sentences, and
JHMDB-Sentences. Without bells and whistles, our OnlineRefer with a Swin-L
backbone achieves 63.5 J&F and 64.8 J&F on Refer-Youtube-VOS and Refer-
DAVIS17, outperforming all other offline methods. Our code is available at
https://github.com/wudongming97/OnlineRefer. ††$\dagger$Corresponding author:
Jianbing Shen. This work was supported in part by the FDCT grants 0154/2022/A3
and SKL-IOTSC(UM)-2021-2023, the MYRG-CRG2022-00013-IOTSC-ICI grant and the
SRG2022-00023-IOTSC grant. $\ddagger$The work is done during the internship at
MEGVII Technology.

##  1 Introduction

Given a natural language expression, the purpose of referring video object
segmentation (RVOS) is to segment the described object in a streaming video.
The emerging task has attracted great attention in the computer vision
community as it provides potential benefits for many applications, _e.g_.,
video editing and human-computer interaction. Its core challenge is
associating all frames with constructing an efficient video representation,
further promoting cross-modal understanding of two modalities, _i.e_., video
and language. Pioneer methods [11, 27] integrate mask propagation into the
referring image segmentation in an online manner, as shown in Fig. 1 (a).
However, the complexity and performance of their model remain far from
satisfactory.

Figure 1: Conceptual comparison on current methods: (a) the mask-propagation
method [11, 27], (b) the offline method [9, 1, 32], and (c) our query-
propagation method.  Figure 2: Visualization of query references and
corresponding results of (a) query-sharing method and (b) our OnlineRefer. The
reference points/boxes are marked red, while the final predictions of mask and
box are marked green.

Recently, the state-of-the-art performance on RVOS has been dominated by
offline methods [29, 9, 1, 3, 12, 32, 42]. They typically follow a clip-level
paradigm, dividing the entire video into multiple non-overlapped clips and
generating referring object masks for each clip, as illustrated in Fig. 1 (b).
In terms of different inter-frame interaction ways within an individual clip,
existing offline methods [29, 9, 1, 3, 12, 32, 42] can be categorized into two
groups: feature association methods and query-sharing methods. The former
feature association methods [29, 9, 12, 30, 42] integrate multi-frame features
into a holistic clip-level visual representation, which is further fused with
text embedding for referring prediction. However, their temporal feature
modeling is commonly complicated and heavy-weighted.

In contrast to the feature association methods, the query-sharing methods [1,
32] provide a simplified pipeline as they build on the query-based Transformer
method [2, 43]. They first construct clip-level cross-modal features and then
use a set of repeated queries to retrieve the same referent object from
different frames. In other words, the cross-frame object correspondence relies
heavily on sharing input queries. The interaction between frames is typically
limited, hindering the association potential of the learned queries. Fig. 2
(a) shows a typical example: all video frames share the same reference points
(or queries), which misses the occluded object. In addition, due to resource
limitations, the referring prediction has to be performed separately on each
clip, which lacks inter-clip association.

In this paper, we propose a new and insightful online referring video object
segmentation framework, OnlineRefer. It goes beyond the intuition of the
online model not working well in RVOS. Its core idea is to take advantage of
the query-based set prediction in Deformable DETR [43] and link all video
frames via continuous query propagation. Specifically, we first provide a
powerful query-based referring segmentation pipeline, which outputs the
embedding representations of the referent object, further generating mask,
box, and category. As these outputs gather rich target information, we propose
a cross-frame query propagation module to transform them as new query inputs
of the next frame. The propagation process has three significant advantages.
First, the referring target is automatically associated with its precursors on
all previous frames. Second, the box information of the last frame provides a
very good spatial regional prior, benefiting the model for accurately
inferring the same object in the current frame (see an example in Fig. 2 (b)).
Third, our architecture avoids complicated temporal modeling or limited cross-
frame association so that the overall training and inference progress is
smooth and effective. Thanks to the remarkable performance, we expect to
contribute the elegant and effective online model as a new baseline to the
community.

To summarize, our main contributions are three-fold:

  * •

We are the first to challenge the widespread belief that only offline models
can deal well with RVOS and make online RVOS great again.

  * •

We propose a simple yet solid online baseline based on query propagation. The
explicit association across video frames facilities temporal target matching
and improves referring prediction accuracy.

  * •

Our method is evaluated on four benchmarks: Refer-Youtube-VOS, Refer-DAVIS17,
A2D-Sentences, and JHMDB-Sentences, outperforming all previous offline methods
and achieving state-of-the-art performance.

##  2 Related Work

Figure 3: Overall architecture of our OnlineRefer. It consists of two main
components: query-based referring segmentation and cross-frame query
propagation. The query-based referring segmentation (§ 3.1) employs a set of
queries to predict the referring object. The outputs contain rich target-aware
information, so the cross-frame query propagation (§ 3.2) transforms them into
a new query of the next frame. By repeating the two parts, we complete the
RVOS task online frame-by-frame. OnlineRefer can also be generalized into a
semi-online baseline using a clip-by-clip manner (§ 3.3).

###  2.1 Referring Video Object Segmentation

Referring video object segmentation is to localize a text-referred object
using a mask. Earlier works [11, 27] used the spatial-temporal memory
mechanism [24], which stored the mask of the previous frames to promote
referring image segmentation [40, 38, 16]. Currently, however, most existing
methods [29, 28, 37, 9, 3, 12, 42, 30, 36] concentrated on designing offline
frameworks, _i.e_., clip-in and clip-out. For example, Hui  _et al_. [9]
proposed a two-stream network, one branch being a temporal encoder to
recognize the object motion and another branch being a spatial encoder to
generate accurate referring segmentation. Wu  _et al_. [30] additionally
considered an object-level branch with salient object regions to enhance the
foreground and background discriminability of the model. These methods
inevitably need a complicated spatial-temporal modeling module, which is not
trivial. Several parallel works [1, 32] employed simple query-based
Transformer models, which share the same query set across different frames.
Precisely, MTTR [1] followed an instance-level segmentation pipeline to
predict sequences of all instances and select the sequence that best fits the
referent object. ReferFormer [32] transformed the input expression as the
decoder queries for directly attending to the most relevant regions in the
video frames.

Although our approach employs the basic query-based prediction architecture,
there are two key differences. First, our model is entirely online and has an
inherent advantage in handling long or ongoing videos, while offline methods
fail due to the limitation of computation sources. Second, the target
information of previous frames is explicitly and effectively used to
strengthen cross-frame tracking and frame-wise referring segmentation,
achieving better results.

###  2.2 Query-based Online Models

Employing query to associate cross-frame objects has been recently explored in
several online models, such as TrackFormer [22], MOTR [41], IDOL [33], and
InsPro [6]. They show the effectiveness and potential of query-based object
association. However, they are different from our OnlineRefer from two
perspectives. On the one hand, OnlineRefer does not need to detect multiple
objects due to the guidance of language expression. TrackFormer [22] and MOTR
[41, 31] adopted an extra query subset to detect new-born objects. They
require additional heuristic rules to combine two types of queries _i.e_.,
track query and detect query. IDOL [33] designed a re-identification module as
post-processing to link instances between frames. InsPro [6] kept the fixed
query number and employed empty query to detect the new-born objects. These
query propagation methods are also evaluated in § 4.5, while our method
performs better. On the other hand, IDOL [33] and InsPro [6] designed
additional training strategies with contrast learning to avoid identification
switches or suppress duplicates. In contrast, our framework minimizes the gap
between training and inference of long videos because it avoids the heuristic
rules during the training stage.

##  3 Methodology

Given an input video and a natural language expression, our method aims to
output binary masks of the referred object in a streaming way. The overall
architecture of OnlineRefer is illustrated in Fig. 3. It comprises two
essential parts: query-based referring segmentation and cross-frame query
propagation. The query-based referring segmentation in § 3.1 is an advanced
referring segmentation pipeline conditioned on the query set. The cross-frame
query propagation in § 3.2 is to generate the input query set of the current
frame by updating the outputs from the last frame. In addition, for training
and inference on video-based backbones, OnlineRefer is extended into a semi-
online pattern, which propagates the query across video clips in § 3.3.

###  3.1 Query-based Referring Sementation

Similar to ReferFormer [32], our query-based referring segmentation mainly
follows the Deformable DETR detector [43], and we make several modifications
on it for referring object prediction. It accepts a video frame, a language
expression, and a set of learnable queries as input. Its outputs are the
target box, mask, category, and a set of output embeddings corresponding to
the expression.

In specific, given the $t^{th}$ frame $I_{t}\\!\in\\!\mathbb{R}^{3\times
H\times W}$ and its corresponding expression $S$, we separately utilize visual
and linguistic backbone to extract their features. The two features are mapped
into the same dimension and fed into an encoder to perform cross-modal fusion
using a cross-attention module before encoder layers. The generated cross-
modal features contain critical target awareness, represented by $\bm{M}_{t}$.
In the decoder, we define two types of queries: content query
$\bm{q}_{t}^{c}\\!\in\\!\mathbb{R}^{N_{t}\times d}$ and position query
(_i.e_., position embedding) $\bm{q}_{t}^{p}\\!\in\\!\mathbb{R}^{N_{t}\times
d}$, where $N_{t}$ is the number of queries. Here, the content query follows
the common usage of DETR [2], while the position query is transformed into
base values of output boxes, denoted as
$\bm{b}^{base}_{t}\\!\in\\!\mathbb{R}^{N_{t}\times 4}$, which decreases
prediction difficulty and benefits model convergence. After that, the decoder
transforms the queries and cross-modal features into output embedding
$\bm{E}_{t}\\!\in\\!\mathbb{R}^{N_{t}\times d}$ (see Fig. 4 for more details).

On top of the output embedding, a 3-layer feed-forward network (FFN) is used
to predict box offset $\bm{b}^{offset}_{t}\\!\in\\!\mathbb{R}^{N\times 4}$,
which add on the base box coordinate to formulate the final box predictions,
_i.e_., $\bm{b}_{t}\\!=\\!\bm{b}^{base}_{t}\\!+\bm{b}^{offset}_{t}$. Another
3-layer FFN generates class probabilities
$\bm{c}_{t}\\!\in\\!\mathbb{R}^{N_{t}\times c}$, where $c$ is the category
number. For per-frame mask generation, we first employ a cross-modal FPN [32]
to perform multi-scale interactions between linguistic features and visual
feature maps. A new FFN then encodes the output embedding into parameters of
the mask head, which performs three-layer $1\\!\times\\!1$ convolution on the
generated FPN feature map, producing mask
$\bm{m}_{t}\\!\in\\!\mathbb{R}^{N_{t}\times\frac{H}{4}\times\frac{W}{4}}$.

Since there is only one referent object in the video, we can find the best
prediction as positive sample by minimizing the matching cost between
predictions and ground truth:

$\small\mathcal{L}_{match}=\lambda_{cls}\mathcal{L}_{cls}+\lambda_{box}\mathcal{L}_{box}+\lambda_{mask}\mathcal{L}_{mask},$
(1)

where $\mathcal{L}_{cls}$ is the class-related loss using the focal loss [15].
$\mathcal{L}_{box}$ represents the box-related loss that combines
$\mathcal{L}_{1}$ loss and GIoU loss [26]. $\mathcal{L}_{mask}$ is the mask-
related loss that sums up DICE loss [23] and binary mask focal loss.
$\lambda_{cls}$, $\lambda_{box}$ and $\lambda_{box}$ are the corresponding
loss coefficients. After completing the matching, we optimize the network
using the loss function $\mathcal{L}_{match}$ for positive samples while
letting negative samples predict the $\varnothing$ class.

Figure 4: Illustration of cross-frame query propagation. The propagated
representations consist of three cues: the output boxes, output embeddings,
and position embeddings.

Method | Backbone | Online/Offline | Refer-Youtube-VOS |  Refer-DAVIS17
---|---|---|---|---
$\mathcal{J}$&$\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ |  $\mathcal{J}$&$\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$
CMSA [38] | ResNet-50 | online | 34.9 | 33.3 | 36.5 | 34.7 | 32.1 | 37.2
CMSA + RNN [38] | ResNet-50 | online | 36.4 | 34.7 | 38.0 | 40.2 | 36.9 | 43.4
URVOS [27] | ResNet-50 | online | 47.2 | 45.2 | 49.1 | 51.6 | 47.2 | 55.9
MLSA [30] | ResNet-50 | offline | 49.7 | 48.4 | 50.9 | 57.9 | 53.8 | 62.0
ReferFormer [32] | ResNet-50 | offline | 55.6 | 54.8 | 56.5 | 58.5 | 55.8 | 61.3
OnlineRefer | ResNet-50 | online | 57.3 | 55.6 | 58.9 | 59.3 | 55.7 | 62.9
PMINet + CFBI [4] | Ensemble | offline | 54.2 | 53.0 | 55.5 | - | - | -
CITD [14] | Ensemble | offline | 61.4 | 60.0 | 62.7 | - | - | -
ReferFormer [32] | Swin-L | offline | 62.4 | 60.8 | 64.0 | 60.5 | 57.6 | 63.4
OnlineRefer | Swin-L | online | 63.5 | 61.6 | 65.5 | 64.8 | 61.6 | 67.7
MTTR ($\omega$=12) [1] | Video-Swin-T | offline | 55.3 | 54.0 | 56.6 | - | - | -
ReferFormer ($\omega$=5) [32] | Video-Swin-T | offline | 59.4 | 58.0 | 60.9 | - | - | -
ReferFormer ($\omega$=5) [32] | Video-Swin-B | offline | 62.9 | 61.3 | 64.6 | 61.1 | 58.1 | 64.1
OnlineRefer ($\omega$=2) | Video-Swin-B | semi-online | 62.9 | 61.0 | 64.7 | 62.4 | 59.1 | 65.6
|  |  |  |  |  |  |  |

Table 1: The quantitative evaluation on Refer-Youtube-VOS and Refer-DAVIS17,
with region similarity $\mathcal{J}$, boundary accuracy $\mathcal{F}$, and
average of $\mathcal{J}$&$\mathcal{F}$. The best results are in bold and the
second ones are underlined.

###  3.2 Cross-frame Query Propagation

As described above, the decoder of query-based referring segmentation
progressively refines the base coordinates from the position query into the
final prediction along decoder layers. Inspired by this, we additionally
consider the refinement domain in temporal axis, because the target box
predicted from the last frame can be a better reference coordinate. Therefore,
we propose the cross-frame query propagation, whose pipeline is illustrated in
Fig. 4.

Given the outputs of the last frame, we first filter out informative
representations that contain rich target awareness. In specific, we choose the
query with the highest class score, and its index is denoted as $\hat{n}$:

$\small\vspace{-3pt}\hat{n}=\mathop{\arg\max}\limits_{n\in
N_{t}}(\bm{c}_{n}).\vspace{-1pt}$ (2)

Here, as the first frame follows the original setting [43] and uses
$N_{1}\\!=\\!5$ learned queries, we can determine one query from multiple
ones. This also leads to the subsequent frames containing only $N_{t}\\!=\\!1$
query (_i.e_., $t\\!>\\!1$), which is retained across the entire video.

Once the index is obtained, we propagate three kinds of corresponding target
cues from $t^{th}$ frame to ${(t\\!+\\!1)}^{th}$ frame, including the
prediction box, output embedding, and position embedding. The box and position
embedding represent the explicit position information, which can be
transformed as the base coordinate and position query of ${(t\\!+\\!1)}^{th}$
frame, which is seamlessly inserted query-based referring segmentation. The
output embedding gathers the semantic information of the target, which is
transformed as the content query of ${(t\\!+\\!1)}^{th}$ frame. Formally, the
propagation process is:

$\displaystyle\bm{b}_{t+1}^{base}$
$\displaystyle=\bm{b}_{t,\hat{n}}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\in\mathbb{R}^{1\times
4},$ (3) $\displaystyle\bm{q}_{t+1}^{p}$
$\displaystyle=\bm{q}^{p}_{t,\hat{n}}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\in\mathbb{R}^{1\times
d},$ $\displaystyle\bm{q}_{t+1}^{c}$
$\displaystyle=\mathcal{F}^{FFN}(\bm{E}_{t,\hat{n}})~{}\in\mathbb{R}^{1\times
d},$

where $\mathcal{F}^{FFN}$ refers to one 3-layer FFN. Using the query
propagation in multiple training frames, the matching cost is independently
computed from each frame and the final loss is averaged by the frame number.

Discussion. In a streaming video, it is common for the referent object to
enter our view in the middle frames. In other words, if an object is invisible
in the first frame, our model will generate an empty query that contains no
object for propagation. Despite this, OnlineRefer still handles well the
entrance objects, as shown in the qualitative results of Fig. 6. This
indicates that our cross-frame query propagation aims to provide a good prior
while the cross-modal understanding in the query-based referring segmentation
still plays an important role in referring prediction.

###  3.3 Extension to Semi-Online Model

In addition to the frame-by-frame pattern, we extend OnlineRefer into a more
generalized framework that follows a clip-by-clip paradigm. Its primary
motivation is to be capable of large video-based backbones. Different from the
existing offline methods that independently process each clip, our approach
provides query propagation between clips to achieve cross-clip object
association. In this work, we define the new framework as semi-online method.

Formally, given the $i^{th}$ video clip $V_{i}\\!\in\\!\mathbb{R}^{I\times
3\times H\times W}$, where $I$ represents the clip length, we feed it into the
query-based referring segmentation. Our semi-online model first extracts
multi-frame visual features using a video-based backbone, _i.e_., Video Swin
Transformer [20], and perform cross-modal interaction between visual and
linguistic embedding in the encoder. As the input query of the semi-online
model is the same as our online model in the decoder, we then repeat the input
query by $I$ times to adapt to the multi-frame referring prediction. Thus, the
semi-online model outputs multi-frame boxes
$\bm{b}_{i}\\!\in\\!\mathbb{R}^{I\times N_{i}\times 4}$, masks
$\bm{m}_{i}\\!\in\\!\mathbb{R}^{I\times N_{i}\times H\times W}$, and classes
$\bm{c}_{i}\\!\in\\!\mathbb{R}^{I\times N_{i}\times c}$. The outputs can be
regarded as $N_{i}$ trajectory predictions on $I$ frames. Finally, we find the
positive sequence from $N_{i}$ predictions by calculating the matching cost
Eq. 1 and optimize the network. During the query propagation, we only transfer
the high-score query of the last frame into the next clip.

Method | Backbone | Online/Offline | P0.5 | P0.6 | P0.7 | P0.8 | P0.9 | Overall IoU | Mean IoU
---|---|---|---|---|---|---|---|---|---
Hu _et al_. [8] | VGG-16 | offline | 34.8 | 23.6 | 13.3 | 3.3 | 0.1 | 47.4 | 35.0
Gavrilyuk _et al_. [5] | I3D | offline | 47.5 | 34.7 | 21.1 | 8.0 | 0.2 | 53.6 | 42.1
CMSA + CFSA [39] | ResNet-101 | offline | 48.7 | 43.1 | 35.8 | 23.1 | 5.2 | 61.8 | 43.2
ACAN [29] | I3D | offline | 55.7 | 45.9 | 31.9 | 16.0 | 2.0 | 60.1 | 49.0
CMPC-V [17] | I3D | offline | 65.5 | 59.2 | 50.6 | 34.2 | 9.8 | 65.3 | 57.3
ClawCraneNet [13] | ResNet-50/101 | offline | 70.4 | 67.7 | 61.7 | 48.9 | 17.1 | 63.1 | 59.9
MTTR ($\omega$=10) [1] | Video-Swin-T | offline | 75.4 | 71.2 | 63.8 | 48.5 | 16.9 | 72.0 | 64.0
ReferFormer ($\omega$=5) [32] | Video-Swin-T | offline | 82.8 | 79.2 | 72.3 | 55.3 | 19.3 | 77.6 | 69.6
ReferFormer ($\omega$=5) [32] | Video-Swin-B | offline | 83.1 | 80.4 | 74.1 | 57.9 | 21.2 | 78.6 | 70.3
OnlineRefer ($\omega$=5) | Video-Swin-B | semi-online | 83.1 | 80.2 | 73.4 | 56.8 | 21.7 | 79.6 | 70.5
|  |  |  |  |  |  |  |  |

Table 2: The quantitative evaluation on A2D-Sentences, with Precision@K,
overall IoU and Mean IoU.

Method | Backbone | Online/Offline | P0.5 | P0.6 | P0.7 | P0.8 | P0.9 | Overall IoU | Mean IoU
---|---|---|---|---|---|---|---|---|---
Hu _et al_. [8] | VGG-16 | offline | 63.3 | 35.0 | 8.5 | 0.2 | 0.0 | 54.6 | 52.8
Gavrilyuk _et al_. [5] | I3D | offline | 69.9 | 46.0 | 17.3 | 1.4 | 0.0 | 54.1 | 54.2
CMSA + CFSA [39] | ResNet-101 | offline | 76.4 | 62.5 | 38.9 | 9.0 | 0.1 | 62.8 | 58.1
ACAN [29] | I3D | offline | 75.6 | 56.4 | 28.7 | 3.4 | 0.0 | 57.6 | 58.4
CMPC-V [17] | I3D | offline | 81.3 | 65.7 | 37.1 | 7.0 | 0.0 | 61.6 | 61.7
ClawCraneNet [13] | ResNet-50/101 | offline | 88.0 | 79.6 | 56.6 | 14.7 | 0.2 | 64.4 | 65.6
MTTR ($\omega$=10) [1] | Video-Swin-T | offline | 93.9 | 85.2 | 61.6 | 16.6 | 0.1 | 70.1 | 69.8
ReferFormer ($\omega$=5) [32] | Video-Swin-T | offline | 95.8 | 89.3 | 66.8 | 18.9 | 0.2 | 71.9 | 71.0
ReferFormer ($\omega$=5) [32] | Video-Swin-B | offline | 96.2 | 90.2 | 70.2 | 21.0 | 0.3 | 73.0 | 71.8
OnlineRefer ($\omega$=5) | Video-Swin-B | semi-online | 96.1 | 90.4 | 71.0 | 21.9 | 0.2 | 73.5 | 71.9
|  |  |  |  |  |  |  |  |

Table 3: The quantitative evaluation on JHMDB-Sentences, with Precision@K,
overall IoU and Mean IoU.

##  4 Experiment

###  4.1 Dataset and Metric

Dataset. We evaluate our approach on four popular benchmarks: Refer-Youtube-
VOS [27], Refer-DAVIS17 [11], A2D-Sentences [5], and JHMDB-Sentences [5].
Refer-Youtube-VOS expands the large-scale video object segmentation benchmark
Youtube-VOS [35] using texutal desciptions. It consists of 3,975 videos and
27,899 expressions. Refer-DAVIS17 extends another video object segmentation
benchmark DAVIS17 [25], which has 90 videos (60 for training and 30 for
testing) and more than 1,500 expressions. A2D-Sentences and JHMDB-Sentences
are developed by adding additional textual descriptions on the original action
and actor datasets A2D [34] and JHMDB [10]. A2D-Sentences includes 3,782
videos and 6,655 expressions, where each video has 3-5 frames annotated with
pixel-level segmentation mask. JHMDB-Sentences contains 928 videos, each being
described by a corresponding expression (a total of 928 sentences).

Evaluation Metric. On Refer-Youtube-VOS and Refer-DAVIS17, we employ region
similarity $\mathcal{J}$, contour accuracy $\mathcal{F}$, and their average
value $\mathcal{J}\&\mathcal{F}$ as our metrics. Since ground-truth
annotations of Refer-Youtube-VOS validation are currently inaccessible, our
predictions are uploaded to the official server for evaluation. On
A2D-Sentences and JHMDB-Sentences, we adopt Overall IoU, Mean IoU, and
precision@K to evaluate our method. Overall IoU is the ratio between the total
intersection and the total union area over all the test samples. Mean IoU
computes the averaged IoU over all the test samples. Precision@K measures the
percentage of test samples whose IoU scores are higher than a threshold K,
where K$\in$[0.5, 0.6, 0.7, 0.8, 0.9].

###  4.2 Experiment Details

Model. We implement different visual backbones for feature extraction, such as
ResNet [7], Swin Transformer [19], Video Swin Transformer [20]. RoBERTa [18]
is adopted as the text encoder, while its parameters are frozen during the
entire training stage. The feature maps of the last three stages are used in
the encoder and FPN. We utilize 4 encoder layers and 4 decoder layers with
dimension $d\\!=\\!256$. The query number of the first frame is set to
$N_{1}\\!=\\!5$. The coefficients for losses are $\lambda_{cls}=2$,
$\lambda_{box}$ = 5, $\lambda_{mask}=2$.

Training. AdamW optimizer [21] is used to optimize our model with an initial
learning rate of $1e\\!-\\!5$, except for the visual backbone with a learning
rate of $5e\\!-\\!6$. The training procedure runs for 6 epochs with the
learning rate decays divided by 10 at the $3^{th}$ and $5^{th}$ epoch. The
data augmentation techniques include random horizontal flip, random resize,
random crop, and photometric distortion. Each frame is resized such that the
shorter side at least has a size of 320 and the longer side at most has a size
of 576.

For Refer-Youtube-VOS, we randomly sample 3 frames during training online
models. The inputs of semi-online models are 3 clips, each one containing a
window size of 2 (denoted as $\omega\\!=\\!2$). In order to improve training
stability, we feed 2 frames/clips into the online/semi-online model before the
$4^{th}$ epoch. For Refer-DAVIS17, we directly report the results using the
model trained on Refer-Youtube-VOS without fine-tuning. For A2D-Sentences, we
use 2 training clips and increase into 3 clips at the $4^{th}$ epoch, where
each clip has 5 frames (_i.e_., $\omega=5$) with the annotated target frame in
the middle. For JHMDB-Sentences, the model trained on A2D-Sentences is
directly employed for evaluation without fine-tuning. For a fair comparison,
our model is pre-trained on Ref-COCO [40].

Figure 5: Frame-wise J&F score on Refer-Youtube-VOS. OnlineRefer has similar
J&F accuracy with ReferFormer at the first frame but obtains better
performance in subsequent frames benefiting from a good prior.  Figure 6:
Qualitative results from Refer-Youtube-VOS. OnlineRefer accurately segments
the referent object under various challenging scenes, _e.g_., object occlusion
or exit, appearance and size variation, and visually-similar objects.

Inference. Since there is no gap between training and inference in our method,
we directly output the predicted segmentation masks using the well-trained
model without post-process. The prediction is selected with the highest
positive score. In the semi-online paradigm, the clip numbers remain the same
with the training setup. More experiment details are included in supplementary
materials.

###  4.3 Comparison to State-of-the-art

Refer-Youtube-VOS & Refer-DAVIS17. We compare our method with existing
approaches on Refer-Youtube-VOS and Refer-DAVIS17, as shown in Table 1. Note
that PMINet [4] and CITD [14] are top-2 solutions in 2021 Refer-Youtube-VOS
Challenge. We can see that the earlier works [38, 27] usually employ an online
manner, while the offline methods currently become mainstream due to better
performance, such as MLSA [30], MTTR [1], and ReferFormer [32]. Surprisingly,
our online model outperforms all offline methods on two datasets under all
metrics. In specific, on Refer-Youtube-VOS, OnlineRefer with backbone
ResNet-50 and Swin-L achieves $\mathcal{J}\&\mathcal{F}$ of 57.3 and 63.5.
They are the highest accuracy among the models using the same backbone.
Notably, our online model performance using Swin-L
($\mathcal{J}\&\mathcal{F}$: 63.5) is higher than ReferFormer using Video
Swin-B backbone ($\mathcal{J}\&\mathcal{F}$: 62.4). When the model is directly
evaluated on Refer-DAVIS17, it achieves the best scores
($\mathcal{J}\&\mathcal{F}$: 64.8), which surpasses ReferFormer by a large
margin. Overall, these impressive results sigificiantly demonstrate the
effectiveness of the complete-online pipeline in referring video object
segmentation.

Furthermore, attaching OnlineRefer on top of a Video Swin-B backbone
formulates a semi-online model. The results are displayed in the last row of
Table 1, which show that the new semi-online model leads to promising
performance on two datasets, especially for the contour accuracy
$\mathcal{F}$. This phenomenon also happens on other online models, which
means that the regional prior of propagated boxes benefits producing high-
quality segmentation masks.

In addition, we show the frame-wise $\mathcal{J}\&\mathcal{F}$ score curve on
Refer-Youtube-VOS using ResNet-50 and Swin-L in Fig. 5. Both OnlineRefer and
ReferFormer achieve similar accuracy in the first frame. But in subsequent
frames, our OnlineRefer has a performance gain of around $1\\!\sim\\!2$ point.
This clearly approves that the employment of explicit references can improve
the referring segmentation performance.

   Query Update Position Update $\mathcal{J}$&$\mathcal{F}$ $\mathcal{J}$
$\mathcal{F}$ $\times$ $\times$ 49.8 49.5 50.1 $\checkmark$ $\times$ 57.3 55.8
58.8 $\checkmark$ $\checkmark$ 32.1 29.7 34.6

Table 4: The update strategy of query and position on Refer-Youtube-VOS, in
terms of $\mathcal{J}$, $\mathcal{F}$.

Propagation Method |  $\mathcal{J}$&$\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$
---|---|---|---
w/o Propagation | 49.3 | 47.4 | 51.2
Concatenation | 55.2 | 53.8 | 56.7
Fixed | 55.0 | 53.6 | 56.5
Ours | 57.3 | 55.8 | 58.8
(a) Comparison on propagation method.

Propagation Number |  $\mathcal{J}$&$\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$
---|---|---|---
Top-4 | 55.2 | 53.7 | 56.7
Top-3 | 55.9 | 54.4 | 57.4
Top-2 | 56.3 | 54.9 | 57.7
Top-1 | 57.3 | 55.8 | 58.8
(b) Comparison on propagation number.

Initial Queries |  $\mathcal{J}$&$\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$
---|---|---|---
1 | 54.1 | 52.6 | 55.6
3 | 55.5 | 54.0 | 56.9
5 | 57.3 | 55.8 | 58.8
8 | 56.9 | 55.2 | 58.5
(c) Comparison on sampler frames.

Table 5:  Ablation studies of different propagation designs on Refer-Youtube-
VOS, in terms of region similarity $\mathcal{J}$, boundary accuracy
$\mathcal{F}$, and average of $\mathcal{J}$&$\mathcal{F}$. The best results
are in bold.

A2D-Sentences & JHMDB-Sentences. We further present comparisons on the
A2D-Sentences benchmark in Table 2. As the dataset is only annotated
keyframes, existing methods generally follow an offline paradigm, which
process clip-wise referring prediction without any cross-clip association. In
contrast, our OnlineRefer is able to link all video clips by query
propagation, _i.e_., the semi-online framework. With the backbone Video Swin-B
and window size of $\omega\\!=\\!5$, our semi-online model obviously exceeds
all offline methods over IoU metrics and keeps enough competitiveness over
precision metrics as well.

The semi-online model is directly evaluated on JHMDB-Sentences without
finetuning to demonstrate the generality of our method. In Table 3,
OnlineRefer achieves competitive performance compared to all other offline
methods. In specific, OnlineRefer leads to higher IoU scores but comparable
precision scores. Considering the clip gap on A2D-Sentences and JHMDB-
Sentences, the above results demonstrate the potential of semi-online model.

###  4.4 Qualitative Results

In Fig. 6, we show several typical referring segmentation results of
OnlineRefer from Refer-Youtube-VOS. The first video sequence is more
challenging because the two referring objects become occluded and invisible in
some frames. Taking the walking person as an example, it requires our online
model to avoid the predicted empty box in the first frame causing the target
missing in subsequent frames. Despite the difficulty, our OnlineRefer
successfully segments out the target with sharp boundaries, showing strong
correctness ability. In other scenes, the referring objects also face various
challenges, such as appearance variation, pose deformation, and visually-
similar objects. Otherwise, our OnlineRefer performs well in these difficult
scenarios.

###  4.5 Ablation Study

To offer a deep insight into our OnlineRefer, we conduct ablation studies to
analyze the effectiveness of each component. If not specialized, we report the
online model performance on Refer-Youtube-VOS using ResNet-50.

Importance of query updating. To investigate the effect of updating strategy
in Eq. 3, we perform experiments whether updating query and position embedding
in Table 4. From the first row, discarding both query and position updating
only achieves 49.8 over $\mathcal{J}\&\mathcal{F}$. After that, adding the
query update achieves remarkable performance improvement (+6.7 on
$\mathcal{J}\&\mathcal{F}$), and reaches the best score. However, updating
position embedding largely hinders model performance (-25.2 on
$\mathcal{J}\&\mathcal{F}$), as shown in the last row of Table 4. The problem
demonstrates that fixed position embedding plays an important role in cross-
frame object association.

Different propagation methods. We then analyze different query propagation
methods in Table 5(a). Removing propagation (_i.e_., w/o propagation) leads to
a frame-based referring segmentation pipeline, which results in a large
performance drop (-6.0 on $\mathcal{J}\&\mathcal{F}$). ‘Concatenation’
represents concatenating the one propagated query and the initial query set,
like MOTR [41] and TrackFormer [22]. ‘Fixed’ refers to giving up the query
selection and keeping the initial number of queries, like InsPro [6]. Both two
have a slight performance decrease. These results approve the effectiveness of
our simple and heuristic-free query propagation.

Number of propagation query. It is also of interest to explore the impact of
different propagation query numbers. As shown in Table 5(b), we vary the query
number from top-4 to top-1, where top-1 is our setting in OnlineRefer. Note
that propagating top-5 queries equals the ‘fixed’ method in Table 5(a). It is
obvious that with the top query number decreasing, the performance on
$\mathcal{J}\&\mathcal{F}$ is gradually improved. Overall, applying one query
on cross-frame propagation is enough and significant for the inter-frame
association.

Effect of initial queries. OnlineRefer starts with a set of initial queries in
the first frame, which is further propagated across the entire video. To study
its influence, we use a relatively small number of queries, as shown in Table
5(c). We can see that fewer queries bring fewer proposals, further leading to
lower $\mathcal{J}\&\mathcal{F}$ scores. However, the performance of more
queries becomes flattened after $N\\!=\\!5$. Empirically, in this work, we set
the initial query number as $N\\!=\\!5$.

##  5 Conclusion

In this paper, we proposed a simple, elegant, and strong baseline for online
referring video object segmentation, named OnlineRefer. It broke up the widely
accepted tradition that only offline models can handle well the challenging
referring understanding task. OnlineRefer includes two crucial parts, query-
based referring segmentation, and query propagation. The query-based referring
segmentation outputs box, mask, and category based on input queries, while
query generation part updates the output set as new queries. By iteratively
using two parts, the objects in all video frames are automatically associated
and predicted. To be compatible with video-based backbones, we developed a
semi-online model that associates and predicts referent object clip by clip.
The experiments are conducted on Refer-Youtube-VOS, Refer-DAVIS17,
A2D-Sentences and JHMDB-Sentences and our OnlineRefer shows the state-of-the-
art performance on the four benchmarks.

## References

  * [1] Adam Botach, Evgenii Zheltonozhskii, and Chaim Baskin.  End-to-end referring video object segmentation with multimodal transformers.  In CVPR, 2022.
  * [2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, 2020.
  * [3] Zihan Ding, Tianrui Hui, Junshi Huang, Xiaoming Wei, Jizhong Han, and Si Liu.  Language-bridged spatial-temporal interaction for referring video object segmentation.  In CVPR, 2022.
  * [4] Zihan Ding, Tianrui Hui, Shaofei Huang, Si Liu, Xuan Luo, Junshi Huang, and Xiaoming Wei.  Progressive multimodal interaction network for referring video object segmentation.  The 3rd Large-scale Video Object Segmentation Challenge, 2021.
  * [5] Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, and Cees GM Snoek.  Actor and action video segmentation from a sentence.  In CVPR, 2018.
  * [6] Fei He, Haoyang Zhang, Naiyu Gao, Jian Jia, Yanhu Shan, Xin Zhao, and Kaiqi Huang.  Inspro: Propagating instance query and proposal for online video instance segmentation.  In NeurIPS, 2020.
  * [7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [8] Ronghang Hu, Marcus Rohrbach, and Trevor Darrell.  Segmentation from natural language expressions.  In ECCV, 2016.
  * [9] Tianrui Hui, Shaofei Huang, Si Liu, Zihan Ding, Guanbin Li, Wenguan Wang, Jizhong Han, and Fei Wang.  Collaborative spatial-temporal modeling for language-queried video actor segmentation.  In CVPR, 2021.
  * [10] Hueihan Jhuang, Juergen Gall, Silvia Zuffi, Cordelia Schmid, and Michael J Black.  Towards understanding action recognition.  In ICCV, 2013.
  * [11] Anna Khoreva, Anna Rohrbach, and Bernt Schiele.  Video object segmentation with language referring expressions.  In ACCV, 2019.
  * [12] Dezhuang Li, Ruoqi Li, Lijun Wang, Yifan Wang, Jinqing Qi, Lu Zhang, Ting Liu, Qingquan Xu, and Huchuan Lu.  You only infer once: Cross-modal meta-transfer for referring video object segmentation.  In AAAI, 2022.
  * [13] Chen Liang, Yu Wu, Yawei Luo, and Yi Yang.  Clawcranenet: Leveraging object-level relation for text-based video segmentation.  arXiv preprint arXiv:2103.10702, 2021.
  * [14] Chen Liang, Yu Wu, Tianfei Zhou, Wenguan Wang, Zongxin Yang, Yunchao Wei, and Yi Yang.  Rethinking cross-modal interaction from a top-down perspective for referring video object segmentation.  arXiv preprint arXiv:2106.01061, 2021.
  * [15] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In ICCV, 2017.
  * [16] Chenxi Liu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, and Alan Yuille.  Recurrent multimodal interaction for referring image segmentation.  In ICCV, 2017.
  * [17] Si Liu, Tianrui Hui, Shaofei Huang, Yunchao Wei, Bo Li, and Guanbin Li.  Cross-modal progressive comprehension for referring segmentation.  TPAMI, 2021.
  * [18] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [19] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, 2021.
  * [20] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video swin transformer.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3202–3211, 2022.
  * [21] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2019.
  * [22] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer.  Trackformer: Multi-object tracking with transformers.  In CVPR, 2022.
  * [23] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi.  V-net: Fully convolutional neural networks for volumetric medical image segmentation.  In 3DV, 2016.
  * [24] Seoung Wug Oh, Joon-Young Lee, Ning Xu, and Seon Joo Kim.  Video object segmentation using space-time memory networks.  In ICCV, 2019.
  * [25] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool.  The 2017 davis challenge on video object segmentation.  arXiv preprint arXiv:1704.00675, 2017.
  * [26] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In CVPR, 2019.
  * [27] Seonguk Seo, Joon-Young Lee, and Bohyung Han.  Urvos: Unified referring video object segmentation network with a large-scale benchmark.  In ECCV, 2020.
  * [28] Hao Wang, Cheng Deng, Fan Ma, and Yi Yang.  Context modulated dynamic networks for actor and action video segmentation with language queries.  In AAAI, 2020.
  * [29] Hao Wang, Cheng Deng, Junchi Yan, and Dacheng Tao.  Asymmetric cross-guided attention network for actor and action video segmentation from natural language query.  In ICCV, 2019.
  * [30] Dongming Wu, Xingping Dong, Ling Shao, and Jianbing Shen.  Multi-level representation learning with semantic alignment for referring video object segmentation.  In CVPR, 2022.
  * [31] Dongming Wu, Wencheng Han, Tiancai Wang, Xingping Dong, Xiangyu Zhang, and Jianbing Shen.  Referring multi-object tracking.  In CVPR, 2023.
  * [32] Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo.  Language as queries for referring video object segmentation.  In CVPR, 2022.
  * [33] Junfeng Wu, Qihao Liu, Yi Jiang, Song Bai, Alan Yuille, and Xiang Bai.  In defense of online models for video instance segmentation.  In ECCV, 2022.
  * [34] Chenliang Xu, Shao-Hang Hsieh, Caiming Xiong, and Jason J Corso.  Can humans fly? action understanding with multiple classes of actors.  In CVPR, 2015.
  * [35] Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang.  Youtube-vos: Sequence-to-sequence video object segmentation.  In ECCV, 2018.
  * [36] Shilin Yan, Renrui Zhang, Ziyu Guo, Wenchao Chen, Wei Zhang, Hongyang Li, Yu Qiao, Zhongjiang He, and Peng Gao.  Referred by multi-modality: A unified temporal transformer for video object segmentation.  arXiv preprint arXiv:2305.16318, 2023.
  * [37] Zhao Yang, Yansong Tang, Luca Bertinetto, Hengshuang Zhao, and Philip HS Torr.  Hierarchical interaction network for video object segmentation from referring expressions.  In BMVC, 2021.
  * [38] Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang.  Cross-modal self-attention network for referring image segmentation.  In CVPR, 2019.
  * [39] Linwei Ye, Mrigank Rochan, Zhi Liu, Xiaoqin Zhang, and Yang Wang.  Referring segmentation in images and videos with cross-modal self-attention network.  TPAMI, 2021.
  * [40] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg.  Modeling context in referring expressions.  In ECCV, 2016.
  * [41] Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei.  Motr: End-to-end multiple-object tracking with transformer.  In ECCV, 2022.
  * [42] Wangbo Zhao, Kai Wang, Xiangxiang Chu, Fuzhao Xue, Xinchao Wang, and Yang You.  Modeling motion with multi-modal features for text-based video segmentation.  In CVPR, 2022.
  * [43] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  In ICLR, 2020.


3(1), Misalignments in NGC 1068–References (2022) 11institutetext:
Astronomical Institute, Academy of Sciences, Boční II 1401, Prague, Czech
Republic
11email<EMAIL_ADDRESS>22institutetext: LESIA, Observatoire de
Paris, Université PSL, CNRS, Sorbonne Université, Université de Paris, 5 place
Jules Janssen, 92195
Meudon, France

# Misalignments in NGC 1068

Pierre Vermot 1122

(Submitted on December 1, 2021)

###### Abstract

NGC 1068 is a nearby Active Galactic Nucleus (AGN) of type 2, meaning that its
accretion disk is hidden behind a large amount of foreground extinction.
Observations at several wavelengths have revealed various disk-like structures
around the nucleus, all possibly part of the putative torus responsible for
the obscuration of the AGN. The paper presents results based on GRAVITY/VLTI
interferometric observations in the near-infrared, which provide very high
angular resolution, and gives insights into the geometry of the innermost
region of the torus. The 3D orientation of the structure is surprising in
several aspects, as it is misaligned with other disks present around the
nucleus, and leaves a clear line of sight toward the central source.

###### Key Words.:

NGC 1068 – Active Galactic Nuclei – GRAVITY – dust

###### doi:

10.1146/annurev.aa.31.090193.002353

###### doi:

10.1023/A:1000567831370

###### doi:

10.1086/504899

###### doi:

10.3847/2041-8205/829/1/L7

###### doi:

10.1086/321616

###### doi:

10.3847/2041-8205/823/1/L12

###### doi:

10.1051/0004-6361/201526554

###### doi:

10.1051/0004-6361/201730838

###### doi:

10.1051/0004-6361/201936255

###### doi:

10.1086/310346

###### doi:

10.1051/0004-6361/201731438

###### doi:

10.3847/2041-8213/ab3c64

###### doi:

10.1016/S0273-1177(99)00262-8

###### doi:

10.1086/174225

###### doi:

10.1051/0004-6361:20078515

###### doi:

10.1051/0004-6361/202141349

##  1 Introduction

AGN are observed with various characteristics in terms of luminosity and
spectral properties. A large amount of these differences can be explained by
the Unified Model of AGN originally proposed by Antonucci (1993), which states
that a structure surrounding the central source causes drastic extinction for
some positions of the observer. This parsec-scale obscuring structure
(commonly called the _torus_) is assumed to be located in the equatorial plane
of the jet, and to be the continuation of the large-scale accretion flow.

The AGN of NGC 1068 is among the easiest to observe, as it is relatively close
to us ($D_{L}=14.4$ Mpc, with 1′′ corresponding to 70 pc), and is located in
the center of a face-on spiral galaxy. As a consequence, the object has been
observed by many instruments, and a wealth of information is available at
various wavelengths and spatial scales. It is known to be a type 2 AGN,
meaning that its central source is obscured by a large amount of foreground
material. The following list presents some of the main structures present in
the nucleus of NGC 1068; from large to small spatial scales:

  * •

The Narrow Line Region, which is an ionized outflow observed and characterized
through spatially resolved emission line Doppler shifts in the UV and IR (Das
et al., 2006; Poncelet et al., 2008), extends up to several hundred parsecs
from the nucleus. It appears to have a hollow bicone shape with a revolution
axis almost in the plane of the sky ($i\sim$ 5–10$\degree$) and oriented with
PA $\sim 30\degree$.

  * •

An extended scattering structure, observed through polarimetric imaging with
SPHERE and described by Gratadour et al. (2015). With dimensions of $20\times
60$ pc, it appears to be seen edge-on, and is oriented with PA $\sim
120\degree$.

  * •

A molecular disk, observed through CO and HCN emission lines with ALMA
(García-Burillo et al., 2016; Gallimore et al., 2016; Impellizzeri et al.,
2019). With a typical spatial scale of $\sim 10$ pc, it is aligned along PA
$\sim 110\degree$ and seems to have an inclination comprised between $34$ and
$66\degree$. Interestingly, this disk is counter-rotating with respect to both
the external regions of the galaxy and the inner maser disk (described below).

  * •

An edge-on maser disk observed with the VLBA, with an inner radius
$r_{\mathrm{in}}\sim 0.65$ pc and an outer radius $r_{\mathrm{out}}\sim 1.1$
pc, oriented along PA $\sim 135\degree$ (Greenhill et al., 1996).

The different structures described above are good examples of the general
cylindrical geometry assumed for AGN, with an outflow along the polar axis and
cold, obscuring material in the equatorial plane. However, as one can notice
in the $i$ and PA values given above, these structures are only approximately
aligned with these two preferred directions.

The next section presents the geometrical properties of the hot dust located
within the inner radius of the maser disk, based on the analysis of very high
angular resolution observations obtained with GRAVITY, the near-infrared
instrument of the VLTI. The orientation of this hot dust with respect to other
components of the nucleus will be discussed in the following sections.

##  2 Observation and results

This study is based on a GRAVITY observation of NGC 1068, performed during the
night of November 20, 2018 (Gravity Collaboration et al., 2017, ESO programmes
IDS 0102.B-0667, 0102.C-0205 and 0102.C-0211). GRAVITY is an instrument of the
VLT interferometer providing high angular resolution information in the K
band.

Despite being one of the most luminous AGN in the near-infrared, NGC 1068 is
in the low-end of GRAVITY observable sources and the instrument had to be used
in a peculiar way, with all the flux from the source being injected into the
Fringe Tracker, and the science channel being left empty.

The observation provided several interferometric observables: the squared
visibility, the closure phase; and the total and coherent spectra. With the
Fringe Tracker, these measurements are available with a low spectral
resolution $R\sim 22$, corresponding to four spectral channels in the K band.
The maximum spatial frequency probed by the interferometer is $\sim 60\
M\lambda$, corresponding to an angular resolution slightly above $3$ mas.

To interpret the interferometric measurements, the radiative transfer
simulation code MontAGN (Grosset et al., 2018) was used to produce flux
calibrated images of physically realistic heated dusty disks in the spectral
channels of the instrument. The Fourier Transforms of these images were then
used to compute synthetic interferometric observables: squared visibility
(with wavelength dependence), closure phase (idem), coherent spectrum in
arbitrary units and total calibrated spectrum (sum of the resolved and
unresolved components), which are then directly compared to the observation.

Figure 1: K band image of the best model for the inner hot dust of NGC 1068. 1
parsec corresponds to $\sim 15$ mas, and the resolution of the interferometer
is $\sim 3$ mas.

The hot dust structure was modeled by a thick uniform disk of graphite grains,
whose inner edge is heated up to its sublimation temperature by a central UV-X
source. Several parameters are fully explored – the inner radius, the opening
angle, the inclination, the position angle, and the density – and an optimal
solution is found, which is in fair agreement with all the interferometric
observables. An image of the best model is provided in Fig. 1, a summary of
its parameters in Table 1, and a comparison with the observed visibility and
closure phase in Fig. 2. A more detailed description of the modeling and
comparisons with the observation is provided by Vermot et al. (2021).

Figure 2: Comparison of the model with observed quantities. Top panel: observed and modeled squared visibility. Bottom panel: observed and modeled closure phase. Table 1: Best model parameters. Parameter | Value
---|---
Inner radius |  $0.21_{-0.03}^{+0.02}$ pc
Half-opening angle |  $21_{-8}^{+8}$ deg
Inclination |  $44_{-6}^{+10}$ deg
Density |  $10_{-5}^{+10}$ cm-3
PA |  $150_{-13}^{+8}$ deg

##  3 Non-obscuring hot dust

The first remarkable feature of our best model is its inclination, which
leaves a clear line of sight for the observer toward the central source. This
result is surprising since it is known for a long time that NGC 1068 is a type
2 object, meaning that its accretion disk is obscured by a large amount of
foreground extinction. In the commonly accepted description of AGN, it is
expected to have this obscuring material distributed in an equatorial
structure, orthogonal to the outflow. The hot dust ring does not follow this
trend.

The inner radius of the modeled hot dusty disk is constrained from the
geometric information obtained with the GRAVITY observations. Assuming that
the inner edge of the hot dusty disk is at the sublimation temperature, its
radius constrains the UV-X luminosity of the central source, which will be
absorbed by the dust and re-emitted in the infrared. Hence, the measure of the
inner radius and opening angle of the dusty disk should provide a realistic
estimate of the IR magnitude of the source. However, this is not the case, and
a high foreground extinction $A_{V}\geq 50$ needs to be included in the model
to account for the observed K band magnitude. This result is supported by the
fact that the luminosity of the central source in the model (between
$3.9\times 10^{38}$ W and $6.4\times 10^{38}$ W), constrained mostly by the
size of the inner radius, matches well the previous estimate from independent
measurements, ranging from a few $10^{37}$ W to a few $10^{38}$ W (Pier et
al., 1994; Bland-Hawthorn et al., 1997; Kishimoto, 1999; Gallimore et al.,
2001).

These two conclusions indicate that the inner edge of the dusty torus is not
responsible for the extinction toward the central accretion disk, but is
rather itself surrounded by a large amount of obscuring material.

##  4 Misalignments

The second remarkable feature of this hot dusty ring model is its misalignment
with respect to the previously mentioned structures. Compared to the extended
scattering torus, it is not seen edge-on, and the estimations of their PA
differ by $30\degree$. The inclination of the molecular torus could be
compatible, but their PA differ even more. At last, our model is similar to
the maser disk in terms of PA but greatly differs in terms of inclination,
with the maser detection requiring a disk closely aligned to the line of
sight.

Since these structures are also misaligned between each other, and that
several methods used to model the GRAVITY observations gave similar
geometrical parameters (geometric modeling in Vermot et al. (2021) and image
reconstruction in Gravity Collaboration et al. (2020)), one can reasonably
assume that this new misalignment is not an artifact resulting from the
interpretation of the interferometric data.

Geometrically, these observations at different wavelengths could probe
different radii of a warped large scale-disk. However, this wouldn’t explain
the counter-rotation observed between the molecular and maser disks, and no
sign of warping is present in the maser observations. Hence, it can be
concluded that these different observations are not probing a unique
structure, but several ring-like objects intricated around the nucleus. Such
structures could result from the tidal disruption of molecular clouds
surrounding the nucleus, as seen in the Milky Way.

###### Acknowledgements.

The author is grateful to the referee for a constructive report which helped
to significantly improve the paper, as well as the Astronomical Institute of
the Czech Academy of Sciences for support by the Czech Science Foundation
Grant 19-15480S and by the project RVO:679858.

## References

  * Antonucci (1993) Antonucci R., 1993.  Ann. Rev. Astron. Astrophys., vol. 31, pp. 473–521. .
  * Bland-Hawthorn et al. (1997) Bland-Hawthorn J., Gallimore J.F., Tacconi L.J., et al., 1997.  Astrophys. Space Sci., vol. 248, no. 1-2, pp. 9–19. .
  * Das et al. (2006) Das V., Crenshaw D.M., Kraemer S.B., Deo R.P., 2006.  Astron. J., vol. 132, no. 2, pp. 620–632. (arXiv astro-ph/0603803).
  * Gallimore et al. (2016) Gallimore J.F., Elitzur M., Maiolino R., et al., 2016.  Astrophys. J. Lett., vol. 829, no. 1, L7. (arXiv 1608.02210).
  * Gallimore et al. (2001) Gallimore J.F., Henkel C., Baum S.A., et al., 2001.  Astrophys. J., vol. 556, no. 2, pp. 694–715. (arXiv astro-ph/0104083).
  * García-Burillo et al. (2016) García-Burillo S., Combes F., Ramos Almeida C., et al., 2016.  Astrophys. J. Lett., vol. 823, no. 1, L12. (arXiv 1604.00205).
  * Gratadour et al. (2015) Gratadour D., Rouan D., Grosset L., Boccaletti A., Clénet Y., 2015\.  Astron. Astrophys., vol. 581, L8. .
  * Gravity Collaboration et al. (2017) Gravity Collaboration, Abuter R., Accardo M., et al., 2017.  Astron. Astrophys., vol. 602, A94. (arXiv 1705.02345).
  * Gravity Collaboration et al. (2020) Gravity Collaboration, Pfuhl O., Davies R., et al., 2020.  Astron. Astrophys., vol. 634, A1. (arXiv 1912.01361).
  * Greenhill et al. (1996) Greenhill L.J., Gwinn C.R., Antonucci R., Barvainis R., 1996.  Astrophys. J. Lett., vol. 472, p. L21. (arXiv astro-ph/9609082).
  * Grosset et al. (2018) Grosset L., Rouan D., Gratadour D., et al., 2018.  Astron. Astrophys., vol. 612, A69. (arXiv 1712.02192).
  * Impellizzeri et al. (2019) Impellizzeri C.M.V., Gallimore J.F., Baum S.A., et al., 2019.  Astrophys. J. Lett., vol. 884, no. 2, L28. (arXiv 1908.07981).
  * Kishimoto (1999) Kishimoto M., 1999.  Advances in Space Research, vol. 23, no. 5-6, pp. 899–904. .
  * Pier et al. (1994) Pier E.A., Antonucci R., Hurt T., Kriss G., Krolik J., 1994.  Astrophys. J., vol. 428, pp. 124–129. .
  * Poncelet et al. (2008) Poncelet A., Sol H., Perrin G., 2008.  Astron. Astrophys., vol. 481, no. 2, pp. 305–317. .
  * Vermot et al. (2021) Vermot P., Clénet Y., Gratadour D., et al., 2021.  Astron. Astrophys., vol. 652, A65. (arXiv 2106.04211).


# Input Reconstruction Attack against Vertical Federated
Large Language Models

Fei Zheng

###### Abstract

Recently, large language models (LLMs) have drawn extensive attention from
academia and the public, due to the advent of the ChatGPT. While LLMs show
their astonishing ability in text generation for various tasks, privacy
concerns limit their usage in real-life businesses. More specifically, either
the user’s inputs (the user sends the query to the model-hosting server) or
the model (the user downloads the complete model) itself will be revealed
during the usage. Vertical federated learning (VFL) is a promising solution to
this kind of problem. It protects both the user’s input and the knowledge of
the model by splitting the model into a bottom part and a top part, which is
maintained by the user and the model provider, respectively. However, in this
paper, we demonstrate that in LLMs, VFL fails to protect the user input since
it is simple and cheap to reconstruct the input from the intermediate
embeddings. Experiments show that even with a commercial GPU, the input
sentence can be reconstructed in only one second. We also discuss several
possible solutions to enhance the privacy of vertical federated LLMs.

##  1 Introduction

With the advent of ChatGPT [14], large language models (LLMs) have drawn much
attention from both the public and academia. LLMs have shown great ability in
various tasks, e.g., question answering, reading comprehension, text
summarization, and even mathematical reasoning. However, in real applications,
LLMs still face significant privacy concerns. For example, a typical scenario
of ChatGPT’s usage is that, the user sends his query to the OpenAI server,
then gets the response. In this case, the user’s query is totally exposed to
the LLM provider. This is unacceptable when the query contains sensitive or
valuable information, e.g., the user’s personal information or confidential
data of the company. Another scenario is that the user downloads the complete
LLM model and deploys it himself. Although user privacy is protected in this
case, the model is completely revealed to the user, which violates the model
provider’s privacy considering LLMs are highly valuable assets.

The fundamental problem here is that the data and the model are held by two
different parties and shall not be revealed to each other. One way to tackle
this problem is cryptographic methods. These methods use cryptographic
technologies such as multiparty computation protocols, garbled circuits, and
homomorphic encryptions to implement secure neural network inference or
training. While these methods fully protect both parties’ privacy in a strict
sense, by now they can only be applied to relatively simple models, due to the
expensive computation and communication overhead. A few pioneering works apply
cryptographic methods to realize secure transformers [8, 10, 4]. However, they
require at least several minutes to generate one token with very short input
and under ideal local network settings, using high-performance servers, which
is impractical in real-life applications.

An alternative method to protect privacy in this case is vertical federated
learning (VFL). The typical method for VFL is split learning, whose procedure
can be described as follows:

  1. 1.

the model provider splits the model by a certain layer to get a bottom model
and a top model. He then sends the bottom part of the model (bottom model) to
the user.

  2. 2.

During inference, the user feeds his private input to the bottom model, whose
output is the hidden embeddings (smashed data). Then he sends the hidden
embeddings to the model provider, who feeds it to the top model and gets the
model output. The model provider sends the model output to the user.

Hence, in split learning, only the hidden embeddings and part of the model are
revealed, which protects both parties’ privacy to some degree. Notice that,
using split learning, the response to the user’s query is also obtained by the
model provider, which could also leak user’s privacy. However, in this paper,
we focus on the privacy of user inputs rather than model outputs.

We show that the adversary can easily reconstruct the exact input text using
the hidden embeddings and the parameters of the bottom model. Specifically, we
first reconstruct the word embeddings in the first layer, then we compute the
similarity between the reconstructed word embeddings and the actual word
embeddings in the bottom model, and find the most similar words. This kind of
attack can be very efficient, and the adversary does not need any side
information except for the bottom model. Such a setting is practical since we
assume the adversary has compromised the model provider, who has the complete
model. The experiments are conducted on the open-sourced ChatGLM6B model. We
also investigated adding noise in the hidden embeddings before sending it to
the model provider in order to enhance input privacy. However, experiment
results show that such defense is impractical, since the model performance is
also severely downgraded.

##  2 Related Work

Although few studies investigated the input reconstruction in LLMs,
reconstructing inputs from hidden embeddings is a long-studied problem in deep
learning, and specifically, VFL. It can be classified into two classes: black-
box and white-box. In the black-box scenario, the bottom model used to
generate the embeddings is unknown, while in the white-box scenario, the
adversary maintains the bottom model, including its structure and values of
parameters. In the black-box scenario, a basic reconstruction method is to
train a reconstruction model that takes the hidden embeddings as input and
outputs the original input [22, 9, 18]. Other methods include training a
generator to invert [9, 13], and training both surrogate models and inputs
[6]. Although [13] also applies the inversion attack to language models, it
adopts a black-box setting, i.e., the adversary has no information of the
bottom model, while he can query it as many times as he wants. This requires
the adversary to train a generative model to reconstruct the inputs, which is
time-consuming. Instead, our paper focuses on the white-box setting, which is
more practical. As for the white-box scenario, the most straightforward method
is to directly optimize the input, as the embedding is now differentiable [9,
12, 18]. However, the authors in [18] directly optimize the one-hot word
vector with Softmax relaxation, which leads to suboptimal performance since
the dimension of the one-hot vector is too high. Instead, we optimize the word
embeddings first, then reconstruct the word according to the word embeddings.

##  3 Preliminaries

###  3.1 Large Language Models

Large language models (LLMs) have gain substantial attention recently, due to
their powerful ability in general-purpose language understanding and
generation. Nowadays, LLMs are mainly built upon the transformer architecture
[20, 3] and trained with the autoregressive generation tasks. They usually
contain from billions to trillions parameters so that they are extremely
expensive to train, and require high-performance GPUs for inference. Notable
examples of LLMs include OpenAI’s GPT models (e.g., GPT [15], GPT-2 [16],
GPT-3 [2], GPT-3.5 and GPT-4, used in ChatGPT [14]), Google’s PaLM (used in
Bard), Meta’s LLaMa [19], and ChatGLM [5].

###  3.2 Vertical Federated Learning

Vertical federated learning (VFL) refers to the federated learning case such
that the data is vertically distributed, i.e., different parties have
different features columns (including label) of the same samples. A typical
technique for VFL is split learning [21], in which multiple parties split the
complete model into multiple parts. During the inference, parties exchange
intermediate results in order to get the model output. The simplest split
learning case is that one party ($P_{0}$) has the bottom model ($M_{b}$) and
the other has the top part of the model ($M_{t}$). During inference, $P_{0}$
computes the hidden embedding $h=M_{b}(x)$ and sends it to $P_{1}$. $P_{1}$
computes $y=M_{t}(h)$ to get the model prediction. In our case, the model
provider sends the bottom model to the user, which consists of embedding
layers and several transformer layers, while the top model is made of the rest
of the transformer layers and the linear layer used to predict the next token.
This setting is realistic, since the model provider has the complete model,
and it is usually not possible for the user and the model provider to
collaboratively train a split LLM.

##  4 Input Reconstruction from Embeddings

The inference procedure of a large language model can be expressed as follows:

$\text{LLM}(x)=\text{Linear}(\text{MultiheadAttns}[\text{WordEmbedding}(x)+\text{PosEmbedding}(x)]),$
(1)

i.e., the input (tokenization) first goes through the word embedding layer and
the positional embedding layer to get the first hidden embedding, and then it
goes through multiple multi-head attention layers to get the final hidden
state, after which a linear layer is used to predict logits. In split
learning, the bottom model contains the initial embedding layers and several
multi-head attention layers, and the top model contains the rest of the multi-
head attention layers and the final linear layer:

$\begin{cases}h_{s}=\text{LLM}_{\text{bottom}}(x)=\text{MultiheadAttns}_{\text{bottom}}[\text{WordEmbedding}(x)+\text{PosEmbedding}(x)]\\\
y=\text{LLM}_{\text{top}}(h_{s})=\text{Linear}[\text{MultiheadAttns}_{\text{top}}(h_{s})]\end{cases}$
(2)

The input reconstruction attack can then be expressed as given
$\text{LLM}_{\text{bottom}}(x)$, find a $x^{\prime}$ such that
$\text{LLM}_{\text{bottom}}(x^{\prime})=\text{LLM}_{\text{bottom}}(x)$. We
divide this task into two steps: word embedding reconstruction and token
reconstruction.

###  4.1 Word Embedding Reconstruction

The first step is to reconstruct the word embedding
$h_{w}=\text{WordEmbedding}(x)$. Given that the hidden embeddings have the
same sequence length as the input, we can determine the positional embeddings
(denoted as $h_{p}$) directly from the hidden embeddings:

$\text{argmin}_{h^{\prime}_{w}}\|\text{MultiheadAttns}_{\text{bottom}}(h^{\prime}_{w}+h_{p})-\text{MultiheadAttns}_{\text{bottom}}(h_{w}+h_{p})\|^{2}.$
(3)

For large language models, all the hidden embeddings have the same size, which
indicates that in (3), the number of variables is the same as the number of
equations. This is preferable to us since that $h^{\prime}_{w}$ is less likely
to have infinite solutions than in the case that the number of variables is
less than the number of equations. We use the gradient descent method to solve
(3). The initialization of $h_{w}^{\prime}$ is also important, as we
empirically found that a random initialization can cause computation errors
when run in half precision. Hence, we use actual sentences to generate the
embedding initialization. For example, if the sequence length is $n$, we can
use $\text{WordEmbedding}(\text{`hello'}\times n)$ as the initial embedding.

###  4.2 Token Reconstruction

After the word embeddings
$h^{\prime}_{w}=(w^{\prime}_{1},\cdots,w^{\prime}_{l})$ is reconstructed, for
each $w^{\prime}_{i}$ we find the token with the most similar embedding in the
word embedding table, i.e.,

$t^{\prime}_{i}=\text{argmax}_{t^{\prime}}\dfrac{\text{WordEmbedding}(t^{\prime})\cdot
w^{\prime}_{i}}{\|\text{WordEmbedding}(t^{\prime})\|\cdot\|w^{\prime}_{i}\|}.$
(4)

Thus, the reconstructed input text is
$(t^{\prime}_{1},\cdots,t^{\prime}_{s})$.

###  4.3 Why not Reconstruct Tokens Directly?

In [18], the authors directly reconstruct tokens from embedding by a token
selection vector $v=(v_{1},\cdots,v_{D})$, where $D$ is the vocabulary size.
The word embedding then becomes a weighted sum:
$h\in\mathbb{R}^{d}=\dfrac{1}{\sum_{j=1}^{D}e^{v_{j}}}\sum_{i}w_{i}e^{v_{i}}$,
where Softmax is to ensure the sum of weights is 1, and $d$ is the dimension
of word embedding. However, notice that $D$ could be as large as 100K, and in
most models, $d$ is much smaller than $D$. This leads infinite solutions of
the equation $\dfrac{1}{{\sum_{j=1}^{D}}e^{v_{j}}}\sum_{i}w_{i}e^{v_{i}}=h$,
contradictory to the fact that $v$ is one-hot.

###  4.4 Adding Noise for Defense

A straightforward and commonly used defense strategy is adding noise to the
hidden embeddings before sending it to the model provider. The most common
type of noise is Gaussian noise, which is widely used in deep learning to
achieve differential privacy [1, 7, 13]. Hence, we also investigate adding
Gaussian noise to the hidden embeddings to protect privacy, i.e., for each
element $h_{i}$ in hidden embeddings $h$, we add a random Gaussian variable to
it: $\tilde{h}_{i}=h_{i}+\epsilon,\epsilon\sim\mathcal{N}(0,\sigma^{2})$. The
standard deviation term $\sigma$ is used to control the scale of the Gaussian
noise.

##  5 Experiments

We choose the ChatGLM6B model [23] for our experiments, and all experiments
are run on the server with Nividia RTX 3090 GPUs. We use the SQuAD QA
(Question-Answering) dataset [17] to evaluate the reconstruction accuracy and
the model performance. For the QA task, we use the greedy generation. The
input is the concatenated context and question, with an average number of
tokens around 220. For the reconstruction attack, we perform gradient descent
until 1000 steps or the cosine similarity between the reconstructed hidden
embedding reaches 0.98. To study the attack performance under different input
lengths, we also use the FindSum dataset [11]. Here we only reconstruct the
input based on the first query’s hidden embeddings.

We use accuracy as the metric for both the QA task and the reconstruction
attack. In the QA task, we view a response as correct if the ground-truth
answer string is contained in the models’ response. In the reconstruction
attack, the accuracy is computed in a token-wise manner:

$\text{attack
accuracy}=\dfrac{1}{N}\sum_{i=1}^{N}\dfrac{1}{l_{i}}\sum_{j=1}^{l_{i}}1_{t_{i,j}=t^{\prime}_{i,j}},$
(5)

where $t_{i,1},\cdots,t_{i,l_{i}}$ is the $i$-th input sequence, and
$t^{\prime}_{i,1},\cdots,t^{\prime}_{i,l_{i}}$ is the sequence reconstructed
by the attacker, and $1_{t=t^{\prime}}$ is the indicator function.

###  5.1 Different Text Lengths

We report the attack accuracy and the number of gradient descent steps taken
for different input lengths in Figure 2. When splitting at the 10th layer, the
attack accuracy is constantly around 100% regardless of the input length. On
the other side, when splitting at the 20th layer, higher attack accuracy
appears on middle-sized inputs (length between 50 and 500). The attack
accuracy drops to around 30% when the input length is around 1,000. The number
of steps for the reconstruction attack shows a similar trend with the
accuracy, i.e., fewer steps are needed for middle-sized inputs, and splitting
at the 20th layer requires significantly more steps than at the 10th layer.

Notably, in our experiments, the maximum number of steps is set to 1,000. Thus
the attack accuracy could increase with more reconstruction steps.

###  5.2 Different Noise Levels

We show the QA accuracy and the attack accuracy under different noise levels
in Figure 1. The performance of the QA task drops significantly when $\sigma$
reaches 5, and becomes zero at $\sigma=15$. On the contrary, the accuracy of
the reconstruction attack is close to 100% at first and gradually decreases as
$\sigma$ increases. When we split the model at higher layers, we did observe a
drop in the attack accuracy. However, even splitting at the 20th layer (which
already reveals about 2/3 of total model parameters), the attack accuracy can
be still around 50% when the QA accuracy becomes around zero. Hence, we can
conclude that simply adding noise in the forward embeddings is not a valid
measure to protect input privacy. The large portion of input can still be
recovered even when the noise already totally damages the model utility. This
is unlike the black-box scenario in [13], where the attack performance drops
faster than the original task performance.

(a) Split layer=10

(b) Split layer=20

Figure 1: QA performance vs. reconstruction performance, with different
$\sigma$ (noise std.).

(a) Attack accuracy vs. input length.

(b) Recontruction steps vs. input length.

Figure 2: Reconstruction attack results with different input length.

##  6 Possible Defenses

###  6.1 New Backbone Architecture

One reason for the easy reconstruction of input is that, the hidden embeddings
are too large and contain too much information in the transformer backbone of
LLMs. For example, hidden embeddings in different layers have the same size as
in the first layer, which enables the attacker to solve the equations. Hence,
the defense of input reconstruction attack may require a re-design of the
current LLM backbone. For example, shrinking the size of hidden embeddings
layer by layer, or gradually dropping embeddings of early tokens. However,
these methods require a modification in the LLM backbone, and the verification
can be very expensive.

###  6.2 Split at Training

The reconstruction attack is effective partly because of the white-box
setting. Hence, it is also possible to train the LLM in the split learning
manner at the beginning, so that no party can obtain the complete model.
However, such methods require that the parties involved in split training do
not collude with each other. Also, potential black-box attacks such like [13]
are possible against such defenses.-

##  7 Conclusion

In this paper, we investigate the possibility of applying vertical federated
learning to large language models in a white-box scenario, i.e., the model is
trained by the model provider but deployed using split learning. We
demonstrate that it is rather simple and costless for the attacker to
reconstruct text from the hidden embeddings in a white-box scenario. The
conventional noise-adding method fails to protect input privacy due to the
model utility will be totally damaged before the noise defense starts to work.
Hence, to realize vertical federated large language models is still a
challenging problem. We also provide several possible solutions to enhance
privacy for vertical federated LLMs.

## References

  * [1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang.  Deep learning with differential privacy.  In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318, 2016.
  * [2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics, 2019.
  * [4] Ye Dong, Wen-jie Lu, Yancheng Zheng, Haoqi Wu, Derun Zhao, Jin Tan, Zhicong Huang, Cheng Hong, Tao Wei, and Wenguang Cheng.  Puma: Secure inference of llama-7b in five minutes.  arXiv preprint arXiv:2307.12533, 2023.
  * [5] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang.  Glm: General language model pretraining with autoregressive blank infilling.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335, 2022.
  * [6] Ege Erdogan, Alptekin Küpçü, and A. Ercüment Çiçek.  Unsplit: Data-oblivious model inversion, model stealing, and label inference attacks against split learning.  In Yuan Hong and Lingyu Wang, editors, Proceedings of the 21st Workshop on Privacy in the Electronic Society, WPES2022, Los Angeles, CA, USA, 7 November 2022, pages 115–124. ACM, 2022.
  * [7] Robin C Geyer, Tassilo Klein, and Moin Nabi.  Differentially private federated learning: A client level perspective.  arXiv preprint arXiv:1712.07557, 2017.
  * [8] Meng Hao, Hongwei Li, Hanxiao Chen, Pengzhi Xing, Guowen Xu, and Tianwei Zhang.  Iron: Private inference on transformers.  Advances in Neural Information Processing Systems, 35:15718–15731, 2022.
  * [9] Zecheng He, Tianwei Zhang, and Ruby B. Lee.  Model inversion attacks against collaborative inference.  In David Balenson, editor, Proceedings of the 35th Annual Computer Security Applications Conference, ACSAC 2019, San Juan, PR, USA, December 09-13, 2019, pages 148–162. ACM, 2019.
  * [10] Xiaoyang Hou, Jian Liu, Jingyu Li, Yuhan Li, Wen-jie Lu, Cheng Hong, and Kui Ren.  Ciphergpt: Secure two-party gpt inference.  Cryptology ePrint Archive, 2023.
  * [11] Shuaiqi Liu, Jiannong Cao, Ruosong Yang, and Zhiyuan Wen.  Long text and multi-table summarization: Dataset and method.  In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1995–2010, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
  * [12] Xinjian Luo, Yuncheng Wu, Xiaokui Xiao, and Beng Chin Ooi.  Feature inference attack on model predictions in vertical federated learning.  In 37th IEEE International Conference on Data Engineering, ICDE 2021, Chania, Greece, April 19-22, 2021, pages 181–192. IEEE, 2021.
  * [13] John X. Morris, Volodymyr Kuleshov, Vitaly Shmatikov, and Alexander M. Rush.  Text embeddings reveal (almost) as much as text.  In EMNLP 2023, 2023.
  * [14] OpenAI.  Introducing chatgpt.  OpenAI blog, 2022.
  * [15] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * [16] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [17] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  SQuAD: 100,000+ Questions for Machine Comprehension of Text.  arXiv e-prints, page arXiv:1606.05250, 2016.
  * [18] Congzheng Song and Ananth Raghunathan.  Information leakage in embedding models.  In CCS ’20: 2020 ACM SIGSAC Conference on Computer and Communications Security, Virtual Event, USA, November 9-13, 2020, pages 377–390. ACM, 2020.
  * [19] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [20] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008, 2017.
  * [21] Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar.  Split learning for health: Distributed deep learning without sharing raw patient data, 2018.
  * [22] Praneeth Vepakomma, Abhishek Singh, Otkrist Gupta, and Ramesh Raskar.  Nopeek: Information leakage reduction to share activations in distributed deep learning.  In Giuseppe Di Fatta, Victor S. Sheng, Alfredo Cuzzocrea, Carlo Zaniolo, and Xindong Wu, editors, 20th International Conference on Data Mining Workshops, ICDM Workshops 2020, Sorrento, Italy, November 17-20, 2020, pages 933–942. IEEE, 2020.
  * [23] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al.  Glm-130b: An open bilingual pre-trained model.  arXiv preprint arXiv:2210.02414, 2022.