# CC-3DT: Panoramic 3D Object Tracking via
Cross-Camera Fusion

Tobias Fischer ∗
ETH Zürich
<EMAIL_ADDRESS>
&Yung-Hsu Yang ∗
National Tsing Hua University
<EMAIL_ADDRESS>
&Suryansh Kumar
ETH Zürich
<EMAIL_ADDRESS>
&Min Sun
National Tsing Hua University
<EMAIL_ADDRESS>
&Fisher Yu
ETH Zürich
<EMAIL_ADDRESS>

###### Abstract

To track the 3D locations and trajectories of the other traffic participants
at any given time, modern autonomous vehicles are equipped with multiple
cameras that cover the vehicle’s full surroundings. Yet, camera-based 3D
object tracking methods prioritize optimizing the single-camera setup and
resort to post-hoc fusion in a multi-camera setup. In this paper, we propose a
method for panoramic 3D object tracking, called CC-3DT, that associates and
models object trajectories both temporally and across views, and improves the
overall tracking consistency. In particular, our method fuses 3D detections
from multiple cameras before association, reducing identity switches
significantly and improving motion modeling. Our experiments on large-scale
driving datasets show that fusion before association leads to a large margin
of improvement over post-hoc fusion. We set a new state-of-the-art with 12.6%
improvement in average multi-object tracking accuracy (AMOTA) among all
camera-based methods on the competitive NuScenes 3D tracking benchmark,
outperforming previously published methods by 6.5% in AMOTA with the same 3D
detector. Project page: vis.xyz/pub/cc-3dt.

**footnotetext: Equal contribution

> Keywords: Autonomous driving, 3D multi-object tracking, Cross-camera fusion

##  1 Introduction

Locating and tracking dynamic objects reliably in 3D is vital to safe
autonomous driving systems. Advanced techniques with complex pipelines
generally use multiple 3D sensor modalities ranging from stereo cameras to
LiDAR and radar. Conversely, a high-quality surround-view camera system
provides a low-cost alternative to expensive 3D active sensors to capture a
vehicle’s surroundings with a panoramic view. Many modern consumer vehicles
are equipped with such multi-camera systems. Yet, it is still challenging to
effectively utilize temporal and cross-camera association of moving objects in
3D. Such an approach can help to track objects in the long term as cameras can
locate them from different viewpoints.

Existing camera-based 3D object tracking methods are designed for single-
camera setups [1, 2, 3]. Since those camera-based approaches follow the
tracking-by-detection paradigm [4], the major difference between them lies in
the association of detections over time [5, 6, 7, 8]. For instance, Zhou et
al. [2] relies on 2D motion to associate objects over time, while Chaabane et
al. [3] utilizes object appearance. In contrast, Hu et al. [1] combines
appearance and 3D motion information. Even though these recent methods perform
well on the temporal association of object tracks, they fail to associate
object tracks in 3D across multiple cameras installed on the vehicle.

Therefore, we propose an effective _cross-camera fusion_ method to the multi-
camera 3D tracking problem. We merge 3D detections and their appearance
information from multiple cameras into a single 3D tracking component that
performs both temporal and cross-camera association. Given the cross-camera
association, we leverage a cross-camera motion model to improve the quality of
single-frame 3D detections even when an object has just appeared in a
particular camera’s field of view (FOV), leading to highly accurate tracking
with much smoother trajectory of objects in 3D. We call the resulting
panoramic 3D object tracking method CC-3DT. The flexible model can work with
various 3D object detection methods.

We test CC-3DT on multiple challenging datasets, namely NuScenes [9] and Waymo
Open [10]. CC-3DT can associate objects both temporally and across cameras,
leading to 17% relative improvement over the single-camera baseline on
NuScenes. CC-3DT can exploit the synergy between cross-camera association,
leading to longer cross-camera trajectories, and improved object motion
modeling, leading to further improved object association. Consequently, we
observe significant improvements in both AMOTP and AMOTA scores. Leveraging
the cross-camera information not only provides smoother object trajectories
but also offers a new state-of-the-art for camera-based 3D object tracking on
the competitive NuScenes [9] benchmark, outperforming existing camera-based
methods by a large margin.

Figure 1: Method overview. We first perform 3D object detection and appearance
feature extraction for each camera view independently. Then, we lift the
camera-space detections into world-space and merge them using non-maximum
suppression [11] in 3D. Next, we associate the detections with existing tracks
across all cameras. Finally, we refine the detections given the trajectories
of the associated tracks across cameras.

##  2 Related Work

#### Camera-based 3D detection.

Driven by the success of convolutional neural networks (CNNs) in object
detection [11, 12, 13, 14, 15], recent works on camera-based 3D object
detection have been using CNNs to regress the 3D parameters of objects from an
image [16, 17, 18, 19, 20, 21]. In contrast, another line of works [22, 23,
24] lifts the image to a 3D point cloud using single image depth prediction
and detects objects in the 3D point cloud, similar to LiDAR-based 3D object
detectors [25, 26, 27]. Lately, transformer networks have been explored for
end-to-end object detection [28]. Inspired by this, Wang et al. [29] propose
to use a transformer network for 3D object detection from multiple cameras.
Our work extends beyond single image 3D detection by performing temporal
refinement of the 3D bounding box parameters, improving upon the initial
detection result.

#### 2D multi-object tracking.

Recent works on MOT follow the tracking-by-detection paradigm [4], where per-
frame object detections are associated using different cues, such as 2D motion
[30, 5, 31, 32, 33, 2], visual appearance [34, 35, 6, 8, 36, 3, 37, 38] or 3D
motion [39, 40, 41, 42, 43]. Recently, researchers have explored the use of
transformers in MOT [44, 45, 46]. These works take inspiration from Carion et
al. [28] and use the query-based detection process for temporal query
propagation to track objects across time with a transformer. In addition,
there is a broad range of literature on multi-camera MOT [47, 48, 49, 50, 51].
Those works focus on scenarios with highly overlapping, stationary
surveillance cameras in e.g. smart city applications [52]. Their aim is to
overcome occlusion problems in monocular MOT systems. On the other hand, our
work focuses on multi-camera tracking in the context of autonomous driving,
where the cameras usually have little overlap and are mounted on a moving
platform.

#### Camera-based 3D multi-object tracking.

In the realm of 3D MOT, there are many methods that rely on LiDAR sensors
since they provide both accurate 3D localization and 360-degree coverage of
the environment [39, 53, 54, 55, 56, 57]. On the contrary, camera-based 3D MOT
has been approached by only a handful of works [1, 2, 3]. Those methods are
conceptually similar to 2D MOT systems, employing tracking-by-detection using
motion or appearance cues for association. While sharing the properties of 2D
MOT systems, the camera-based approaches cannot benefit from the 360-degree
coverage of the LiDAR-based 3D MOT methods. Nevertheless, none of the methods
above consider cross-camera association explicitly. Concurrent to our work,
Zhang et al. [58] propose MUTR3D, a method that combines Wang et al. [29] with
the ideas in [44, 45, 46] into a transformer-based multi-camera 3D tracking
method. However, our work differs significantly from MUTR3D in that we do not
rely on a transformer for joint detection and association, but fuse the
detections from multiple cameras into a single tracking component. Moreover,
our method outperforms MUTR3D significantly in experimental evaluation, even
with a weaker 3D detector.

##  3 Method

We first introduce our panoramic 3D tracking pipeline inspired by Hu et al.
[1, 59]. Afterward, we detail our cross-camera association and motion
modeling. Our method is illustrated in Figure 1.

###  3.1 Overview

Assume we are given a set of input images $I_{t}$ from cameras $M$ and the
corresponding camera locations in world-frame $\xi_{t}$ in a sequence of time
steps $T$. We aim to find the set of tracks
$\mathcal{T}=\\{\mathbf{\tau}^{1},\ldots,\mathbf{\tau}^{N}\\}$ corresponding
to the objects in the scene. Each track $i\in[1,N]$ consists of a set of
states $\langle\tau_{t}^{i},\tau_{t+1}^{i},...\rangle$ depending on the time
steps that the object is visible in. Each state
$\tau_{t}^{i}=\langle\mathbf{f}_{t}^{\tau^{i}},\mathbf{b}_{t}^{\tau^{i}},c_{t}^{\tau^{i}}\rangle$
at a specific time $t\in T$ consists of an appearance embedding
$\mathbf{f}_{t}^{\tau^{i}}\in\mathbb{R}^{256}$, a 3D bounding box
$\mathbf{b}_{t}^{\tau^{i}}\in\mathbb{R}^{7}$ and a confidence value
$c_{t}^{\tau^{i}}\in[0,1]$. The 3D bounding box is defined as
$\mathbf{b}_{t}^{\tau^{i}}=[x,y,z,\theta,l,w,h]$ where $(x,y,z,\theta)$ define
the 3D center of the object and its orientation around the vertical axis in 3D
world-frame, and $(l,w,h)$ define the object dimensions.

#### 3D estimation and merging.

We use a 3D estimation network to extract detections $\mathcal{D}_{t}^{m}$
from an input image $I_{t}^{m}$ of camera $m$ at time $t$. We lift all
detections $\mathcal{D}_{t}^{m}$ from all cameras $m\in M$ to world space with
$\xi_{t}^{m}$ and merge them with non-maximum suppression (NMS) in 3D,
obtaining multi-camera detections $\mathcal{D}_{t}$. The detections
$d\in\mathcal{D}_{t}$ share the same state space with tracks $\mathcal{T}$,
i.e. $d_{t}=\langle\mathbf{f}_{t}^{d},\mathbf{b}_{t}^{d},c_{t}^{d}\rangle$.

#### Data association.

We associate the detections $\mathcal{D}_{t}$ with active tracks
$\mathcal{T}_{t}$ at time $t$ by greedy assignment given an affinity matrix
$\mathbf{A}(\mathcal{T}_{t},\mathcal{D}_{t})\in\mathbb{R}^{||\mathcal{T}_{t}||\times||\mathcal{D}_{t}||}$
that combines appearance, location, and velocity correlation. We compute the
appearance correlation based on the appearance embedding similarity in [37,
38] as

$\mathbf{a}_{\mathtt{deep}}(\tau_{t},d_{t})=\frac{1}{2}\left[\frac{\text{exp}(\mathbf{f}_{t}^{d}\cdot\mathbf{f}_{t}^{\tau})}{\sum_{d\in\mathcal{D}}\text{exp}(\mathbf{f}_{t}^{d}\cdot\mathbf{f}_{t}^{\tau})}+\frac{\text{exp}(\mathbf{f}_{t}^{d}\cdot\mathbf{f}_{t}^{\tau})}{\sum_{\tau\in\mathcal{T}}\text{exp}(\mathbf{f}_{t}^{d}\cdot\mathbf{f}_{t}^{\tau})}\right].$
(1)

The location correlation is based on the absolute difference between the
predicted and observed 3D bounding box parameters as

$\mathbf{a}_{\mathtt{loc}}(\tau_{t},d_{t})=\exp(\frac{-|\mathbf{b}^{\mathbf{\tau}}_{t}-\mathbf{b}^{d}_{t}|}{r}),$
(2)

where $r$ is a constant scalar. The motion correlation considers the absolute
difference between the predicted motion
$V_{\tau_{t}}=\overrightarrow{P_{\tau_{t-1}}P_{\tau_{t}}}$ and observed pseudo
motion $V_{d_{t}}=\overrightarrow{P_{\tau_{t-1}}P_{d_{t}}}$ and the centroid
distance $|P_{\tau_{t}}-P_{d_{t}}|$ as

$\begin{split}\mathbf{a}_{\mathtt{motion}}(\tau_{t},d_{t})&=w_{\mathtt{cos}}\,\mathbf{a}_{\mathtt{centroid}}+(1-w_{\mathtt{cos}})\,\mathbf{a}_{\mathtt{pseudo}}\\\
\text{where},~{}\mathbf{a}_{\mathtt{centroid}}&=\exp(\frac{-|P_{\tau_{t}}-P_{d_{t}}|}{r}),~{}\mathbf{a}_{\mathtt{pseudo}}=\exp(\frac{-|V_{\tau_{t}}-V_{d_{t}}|}{r}),~{}w_{\mathtt{cos}}=\angle(V_{\tau_{t}},V_{d_{t}}).\end{split}$
(3)

$w_{\mathtt{cos}}$ is defined as cosine similarity of two motion vectors to
favor centroid distance if the motion vectors are in the same hemisphere and
pseudo motion otherwise, so that we do not match a pair with conflicting
motions. Note that we transform object center $P$ into world-frame using $\xi$
to account for ego-motion. Given the individual affinities,
$\mathbf{A}(\mathcal{T}_{t},\mathcal{D}_{t})$ is computed as

$\begin{split}\mathbf{A}(\mathcal{T}_{t},\mathcal{D}_{t})&=w_{\mathtt{deep}}\mathbf{A}_{\mathtt{deep}}(\mathcal{T}_{t},\mathcal{D}_{t})+(1-w_{\mathtt{deep}})\mathbf{A}_{\mathtt{motion}}(\mathcal{T}_{t},\mathcal{D}_{t})\cdot\mathbf{A}_{\mathtt{loc}}(\mathcal{T}_{t},\mathcal{D}_{t}),\end{split}$
(4)

where $w_{\mathtt{deep}}$ is a constant that weighs appearance and a hybrid of
motion and location correlation.

#### Motion model.

We employ two LSTM networks $\Phi_{\mathtt{pred}}$ and
$\Phi_{\mathtt{update}}$ to handle the extrapolation and update of
trajectories given track histories and new observations, respectively. Before
data association, we use $\Phi_{\mathtt{pred}}$ to propagate the previous
object state $\mathbf{b}_{t-1}$ one time step into the future to compute the
location and motion correlation. We input the previous $n-1=5$ object
velocities $\mathbf{v}_{t-n:t-1}^{\tau}$ and the previous LSTM hidden state
$\mathbf{h}^{\mathtt{pred}}_{t-1}$ into the network, where an object velocity
is defined as
$\mathbf{v}_{t}^{\tau}=\mathbf{b}_{t}^{\tau}-\mathbf{b}_{t-1}^{\tau}$. The
network outputs the predicted velocity
$\mathbf{\hat{v}}_{t}^{\tau},\mathbf{h}_{t}^{\mathtt{pred}}=\Phi_{\mathtt{pred}}(\mathbf{v}_{t-n:t-1}^{\tau},\mathbf{h}^{\mathtt{pred}}_{t-1})$.
After data association, we use $\Phi_{\mathtt{update}}$ to estimate a velocity
refinement given the predicted velocity $\mathbf{\hat{v}}_{t}^{\tau}$,
observed velocity $\mathbf{v^{\prime}}_{t}^{\tau}$, confidence of the
associated detection $c_{t}^{\tau}$ and previous hidden state
$\mathbf{h}^{\mathtt{pred}}_{t-1}$. The velocity refinement
$\mathbf{v}_{t}^{\tau},\mathbf{h}^{\mathtt{update}}_{t}=\Phi_{\mathtt{update}}([\mathbf{\hat{v}}_{t}^{\tau},\mathbf{v^{\prime}}_{t}^{\tau},c_{t}^{\tau}],\mathbf{h}^{\mathtt{update}}_{t-1})$
in turn refines the object state
$\mathbf{b}_{t}^{\tau}=\mathbf{b}_{t-1}^{\tau}+\mathbf{v}_{t}^{\tau}$. Note
that for both networks, we first use a linear projection to encode each input
into a $64$-dimensional vector which we omit for ease of notation.

#### Track management.

In order to handle newly appearing, disappearing, and reappearing objects
during inference, we keep an online track memory where each track
$\tau\in\mathcal{T}$ is assigned one of three possible states
$\\{\mathtt{active},\mathtt{inactive},\mathtt{dead}\\}$. At each time step
$t$, all tracks currently marked as $\mathtt{active}$ or $\mathtt{inactive}$
are used to calculate their affinity with the current detections
$\mathcal{D_{t}}$. If a detection $d_{t}\in\mathcal{D_{t}}$ matches with a
track $\tau\in\mathcal{T}$, $d_{t}$, it will be assigned to $\tau$. If a track
$\tau$ has no matching detection, it will transition into state
$\mathtt{inactive}$. If $\tau$ did not have a matching detection for $L$ time
steps, $\tau$ transitions into state $\mathtt{dead}$ and will no longer be
considered. Note that we propagate the position of inactive tracks forward in
time with $\Phi_{\mathtt{pred}}$ to properly match tracks in cases of
occlusion. Each detection without matching track spawns a new track in
$\mathtt{active}$ state.

###  3.2 Cross-camera association

Hu et al. [1] employs a separate tracker for each camera $m\in M$. In
particular, the data association of each camera happens independently, as well
as the track management and motion modeling. Since each camera has only a
limited FOV, there are many disappearing and re-appearing tracks, making the
association prone to error and the trajectories less stable. In addition,
object identity will not be preserved across camera views. We explore how a
panoramic multi-camera tracker can overcome those limitations with a few
simple modifications to the tracking pipeline.

First, we investigate how performance is affected if we use NMS in 3D to
naively combine the existing tracks from single-camera trackers. We call this
approach detect $\rightarrow$ track $\rightarrow$ merge. Next, we introduce
the detect $\rightarrow$ merge $\rightarrow$ track paradigm, i.e. instead of
aggregating the tracking result of camera-dependent trackers, we first
aggregate the detection results from all cameras $m\in M$ and subsequently
apply global data association, motion modeling, and track management. The
advantages of this are threefold. (i) In the regions where the cameras have an
overlapping FOV, the multi-camera tracker has access to more detection
candidates, meaning that an object that is highly truncated in one camera can
be better recognized in a neighboring camera. (ii) Cross-camera data
association results in long trajectories that span across multiple cameras,
which allows for a richer track history and thus better motion modeling. (iii)
Our data association scheme leverages appearance, location and velocity
information to follow tracks more effectively across cameras compared to 3D
NMS based merging of single-camera tracking results.

###  3.3 Cross-camera motion modeling

A crucial advantage of our panoramic tracker is that we can exploit longer
trajectories across camera views for motion modeling. This leads to an
improved multi-frame refinement via $\Phi_{\mathtt{update}}$ and more accurate
location prediction via $\Phi_{\mathtt{pred}}$, which in turn benefits the
data association since we can expect better estimates for the affinities
$\mathbf{a}_{\mathtt{loc}}(\tau_{t},d_{t})$ and
$\mathbf{a}_{\mathtt{motion}}(\tau_{t},d_{t})$. In addition, we propose to
integrate cross-camera trajectories into the training process of the motion
model. The intuition is that due to object truncation, single-view 3D
localization at image borders is usually brittle. Thus, if we train the motion
model on these trajectories, it can handle those situations better. In
contrast to Hu et al. [1], who create the trajectory training data per camera
and train on the union of the single-camera trajectories, we create a cross-
camera trajectory dataset that matches ground truth trajectories spanning
multiple cameras to the aggregated set of detections from all cameras. To
match detections to a ground truth trajectory, we use birds-eye-view (BEV)
distance instead of Intersection-over-Union (IoU) of 2D bounding boxes in Hu
et al. [1].

To train $\Phi_{\mathtt{pred}}$ and $\Phi_{\mathtt{update}}$, we randomly
sample a ground truth trajectory and extract the matched ground truth and
detection pairs in a random temporal window $\Delta t$ of fixed size, possibly
spanning across multiple camera views. We use the set of detections to
simulate inference, i.e. we obtain the predicted velocities
$\mathbf{\hat{v}}_{\Delta t}$ from $\Phi_{\mathtt{pred}}$ given the track
history and the refined velocities $\mathbf{v}_{\Delta t}$ from
$\Phi_{\mathtt{update}}$ given the predicted velocity, the observed velocity,
and the detection confidence. We use the ground truth trajectory to calculate
the ground truth velocities $\mathbf{\tilde{v}}_{\Delta t}$ and the trajectory
loss

$\mathcal{L}_{\mathtt{traj}}(\mathbf{v}_{\Delta t},\mathbf{\hat{v}}_{\Delta
t},\mathbf{\tilde{v}}_{\Delta t})=\frac{1}{||\Delta t||}\sum_{t\in\Delta
t}\delta(\mathbf{\hat{v}}_{t},\mathbf{\tilde{v}}_{t})+\delta(\mathbf{v}_{t},\mathbf{\tilde{v}}_{t}),$
(5)

where $\delta(\cdot,\cdot)$ is the Huber loss [60]. Moreover, we add a
regularization term that encourages $\Phi_{\mathtt{pred}}$ to predict a smooth
trajectory over $\Delta t$:

$\mathcal{L}_{\mathtt{linear}}(\mathbf{\hat{v}}_{\Delta t})=\frac{1}{||\Delta
t||}\sum_{t\in\Delta
t}||(\mathbf{\hat{v}}_{t+1}-\mathbf{\hat{v}}_{t})-(\mathbf{\hat{v}}_{t}-\mathbf{\hat{v}}_{t-1})||_{1}.$
(6)

Finally, we calculate the overall loss $\mathcal{L}_{\mathtt{motion}}$ as a
weighted combination of the above terms

$\mathcal{L}_{\mathtt{motion}}(\mathbf{v}_{\Delta t},\mathbf{\hat{v}}_{\Delta
t},\mathbf{\tilde{v}}_{\Delta
t})=\mathcal{L}_{\mathtt{traj}}(\mathbf{v}_{\Delta t},\mathbf{\hat{v}}_{\Delta
t},\mathbf{\tilde{v}}_{\Delta
t})+w_{\mathtt{linear}}\mathcal{L}_{\mathtt{linear}}(\mathbf{\hat{v}}_{\Delta
t}).$ (7)

##  4 Experiments

#### Datasets.

We evaluate our method on two popular 3D MOT benchmarks. First, we experiment
on the NuScenes benchmark [9] which consists of street scenes captured in
varying scenery and geographic areas in Boston and Singapore. The scenes are
captured from a moving vehicle with a multiple sensors attached. It provides
six different camera views with a 360-degree coverage of the environment. In
total, the dataset contains 1000 sequences sampled at 2 Hz. Second, we
evaluate our method on the Waymo Open benchmark [10], which has similar
characteristics to the NuScenes benchmark. However, the sampling rate is 10
Hz, and the five cameras mounted on the vehicle cover only about a $5/8$
azimuth angle of the horizon.

#### Evaluation metrics.

We follow the evaluation protocol in NuScenes using the metrics in Weng et al.
[53]. The main metrics are average multi object tracking accuracy (AMOTA) and
average multi object tracking precision (AMOTP) which are integrals over MOTA
/ MOTP [61] scores computed via n-point interpolation. AMOTA is defined as
$\mathrm{AMOTA}=\frac{1}{n-1}\sum_{r\in\\{\frac{1}{n-1},\frac{2}{n-1},\ldots,1\\}}\mathrm{MOTA}_{r}$,
where
$\mathrm{MOTA}_{r}=\max{(0,1-\frac{\mathrm{IDS_{r}}+\mathrm{FP}_{r}+\mathrm{FN}_{r}+-(1-r)\mathrm{P}}{r\mathrm{P}})}$.
Specifically, $\mathrm{MOTA}_{r}$ is a recall normalized MOTA and
$\mathrm{IDS}_{r}$, $\mathrm{FP}_{r}$, $\mathrm{FN}_{r}$ are number of ID
switches, false positives and false negatives at recall $r$. $\mathrm{P}$ is
the total number of ground truth positives. AMOTP is defined as
$\mathrm{AMOTP}=\frac{1}{n-1}\sum_{r\in\\{\frac{1}{n-1},\frac{2}{n-1},\ldots,1\\}}\frac{\sum_{i,t}d_{i,t}}{\sum_{t}\mathrm{TP}_{t}}$
where $d_{i,t}$ is the BEV distance between track $i$ and its ground truth
track at time $t$ and $TP_{t}$ denotes the number of matched ground truth
tracks at time $t$.

#### Implementation details.

We train the 3D estimation network using ResNet-101 [62] as backbone with
batch size of 32 for 24 epochs with random image flipping as data augmentation
on 8 V100 GPUs. The initial learning rate is set to $0.01$ with a linear
learning rate warm-up in the first 1000 iterations. We decay the learning rate
by a factor of $10$ at epochs $16$ and $22$, respectively. We use SGD [63]
with a momentum of $0.9$ and a weight decay of $0.0001$. For NuScenes, we use
$1600\times 900$ resolution and train on the training set only. On Waymo Open,
we scale the images to $1280\times 854$ resolution and use batch size of $16$
during training. We decay the learning rate by a factor of $5$ at epochs
$8,12,16,20,22$, respectively. To merge 3D detections across cameras, we use
NMS with a 3D IoU threshold of $0.1$. To generate the cross-camera trajectory
dataset, we use a matching threshold of $2$ meters BEV distance. We train
$\Phi_{\mathtt{pred}}$ and $\Phi_{\mathtt{update}}$ with batch size of $128$
for $100$ epochs. We use $w_{\mathtt{linear}}=0.001$ and Adam [64] with
AMSGrad and a weight decay of $0.0001$.

Table 1: Pipeline and motion modeling ablation study. We compare the tracking
performance of our method with the single-camera baseline [1] and the detect
$\rightarrow$ track $\rightarrow$ merge paradigm on the NuScenes validation
split. We further analyze the benefit of motion modeling across all pipelines,
both with LSTM motion model and a Kalman Filter baseline (KF3D).

Pipeline | Motion Model | AMOTA $\uparrow$ | AMOTP $\downarrow$ | RECALL $\uparrow$ | MOTA $\uparrow$ | IDS $\downarrow$
---|---|---|---|---|---|---
QD-3DT [1] | - | 0.221 | 1.529 | 0.371 | 0.193 | 5256
KF3D | 0.232 | 1.530 | 0.393 | 0.206 | 5574
LSTM | 0.247 | 1.507 | 0.399 | 0.218 | 5919
Detect $\rightarrow$ Track $\rightarrow$ Merge | - | 0.248 | 1.500 | 0.386 | 0.212 | 4451
KF3D | 0.256 | 1.506 | 0.408 | 0.225 | 4256
LSTM | 0.264 | 1.490 | 0.413 | 0.235 | 4470
Detect $\rightarrow$ Merge $\rightarrow$ Track | - | 0.249 | 1.490 | 0.405 | 0.212 | 2399
KF3D | 0.279 | 1.501 | 0.415 | 0.252 | 1982
LSTM | 0.283 | 1.490 | 0.444 | 0.262 | 2131

Table 2: Motion model training ablation study. We compare our modified training scheme for the LSTM motion model with cross-camera trajectories to the single-camera baseline on the NuScenes validation split. $\dagger$ denotes using new training scheme for 3D estimation network. Method | Trajectory data | matching | Total Trajectories | AMOTA $\uparrow$ | AMOTP $\downarrow$ | IDS $\downarrow$
---|---|---|---|---|---|---
CC-3DT | Single-camera | 2D IoU | 29527 | 0.283 | 1.490 | 2131
CC-3DT | Cross-camera | BEV Distance | 36112 | 0.289 | 1.483 | 2129
$\text{CC-3DT}\dagger$ | 0.311 | 1.433 | 2536

###  4.1 Ablation studies

#### Cross-camera association.

In Table 1, we compare the pipeline in Hu et al. [1], naive merging of single
camera results (detect $\rightarrow$ track $\rightarrow$ merge) and our
pipeline described in section 3.2 using the LSTM motion model, a Kalman Filter
baseline (KF3D) and no motion model. Note that all individual components are
consistent while only changing the high-level pipeline to provide a fair
comparison. We observe that naive merging helps the overall performance and
results in a gain of $1.7$% in AMOTA using the LSTM motion model. With our
cross-camera association (detect $\rightarrow$ merge $\rightarrow$ track), we
observe another $1.9$% gain in AMOTA while the IDS are cut in half,
highlighting the effectiveness of cross-camera association. Note that while
KF3D has slightly less IDS than LSTM, its recall level is also lower, i.e. IDS
are lower due to lower recall not better association. Interestingly, the gain
of cross-camera association is very small when not using a motion model
($0.1$%), supporting our hypothesis that the motion model benefits greatly
from cross-camera association. Moreover, the influence of the motion model on
the final tracking performance is the highest when using cross-camera
association, improving from $24.9\%$ to $28.3\%$ AMOTA. Our results underline
the importance of cross-camera association to achieve consistent tracking
results.

#### Cross-camera motion modeling.

Table 1 also shows that motion modeling improves through the use of cross-
camera trajectories, i.e. we observe a higher gain in AMOTA of 3.4% (LSTM) and
2.0% (KF3D) versus using no motion model compared to only 2.6% (LSTM) and 1.1%
(KF3D) when using the pipeline in Hu et al. [1]. The effect of training the
motion model with cross-camera trajectories (see section 3.3) is illustrated
in Table 2. We observe another improvement of 0.6% in AMOTA compared to
utilizing cross-camera trajectories only during inference. Moreover, Table 2
shows that the total number of trajectories for training increases, which can
be attributed to the fact that for a trajectory to be considered for training,
it needs to be at least $||\Delta t||$ time steps long. Cross-camera
trajectories help to increase training data diversity. In sum, we achieve a
gain of 4.2% in AMOTA, equating to a 17% relative improvement compared to the
single-camera baseline.

#### Qualitative comparison.

Figure 2 illustrates CC-3DT results compared to the baselines in Table 1.
While our proposed CC-3DT preserves object identity and exhibits a smooth,
accurate trajectory for the object moving across camera views, the baselines
struggle with object localization and object association. Specifically, QD-3DT
[1] cannot preserve the object identity while naive merging (detect
$\rightarrow$ track $\rightarrow$ merge) results in poor localization, showing
the benefit of our method.

Table 3: Comparison to MUTR3D [58] using different motion models. We report tracking performance on the NuScenes validation split. We use DETR3D [29] as detector for fair comparison. Method | Motion model | AMOTA $\uparrow$ | AMOTP $\downarrow$ | RECALL $\uparrow$ | MOTA $\uparrow$ | IDS $\downarrow$
---|---|---|---|---|---|---
MUTR3D [58] | SimpleTrack | 0.293 | 1.307 | 0.418 | 0.263 | 1695
FFN | 0.294 | 1.498 | 0.427 | 0.267 | 3822
CC-3DT | - | 0.307 | 1.394 | 0.434 | 0.263 | 2419
KF3D | 0.353 | 1.379 | 0.478 | 0.316 | 2050
LSTM | 0.359 | 1.361 | 0.498 | 0.326 | 2152

| QD-3DT | Detect$\rightarrow$Track$\rightarrow$Merge | CC-3DT
---|---|---|---
Back ($t$) |  |  |
Back Left ($t+1$) |  |  |
Back Left ($t+2$) |  |  |
3D View |  |  |

Figure 2: Qualitative comparison. Different types of cross-camera association
on NuScenes validation split. We plot the 3D car states in camera and 3D views
and depict car identity as color. Clearly, CC-3DT provides a smooth and
precise trajectory across cameras (green). Detect $\rightarrow$ Track
$\rightarrow$ Merge shows awry car localization (purple), while QD-3DT lacks
car identity (violet).

#### 3D estimation network training scheme.

Following [29] and different from [1], we optimize the 3D estimation network
training scheme by increasing the learning rate of the 3D detection head by a
factor of 10. As shown in Table 2, this improves the overall performance by
$2.2\%$ in AMOTA. Note that IDS increase due to increased recall of our
detector.

###  4.2 Comparison to state-of-the-art

Table 4: Comparison to state-of-the-art. We compare the tracking performance of our method with existing camera-based methods on the NuScenes validation and testing sets.  | Method | Detector | AMOTA $\uparrow$ | AMOTP $\downarrow$ | RECALL $\uparrow$ | MOTA $\uparrow$ | IDS $\downarrow$
---|---|---|---|---|---|---|---
Validation | DEFT [3] | CenterNet [65] | 0.201 | N/A | N/A | 0.171 | N/A
QD-3DT [1] | Faster-RCNN [11] | 0.247 | 1.507 | 0.405 | 0.221 | 5919
MUTR3D [58] | DETR3D [29] | 0.294 | 1.498 | 0.427 | 0.267 | 3822
CC-3DT | Faster-RCNN [11] | 0.311 | 1.433 | 0.472 | 0.278 | 2536
DETR3D [29] | 0.359 | 1.361 | 0.498 | 0.326 | 2152
BEVFormer [66] | 0.429 | 1.257 | 0.534 | 0.385 | 2219
Testing | CenterTrack [2] | CenterNet [65] | 0.046 | 1.543 | 0.233 | 0.043 | 3807
DEFT [3] | CenterNet [65] | 0.177 | 1.564 | 0.338 | 0.156 | 6901
QD-3DT [1] | Faster-RCNN [11] | 0.217 | 1.550 | 0.375 | 0.198 | 6856
MUTR3D [58] | DETR3D [29] | 0.270 | 1.494 | 0.411 | 0.245 | 6018
CC-3DT | Faster-RCNN [11] | 0.284 | 1.470 | 0.427 | 0.251 | 2945
DETR3D [29] | 0.357 | 1.343 | 0.472 | 0.317 | 3070
BEVFormer [66] | 0.410 | 1.274 | 0.538 | 0.357 | 3334

#### Comparison on the NuScenes benchmark.

We evaluate our method the NuScenes benchmark in Table 4. On the validation
set, we outperform the state-of-the-art method MUTR3D by 1.7% with a weaker
detector, by 6.5% in AMOTA with the same detector (DETR3D [29]), and by 13.5%
with a stronger detector (BEVFormer [66]). Our AMOTP is significantly lower
(1.361 vs. 1.498) with the same detector, meaning that our trajectories are
more precise. We also have the lowest IDS, meaning that our association works
best. The testing set results corroborate our findings, i.e. we outperform
MUTR3D by as much as 12.4% in AMOTA and achieve the lowest AMOTP and IDS. We
set a new state-of-the-art of 42.9% and 41.0% AMOTA on the validation and
testing sets, respectively.

#### Comparison to MUTR3D.

Table 3 provides additional comparison to MUTR3D using different motion
models. MUTR3D reports results with a Kalman Filter baseline [67] and a feed-
forward network (FFN). While the FFN can only improve marginally upon the
Kalman Filter baseline, our motion model significantly outperforms the Kalman
Filter baseline (KF3D). Also, our cross-camera association is more effective
than the transformer-based one in MUTR3D (lower IDS). Note that while MUTR3D
with Kalman Filter has the lowest IDS, the detection recall is also much
lower.

Table 5: Comparison on Waymo Open. We compare our 3D Vehicle Tracking results
with QD-3DT on the validation set of the Waymo Open benchmark (Level 2
difficulty).

Method | 3D IoU | MOTA $\uparrow$ | Mismatch $\downarrow$
---|---|---|---
QD-3DT [1] | 0.3 | 0.1867 | 0.0134
0.5 | 0.0308 | 0.0055
CC-3DT | 0.3 | 0.2032 | 0.0069
0.5 | 0.0480 | 0.0018

#### Comparison on the Waymo Open benchmark.

We compare with [1] on the validation split of Waymo Open. The comparison in
Table 5 shows that our method performs better in MOTA than QD-3DT, while also
having a lower Mismatch ratio, meaning that we have both better overall and
association performance. We outperform QD-3DT significantly on each of the 3D
IoU thresholds. Note that we do not evaluate with an IoU threshold of $0.7$,
as it is too restrictive for camera-based methods to achieve a meaningful
score.

###  4.3 Limitations

Our method could fail if the predicted 3D detection is amiss. Specifically, in
cases of high occlusion or truncation, the object’s 3D properties are often
ambiguous to the observer, e.g. an object’s 3D center could even be out of the
camera FOV. This is especially pronounced for big or close objects that are
often only partially visible. While combining information from multiple time
steps in our motion model alleviates those issues, integrating this
information during the detection stage could further improve the results.
These challenges can also influence the classification performance of the 3D
detector. If an object is highly occluded or truncated, it is often
challenging to infer its category from a single image, especially for
semantically similar classes (e.g. bus and truck). However, if the category is
estimated incorrectly, object identity may not be preserved.

##  5 Conclusion

This paper presents an effective 3D object tracking method in a multi-camera
setting. The key idea we present is to merge 3D detections of multiple cameras
into a single 3D tracking component that handles both temporal and cross-
camera association. By leveraging cross-camera association, we observe
improved motion model quality, leading to enhanced object association and
smoother trajectories across camera views. As a result, the suggested approach
outperforms existing methods by a large margin achieving state-of-the-art
results on the widely-used NuScenes benchmark.

##  6 Acknowledgments

This work is supported by Ministry of Science and Technology of Taiwan (MOST
110-2634-F-002-051). We gratefully thank National Center for High-performance
(NCHC) for the storage and computational resource.

## References

  * Hu et al. [2022] H.-N. Hu, Y.-H. Yang, T. Fischer, T. Darrell, F. Yu, and M. Sun.  Monocular quasi-dense 3d object tracking.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * Zhou et al. [2020] X. Zhou, V. Koltun, and P. Krähenbühl.  Tracking objects as points.  In _European Conference on Computer Vision_ , 2020.
  * Chaabane et al. [2021] M. Chaabane, P. Zhang, J. R. Beveridge, and S. O’Hara.  Deft: Detection embeddings for tracking.  _arXiv preprint arXiv:2102.02267_ , 2021.
  * Ramanan and Forsyth [2003] D. Ramanan and D. A. Forsyth.  Finding and tracking people from the bottom up.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2003\.
  * Yu et al. [2016] F. Yu, W. Li, Q. Li, Y. Liu, X. Shi, and J. Yan.  POI: multiple object tracking with high performance detection and appearance feature.  In _European Conference on Computer Vision Workshop_ , 2016.
  * Wojke et al. [2017] N. Wojke, A. Bewley, and D. Paulus.  Simple online and realtime tracking with a deep association metric.  In _International Conference on Image Processing_ , 2017.
  * Bergmann et al. [2019] P. Bergmann, T. Meinhardt, and L. Leal-Taixé.  Tracking without bells and whistles.  _arXiv preprint arXiv:1903.05625_ , 2019.
  * Lu et al. [2020] Z. Lu, V. Rathod, R. Votel, and J. Huang.  Retinatrack: Online single stage joint detection and tracking.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2020\.
  * Caesar et al. [2020] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2020\.
  * Sun et al. [2020] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2020\.
  * Ren et al. [2017] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2017.
  * Liu et al. [2016] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg.  Ssd: Single shot multibox detector.  In _European Conference on Computer Vision_ , 2016.
  * Redmon et al. [2016] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi.  You only look once: Unified, real-time object detection.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2016\.
  * Redmon and Farhadi [2017] J. Redmon and A. Farhadi.  Yolo9000: Better, faster, stronger.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2017\.
  * Lin et al. [2017] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár.  Focal loss for dense object detection.  In _IEEE International Conference on Computer Vision_ , 2017.
  * Chen et al. [2015] X. Chen, K. Kundu, Y. Zhu, A. G. Berneshawi, H. Ma, S. Fidler, and R. Urtasun.  3D object proposals for accurate object class detection.  In _Advances in Neural Information Processing Systems_ , 2015.
  * Chen et al. [2016] X. Chen, K. Kundu, Z. Zhang, H. Ma, S. Fidler, and R. Urtasun.  Monocular 3D object detection for autonomous driving.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2016\.
  * Mousavian et al. [2017] A. Mousavian, D. Anguelov, J. Flynn, and J. Košecká.  3D bounding box estimation using deep learning and geometry.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2017\.
  * Brazil and Liu [2019] G. Brazil and X. Liu.  M3D-RPN: Monocular 3D region proposal network for object detection.  In _IEEE International Conference on Computer Vision_ , 2019.
  * Simonelli et al. [2019] A. Simonelli, S. R. Bulo, L. Porzi, M. López-Antequera, and P. Kontschieder.  Disentangling monocular 3D object detection.  In _IEEE International Conference on Computer Vision_ , 2019.
  * Park et al. [2021] D. Park, R. Ambrus, V. Guizilini, J. Li, and A. Gaidon.  Is pseudo-lidar needed for monocular 3d object detection?  In _IEEE International Conference on Computer Vision_ , 2021.
  * Wang et al. [2019] Y. Wang, W.-L. Chao, D. Garg, B. Hariharan, M. Campbell, and K. Q. Weinberger.  Pseudo-LiDAR from visual depth estimation: Bridging the gap in 3D object detection for autonomous driving.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019\.
  * Ma et al. [2019] X. Ma, Z. Wang, H. Li, P. Zhang, W. Ouyang, and X. Fan.  Accurate monocular 3D object detection via color-embedded 3D reconstruction for autonomous driving.  In _IEEE International Conference on Computer Vision_ , 2019.
  * Reading et al. [2021] C. Reading, A. Harakeh, J. Chae, and S. L. Waslander.  Categorical depth distribution network for monocular 3d object detection.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2021\.
  * Zhou and Tuzel [2018] Y. Zhou and O. Tuzel.  Voxelnet: End-to-end learning for point cloud based 3d object detection.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2018\.
  * Shi et al. [2019] S. Shi, X. Wang, and H. Li.  PointRCNN: 3D object proposal generation and detection from point cloud.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019\.
  * Lang et al. [2019] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom.  PointPillars: Fast encoders for object detection from point clouds.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019\.
  * Carion et al. [2020] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko.  End-to-end object detection with transformers.  In _European Conference on Computer Vision_ , 2020.
  * Wang et al. [2021] Y. Wang, V. Guizilini, T. Zhang, Y. Wang, H. Zhao, , and J. M. Solomon.  Detr3d: 3d object detection from multi-view images via 3d-to-2d queries.  In _The Conference on Robot Learning (CoRL)_ , 2021.
  * Bewley et al. [2016] A. Bewley, Z. Ge, L. Ott, F. T. Ramos, and B. Upcroft.  Simple online and realtime tracking.  In _International Conference on Image Processing_ , 2016.
  * Xiao et al. [2018] B. Xiao, H. Wu, and Y. Wei.  Simple baselines for human pose estimation and tracking.  In _European Conference on Computer Vision_ , 2018.
  * Held et al. [2016] D. Held, S. Thrun, and S. Savarese.  Learning to track at 100 FPS with deep regression networks.  In _European Conference on Computer Vision_ , 2016.
  * Feichtenhofer et al. [2017] C. Feichtenhofer, A. Pinz, and A. Zisserman.  Detect to track and track to detect.  In _IEEE International Conference on Computer Vision_ , 2017.
  * Kim et al. [2015] C. Kim, F. Li, A. Ciptadi, and J. M. Rehg.  Multiple hypothesis tracking revisited.  In _IEEE International Conference on Computer Vision_ , 2015.
  * Leal-Taixé et al. [2016] L. Leal-Taixé, C. Canton-Ferrer, and K. Schindler.  Learning by tracking: Siamese CNN for robust target association.  In _IEEE Conference on Computer Vision and Pattern Recognition Workshop_ , 2016.
  * Zhang et al. [2020] Y. Zhang, C. Wang, X. Wang, W. Zeng, and W. Liu.  Fairmot: On the fairness of detection and re-identification in multiple object tracking.  _arXiv preprint arXiv:2004.01888_ , 2020.
  * Pang et al. [2021] J. Pang, L. Qiu, X. Li, H. Chen, Q. Li, T. Darrell, and F. Yu.  Quasi-dense similarity learning for multiple object tracking.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2021\.
  * Fischer et al. [2022] T. Fischer, J. Pang, T. E. Huang, L. Qiu, H. Chen, T. Darrell, and F. Yu.  Qdtrack: Quasi-dense similarity learning for appearance-only multiple object tracking.  _arXiv preprint arXiv:2210.06984_ , 2022.
  * Held et al. [2013] D. Held, J. Levinson, and S. Thrun.  Precision tracking with sparse 3d and dense color 2d data.  In _IEEE International Conference on Robotics and Automation_ , 2013\.
  * Mitzel and Leibe [2012] D. Mitzel and B. Leibe.  Taking mobile multi-object tracking to the next level: People, unknown objects, and carried items.  In _European Conference on Computer Vision_ , 2012.
  * Osep et al. [2017] A. Osep, W. Mehner, M. Mathias, and B. Leibe.  Combined image-and world-space tracking in traffic scenes.  In _IEEE International Conference on Robotics and Automation_ , 2017\.
  * Sharma et al. [2018] S. Sharma, J. A. Ansari, J. K. Murthy, and K. M. Krishna.  Beyond pixels: Leveraging geometry and shape cues for online multi-object tracking.  In _IEEE International Conference on Robotics and Automation_ , 2018\.
  * Luiten et al. [2020] J. Luiten, T. Fischer, and B. Leibe.  Track to reconstruct and reconstruct to track.  _IEEE Robotics and Automation Letters_ , 2020.
  * Meinhardt et al. [2021] T. Meinhardt, A. Kirillov, L. Leal-Taixe, and C. Feichtenhofer.  Trackformer: Multi-object tracking with transformers.  _arXiv preprint arXiv:2101.02702_ , 2021.
  * Sun et al. [2020] P. Sun, J. Cao, Y. Jiang, R. Zhang, E. Xie, Z. Yuan, C. Wang, and P. Luo.  Transtrack: Multiple-object tracking with transformer.  _arXiv preprint arXiv: 2012.15460_ , 2020.
  * Zeng et al. [2021] F. Zeng, B. Dong, T. Wang, X. Zhang, and Y. Wei.  Motr: End-to-end multiple-object tracking with transformer.  _arXiv preprint arXiv:2105.03247_ , 2021.
  * Tang et al. [2019] Z. Tang, M. Naphade, M.-Y. Liu, X. Yang, S. Birchfield, S. Wang, R. Kumar, D. Anastasiu, and J.-N. Hwang.  Cityflow: A city-scale benchmark for multi-target multi-camera vehicle tracking and re-identification.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019\.
  * Han et al. [2021] X. Han, Q. You, C. Wang, Z. Zhang, P. Chu, H. Hu, J. Wang, and Z. Liu.  Mmptrack: Large-scale densely annotated multi-camera multiple people tracking benchmark.  _arXiv preprint arXiv:2111.15157_ , 2021.
  * Ristani and Tomasi [2018] E. Ristani and C. Tomasi.  Features for multi-target multi-camera tracking and re-identification.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2018\.
  * He et al. [2020] Y. He, X. Wei, X. Hong, W. Shi, and Y. Gong.  Multi-target multi-camera tracking by tracklet-to-target assignment.  _IEEE Transactions on Image Processing_ , 2020.
  * Shim et al. [2021] K. Shim, S. Yoon, K. Ko, and C. Kim.  Multi-target multi-camera vehicle tracking for city-scale traffic management.  In _IEEE Conference on Computer Vision and Pattern Recognition Workshop_ , 2021.
  * Naphade et al. [2021] M. Naphade, S. Wang, D. C. Anastasiu, Z. Tang, M.-C. Chang, X. Yang, Y. Yao, L. Zheng, P. Chakraborty, C. E. Lopez, A. Sharma, Q. Feng, V. Ablavsky, and S. Sclaroff.  The 5th ai city challenge.  In _IEEE Conference on Computer Vision and Pattern Recognition Workshop_ , 2021.
  * Weng et al. [2020] X. Weng, J. Wang, D. Held, and K. Kitani.  3d multi-object tracking: A baseline and new evaluation metrics.  In _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2020.
  * Yin et al. [2021] T. Yin, X. Zhou, and P. Krähenbühl.  Center-based 3d object detection and tracking.  _IEEE Conference on Computer Vision and Pattern Recognition_ , 2021\.
  * Luo et al. [2021] C. Luo, X. Yang, and A. Yuille.  Exploring simple 3d multi-object tracking for autonomous driving.  In _IEEE International Conference on Computer Vision_ , 2021.
  * Wang et al. [2021] Y. Wang, K. Kitani, and X. Weng.  Joint object detection and multi-object tracking with graph neural networks.  In _IEEE International Conference on Robotics and Automation_ , 2021\.
  * Kim et al. [2021] A. Kim, A. Ošep, and L. Leal-Taixé.  Eagermot: 3d multi-object tracking via sensor fusion.  In _IEEE International Conference on Robotics and Automation_ , 2021\.
  * Zhang et al. [2022] T. Zhang, X. Chen, Y. Wang, Y. Wang, and H. Zhao.  Mutr3d: A multi-camera tracking framework via 3d-to-2d queries.  _arXiv preprint arXiv:2205.00613_ , 2022.
  * Hu et al. [2019] H.-N. Hu, Q.-Z. Cai, D. Wang, J. Lin, M. Sun, P. Krähenbühl, T. Darrell, and F. Yu.  Joint monocular 3d vehicle detection and tracking.  2019\.
  * Huber [1964] P. J. Huber.  Robust estimation of a location parameter.  _The Annals of Mathematical Statistics_ , 1964.
  * Bernardin and Stiefelhagen [2008] K. Bernardin and R. Stiefelhagen.  Evaluating multiple object tracking performance: The clear mot metrics.  _EURASIP Journal on Image and Video Processing_ , 2008, 01 2008.  doi:10.1155/2008/246309.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Robbins and Monro [1951] H. Robbins and S. Monro.  A stochastic approximation method.  _The annals of mathematical statistics_ , pages 400–407, 1951.
  * Kingma and Ba [2014] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Zhou et al. [2019] X. Zhou, D. Wang, and P. Krähenbühl.  Objects as points.  _arXiv preprint arXiv:1904.07850_ , 2019.
  * Li et al. [2022] Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Q. Yu, and J. Dai.  Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.  _arXiv preprint arXiv:2203.17270_ , 2022.
  * Pang et al. [2021] Z. Pang, Z. Li, and N. Wang.  Simpletrack: Understanding and rethinking 3d multi-object tracking.  _arXiv preprint arXiv:2111.09621_ , 2021.

In this appendix, we provide additional results on 3D detection performance of
CC-3DT on NuScenes [9], an ablation study on the effect of different 3D
detectors, technical details about the 3D estimation model, more qualitative
results and a runtime analysis.

##  Appendix A 3D detection results on NuScenes

We evaluate the 3D detection results of CC-3DT with DETR3D [29] as 3D detector
on the NuScenes validation split. As shown in Table 6, our 3D tracking
framework with cross-camera motion model can also improve the overall 3D
detection results. Compared to our Kalman Filter baseline (KF3D), the LSTM
motion model refines the 3D estimation better and achieves higher mAP and
lower mATE (mean Average Translation Error). By learning the cross-camera
motion, our framework estimates more accurate object velocity and achieves
lower mAVE (mean Average Velocity Error). Overall, CC-3DT improves not only 3D
tracking performance but also benefits the 3D detection results.

##  Appendix B Effect of different 3D detectors

We evaluate the overall performance with different 3D detectors with our
CC-3DT on the NuScenes dataset. In Table 7, among of all methods, our proposed
CC-3DT achieves the best NDS and AMOTA when the most accurate detection
results in BEVFormer [66] are used. Thus, we show that our method is robust to
the 3D detector being used and that it can benefit from more accurate 3D
detections.

##  Appendix C 3D estimation model

The 3D estimation model in CC-3DT consists of a 3D detector and an appearance
feature extractor. The appearance features are extracted by the similarity
head with the generated 2D bounding boxes as region proposal. We use 4
convolutional layers followed by one fully-connected layer as the similarity
head following QDTrack [37, 38], and train the 3D estimation network using the
same 3D detector as in QD-3DT [1].

###  C.1 Similarity head training

We use quasi-dense similarity learning [37, 1, 38] to train our similarity
head. Given a key frame at time $t$, we sample a reference frame within a
temporal interval $n$, where $n\in[-2,2]$ for NuScenes [9] due to its low
sampling frequency and $n\in[-3,3]$ for Waymo Open [10]. For the estimated 3D
bounding boxes $\mathbf{b}_{t}$, each has its corresponding 2D bounding box
$\mathbf{e}^{d}_{t}$ on the image $\mathbf{I}^{m}_{t}$. We optimize the
appearance embedding $\mathbf{f}^{d}_{t}$ for each 2D proposal
$\mathbf{e}^{d}_{t}$ using a multi-positive cross-entropy loss as

$\displaystyle\mathcal{L}_{\mathtt{embed}}$
$\displaystyle=\log[1+\sum_{\mathbf{p}^{d}_{t+n}}\sum_{\mathbf{n}^{d}_{t+n}}\text{exp}(\mathbf{f}^{d}_{t}\cdot\mathbf{n}^{d}_{t+n}-\mathbf{f}^{d}_{t}\cdot\mathbf{p}^{d}_{t+n})],$
(8)

where the appearance embedding $\mathbf{f}^{d}_{t}$ should be similar to its
positive reference embeddings $\mathbf{p}^{d}_{t+n}$, and dissimilar to all
its negative reference embeddings $\mathbf{n}^{d}_{t+n}$. We apply an
auxiliary loss based on the cosine similarity of the appearance embedding
$\mathbf{f}^{d}_{t}$ in the key frame and its corresponding embedding in the
reference frame $\mathbf{f}^{d}_{t+n}$ as

$\displaystyle\mathcal{L}_{\mathtt{aux}}=(\frac{\mathbf{f}^{d}_{t}\cdot\mathbf{f}^{d}_{t+n}}{||\mathbf{f}^{d}_{t}||\cdot||\mathbf{f}^{d}_{t+n}||}-\mathds{1}({\mathbf{e}^{d}_{t},\mathbf{e}^{d}_{t+n}}))^{2},$
(9)

where $\mathds{1}({\mathbf{e}^{d}_{t},\mathbf{e}^{d}_{t+n}})$ is $1$ if
$\mathbf{e}^{d}_{t}$ and $\mathbf{e}^{d}_{t+n}$ are matched to the same ground
truth object and $0$ otherwise. The overall loss for similarity head training
is

$\displaystyle\mathcal{L}_{\mathtt{similarity}}$
$\displaystyle=\lambda_{\mathtt{embed}}\,\mathcal{L}_{\mathtt{embed}}+\mathcal{L}_{\mathtt{aux}},$
(10)

and we use $\lambda_{\mathtt{embed}}$ as 0.25 for training on both datasets.

###  C.2 Baseline 3D detector

We develop the baseline 3D detector based on Faster R-CNN [11] as in [1]. We
estimate the 3D center $(x,y,z)$ by regressing a logarithmic depth value, and
an offset from the Faster R-CNN generated 2D bounding box center to the
projected 3D bounding box center. For the object confidence $c$, we regress
the score by generating the target confidence score as an exponential of
negative L1 distance between depth estimation and ground truth. We estimate
the object dimensions $(l,w,h)$ by regressing a logarithmic value, and
estimate the object orientation $\theta$ following Mousavian et al. [18], i.e.
we learn to classify the angle into $2$ bins with binary cross-entropy loss
and regress the residual relative to the belonging bin center. We us Huber
loss [60] for the 3D box regression losses.

Table 6: Motion model ablation study. 3D detection results of CC-3DT with different motion models on the NuScenes validation split. 3D Detector | Motion Model | NDS $\uparrow$ | mAP $\uparrow$ | mATE $\downarrow$ | mAVE $\downarrow$
---|---|---|---|---|---
DETR3D [29] | - | 0.4138 | 0.3228 | 0.6803 | 0.8545
KF3D | 0.4189 | 0.3244 | 0.6846 | 0.8506
VeloLSTM | 0.4326 | 0.3304 | 0.6594 | 0.7716
Table 7: Detector ablation study. Comparison of 3D tracking performance of CC-3DT using different 3D detectors on the NuScenes validation split. 3D Detector | NDS$\uparrow$ | AMOTA $\uparrow$ | AMOTP $\downarrow$ | RECALL $\uparrow$ | MOTA $\uparrow$ | IDS $\downarrow$
---|---|---|---|---|---|---
Baseline | 0.3868 | 0.311 | 1.433 | 0.472 | 0.278 | 2536
DETR3D [29] | 0.4326 | 0.359 | 1.361 | 0.498 | 0.326 | 2152
BEVFormer [66] | 0.4781 | 0.429 | 1.257 | 0.534 | 0.385 | 2219

###  C.3 Different 3D detectors

As shown in Table 7, our proposed CC-3DT works with different existing 3D
detectors. To extract appearance embeddings given only 3D bounding boxes, we
generate the corresponding 2D bounding boxes by projecting the 3D bounding
boxes to the images. We use these 2D bounding boxes to extract the appearance
embedding from the image where the 3D bounding box is visible in. In addition,
we use the 3D score estimated by the 3D detector as our object confidence for
the motion model and only keep the boxes if the score is greater than $0.05$.

###  C.4 3D tracker settings

We use $\mathbf{w}_{\mathtt{deep}}=0.5$ for the affinity matrix $\mathbf{A}$
and $0.5$ as the matching score threshold. $0.8$ is set as the score threshold
to start a new track for the baseline 3D detector, while $0.1$ and $0.2$ are
used for DETR3D [29] and BEVFormer [66] considering the higher quality of the
3D detections. We use $0.5$ as the score threshold to continue a track for the
baseline 3D detector and $0.05$ and $0.1$ for DETR3D and BEVFormer. We keep
$10$ frames for the tracks and $1$ frame for the backdrops. Because we use
lower score threshold for DETR3D and BEVFormer 3D detectors, we do not keep
the backdrops for them. Following [37, 1, 38], we apply $0.8$ as the momentum
to update the appearance features inside the tracks We use $0.7$ and $0.3$ 2D
IoU threshold for duplicate removal on each camera for the new detections and
the backdrops.

Table 8: Runtime analysis. We analyse the contribution of the different components in our pipeline to the total runtime in seconds on the NuScenes dataset. We measure the runtime of processing a complete sample consisting of 6 images at full resolution ($900\times 1600$ pixels). Motion model | 3D Detection | Appearance feature | Data association | Motion model | Total
---|---|---|---|---|---
KF3D | 0.381 | 0.002 | 0.037 | 0.016 | 0.436
LSTM | 0.381 | 0.002 | 0.029 | 0.072 | 0.484

##  Appendix D Runtime analysis

We provide a breakdown of the runtime of our pipeline in Table 8. We measure
the runtime using a single RTX 3090 graphics card at batch size 1 on the
NuScenes dataset. One batch element consists of 6 images at $900\times 1600$
pixel resolution. Note that the setup can differ between datasets and that
this may affect the runtime, depending on how many images need to be processed
simultaneously. We compare the runtime of our LSTM motion model with the
Kalman Filter (KF3D) baseline, and find that the LSTM is slower ($0.072$s vs.
$0.016$s). However, we observe that the majority of the total runtime is
occupied by the 3D detection method, while our data association and motion
modeling is responsible only for a minor part of the total runtime. We ablate
the effect of using different detectors with our method in section B. Note
that the motion model used affects the association results, which influences
the runtime of the data association.

Front Left | Front | Front Right
---|---|---
|  |
|  |
|  |
|  |
|  |
Figure 3: Qualitative results of our tracker on the Waymo Open validation
split. Note the consistent identity of objects moving along camera borders.

##  Appendix E Qualitative results

We show qualitative results on the Waymo Open and NuScenes datasets in Figure
3, Figure 4 and Figure 5, respectively. We plot the 3D bounding boxes in the
image view and depict object identity as box color. For NuScenes, we provide
both front and back cameras results and show that our CC-3DT can associate the
objects across camera borders.

Front Left | Front | Front Right
---|---|---
|  |
|  |
|  |
|  |
|  |
Figure 4: Qualitative results of our tracker on the front camera of the NuScenes validation split. Note the consistent identity of objects moving along camera borders. Back Left | Back | Back Right
---|---|---
|  |
|  |
|  |
|  |
|  |
Figure 5: Qualitative results of our tracker on the back cameras of the
NuScenes validation split. Note the consistent identity of objects moving
along camera borders.


# Exploring the Relationship between In-Context Learning and Instruction
Tuning

Hanyu Duan,1 Yixuan Tang,1 Yi Yang,1 Ahmed Abbasi,2 Kar Yan Tam1
Department of Information Systems, Business Statistics and Operations
Management, HKUST 1
Department of IT, Analytics, and Operations, University of Notre Dame 2
<EMAIL_ADDRESS>{yixuantang<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

In-Context Learning (ICL) and Instruction Tuning (IT) are two primary
paradigms of adopting Large Language Models (LLMs) to downstream applications.
However, they are significantly different. In ICL, a set of demonstrations are
provided at inference time but the LLM’s parameters are not updated. In IT, a
set of demonstrations are used to tune LLM’s parameters in training time but
no demonstrations are used at inference time. Although a growing body of
literature has explored ICL and IT, studies on these topics have largely been
conducted in isolation, leading to a disconnect between these two paradigms.
In this work, we explore the relationship between ICL and IT by examining how
the hidden states of LLMs change in these two paradigms. Through carefully
designed experiments conducted with LLaMA-2 (7B and 13B), we find that ICL is
implicit IT. In other words, ICL changes an LLM’s hidden states as if the
demonstrations were used to instructionally tune the model. Furthermore, the
convergence between ICL and IT is largely contingent upon several factors
related to the provided demonstrations. Overall, this work offers a unique
perspective to explore the connection between ICL and IT and sheds light on
understanding the behaviors of LLM.

##  1 Introduction

Large language models (LLMs), such as ChatGPT 111https://openai.com/chatgpt,
GPT-4 (OpenAI, 2023), PaLM (Chowdhery et al., 2022), and LLaMA-2 (Touvron et
al., 2023), have significantly changed the paradigm of natural language
processing and hold great potential for artificial general intelligence
(Bubeck et al., 2023). In real-world applications, the success of deploying
Large Language Models (LLMs) can largely be attributed to the effectiveness of
two primary learning paradigms: 1) In-Context Learning (ICL) and 2)
Instruction Tuning (IT). ICL, a paradigm introduced in the GPT-3 paper,
involves utilizing a set of demonstrations are provided at inference time to
guide the model’s responses, but the model’s parameters are not updated during
this process. In contrast, IT refers to the process of further training LLMs
on input, output, along with instructions in a supervised fashion. IT has been
shown to be effective in enhancing an LLM’s generalizability on unseen tasks
(Longpre et al., 2023) and a viable strategy for LLM alignment (Taori et al.,
2023; Zhou et al., 2023). Figure 1 illustrates ICL and IT using sentiment
analysis as an example.

A growing body of literature has examined the mechanisms of ICL and IT, such
as identifying the conditions under which ICL emerges in LLMs (Liu et al.,
2021; Lu et al., 2021; Su et al., 2022; Wang et al., 2023; Chan et al., 2022;
Xie et al., 2021), and determining how to design data and tasks for effective
instruction tuning to enhance the zero-shot generalizability of LLMs (Longpre
et al., 2023). However, while ICL and IT are two primary methods for enhancing
the capabilities of LLMs, studies on ICL and IT have been conducted in
isolation. This has led to a research question: What are the connections
between ICL and IT, and in which way do they enhance an LLM’s capability.

Figure 1: Illustrations for ICL and IT using sentiment analysis as an example.
Through ICL, the LLM infers a ”Negative” sentiment for ”Many pointless.”
conditioned on a set of demonstrations (Left). In contrast, IT involves
further tuning the LLM’s parameters with the IT training data, and the tuned
LLM is then used at inference time (Right).

In this work, we examine the connection between ICL and IT via the hidden
state of the input sequence’s last token. In an autoregressive model, the
hidden state of the input sequence’s last token summarizes the information of
the entire input sequence and determines the logit vector for the next word
prediction. In the context of ICL and IT, three situations arise, each
producing different hidden states. The first situation involves zero-shot
learning for an LLM. In this case, the hidden state of the last token in the
input sequence is determined by the LLM, conditioned on the inference example.
Since this is the basic case—where no demonstrations are provided and the
LLM’s parameters are not updated—we denote this as the anchor hidden state,
$h_{anchor}$. The second situation is ICL, where demonstrations are provided
to guide the LLM’s response. Since ICL does not tune the LLM’s parameters, the
hidden state is determined by the LLM, conditioned on the provided
demonstrations and the inference sample. We denote this hidden state as
$h_{ICL}$. The third situation is IT, where demonstrations are used to tune
the LLM’s parameters, transforming the LLM into a tuned-LLM. Here, the hidden
state is determined by the tuned-LLM, conditioned on the inference sample, and
we denote this hidden state as $h_{IT}$. Comparing the similarity between
$h_{anchor}$ and $h_{ICL}$ allows to quantify the effect of a demonstration in
ICL, while comparing the similarity between $h_{anchor}$ and $h_{IT}$ allows
to quantify the effect of IT with the demonstration. If a demonstration is
effective for ICL and IT, we would observe a small similarity score because
the demonstration gears the LLM to produce a guided (either through ICL or
through tuning) response. Moreover, examining the similarity between $h_{ICL}$
and $h_{IT}$ allows us to directly quantify the extent to which ICL and IT on
LLM converge, conditioned on the demonstrations. Figure 2 illustrates the
analysis framework.

In the experiment, we select LLaMA-2 (7B) (Touvron et al., 2023) as the
foundational LLM. We compile a demonstration dataset for sentiment analysis,
consisting of tuples of $<$instruction, example, label$>$. Subsequently, we
apply ICL and IT to LLaMA-2 using the same demonstration and examine the
similarities between $h_{\text{anchor}}$, $h_{\text{ICL}}$, and
$h_{\text{IT}}$. We repeat the experiment with variations in the wording of
the instruction and demonstration examples. The results reveal a high
similarity between $h_{\text{ICL}}$ and $h_{\text{IT}}$, while the similarity
of these two hidden states with $h_{\text{anchor}}$ is low. This suggests that
ICL and IT essentially guide the LLM to a similar status, although IT tunes
the LLM’s parameters while ICL does not. To further investigate, we vary the
demonstrations used in ICL and IT and quantify the extent of similarity
between ICL and IT conditioned on the demonstrations. For instance, we
manipulate the number of demonstrations (from one-shot ICL to few-shot ICL),
alter the semantic similarity between demonstration examples and inference
examples, use a wrong label for the demonstration example, and employ
different tasks as demonstrations. The results consistently support the
finding that using a demonstration in ICL has a similar effect as using the
demonstration to instructionally tune the LLM. In additional analyses
examining the robustness of our findings, we change the inference task to a
machine translation task and replace LLaMA-2 (7B) with LLaMA-2 (13B); the
results remain consistent.

In summary, this work makes two contributions. First, we provide empirical
evidence that ICL and IT are closely related. Although ICL does not alter
model parameters—unlike IT—the instructions and demonstrations they employ
drive the model towards convergent hidden states. Second, this study sheds
light on how to design effective datasets and tasks for ICL and IT,
potentially advancing the development and alignment of foundation models for
downstream applications. We will make the experimental codes available for
replication.

##  2 Analysis Framework

We illustrate our analysis framework in Figure 2, using sentiment analysis on
reviews as an example. In this framework, we examine the impact of different
demonstrations (zero-shot vs. few-shot ICL) and different paradigms (ICL vs.
IT) on the model’s hidden states separately. Although LLMs maintain hidden
states for every input token, we primarily focus on the hidden states
associated with the last input token of the sequence in this study. This focus
is due to the hidden state of the last token of the last layer summarizing the
information of the entire input sequence and determining the logit vector for
the next word prediction.

Figure 2: Analysis framework using sentiment analysis on reviews as an
example. Our framework has variations by manipulating the demonstrations,
changing the LLM, altering the input template, and adapting to different
natural language tasks.

We denote instruction as $X$ (such as, what is the sentiment of this review?),
demonstration as $A$=(Text A, Label A) (such as, Review: This is a wonderful
movie. Sentiment: Positive), and inference text as $B$=(Text B) (such as,
Review: I like this movie.). We then consider the following three situations.

Basic situation. This is the basic zero-shot learning setting where no
demonstrations are provided to guide the model inference. In this situation,
we concatenate instruction with the inference example (i.e., Instruction $X$
\+ Text $B$) and feed into an LLM. We collect the final hidden state of the
last token of the input sequence, denoted as $h_{anchor}$.

ICL situation. In ICL, demonstrations, along with the inference example (i.e.,
Instruction $X$ \+ Text $A$ \+ Label $A$ \+ Text $B$), are provided as input
to the LLM, which then directly infers the distribution of the last token. We
collect the final hidden state of the last token of the input sequence,
denoted as $h_{ICL}$. Comparing the similarity between $h_{anchor}$ and
$h_{ICL}$ allows us to examine the effect of the provided demonstration. If
the similarity is low, it indicates that the demonstration information are
incorporated by the LLM so that the final hidden states are geared away.

IT situation. In IT, unlike the ICL situation where the demonstration is used
as a part of input sequence, we instead use the demonstration (i.e.,
Instruction $X$ \+ Text $A$ \+ Label $A$) to instructionally tune the LLM,
leading to a tuned LLM. We then send the inference example (i.e., Instruction
$X$ \+ Text $B$) to the tuned LLM and obtain final hidden state of the last
token, denoted as $h_{IT}$. Note that the input sequence to the final LLM are
exactly the same (i.e., Instruction $X$ \+ Text $B$) in both the basic
situation and the IT situation. The only difference is that the basic
situation involves the vanilla LLM while the IT situation involves the
instruction-tuned LLM. Therefore, by comparing $h_{anchor}$ with $h_{IT}$, we
can quantify the effect of IT with the demonstration.

Since the same demonstration is used in both ICL and IT, we can precisely
quantify the effect of the demonstration. By varying the provided
demonstrations, we can also determine the extent to which ICL is related to
IT, conditioned on the demonstrations. In the analysis, we further denote
$s_{anchor-ICL}$ as the similarity between $h_{anchor}$ and $h_{ICL}$, and
denote $s_{anchor-IT}$ as the similarity between $h_{anchor}$ and $h_{IT}$. We
also measure the similarity between $h_{ICL}$ and $h_{IT}$, denoted as
$s_{ICL-IT}$, which quantifies the extent to which ICL and IT converge. If the
$s_{ICL-IT}$ is very high, it indicates ICL and IT guide the model status
towards the same direction although the model parameters are not updated in
ICL but tuned in IT.

##  3 Experiments

###  3.1 Experiment Setup

Datasets: In the experiment, we use the SST2 for sentiment analysis (Socher et
al., 2013) and EN-CS of WMT16 for English-Czech translation (Bojar et al.,
2016). For each of the tasks, we manually craft a pool of instructions and
randomly choose instruction in the repeated experiment, alleviating the
concern that the experiment results are driven by a specific instruction.
Instructions used for each task are presented in Appendix A.

LLMs: We use LLaMA-2-base as the foundation model (Touvron et al., 2023),
including 7B (32 layers with a hidden size of 4,096) and 13B (40 layers with a
hidden size of 5,120). We download the models following the instructions from
Meta AI 222https://github.com/facebookresearch/llama, and implement them using
the transformers library 333https://github.com/huggingface/transformers.

Instruction tuning: We use the LoRA technique (Hu et al., 2021) to
instruction-tune the LLaMA-2 model due to its efficiency. Specifically, we
target modules $Q$ and $V$, and use a dropout probability 0.05, learning rate
1e-4, scaling factor 32, and a rank of 8. We use AdamW optimizer (Loshchilov &
Hutter, 2017). Without further specification, we tune the model with 10 epochs
and use bf16 precision.

Repeated experiment: In the following analysis, we randomly choose an
instruction, a demonstration and an inference example from the dataset for ICL
and IT. We repeat the procedure for 30 runs with different random seeds.

###  3.2 Empirical Findings

We present the empirical findings as follows.

ICL and IT convergence: In-Context Learning (ICL) and Instruction Tuning (IT)
result in a converged model state. We present the hidden state similarities in
Figure 3(a). Firstly, we observe that the similarity between $h_{anchor}$ and
either $h_{ICL}$ or $h_{IT}$ is almost zero, indicating that the model
undergoes significant changes in its hidden representations when exposed to
in-context demonstrations or when tuned by the demonstrations. Furthermore,
the high similarity between $h_{ICL}$ and $h_{IT}$ (approximately 0.9)
demonstrates that the model is indeed oriented toward a similar state in ICL
and IT. This provides a first evidence that ICL is implicit IT.

(a) ICL and IT convergence.

(b) Wrong demonstration labels

(c) Different ICL task

Figure 3: Similarities between different hidden states. We use the box plots
to show the distribution of scores in the repeated experiments.

Demonstration-inference similarity: The convergence between ICL and IT is
positively correlated with the semantic similarity between the demonstration
and the inference example. We further investigate how the semantic similarity
between the demonstration (i.e., Text $A$ in Figure 2) and the inference
example (i.e., Text $B$) affects the ICL-IT convergence. To do this, we use a
sentence-transformer model ”all-MiniLM-L6-v2” 444https://www.sbert.net to
measure the demonstration-inference similarity (Reimers & Gurevych, 2019). We
consider 10 levels of similarity ranging from 0 to 1. For each inference
example, we identify demonstrations in the dataset that fall within a specific
similarity range. In each repeated experiment involving different similarity
levels, we randomize the input but use the same set of inference examples
across these cases to facilitate a fair comparison. The results are shown in
Figure 4. Clearly, the similarity between ICL and IT increases as the
similarity between the demonstration and the inference example increases
(Figure 4(c)). A possible explanation is that a demonstration that is more
similar to the inference example can better enhance the model’s ICL ability
and is also more helpful for IT, resulting in higher convergence. It is worth
noting that the range of the degree of convergence between ICL and IT is quite
large, ranging from around 0.4 when they are entirely different
(demonstration-inference similarity is 0) to 0.8 when they are exactly the
same (demonstration-inference similarity is 1).

In contrast, the similarity between $h_{anchor}$ and $h_{IT}$ exhibits an
opposite trend, as shown in Figure 4(a), suggesting that a demonstration that
is more similar to the inference example can change the model’s state to a
greater extent. This finding aligns with prior literature, which has
demonstrated that instruction tuning with similar examples is more effective
(Gudibande et al., 2023). Put it another way, fine-tuning the model with
semantically different examples does not substantially alter the model’s
inference capability.

Interestingly, we observe that the similarity between $h_{anchor}$ and
$h_{ICL}$ remains consistently low, regardless of the demonstration-inference
similarity, as illustrated in Figure 4(b). This suggests that incorporating
demonstrations into the ICL input can consistently and significantly impact
the model’s inference. Previous studies on ICL have indicated that higher
demonstration-inference similarity leads to improved inference accuracy. It’s
important to emphasize that Figure 4(b) does not contradict this finding, as
it measures the similarity between $h_{anchor}$ and $h_{ICL}$.

(a) Anchor-IT

(b) Anchor-ICL

(c) ICL-IT

Figure 4: Averaged hidden state similarities across demonstration-inference
similarity levels.

Number of demonstrations: The convergence between ICL and IT increases as the
number of demonstration increases. In the previous analysis, we used a single
demonstration in ICL and IT. In this experiment, we vary the number of
demonstrations (i.e., few-shot learning) in ICL and IT. Specifically, we
consider 1-shot, 2-shot, 5-shot, and 10-shot scenarios. To ensure a fair
assessment, we maintain consistent parameters update times and instruction-
tune the model with 10, 5, 2, and 1 epoch(s), respectively. For each repeated
experiment in the various few-shot cases, we randomize the input but use the
same set of inference examples across these cases to enable a fair comparison.

We present the results in Figure 5. We observe a clear increasing trend in the
convergence between ICL and IT as we incorporate more demonstrations. This
trend is intuitive since ICL with multiple demonstrations (i.e., few-shot
learning) can help the model discover patterns in the context and quickly
adapt to the task. Similarly, IT using more examples related to the same task
can better tune the model for that specific task, leading to a higher level of
convergence between ICL and IT.

Figure 5: ICL-IT convergence across different numbers of demonstrations.

Wrong label: Demonstration with wrong label slightly affects the ICL-IT
convergence. Prior studies in ICL have shown that the correctness of
demonstration’s label does not matter much and only the task format is
important for ICL (Min et al., 2022). Therefore, it motivates us to examine
how the label correctness affects the ICL-IT convergence. In this experiment,
we reverse the labels of demonstrations (e.g., changing ”Positive” to
”Negative”), and conduct the ICL and IT procedure again. The results are shown
in Figure 3(b).

Interestingly, we find that while ICL and IT still exhibit a high level of
convergence, the degree is slightly lower than its counterpart when using
correct labels as compared to Figure 3(a). Besides, the variation of the
degree of ICL-IT convergence significantly increases, as evidenced by the
larger interquartile range and longer whiskers of the box plot.

As a sanity check, we examine if using wrong labels to do IT hurts the model
performance, and present the results in Figure 6. Surprisingly, although we do
observe a performance drop, the decrease is not statistically significant,
which appears to be well aligned with previous observations in (Kung & Peng,
2023).

Figure 6: Prediction accuracies of using wrong demonstration labels vs. right.
We perform one-tailed Wilcoxon signed-rank test, and the null hypothesis is
the difference between paired observations (right-wrong) is greater than zero.

Different task: Different demonstration task would not affect the ICL-IT
convergence. In the previous experiments, the demonstration task and the
inference task are the same (i.e., sentiment analysis). This experiment
differs in that we change the demonstration task to machine translation using
the EN-CS subset of WMT16 translating English to Czech 555We use the following
template: ”Instruction X. English: English A. Czech: Czech A. Instruction X.
English: English B. Czech:”. , but the sentiment analysis remains the
inference task. We present the results in Figure 3(c). Clearly, the high level
of convergence in similarities between ICL-IT, Anchor-ICL, and Anchor-IT
indicates that the demonstrations involving the machine translation task do
not impact the model’s inference capability for the sentiment analysis task.

Intermediate layers: The convergence between ICL and IT starts to increase at
later layers. In this experiment, we examine the hidden states of the last
token of the input sequence in all layers of the LLM. The results are shown in
Figure 7. Interestingly, we observe an U shape across different layers. The
high similarity between ICL and IT in the lower layer is primarily due to the
fact that the hidden states are all similar to the anchor hidden states,
meaning they are not significantly impacted by the demonstrations. The LLM’s
intermediate layers are gradually influenced by the demonstrations, resulting
in the low similarity between ICL and IT in the middle layers. Eventually, as
the input approaches the higher layers that are closer to the final output,
the hidden states of ICL and IT start to converge.

Figure 7: ICL-IT convergence scores of all layers.

##  4 additional analysis

(a) LLaMA-2-13B.

(b) Classic supervised learning.

(c) Machine translation task.

Figure 8: Similarities between different hidden states (additional analysis).

###  4.1 LLaMA-2-13B

In this study, we examine if ICL and IT still converges in a larger LLM. We
choose LLaMA-2-13B as the foundation model and repeat the same analysis
procedure to quantify the similarity between Anchor-IT, Anchor-ICL and ICL-IT.
The results are shown in Figure 8(a), indicating that ICL-IT convergence
remains high. However, Anchor-IT and Anchor-ICL also achieve a high level of
convergence, indicating that larger model is more capable of understanding the
task even without any demonstrations provided (note that in the basic
situation, an instruction is provided which could provide sufficient
information for the larger LLM to do zero-shot learning).

###  4.2 Supervised Learning

Instruction tuning differs from classic supervised learning in that the former
employs additional instructions to enhance an LLM’s generalizability, while
supervised learning typically teaches the LLM to specialize in a specific
task.

To further understand the role of instructions in IT, we conduct classic
supervised learning for the LLM. In this setup, we remove Instruction $X$ from
the training input and solely use task examples to fine-tune the LLM. We
denote this supervised situation as SL. We repeat the same analysis procedure
and measure the similarity between Anchor-SL, Anchor-ICL, and ICL-SL. We
present the results in Figure 8(b). Clearly, while the convergence between ICL
and SL still exists, the convergence score is significantly lower than that of
its IT counterparts, as shown in Figure 3(a). This observation underscores the
critical role of instructions in driving the convergence between ICL and IT in
LLMs’ hidden states.

###  4.3 Understanding Instruction Tuning from In-Context Learning

Evidences discussed above suggest that ICL is essentially doing IT via
demonstrations. In this section, we aim to understand IT through the lens of
ICL. Specifically, instead of focusing on the hidden states, we calculate the
change of per token loss of the LLM. We define per token loss as the cross-
entropy loss between each output token and the corresponding ground truth
token in a sequence (Olsson et al., 2022). We illustrate the procedures of the
experiment in Figure 9. The major steps are as follows. Firstly, we randomly
sample an instruction $X$ and an example $A$. We then construct the input
using the template shown in Figure 2 as: ”Instruction $X$. Review: Text $A$.
Sentiment: Label $A$.”. Next, we send the input to LLaMA-2-7B and collect the
per token loss. After that, we instruction-tune the language model using this
example. After tuning, we send the same input again to the tuned model and
collect the per token loss. We then calculate the loss decrease for each token
and average the per token loss decrease by token’s identity (i.e,
”Instruction” or ”Example”). We conduct 30 independent experiments using
different seed values. The results are shown in Figure 10. Clearly, we observe
a more significant loss decrease for the ”Example” component compared to the
”Instruction” component, suggesting the tuned model is more likely to
reproduce task relevant examples given an instruction. In other words, the
instruction is somehow substituted by the examples it associates at inference
time, leading to a similar input format as ICL.

Figure 9: Illustration: The decreased loss indicates instruction can help the
model associate relevant examples at inference time.

###  4.4 Robustness Check: Machine Translation

As a robustness check, we replace the sentiment analysis task (a natural
language inference task) with the machine translation task (a natural language
generation task), and conduct the same procedure to examine if the connection
between ICL and IT still holds. We choose a machine translation task that
translates English text into Czech using the EN-CS subset of WMT16 dataset
(Bojar et al., 2016). We present the results in Figure 8(c). It is interesting
to note that the similarity between ICL and IT is remarkably high. Recall that
the input examples for ICL and IT are very different. The substantial
similarity between ICL and IT supports the earlier findings that ICL, when
using demonstrations, significantly alters an LLM’s inference capability, akin
to how demonstrations are used to fine-tune the LLM.

Unlike sentiment analysis, where the similarity between Anchor-IT and Anchor-
ICL is as low as zero, the similarity is higher in the machine translation
task. However, a statistical test reveals that the similarity between ICL and
IT is statistically greater than that between Anchor-IT and Anchor-ICL666We
conducted a one-tailed Wilcoxon signed-rank test between each pair of them.
The p-value is $2.79e-9$ for Anchor-IT and ICL-IT, and $9.31e-10$ for Anchor-
ICL and ICL-IT. The sample size is 30, and the null hypothesis is that the
difference between paired observations ($s_{ICL-IT}-s_{anchor-IT}$ and
$s_{ICL-IT}-s_{anchor-ICL}$) is greater than zero.. This rules out the
possibility that all three hidden states are very similar to each other.

Figure 10: Per-token loss decrease due to instruction tuning.

##  5 Related work

In-Context Learning (ICL) is a phenomenon emerged in large language models
(Brown et al., 2020). A growing body of literature has investigated the ICL
phenomenon in LLMs. Some studies have focused on identifying the conditions
under which ICL emerges in LLMs, predominantly by finding good demonstrations
(Liu et al., 2021; Lu et al., 2021; Su et al., 2022; Wang et al., 2023) and
identifying pre-training data distributions that can lead to the emergence of
ICL (Chan et al., 2022; Xie et al., 2021). Another line of research aims to
explain ICL through building the relationship with the model training stage
(Akyürek et al., 2022; Dai et al., 2022; Li et al., 2023; Von Oswald et al.,
2023). For instance, Akyürek et al. (2022) find ICL implicitly updates smaller
models encoded in the activations. Olsson et al. (2022) provide evidence that
the so-called ”induction heads” contribute to the majority of the ICL
behaviors in LLMs.

Our work differs from existing studies in two ways. First, we attempt to
understand ICL by investigating its connection with IT, which is new and opens
up the possibilities for harnessing the complementary knowledge of ICL and IT.
Second, we empirically study off-the-shelf LLMs with much more complex model
structures (LLaMA-2 7B and 13B), whereas most prior works conduct experiments
using more simplified models (Li et al., 2023).

Instruction Tuning (IT) is an efficient technique to adapt LLMs to downstream
tasks by further tuning the model on (”input”, ”output”) pairs with
instructions in a supervised manner. The intuition behind IT is to bridge the
gap between the language modeling objective in pre-training and the users’
objective in downstream tasks, such that the model can follow the instructions
from users. The effectiveness of IT is well-demonstrated by a variety of
instruction-tuned LLMs, with representatives such as InstructGPT (Ouyang et
al., 2022), Alpaca (Taori et al., 2023), Flan-T5 (Longpre et al., 2023), and
Vicuna 777https://lmsys.org/blog/2023-03-30-vicuna/. A growing body of
literature focuses on designing tasks and datasets for effective instruction
tuning. For example, LIMA (Zhou et al., 2023) shows that a small set of high-
quality instruction datasets is sufficient for foundation model alignment. Our
work aims to provide empirical evidence to further understand IT, through the
lens of its connection with ICL.

##  6 Conclusions

In this work, we explore the connection between in-context learning (ICL) and
instruction tuning (IT). Through carefully designed experiments, we provide
strong evidences suggesting ICL is implicitly IT. In other words, ICL changes
an LLM’s hidden states as if the demonstrations were used in IT. This finding
sheds light on the behaviors of two very different learning paradigms of LLM
(ICL vs. IT), potentially benefiting the development and alignment of
foundation LLMs to downstream real-world applications.

## References

  * Akyürek et al. (2022) Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou.  What learning algorithm is in-context learning? investigations with linear models.  _arXiv preprint arXiv:2211.15661_ , 2022.
  * Bojar et al. (2016) Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al.  Findings of the 2016 conference on machine translation.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pp. 131–198, 2016.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_ , 2023.
  * Chan et al. (2022) Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill.  Data distributional properties drive emergent in-context learning in transformers.  _Advances in Neural Information Processing Systems_ , 35:18878–18891, 2022.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Dai et al. (2022) Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei.  Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers.  _arXiv preprint arXiv:2212.10559_ , 2022.
  * Gudibande et al. (2023) Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song.  The false promise of imitating proprietary llms.  _arXiv preprint arXiv:2305.15717_ , 2023.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.
  * Kung & Peng (2023) Po-Nien Kung and Nanyun Peng.  Do models really learn to follow instructions? an empirical study of instruction tuning.  _arXiv preprint arXiv:2305.11383_ , 2023.
  * Li et al. (2023) Yingcong Li, M Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak.  Transformers as algorithms: Generalization and implicit model selection in in-context learning.  _arXiv preprint arXiv:2301.07067_ , 2023.
  * Liu et al. (2021) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen.  What makes good in-context examples for gpt-$3$?  _arXiv preprint arXiv:2101.06804_ , 2021.
  * Longpre et al. (2023) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al.  The flan collection: Designing data and methods for effective instruction tuning.  _arXiv preprint arXiv:2301.13688_ , 2023.
  * Loshchilov & Hutter (2017) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Lu et al. (2021) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  _arXiv preprint arXiv:2104.08786_ , 2021.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer.  Rethinking the role of demonstrations: What makes in-context learning work?  _arXiv preprint arXiv:2202.12837_ , 2022.
  * Olsson et al. (2022) Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al.  In-context learning and induction heads.  _arXiv preprint arXiv:2209.11895_ , 2022.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report, 2023.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Reimers & Gurevych (2019) Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ , 2019.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pp. 1631–1642, 2013.
  * Su et al. (2022) Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al.  Selective annotation makes language models better few-shot learners.  _arXiv preprint arXiv:2209.01975_ , 2022.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Von Oswald et al. (2023) Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov.  Transformers learn in-context by gradient descent.  In _International Conference on Machine Learning_ , pp. 35151–35174. PMLR, 2023.
  * Wang et al. (2023) Xinyi Wang, Wanrong Zhu, and William Yang Wang.  Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning.  _arXiv preprint arXiv:2301.11916_ , 2023.
  * Xie et al. (2021) Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma.  An explanation of in-context learning as implicit bayesian inference.  _arXiv preprint arXiv:2111.02080_ , 2021.
  * Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al.  Lima: Less is more for alignment.  _arXiv preprint arXiv:2305.11206_ , 2023.

##  Appendix A Instruction Sets

What is the sentiment of the movie review below? Is it negative or positive?
---
Determine whether the sentiment expressed in this movie review is negative or
positive:
Identify whether this movie review contains negative or positive opinions.
Classify whether this movie review conveys negative or positive opinions.
Rate whether the viewpoint on the costumes is more negative or positive.
Based on the review content, would you say the sentiment is negative or
positive?
Analyze the sentiment expressed in this movie review. Is it positive or
negative?
Identify negative or positive of the content.
Evaluate the sentiment of this movie critique. Is it negative or positive?
Determine the sentiment conveyed in this movie review. Is it negative or
positive?
Classify the overall sentiment of this movie review as negative or positive.
Determine if the tone of this movie review is negative or positive.
Assess if the tone of this movie review is negative or positive.
Detect whether this movie review contains negative or positive sentiment.
Determine whether this movie review expresses negative or positive sentiment.
Identify whether the sentiment expressed in this movie review is negative or
positive.
Distinguish whether the evaluation in this movie review is negative or
positive.Provide your answer as either negative or positive:
Infer whether the tone of this movie review is negative or positive.
Grade if the perspective in this movie review is negative or positive.Provide
your answer as either negative or positive:
What’s the emotional tone of this movie review? Would you describe it as
negative or positive?
Infer whether this movie review expresses negative or positive emotion.
Estimate if the analysis in this movie review is negative or positive.Provide
your answer as either negative or positive:
Determine whether the opinions in this movie review are negative or positive.
Identify the sentiment of the following movie review text. Is it negative or
positive?
Assess the sentiment expressed in the following movie review. Is it positive
or negative?
Determine the sentiment expressed in this movie review. Negative or positive?

Table 1: Instructions for sentiment analysis.

Can you express this English phrase in Czech?
---
Can you present this English sentence in Czech?
Please make this English sentence into a Czech sentence.
Please convert this English text into Czech.
Help me interpret this English phrase in Czech.
Translate this English sentence into Czech.
Please provide a Czech translation for this English sentence.
I need your help to change this English sentence into Czech.
Could you help convert this English phrase into Czech?
Could you translate this English text into Czech?
Please, translate the following English sentence into Czech.
Rephrase this English sentence in Czech for me, please.
Please give me the Czech version of this English sentence.
Can you assist in translating this English sentence into Czech?
Can you change this English sentence into Czech?
How would you say this English sentence in Czech?
Please convert this English phrase into Czech.
Can you convert this English sentence into Czech, please?
Please interpret this English sentence into Czech for me.
Please provide a Czech version of this English sentence.
Can you give me a Czech translation of this English text?
Could you kindly convert this English text into Czech?
Could you rewrite this English phrase in Czech?
I require this English sentence to be translated to Czech.
I need this English phrase translated to Czech.
Translate this English content into the Czech language, please.
Translate this English phrase into Czech for me, please.
Can you provide a Czech interpretation of this English sentence?
Can you render this in the Czech language, please?
Can you transcribe this English text into Czech?

Table 2: Instructions for machine translation.