# Tracking Objects with 3D Representation from Videos

Jiawei He1 Lue Fan1 Yuqi Wang1 Yuntao Chen2
Zehao Huang3 Naiyan Wang3 Zhaoxiang Zhang1,2
1 CASIA 2 CAIR, HKISI_CAS 3 TuSimple
{hejiawei2019, fanlue2019, wangyuqi2020<EMAIL_ADDRESS>
{chenyuntao08, zehaohuang18<EMAIL_ADDRESS>
###### Abstract

Data association is a knotty problem for 2D Multiple Object Tracking due to
the object occlusion. However, in 3D space, data association is not so hard.
Only with a 3D Kalman Filter, the online object tracker can associate the
detections from LiDAR. In this paper, we rethink the data association in 2D
MOT and utilize the 3D object representation to separate each object in the
feature space. Unlike the existing depth-based MOT methods, the 3D object
representation can be jointly learned with the object association module.
Besides, the object’s 3D representation is learned from the video and
supervised by the 2D tracking labels without additional manual annotations
from LiDAR or pretrained depth estimator. With 3D object representation
learning from Pseudo 3D object labels in monocular videos, we propose a new 2D
MOT paradigm, called P3DTrack. Extensive experiments show the effectiveness of
our method. We achieve new state-of-the-art performance on the large-scale
Waymo Open Dataset.

##  1 Introduction

Multiple Object Tracking (MOT) is the core component of the perception system
for many applications, such as autonomous driving and video surveillance. In
the deep learning era, metric learning helps the network to learn better
object affinity between frames for object association in MOT [56, 3, 16].
Another hot trend is jointly learning object detection and association, termed
as end-to-end MOT [32, 59, 68]. In these methods, they use the shared query to
generate the object’s bounding box in each frame belonging to the same track.
This kind of design makes the neural network jointly learn object
representation and data association across frames. Previous attempts
demonstrate that precise association is crucial in MOT. However, in 2D MOT,
object association remains a significant challenge due to object occlusion.
The presence of many partially visible objects in congested scenarios like
shopping malls and traffic jams makes incorrect association nearly impossible
to prevent. Several approaches have aided the data association module with
complex appearance models and image-space motion models to address the
challenging 2D data association. Despite the efficacy of these techniques,
they do not target the main problem of object association, that is, trying to
associate 3D objects in 2D image space.

Conversely, in 3D MOT, many works demonstrate that object association is
nearly a trivial problem, even with a simple motion model. ImmortalTracker
[51], in particular, reveals that using the 3D Kalman Filter to model motion
from the LiDAR 3D bounding boxes, the wrong association only occurs _once_ in
the entire Waymo Open Dataset (WOD) dataset. This significant gap between 2D
and 3D MOT reveals that association in a higher dimension space is much
simpler than in a low-dimensional space. Therefore, inspired by this
observation, this paper aims to address the 2D object association problem in a
3D space.

Figure 1: Three streams in the field of 2D MOT. Left: Association by
appearance feature learning. Middle: Association by lifted 3D objects from the
pretrained depth model. Right: Association by learnable 3D representation from
pseudo 3D object labels.

Recent works [23, 7] explore the most straightforward way to lift 2D
association to 3D space, that is utilizing the off-the-shelf depth model.
However, such methods are not effective enough for three reasons. (1) It is
hard to estimate the scene-level depth from the monocular image. (2) The
camera’s intrinsic parameters are different, so the pretrained depth
estimation model has limited generalization ability in the tracking scenes.
(3) Association with explicit depth is sub-optimal since the depth estimation
and the association part are isolated without joint optimization. Meanwhile,
without joint optimization, the association is also sensitive to the noise of
explicit depth.

Distinct from these works, we want to learn the representation containing 3D
position information for the objects, and the representation can be jointly
learned with the association module, as shown in Fig. 1. Besides, due to the
expensive cost and the additional sensors (e.g., LiDAR) to obtain the 3D
annotations, we want to dig the 3D object representation only from 2D tracklet
labels annotated in the videos. In this paper, we propose a new video-based 3D
representation learning framework and the 2D Multiple Object Tracking
algorithm with the 3D object representation, called P3DTrack. P3DTrack mainly
consists of three parts: (1) pseudo 3D object label generation, (2) jointly
learning framework including 3D object representation and object association
learning, (3) a simple yet effective online tracker with learned 3D object
representation. As for pseudo 3D object label generation, inspired by
Structure-from-Motion [44], the scene can be reconstructed in 3D space from
the camera motion. The 3D object is located as a part of the reconstructed
scene. Fortunately, with the 2D bounding boxes in video frames, we can find
the 3D object’s reconstructed point clouds lie in the intersection area of the
object frustums of multiple frames. By finding the main cluster of the
reconstructed object point clouds, the 3D object position can be annotated as
the center of the cluster. After that, the pseudo label offers P3DTrack
supervision to learn the 3D representation of objects. The representations are
then fed into a graph-matching-based object association module for joint
optimization. Then the tracker can track the objects frame-by-frame with the
robustness of heavy occlusion and similar appearance. In summary, our work has
the following contributions:

  * •

We propose a new online 2D MOT paradigm, called P3DTrack. P3DTrack utilizes
the jointly learned 3D object representation and object association.

  * •

We design the 3D object representation learning module, called 3DRL. With
3DRL, the object’s 3D representation is learned from the video and supervised
by the 2D tracking labels without any additional annotations from LiDAR or
pretrained depth estimator.

  * •

The extensive experiments on the large-scale Waymo Open Dataset show that we
achieve new state-of-the-art performance.

##  2 Related Work

Figure 2: Illustration of pseudo 3D object label generation process. After
reconstructing the scene with Structure-from-Motion, intra-frame and inter-
frame point clustering are adopted. The pseudo 3D object label can be defined
as the barycenter of the cluster.

###  2.1 Multiple Object Tracking

Multiple Object Tracking (MOT) aims to associate the same object in a video.
To track the object, linear motion models, like Kalman Filter (KF) [21], and
appearance models [56] from re-identification are the core components in the
tracker. However, the 2D MOT task suffers from object occlusion, camera shake,
and similar appearance. Many methods tend to use graph neural networks [3, 16]
and attention mechanism [59, 32, 68] to aggregate the object features across
intra- and inter-frames. Some researchers reveal that learning object features
and object association jointly [57, 16] can help to obtain more discriminative
object features. Transformer-based end-to-end MOT [59, 32] build a new
paradigm with track query to represent a whole track. Score-based detection
selection, such as ArTIST [41] and ByteTrack [62], also helps the tracker to
keep high-quality detections and tracks. In 3D MOT, 3D Kalman Filter is a very
common practice. AB3DMOT [55] propose a simple KF-based online tracker. Recent
work [51] has revealed that with only 3D KF, LiDAR-based 3D MOT is almost
association error-free evaluated on the mainstream autonomous driving
datasets.

Recently, with the development of monocular 3D object detection [36, 61, 53]
and multi-camera 3D object detection [54, 20, 26], camera-based 3D multiple
object tracking is emerging. In the early years, Mono3DT [18] is the first to
learn monocular 3D object detection and 3D multiple object tracking. MUTR3D
[60] extends the transformer-based tracker to camera-based 3D MOT. PF-Track
[35] is the new state-of-the-art camera-based 3D MOT method utilizing past and
future trajectory reasoning. Although these methods are a little similar to
ours on the 3D representation learning level, we focus on different key
problems. We care more about whether the hidden 3D information in the video
can help the association in 2D MOT, so we do not use the manually annotated 3D
object labels.

###  2.2 3D Representation from Video

In this subsection, we visit the existing work representing objects in the 3D
space, especially for learning from videos. Generally speaking, recovering the
3D representation from a video can be divided into scene reconstruction and
representing objects in 3D space. With the help of multi-view geometry,
Structure from Motion (SfM) [44] is a practical algorithm to estimate the
depth of each keypoint and recover the sparse 3D structure of the static scene
from a movable camera. Multi-view Stereo (MVS) [45] is the dense scene
reconstruction method from every pixel. In robotics, Simultaneous Localization
and Mapping (SLAM) [6, 34, 33] utilizes a continuous video clip to estimate
the robot’s ego-pose and construct a sparse or dense map for planning and
controlling.

In the perception field, a common practice is representing 3D objects with
cuboid bounding boxes, defined as the 3D object detection task [46, 11].
However, most vision-based 3D object detection methods [36, 61] leverage the
neural network to fit the cuboid labels from _LiDAR_ annotations. As for 3D
video object detection, inspired by scene reconstruction, DfM [52] and
BEVStereo [25] refine the object depth from the video. BA-Det [15] is the
first to represent the consistent 3D object explicitly in the video via
object-centric bundle adjustment.

Another kind of 3D representation from the video is to learn video consistent
depth. SfMLearner [65] is a pioneer work taking view synthesis as supervision.
[31, 64] are the methods of learning consistent video depth and smooth scene
flow with test-time training. These methods learn from traditional Structure-
from-Motion and treat temporal warping consistency as geometric constraints.
However, our method learns object 3D representation rather than the dense
scene-level depth.

##  3 Preliminary: Structure-from-Motion

Given a series of images$\\{\mathbf{I}^{t}\\}$, Structure-from-Motion is to
reconstruct the 3D scene with 3D points $\\{\mathbf{P}_{i}\\}$ amd estimate
camera motion $\\{\mathbf{T}^{t}\\}$ in the global frame. In the SfM system,
local features $\\{\mathbf{f}^{t}_{i}\\}$ are extracted at the location of
keypoints $\\{\mathbf{p}^{t}_{i}\\}$ in each frame firstly using traditional
feature extractor SIFT [28], or deep learning-based feature extractor D2-Net
[9] and SuperPoint [8]. Then the extracted local features are matched as pairs
$\\{(\mathbf{p}^{t}_{i},\mathbf{p}^{t^{\prime}}_{i})\\}$ between different
frames and the keypoint tracklets
$\\{(\mathbf{p}^{1}_{i},\cdots,\mathbf{p}^{T}_{i})\\}$ are generated. With
triangulation, the initial 3D location of matched keypoint pairs can be
obtained. And utilizing bundle adjustment in multi-frames, the camera poses
$\\{\mathbf{T}^{t*}\\}$ in each frame and the global locations
$\\{\mathbf{P}_{i}^{*}\\}$ of each keypoint are optimized by minimizing the
reprojection errors of the extracted keypoints. In practice, COLMAP [44] is a
fast and widely-used SfM framework.

##  4 Methodology

In this section, we introduce the new 2D MOT paradigm with 3D object
representation. The pipeline contains three parts: (1) a CenterNet [67]-based
2D object detection branch; (2) a joint learning framework to learn 3D object
representation and object association; (3) an online 2D tracker with 3D object
representation in the inference stage, called P3DTrack.

The 2D object detection branch is mainly from CenterNet [67], with DLA-34 [58]
backbone fusing multi-scale features. The output feature map of the backbone
is with stride 4. The detection heads include the center heatmap prediction
head and the bounding box regression head. Following FCOS [47], the regression
target is $(l^{*},t^{*},r^{*},b^{*})$, corresponding to the distance from the
regression center to the sides of the bounding box. In the inference stage,
non-maximum suppression (NMS) is performed as a post-process, different from
the original CenterNet.

Figure 3: Pipeline of P3DTrack. The object detector is based on CenterNet. We
learn the 3D object representation and data association module jointly from
the pseudo 3D object labels.

###  4.1 Generating Pseudo 3D Object Labels

The existing works, which either utilize the pretrained depth estimator
supervised by LiDAR or learn the object representation with human-annotated 3D
labels, are not pure 2D MOT paradigms. We want to learn the 3D object
representation only from the monocular video. In this section, as shown in
Alg. 1 and Fig. 2, we introduce how to generate the pseudo 3D object labels
from the video clips taken from the surrounding cameras based on the
Structure-from-Motion technique.

Given a video clip $\mathcal{V}=\\{\mathbf{I}_{t}^{k}|t=1,2\cdots,T\\}$ taken
from the surrounding cameras $\\{c_{k}\\}$, we first extract the 2D keypoints
$\mathbf{p}_{t,k}^{i}=[u_{i},v_{i}]^{\top},(i=1,2,\cdots,n)$ and the
corresponding local features $\mathcal{F}_{t}=\\{\mathbf{f}_{t,k}^{i}\\}$
utilizing SuperPoint [8] network. The local feature matching network is from
SuperGlue [43], a GNN-style network aggregating spatial and temporal
information to make local features discriminative. Suppose that intrinsic of
each camera $\mathbf{K}_{c_{k}}$ and the camera pose
$\mathbf{T}_{t}^{k}=[\mathbf{R}_{t}^{k}|\mathbf{t}_{t}^{k}]$ at time $t$ are
given, then SfM system can be applied to obtain the 3D position of each
keypoint $\mathbf{P}_{i}$ in the global frame by solving the nonlinear least
square problem

$\\{{\mathbf{P}}^{*}_{i}\\}_{i=1}^{n}=\underset{\\{\mathbf{P}_{i}\\}_{i=1}^{n}}{\arg\min}\frac{1}{2}\sum_{i=1}^{n}\sum_{t=1}^{T}||\mathbf{p}_{i}^{t}-\Pi(\mathbf{T}^{t},\mathbf{P}_{i},\mathbf{K})||^{2}.$
(1)

where $\Pi(\cdot)$ is the reprojection function from the world frame to the
image frame with the camera pose and the intrinsic parameters. Note that not
all scenes can be reconstructed well. When the camera movement is relatively
small, the reconstruction performance is bad. So, we filter the videos with a
low speed of ego-motion. Besides, in the traditional SfM system, the static
scene can be well-reconstructed but the moving objects can not be
reconstructed because their movement is not the same as the ego-motion and are
filtered in the outlier filtering step by RANSAC [13]. Thanks to the
generalization capability of the deep neural network, these inherent problems
of traditional SfM are easily solved. Please refer to Sec. 4.2 for more
details.

Input: video clip $\mathcal{V}$, camera intrinsic $\\{\mathbf{K}_{c_{k}}\\}$,
camera pose $\\{\mathbf{T}_{t}^{k}\\}$, 2D bounding box with id
$\\{\mathbf{B}_{j}^{t}\\}_{j=1}^{n_{d}}$

Output: pseudo 3D object label $\\{\mathbf{o}^{t*}_{j}\\}$

1

2$\\{{\mathbf{P}}^{*}_{i}\\}_{i=1}^{n}\leftarrow\mathtt{SfM}(\mathcal{V},\\{\mathbf{K}_{c_{k}}\\},\\{\mathbf{T}_{t}^{k}\\})$

3 $\mathcal{P}\leftarrow\varnothing$

4 // IntraPC for each object

5 for  _$j\in[1,n_{d}]$_ do

6 for  _$t\in[1,T]$_ do

7
$\mathcal{P}_{j}^{t}\leftarrow\mathtt{IntraPC}(\\{{\mathbf{P}}^{*}_{i}\\},\mathbf{B}_{j}^{t})$

8 $\mathcal{P}\leftarrow\mathcal{P}\cup\mathcal{P}_{j}^{t}$

9

10

11// InterPC for all foreground points $\mathcal{P}$ to generate
$n_{d}^{\prime}$ clusters

12
$\\{\mathcal{P}_{j^{\prime}}\\}_{j^{\prime}=1}^{n^{\prime}_{d}}\leftarrow\mathtt{InterPC}(\mathcal{P})$

13 // Assign ID to the point clusters

14
$\\{\mathcal{P}_{j}^{*}\\}_{j=1}^{n_{d}}\leftarrow\mathtt{MatchID}(\\{\mathcal{P}_{j^{\prime}}\\},\\{\mathbf{B}_{j}^{t}\\})$

15 // Find barycenter for each cluster and transform to each camera frame

16
$\\{\mathbf{o}^{t*}_{j}\\}\leftarrow\mathtt{FindCenter\&Transform}(\\{\mathcal{P}_{j}^{*}\\},\\{\mathbf{T}_{t}^{k}\\})$

17 return _$\\{\mathbf{o}^{t*}_{j}\\}$_

18

Algorithm 1 Generating pseudo 3D object labels from a video.

After reconstructing the 3D points in the global frame, we filter the points
that can be projected to the foreground regions in the images, and then we
perform the Intra-frame Point Clustering (Intra-PC) to select the main
connected component of the 3D keypoints for each object $\mathbf{B}_{j}^{t}$:

$\mathcal{P}_{j}^{t}=\\{{\mathbf{P}}_{i}\\}_{i=1}^{m}=\mathtt{IntraPC}(\\{\mathbf{P}_{i}^{*}|\Pi(\mathbf{T}_{t}^{k},\mathbf{P}_{i}^{*},\mathbf{K}_{c_{k}})\in\mathbf{b}_{j}^{t}\\}),$
(2)

where $\mathbf{b}_{j}^{t}$ is the region within the ground truth 2D bounding
box $\mathbf{B}_{j}^{t}$. The index $j\in[1,n_{d}]$ means the identity of the
object. The distance threshold of a cluster is $\delta$. That means we only
consider the biggest cluster belonging to the object, which can filter the
noise points in the background regions or caused by keypoint mismatching.

Besides, we cluster the 3D keypoints for the second stage, called Inter-frame
Point Clustering (Inter-PC), to further filter the noise points in the global
frame for all objects together:

$\\{\mathcal{P}_{j^{\prime}}\\}_{j^{\prime}=1}^{n^{\prime}_{d}}=\mathtt{InterPC}(\bigcup_{t,j}\mathcal{P}_{j}^{t}).$
(3)

We finally generate $n_{d}^{\prime}$ object clusters. In each cluster, the
number of points must exceed the threshold $\kappa$. In general, the cluster
number is less than the number of tracks, so some tracks will not be assigned
to the 3D object labels. In 3D object representation learning, these objects
are only supervised by 2D representation labels and be ignored in 3D.

After clustering the 3D points, we should match the cluster with the maximum
probability corresponding to the tracklet. We define the matching strategy
that if the cluster can be only projected into _one_ bounding box
$\mathbf{B}_{j}$ with id $j$, we match the cluster to the tracklet $j$. If
not, we assign the cluster to the tracklet whose 2D bounding box can involve
the maximum number of the reprojected points. Note that we only label static
objects and this matching strategy guarantees the same 3D object position in
the global frame for a tracklet, so the 3D object representation learning can
keep _consistent_ in a tracklet and the association can be learned well.

To supervise the 2D bounding box with 3D representation, we need to define the
target 3D location for each ground truth 2D bounding box. Two kinds of
definitions are common in practice. The first is the barycenter, which is the
average 3D position of the cluster in the global frame. The second definition
is the minimum circumscribed cuboid for the cluster. In the experiments, we
find that with these two definitions, tracking performance is nearly the same.
So, we finally choose the barycenter
$\mathbf{o}_{j}^{t*}=[x_{j}^{t*},y_{j}^{t*},z_{j}^{t*}]^{\top}$ in the camera
frame as the 3D position target for the object. In summary, we obtain the
initial 3D representation with a cluster of 3D points for each tracklet. Then
we need to learn the generalized 3D representation for each object and utilize
the 3D object representation to associate the objects between frames.

###  4.2 3D Representation and Association Learning

3D object representation learning. In this subsection, we introduce the
proposed 3D object representation learning module, called the 3DRL, shown in
Fig. 3. 3DRL module is an additional head of the object detection model
CenterNet to represent 3D attributes of the objects, with a shared DLA-34
backbone. We denote the output of the backbone feature as
$\mathbf{F}_{t}\in\mathbb{R}^{H\times W\times C}$. Like FairMOT [63], we
choose the 2D object center position to extract the 3D feature
${{}^{(3D)}}\mathbf{f}_{j}^{t}$ of the object $\mathbf{B}_{j}^{t}$, which will
be further utilized for object tracking. In the following, we will explain how
to extract the 3D object feature. The 3D branch first uses MLPs as the feature
encoder to encode the backbone features, and the 3D object feature is

$\mathbf{o}_{j}^{t}=\mathtt{MLP}(\mathbf{F}_{t})[c_{j}(u),c_{j}(v)],$ (4)

where $[c_{j}(u),c_{j}(v)]$ is the position of 2D object center on the feature
map $\mathbf{F}_{t}$, and
$\mathbf{o}_{j}^{t}=[x_{j}^{t},y_{j}^{t},z_{j}^{t}]^{\top}$ is the 3D position
of the object center in the camera frame. Learned from the generated pseudo 3D
object labels, we define the 3D object position auxiliary loss with
uncertainty following [22, 61],

$\mathcal{L}_{3D}=\frac{||\mathbf{o}_{j}^{t}-\mathbf{o}_{j}^{t*}||_{1}}{\sigma^{2}}+\log(\sigma^{2}),$
(5)

where $\sigma^{2}$ is the learnable 3D position uncertainty for the object
$j$.

Then the 3D object representation is predicted by an additional fully-
connected layer

${{}^{(3D)}}\mathbf{f}_{j}^{t}=\mathtt{fc}(\mathbf{o}_{j}^{t}).$ (6)

Note that even though only the position of the static object is supervised,
the network can easily make the 3D position prediction of the moving object,
because the network does not distinguish the object movement in one frame. So,
whether the object is moving does not affect 3D object representation
learning.

Appearance model. Besides the 3D object representation, following the
mainstream 2D MOT methods, we use an object re-identification network as the
appearance model:

${{}^{(2D)}}\mathbf{f}_{j}^{t}=\mathtt{ReID}(\mathbf{I}^{t},\mathbf{B}_{j}^{t}).$
(7)

The architecture of the appearance model is the ResNet50 backbone followed by
a global average pooling layer and a fully connected layer with 512 channels.
Triplet loss [17] is used for robust appearance feature learning. The
appearance model is trained on WOD training set for 20 epochs and the feature
is also used for training the object association module, but will not be
updated.

Object association. The proposed object association model contains two parts:
a GNN-based feature aggregation module and a differentiable matching module.
Given the object’s 3D representation and appearance feature, we define the
object feature for

${{}^{(0)}}\mathbf{f}_{j}^{t}=[{{}^{(2D)}}\mathbf{f}_{j}^{t},{{}^{(3D)}}\mathbf{f}_{j}^{t}],\\\
$ (8)

where $[\cdot]$ is the concatenation operation and 3D representation and
appearance features are $L_{2}$-normalized first. We adopt an $L$-layer cross-
frame GNN to aggregate the object features from different frames:

${{}^{(l+1)}}\mathbf{f}_{j}^{t}=\mathtt{MLP}({{}^{(l)}}\mathbf{f}_{j}^{t}+\frac{||{{}^{(l)}}\mathbf{f}_{j}^{t}||_{2}{{}^{(l)}}\mathbf{m}_{j}^{t-1}}{||{{}^{(l)}}\mathbf{m}_{j}^{t-1}||_{2}}),l\in[0,L-1],$
(9)

where the aggregated message $\mathbf{m}_{j}^{t-1}$ is calculated from

${{}^{(l)}}\mathbf{m}_{j}^{t-1}=\sum_{j^{\prime}=1}^{n_{t-1}}{{}^{(l)}}w_{j,j^{\prime}}{{}^{(l)}}\mathbf{f}_{j^{\prime}}^{t-1}.$
(10)

Averaging weight $w_{j,j^{\prime}}$ is cosine similarity between the
normalized features

${{}^{(l)}}w_{j,j^{\prime}}=\cos({{}^{(l)}}\mathbf{f}_{j}^{t},{{}^{(l)}}\mathbf{f}_{j^{\prime}}^{t-1}).$
(11)

The matching layer is defined as a differentiable Quadratic Problem [16], and
the association loss can be the binary cross-entropy loss between the matching
results and the matching ground truth.

###  4.3 Tracking with 3D Object Representation

After obtaining the 3D object representation and training the object
association model, we propose our online tracker P3DTrack. In this section, we
explain how the tracker infers frame by frame. Similar to DeepSORT [56], we
match detections with existing tracklets in each frame. The tracklet’s
appearance feature is the average of appearance features in each frame. As for
3D object representation, we utilize a 3D Kalman Filter to model the 3D object
motion of the tracklet. The 3D representation of the tracklet is from the
predicted 3D position using 3D KF and encoded with Eq. 6. During the object
association stage, we use the association model in Sec. 4.2. Besides, we
balance the 3D representation and appearance similarity with weight $\alpha$
and $1-\alpha$, respectively.

##  5 Experiments

Method | Backbone | Split | Category | MOTA $\uparrow$ | IDF1 $\uparrow$ | FN $\downarrow$ | FP $\downarrow$ | ID Sw. $\downarrow$ | MT $\uparrow$ | ML $\downarrow$
---|---|---|---|---|---|---|---|---|---|---
IoU baseline [29] | ResNet-50 | val | Vehicle | 38.25 | - | - | - | - | - | -
Tracktor++ [1, 29] | ResNet-50 | val | Vehicle | 42.62 | - | - | - | - | - | -
RetinaTrack [29] | ResNet-50 | val | Vehicle | 44.92 | - | - | - | - | - | -
QDTrack [12] | ResNet-50 | val | Vehicle | 55.6 | 66.2 | 514548 | 214998 | 24309 | 17595 | 5559
P3DTrack (Ours) | DLA-34 | val | Vehicle | 55.9 | 65.6 | 638390 | 74802 | 37478 | 13643 | 8583

Table 1: Results on WOD tracking val set using py-motmetrics library.

###  5.1 Dataset and Evaluation Metrics

We conduct our experiments mainly on the large-scale autonomous driving
dataset, Waymo Open Dataset (WOD). WOD has 798 sequences for training, 202 for
validation, and 150 for testing. More than 1 million images are taken from 5
surrounding cameras. Such a big dataset is suitable for more general feature
learning in MOT. The official evaluation metrics are CLEAR MOT metrics [2].
MOTA is the main metric to evaluate the number of false positives, false
negatives, and mismatches. The number of mismatches is the most important to
evaluate the data association performance. Besides, to evaluate more detailed
data association performance, we also report the IDF1 metric [40] using py-
motmetrics111https://github.com/cheind/py-motmetrics library. The ablation
study is conducted with the FRONT camera on WOD val set. To verify the
performance of our method in different datasets, we also report the results on
KITTI [14] dataset in the appendix.

###  5.2 Implementation Details

Training. We use a DLA-34 [58] as the backbone without any neck, and the head
is with 2 layers of 3*3 convolutions and MLP. The resolution of the input
images is 1920*1280. If the input size is smaller than it, we will use zero
padding to complete the image. Our implementation is based on PyTorch [37]
framework. We train our model on 8 NVIDIA A40 GPUs. Adam [24] optimizer is
applied with $\beta_{1}=0.9$ and $\beta_{2}=0.999$. The learning rate is
8$\times$10-5 and weight decay is 10-5. We train 6 epochs for 2D object
detection and an additional epoch for 3D object representation learning. The
object association module is learned for 4 additional epochs. The cosine
learning rate scheduler is adopted. The warm-up stage is the first 500
iterations. In the label generation stage, threshold $\delta$ and $\kappa$ are
0.5 and 30.

Inference. We set the detection threshold to 0.5 and the bounding box whose
score is below the threshold will not create a new tracklet. Similar to
ByteTrack [62], the detections with lower scores are matched to the tracklet
after the high-quality detections. We set the appearance threshold to 0.6 in
data association, and we do not match the pair whose appearance similarity is
below the threshold. Like DeepSORT [56], we add an additional 2D IoU-based
matching for the remaining detections and tracklets. 3D similarity weight
$\alpha$ in matching is set to 0.4. The time interval before we terminate a
tracklet, called max-age, is 30 frames.

###  5.3 Comparisons with State-of-the-art Methods

In Table 1, we show the results compared with SOTA methods on the WOD val set.
QDTrack is the previous SOTA method with a FasterRCNN-based detector and the
quasi-dense matching learning module. We outperform QDTrack by 0.3 MOTA with a
worse DLA-34 backbone. The strict matching strategy and thresholds make the
number of FP much lower than QDTrack. The CenterNet-based detector has a lower
recall than the FasterRCNN detector in QDTrack, so the FN is much higher than
QDTrack and our IDF1 is also suffering from the lower ID Recall problem.

| MOTA $\uparrow$ | IDF1 $\uparrow$ | FP $\downarrow$ | FN $\downarrow$ | ID Sw. $\downarrow$
---|---|---|---|---|---
Baseline | 51.0 | 62.3 | 8709 | 331056 | 9100
+low-quality dets [62] | 53.4 | 64.3 | 13058 | 309653 | 9005
+GNN | 56.5 | 66.5 | 20381 | 278752 | 10129
+3D representation | 57.6 | 68.1 | 33587 | 258066 | 9920

Table 2: Ablation study of P3DTrack.

###  5.4 Ablation Study

We ablate each component in P3DTrack, shown in Table 2. Our baseline is
modified DeepSORT [56], but without delayed tracklet initialization.
Considering the low-quality detections like ByteTrack [62], the MOTA increases
by 2.4. With association learning with GNN and jointly learning 3D
representation, MOTA and IDF1 metrics can improve by 4.2 and 3.8.

Methods | MOTA $\uparrow$ | IDF1 $\uparrow$ | FP $\downarrow$ | FN $\downarrow$ | ID Sw. $\downarrow$
---|---|---|---|---|---
2D + 3D feature (Ours) | 57.6 | 68.1 | 33587 | 258066 | 9920
2D feature | 56.5 | 66.5 | 20381 | 278752 | 10129
2D feature + 2D motion | 55.5 | 58.6 | 29586 | 263370 | 23979
2D feature + 3D motion | 57.3 | 63.3 | 33048 | 257936 | 13026

Table 3: Different 2D and 3D representations in P3DTrack.

3D rep from | MOTA $\uparrow$ | IDF1 $\uparrow$ | FP $\downarrow$ | FN $\downarrow$ | ID Sw. $\downarrow$
---|---|---|---|---|---
P3DTrack (Ours) | 57.6 | 68.1 | 33587 | 258066 | 9920
SfM [44] | 55.5 | 66.8 | 55823 | 249935 | 11145
MiDaS v3 [38] | 55.7 | 63.4 | 34514 | 259471 | 21058

Table 4: Comparisons with different 3D representation.

det thres | app thres | MOTA $\uparrow$ | IDF1 $\uparrow$ | FP $\downarrow$ | FN $\downarrow$ | ID Sw. $\downarrow$
---|---|---|---|---|---|---
0.4 | 0.6 | 51.4 | 64.9 | 117703 | 210305 | 17933
0.5 | 0.6 | 57.6 | 68.1 | 33587 | 258066 | 9920
0.6 | 0.6 | 50.1 | 63.0 | 12885 | 337208 | 4864
0.5 | 0.5 | 56.1 | 66.6 | 41097 | 257660 | 13359
0.5 | 0.7 | 58.0 | 66.9 | 28798 | 260921 | 9139

Table 5: Different detection and appearance thresholds.

3D weight $\alpha$ | MOTA $\uparrow$ | IDF1 $\uparrow$ | FP $\downarrow$ | FN $\downarrow$ | ID Sw. $\downarrow$
---|---|---|---|---|---
0.0 (2D feature) | 56.5 | 66.5 | 20381 | 278752 | 10129
0.4 | 57.6 | 68.1 | 33587 | 258066 | 9920
0.5 | 57.3 | 67.2 | 34091 | 258324 | 11733
0.6 | 56.8 | 66.0 | 34213 | 258583 | 14431
1.0 (3D feature) | 52.1 | 54.2 | 33366 | 265203 | 42443

Table 6: Different 3D representation similarity weight.

Methods | Backbone | Latency | Memory | #Param.
---|---|---|---|---
Baseline | DLA-34 | 219.8ms | 13.29GB | 22.65M
P3DTrack (Ours) | DLA-34 | 231.5ms | 13.33GB | 24.42M

Table 7: Inference latency, training memory, and the number of parameters
compared with the baseline model.

(a) Heavy occlusion. (b) Similar appearance.

Figure 4: Qualitative results. Top: Baseline model without 3D object
representation. Bottom: P3DTrack (Ours).

###  5.5 Discussions

Influence of 3D representation. In P3DTrack, we consider the appearance
feature similarity and 3D representation similarity and weight them with
$\alpha$. As shown in Table 3, we conduct experiments to verify this kind of
design is better than the method only with appearance features and with 2D
Kalman Filter and 3D Kalman Filter motion models. Compared with appearance
features only, we have 1.1 MOTA and 1.6 IDF1 improvement. If we use 2D KF to
model the motion of bounding boxes, the performance is worse than with
appearance features only. That is because the 2D motion cannot be modeled as a
linear system, especially for the cars parking on the roadside. The 3D KF is a
better motion model. However, without jointly learned 3D representation, the
association is still not well enough, especially for the IDF1 metric.

Different 3D representation. We also compare different 3D representation in
Table 4. Compared with learning-free Structure-from-Motion 3D positions, our
method has the generalization ability to predict the moving objects, but only
with SfM the 3D position of the moving object is incorrect. Besides, the pre-
trained depth model MiDaS v3 [38] can not estimate video-consistent depth for
the object. Based on the relative depth, the 3D motion model can not predict
the correct depth in the next frame, which is important to obtain a good 3D
similarity.

Influence of detection and appearance thresholds. The detection threshold is
to decide the quality of the detections. And the appearance threshold is to
avoid the mismatch between the detections and the tracklets. The experiments
in Table 5 show these thresholds are necessary for our method. And finally, we
choose 0.5 and 0.6 for detection and appearance thresholds to balance the MOTA
and IDF1 metrics.

Influence of 3D similarity weight. As mentioned in Sec. 4.3, we weight the 3D
similarity and appearance similarity with $\alpha$. In Table 6, we show the
tracking results with different weights. When $\alpha=0$, we only use the
appearance feature to associate the objects. Note that even only with the 3D
representation learned from the video, not depending on any image features,
the tracking results are not so bad, with 5.1 MOTA and 13.9 IDF1 decreases
compared with the best performance. However, with the combination of
appearance similarity and 3D representation similarity, the performance is
better.

Additional cost compared with baseline. Although we generate the pseudo 3D
object labels and train an additional 3D object representation learning
module, we do not add much extra computation overhead and memory cost. As
shown in Table 7, the inference latency only increases by 12 ms, about 5.5%
relatively slower than the baseline. The training memory and the number of
network parameters also indicate the efficiency of our method.

###  5.6 Qualitative Results

We show the qualitative results of baseline and P3DTrack in Fig. 4. When the
object is heavily occluded, with 3D representation, P3DTrack can keep the
tracklet and avoid mismatching. 3D representation is also useful for similar
appearances, the new object will not match the existing tracklet, due to
different 3D locations.

##  6 Conclusion

In this paper, we propose a new 2D MOT framework with the help of 3D object
representation from the video itself, called P3DTrack. We design a pseudo 3D
object label generation method from the reconstructed scene and 2D bounding
boxes with identity. Supervised by the pseudo 3D object labels, the 3D object
representation can be jointly learned with 2D object detection and object
association. So, an online tracker can be easily achieved with the learnable
association module. We conduct experiments mainly on the large-scale Waymo
Open Dataset. Extensive experiments show the effectiveness of our method.

## Appendix

In the appendix, we describe the detailed network architectures and
implementation details, perform additional experiments, and provide more
analyses. The contents are as below:

  * •

In Sec. A and B, we show the details of P3DTrack to help understand our
method.

  * •

In Sec. C, there are additional experiments on KITTI and analyses about our 3D
object learning module.

  * •

In Sec. D, visualization shows the success and failure cases.

  * •

In Sec. E, we discuss the limitation and future work.

  * •

In Sec. F, the reproducibility statement helps reproduce our work.

##  A Network Architecture

Module | Keys | Values
---|---|---
DLA-34 | levels | 1, 1, 1, 2, 2, 1
channels | 16, 32, 64, 128, 256, 512
DLA-up | channels | 64, 128, 256, 512
scales | 1, 2, 4, 8
Cls-head-pre | Conv | 2* (3*3),64,256
Reg-head-pre | Conv | 2* (3*3),64,256
Cls-head | MLP | 256,3
Reg-head | MLP | 256,4
Offset-head | MLP | 256,2
Depth-head | MLP | 256,1
Uncer-head | MLP | 256,1
App-model | backbone | ResNet50
App-feature | channels | 512
3D-feature | channels | 512
Obj-feature | fusion | concat&norm
GNN | #layer | 1

Table A: Network architecture of P3DTrack

In this section, we explain the detailed network design of P3DTrack. The
parameters of the network architecture are shown in Table A. The backbone is a
standard ‘dla-seg’ architecture, please refer to [67] for more details. The
heads include classification head (Cls-head), 2D regression head (Reg-head),
offset head (Offset-head), depth and depth uncertainty head (Depth-head &
Uncer-head). The appearance model is the ResNet50 backbone followed by a
global average pooling layer and a fully connected layer with 512 channels.
The 3D object feature is 512 channels predicted by a fully-connected layer
from the 3D object position regression. The object feature is fused by
concatenating the appearance feature and 3D feature and normalizing it. The
object feature is aggregated by 1 layer GNN.

##  B Training Details

In Table B, we list the training config of P3DTrack. In P3DTrack, 2D object
detection is first learned for 6 epochs and 3D object representation is
learned for 1 epochs. Then 3D object representation and object association can
be jointly learned for 4 epochs. The appearance model is pre-trained and
frozen in the later training stage.

Configuration | Values
---|---
optimizer | Adam
optimizer momentum | $\beta_{1}=0.9,\beta_{2}=0.999$
weight decay | $1\times 10^{-5}$
learning rate | $8\times 10^{-5}$
learning rate scheduler | cosine decay
warmup iterations | 500
epochs | 6+1+4
augmentation | random flip
aug. probability | 0.5
batch size | 8
gradient clip | 5
image size | 1920 $\times$1280 (zero padding)
loss name | heatmap loss, bbox loss, offset loss
depth loss, reid loss, association loss
loss weight | 1.0, 1.0, 0.2, 0.1, NA (pretrained), 0.5
sync BN | True

Table B: Training config of P3DTrack.

Method | +Label | +Data | HOTA | AssA | ID Sw. | DetA | DetRe | DetPr | AssRe | AssPr | LocA | MOTA
---|---|---|---|---|---|---|---|---|---|---|---|---
QD-3DT [19] | 3D GT |  | 72.77 | 72.19 | 206 | 74.09 | 78.13 | 85.48 | 74.87 | 89.21 | 87.16 | 85.94
Mono3DT [18] | 3D GT |  | 73.16 | 74.18 | 379 | 72.73 | 76.51 | 85.28 | 77.18 | 87.77 | 86.88 | 84.28
OC-SORT [4] |  | PD | 76.54 | 76.39 | 250 | 77.25 | 80.64 | 86.36 | 80.33 | 87.17 | 87.01 | 90.28
PermaTrack [49] |  | PD | 78.03 | 78.41 | 258 | 78.29 | 81.71 | 86.54 | 81.14 | 89.49 | 87.10 | 91.33
RAM [48] |  | PD | 79.53 | 80.94 | 210 | 78.79 | 82.54 | 86.33 | 84.21 | 88.77 | 87.15 | 91.61
QDTrack [12] |  |  | 68.45 | 65.49 | 313 | 72.44 | 76.01 | 85.37 | 68.28 | 88.53 | 86.50 | 84.93
TrackMPNN [39] |  |  | 72.30 | 70.63 | 481 | 74.69 | 80.02 | 83.11 | 73.58 | 87.14 | 86.14 | 87.33
CenterTrack [66] |  |  | 73.02 | 71.20 | 254 | 75.62 | 80.10 | 84.56 | 73.84 | 89.00 | 86.52 | 88.83
LGM [50] |  |  | 73.14 | 72.31 | 448 | 74.61 | 80.53 | 82.16 | 76.38 | 84.74 | 85.85 | 87.60
DEFT [5] |  |  | 74.23 | 73.79 | 344 | 75.33 | 79.96 | 83.97 | 78.30 | 85.19 | 86.14 | 88.38
P3DTrack (Ours) |  |  | 74.59 | 76.86 | 219 | 72.88 | 78.09 | 83.21 | 80.66 | 86.67 | 86.28 | 85.60

Table C: Tracking results on KITTI _test_ set.

| $\delta<1.25\uparrow$ | $\delta<1.25^{2}\uparrow$ | $\delta<1.25^{3}\uparrow$ | Abs Rel$\downarrow$ | Sq Rel$\downarrow$ | RMSE$\downarrow$ | RMSE log$\downarrow$
---|---|---|---|---|---|---|---
SfM [44] init. | 0.945 | 0.962 | 0.967 | 0.112 | 1.183 | 4.744 | 0.203
P3DTrack (Ours) | 0.974 | 0.983 | 0.985 | 0.049 | 0.456 | 3.372 | 0.110
training w/ 3D GT | 0.991 | 0.994 | 0.995 | 0.025 | 0.142 | 1.999 | 0.059

Table D: 3D object depth accuracy analyses on KITTI _training_ set. We ignore
the predictions with depth$>$75m.

| $\delta<1.25\uparrow$ | $\delta<1.25^{2}\uparrow$ | $\delta<1.25^{3}\uparrow$ | Abs Rel$\downarrow$ | Sq Rel$\downarrow$ | RMSE$\downarrow$ | RMSE log$\downarrow$
---|---|---|---|---|---|---|---
SfM [44] init. | 0.981 | 0.988 | 0.990 | 0.106 | 0.831 | 3.973 | 0.473
P3DTrack (Ours) | 0.987 | 0.992 | 0.993 | 0.084 | 0.653 | 2.940 | 0.145
training w/ 3D GT | 0.996 | 0.998 | 0.998 | 0.022 | 0.233 | 1.569 | 0.061

Table E: 3D object depth accuracy analyses on WOD _training_ set. We ignore
the predictions with depth$>$75m. Note that 3D GT is annotated on LiDAR, so
some objects do not have the 3D label. True depth error is lower than the
values in table.

| ratio | $\delta<1.25\uparrow$ | $\delta<1.25^{2}\uparrow$ | $\delta<1.25^{3}\uparrow$ | Abs Rel$\downarrow$ | Sq Rel$\downarrow$ | RMSE$\downarrow$ | RMSE log$\downarrow$
---|---|---|---|---|---|---|---|---
Dynamic | 56.4% | 0.985 | 0.990 | 0.992 | 0.087 | 0.922 | 3.349 | 0.152
Static | 43.6% | 0.988 | 0.994 | 0.995 | 0.081 | 0.305 | 2.305 | 0.135
All | 100% | 0.987 | 0.992 | 0.993 | 0.084 | 0.653 | 2.940 | 0.145

Table F: Dynamic/static objects’ 3D location performance on WOD.

| MOTA $\uparrow$ | IDF1 $\uparrow$ | FP $\downarrow$ | FN $\downarrow$ | ID Sw. $\downarrow$
---|---|---|---|---|---
FRONT | 57.6 | 68.1 | 33587 | 258066 | 9920
FRONT LEFT | 58.5 | 66.6 | 11782 | 105092 | 6874
FRONT RIGHT | 55.6 | 64.1 | 8566 | 79516 | 4765
SIDE LEFT | 52.8 | 61.5 | 12741 | 112641 | 8506
SIDE RIGHT | 49.4 | 59.1 | 8126 | 83075 | 7413

Table G: Tracking results for each camera on WOD _val_ set.

##  C Additional Experiments and Analyses

###  C.1 Results on KITTI Dataset

We conduct some additional experiments on KITTI [14] dataset. KITTI MOT
dataset has 21 sequences for training and 29 sequences for testing. The
official metrics are mainly based on HOTA [30], including the detection
metrics DetA, LocA, and the association metric AssA. KITTI also evaluates the
CLEAR MOT [2] metrics. The implementation details of P3DTrack are mostly the
same as WOD dataset. However, there are some different settings. We train the
2D detection part for 40 epochs and then train 3D representations for 20
epochs and jointly train the association module for 5 epochs. We use the pre-
trained 2D detection model on the general object detection COCO [27] dataset,
the same as the other methods evaluated on KITTI. The 3D weight $\alpha$ is
set to 0.3, and the detection threshold is 0.2. The basic image resolution is
1280$\times$384\. We use additional random resize augmentation. The
augmentation probability is 0.5, and the resize scale is random in [0.5,1.5].
We report the tracking results of vehicles.

As shown in Table C, we achieve new state-of-the-art performance compared with
the methods without any additional human annotations and pre-trained models on
the autonomous driving datasets. Especially for the association metrics, we
improve AssA by 3.0 and have the lowest number of ID Switches. Because we do
not train our model on other autonomous driving datasets, the detection
performance is not the best and we do not outperform the methods with a pre-
trained model on the PD dataset. However, compared with the monocular 3D MOT
methods that use the additional 3D ground truth, we still outperform them.

###  C.2 3D Object Location Accuracy Analyses

In Table D and Table E, we discuss the object depth accuracy of our P3DTrack
on KITTI and WOD. The metrics are from the general depth estimation metrics
[10]. Note that we only calculate the depth accuracy of the object centers. We
can find that our P3DTrack, which trains 3D object representation from SfM
initialization, has more accurate object depth estimation than the initialized
pseudo labels. Note that we only consider the objects that have been assigned
3D pseudo labels when evaluating the SfM initialization. However, for
P3DTrack, we evaluate depth of all detected objects, including the generalized
objects without initial 3D pseudo labels. Even though, P3DTrack still
outperforms the SfM initialization in object 3D location on both KITTI and WOD
datasets.

###  C.3 Gap between Ours and Using 3D GT Labels

Table D and Table E also show the 3D object location accuracy gap between ours
and the model trained with ground truth 3D object labels. We find that
learning 3D object representation from SfM-initialized pseudo labels is a
little worse than learning from GT 3D labels. However, the gap in 3D location
accuracy is relatively small, which means our pseudo labels are practical for
estimating the object’s 3D location. The ablation study results in the main
paper also show that 3D object representation can help object association. In
Table C, the comparison with monocular 3D MOT, QD-3DT and Mono3DT, shows our
advantages.

###  C.4 Dynamic Object 3D Estimation Analyses

As mentioned in Sec. 4.1 and 4.2 of the main paper, we can not assign labels
for some objects because they are filtered in pseudo 3D object label
generation stage, especially for the dynamic objects that have the different
moving directions from the camera. The 3D location of these objects can only
be learned by our 3DRL module. In Table F, we validate the performance of
generalized 3D location learning for dynamic objects. Most objects are not
generated 3D object labels and only 43.6% of the objects (static objects) have
pseudo labels. The small gap between dynamic and static objects in results
show the generalization of our 3DRL module to the unlabeled dynamic objects.

###  C.5 Tracking Results of Different Cameras

We report the tracking results conditioned on the camera in Table G. It is
obvious that the main FRONT camera has the best performance. The FRONT LEFT
and FRONT RIGHT camera has a relatively well performance on both MOTA and
IDF1. However, the results of side cameras are not as good as front cameras,
because of the rapid movement of the camera relative to the objects and the
crowded vehicles in parking lots that are often on the sides. To keep the
generalizability and scalability of the algorithm, we do not set specific
parameters for the side cameras.

##  D Qualitative Results

We provide videos to show the tracking results of P3DTrack on WOD and KITTI
datasets. The sequences are ‘segment-7650923902987369309_2380_000_2400_000’ in
WOD val set and ‘0011’ in KITTI test set. Please refer to the attachments in
the compressed zip file.

##  E Limitations and Future Work

In this paper, we estimate the object’s 3D location and represent it as a 3D
feature. However, because the monocular depth estimation is an ill-posed
problem, the learned object 3D location is not so good, especially for the
object far away ($>$100m). We will try to learn 3D object representation from
multi-frames like recent work [15, 25, 52].

##  F Reproducibility Statement

We will release the training and inference codes to help reproduce our work
and the documents will be clearly written. Pseudo 3D object labeling tools
that are based on open-source packages COLMAP
[44]222https://github.com/colmap/colmap and hloc
[42]333https://github.com/cvg/Hierarchical-Localization will be released. The
checkpoint trained on the KITTI dataset will also be released. Limited by the
license of WOD, the checkpoint of the model trained on WOD cannot be publicly
available. However, we will provide it by email if needed. The implementation
details, including the implementation details, network architecture, and
different settings on KITTI are mentioned in Sec. 4 and 5.2 in the main paper
and Sec. A, B and C in the appendix. The best hyperparameters of the
experiments are listed in Sec. 5.2 in the main paper and Sec. C in appendix,
and the influences of hyperparameters are shown in Table 5 and Table 6. The
inference latency and computational cost are in Table 7 in the main paper. The
data and annotations of WOD444https://waymo.com/open/ and
KITTI555https://www.cvlibs.net/datasets/kitti/eval_tracking.php are publicly
available.

## References

  * [1] Philipp Bergmann, Tim Meinhardt, and Laura Leal-Taixe.  Tracking without bells and whistles.  In ICCV, 2019.
  * [2] Keni Bernardin and Rainer Stiefelhagen.  Evaluating multiple object tracking performance: the clear mot metrics.  EURASIP Journal on Image and Video Processing, 2008:1–10, 2008\.
  * [3] Guillem Brasó and Laura Leal-Taixé.  Learning a neural solver for multiple object tracking.  In CVPR, 2020.
  * [4] Jinkun Cao, Xinshuo Weng, Rawal Khirodkar, Jiangmiao Pang, and Kris Kitani.  Observation-centric sort: Rethinking sort for robust multi-object tracking.  In CVPR, 2023.
  * [5] Mohamed Chaabane, Peter Zhang, J Ross Beveridge, and Stephen O’Hara.  Deft: Detection embeddings for tracking.  In CVPRW, 2021.
  * [6] Andrew J Davison.  Real-time simultaneous localisation and mapping with a single camera.  In ICCV, 2003.
  * [7] Patrick Dendorfer, Vladimir Yugay, Aljosa Osep, and Laura Leal-Taixé.  Quo vadis: Is trajectory forecasting the key towards long-term multi-object tracking?  In NeurIPS, 2022.
  * [8] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superpoint: Self-supervised interest point detection and description.  In CVPRW, 2018.
  * [9] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler.  D2-net: A trainable cnn for joint detection and description of local features.  arXiv preprint arXiv:1905.03561, 2019.
  * [10] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  In NeurIPS, 2014.
  * [11] Lue Fan, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang.  Fully sparse 3d object detection.  In NeurIPS, 2022.
  * [12] Tobias Fischer, Jiangmiao Pang, Thomas E Huang, Linlu Qiu, Haofeng Chen, Trevor Darrell, and Fisher Yu.  Qdtrack: Quasi-dense similarity learning for appearance-only multiple object tracking.  arXiv preprint arXiv:2210.06984, 2022.
  * [13] Martin A Fischler and Robert C Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  Communications of the ACM, 24(6):381–395, 1981.
  * [14] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In CVPR, 2012.
  * [15] Jiawei He, Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang.  3d video object detection with learnable object-centric global optimization.  In CVPR, 2023.
  * [16] Jiawei He, Zehao Huang, Naiyan Wang, and Zhaoxiang Zhang.  Learnable graph matching: Incorporating graph partitioning with deep feature learning for multiple object tracking.  In CVPR, 2021.
  * [17] Alexander Hermans, Lucas Beyer, and Bastian Leibe.  In defense of the triplet loss for person re-identification.  arXiv preprint arXiv:1703.07737, 2017.
  * [18] Hou-Ning Hu, Qi-Zhi Cai, Dequan Wang, Ji Lin, Min Sun, Philipp Krahenbuhl, Trevor Darrell, and Fisher Yu.  Joint monocular 3d vehicle detection and tracking.  In ICCV, 2019.
  * [19] Hou-Ning Hu, Yung-Hsu Yang, Tobias Fischer, Trevor Darrell, Fisher Yu, and Min Sun.  Monocular quasi-dense 3d object tracking.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):1992–2008, 2022.
  * [20] Junjie Huang, Guan Huang, Zheng Zhu, and Dalong Du.  Bevdet: High-performance multi-camera 3d object detection in bird-eye-view.  arXiv preprint arXiv:2112.11790, 2021.
  * [21] Rudolph Emil Kalman.  A new approach to linear filtering and prediction problems.  1960\.
  * [22] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  In NeurIPS, 2017.
  * [23] Tarasha Khurana, Achal Dave, and Deva Ramanan.  Detecting invisible people.  In ICCV, 2021.
  * [24] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [25] Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li.  Bevstereo: Enhancing depth estimation in multi-view 3d object detection with dynamic temporal stereo.  arXiv preprint arXiv:2209.10248, 2022.
  * [26] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai.  Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.  In ECCV, 2022.
  * [27] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [28] David G Lowe.  Distinctive image features from scale-invariant keypoints.  International journal of computer vision, 60(2):91–110, 2004.
  * [29] Zhichao Lu, Vivek Rathod, Ronny Votel, and Jonathan Huang.  Retinatrack: Online single stage joint detection and tracking.  In CVPR, 2020.
  * [30] Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taixé, and Bastian Leibe.  Hota: A higher order metric for evaluating multi-object tracking.  International journal of computer vision, 129:548–578, 2021.
  * [31] Xuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopf.  Consistent video depth estimation.  ACM Transactions on Graphics (ToG), 39(4):71–1, 2020.
  * [32] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer.  Trackformer: Multi-object tracking with transformers.  In CVPR, 2022.
  * [33] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos.  Orb-slam: a versatile and accurate monocular slam system.  IEEE transactions on robotics, 31(5):1147–1163, 2015.
  * [34] Richard A Newcombe, Steven J Lovegrove, and Andrew J Davison.  Dtam: Dense tracking and mapping in real-time.  In ICCV, 2011.
  * [35] Ziqi Pang, Jie Li, Pavel Tokmakov, Dian Chen, Sergey Zagoruyko, and Yu-Xiong Wang.  Standing between past and future: Spatio-temporal modeling for multi-camera 3d multi-object tracking.  In CVPR, 2023.
  * [36] Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon.  Is pseudo-lidar needed for monocular 3d object detection?  In ICCV, 2021.
  * [37] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In NeurIPS, 2019.
  * [38] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3), 2022.
  * [39] Akshay Rangesh, Pranav Maheshwari, Mez Gebre, Siddhesh Mhatre, Vahid Ramezani, and Mohan M Trivedi.  Trackmpnn: A message passing graph neural architecture for multi-object tracking.  arXiv preprint arXiv:2101.04206, 2021.
  * [40] Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi.  Performance measures and a data set for multi-target, multi-camera tracking.  In ECCV, 2016.
  * [41] Fatemeh Saleh, Sadegh Aliakbarian, Hamid Rezatofighi, Mathieu Salzmann, and Stephen Gould.  Probabilistic tracklet scoring and inpainting for multiple object tracking.  In CVPR, 2021.
  * [42] Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk.  From coarse to fine: Robust hierarchical localization at large scale.  In CVPR, 2019.
  * [43] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In CVPR, 2020.
  * [44] Johannes Lutz Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In CVPR, 2016.
  * [45] Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm.  Pixelwise view selection for unstructured multi-view stereo.  In ECCV, 2016.
  * [46] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  Pv-rcnn: Point-voxel feature set abstraction for 3d object detection.  In CVPR, 2020.
  * [47] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: Fully convolutional one-stage object detection.  In ICCV, 2019.
  * [48] Pavel Tokmakov, Allan Jabri, Jie Li, and Adrien Gaidon.  Object permanence emerges in a random walk along memory.  In ICML, 2022.
  * [49] Pavel Tokmakov, Jie Li, Wolfram Burgard, and Adrien Gaidon.  Learning to track with object permanence.  In ICCV, 2021.
  * [50] Gaoang Wang, Renshu Gu, Zuozhu Liu, Weijie Hu, Mingli Song, and Jenq-Neng Hwang.  Track without appearance: Learn box and tracklet embedding with local and global motion patterns for vehicle tracking.  In ICCV, 2021.
  * [51] Qitai Wang, Yuntao Chen, Ziqi Pang, Naiyan Wang, and Zhaoxiang Zhang.  Immortal tracker: Tracklet never dies.  arXiv preprint arXiv:2111.13672, 2021.
  * [52] Tai Wang, Jiangmiao Pang, and Dahua Lin.  Monocular 3d object detection with depth from motion.  In ECCV, 2022.
  * [53] Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin.  Fcos3d: Fully convolutional one-stage monocular 3d object detection.  In ICCV, 2021.
  * [54] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon.  Detr3d: 3d object detection from multi-view images via 3d-to-2d queries.  In CoRL, 2022.
  * [55] Xinshuo Weng, Jianren Wang, David Held, and Kris Kitani.  Ab3dmot: A baseline for 3d multi-object tracking and new evaluation metrics.  arXiv preprint arXiv:2008.08063, 2020.
  * [56] Nicolai Wojke, Alex Bewley, and Dietrich Paulus.  Simple online and realtime tracking with a deep association metric.  In ICIP, 2017.
  * [57] Yihong Xu, Aljosa Osep, Yutong Ban, Radu Horaud, Laura Leal-Taixé, and Xavier Alameda-Pineda.  How to train your deep multi-object tracker.  In CVPR, 2020.
  * [58] Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell.  Deep layer aggregation.  In CVPR, 2018.
  * [59] Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei.  Motr: End-to-end multiple-object tracking with transformer.  In ECCV, 2022.
  * [60] Tianyuan Zhang, Xuanyao Chen, Yue Wang, Yilun Wang, and Hang Zhao.  Mutr3d: A multi-camera tracking framework via 3d-to-2d queries.  In CVPR, 2022.
  * [61] Yunpeng Zhang, Jiwen Lu, and Jie Zhou.  Objects are different: Flexible monocular 3d object detection.  In CVPR, 2021.
  * [62] Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang.  Bytetrack: Multi-object tracking by associating every detection box.  In ECCV, 2022.
  * [63] Yifu Zhang, Chunyu Wang, Xinggang Wang, Wenjun Zeng, and Wenyu Liu.  Fairmot: On the fairness of detection and re-identification in multiple object tracking.  International Journal of Computer Vision, 129(11):3069–3087, 2021\.
  * [64] Zhoutong Zhang, Forrester Cole, Richard Tucker, William T Freeman, and Tali Dekel.  Consistent depth of moving objects in video.  ACM Transactions on Graphics (TOG), 40(4):1–12, 2021.
  * [65] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe.  Unsupervised learning of depth and ego-motion from video.  In CVPR, 2017.
  * [66] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Tracking objects as points.  In ECCV, 2020.
  * [67] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  arXiv preprint arXiv:1904.07850, 2019.
  * [68] Xingyi Zhou, Tianwei Yin, Vladlen Koltun, and Philipp Krähenbühl.  Global tracking transformers.  In CVPR, 2022.


††thanks: corresponding author<EMAIL_ADDRESS>
# Superconductivity in hyperdoped Ge by molecular beam epitaxy

Patrick J. Strohbeen  Aurelia M. Brook  Center for Quantum Information
Physics, Department of Physics, New York University, New York, NY 10003 USA
Wendy L. Sarney  Army Research Directorate, DEVCOM Army Research Laboratory,
Adelphi, MD 20783 USA  Javad Shabani  Center for Quantum Information Physics,
Department of Physics, New York University, New York, NY 10003 USA

###### Abstract

Superconducting germanium films are an intriguing material for possible
applications in fields such as cryogenic electronics and quantum bits.
Recently, there has been great deal of progress in hyperdoping of Ga doped Ge
using ion implantation. The thin film growths would be advantageous allowing
homoepitaxy of doped and undoped Ge films opening possibilities for vertical
Josephson junctions. Here, we present our studies on the growth of one layer
of hyperdoped superconducting germanium thin film via molecular beam epitaxy.
We observe a fragile superconducting phase which is extremely sensitive to
processing conditions and can easily phase-segregate, forming a percolated
network of pure gallium metal. By suppressing phase segregation through
temperature control we find a superconducting phase that is unique and appears
coherent to the underlying Ge substrate.

††preprint: AIP/123-QED

High-quality thin films of covalent superconducting material is enticing for
pushing towards up-scaling current state-of-the-art quantum devices. Proposals
calling for highly scalable device platforms such as cryogenic CMOS circuitry
Patra  _et al._ (2018) or qubit architectures Shim and Tahan (2016); Zhao  _et
al._ (2020) have emerged as a new challenge in finding materials such as Si or
Ge for devices with superior performance at cryogenic temperatures Shim and
Tahan (2014). Additionally, covalent superconductors offer a unique aspect in
that high-quality fully homoepitaxial Josephson junctions can be created in
the growth direction Zhao  _et al._ (2020); McRae  _et al._ (2021). Thus,
drawing inspiration from the original superconducting qubits Nakamura  _et
al._ (1999); Vion  _et al._ (2002) one can begin to consider qubit designs
that make use of the out-of-plane degree of freedom such as the recently
studied merged-element transmon, or “Mergemon” Zhao  _et al._ (2020). Such
designs have already shown through preliminary evaluations the impact of the
superconductor-semiconductor interface on the observed information coherence
times Zhao  _et al._ (2020); McRae  _et al._ (2021). It is projected that high
quality epitaxial interfaces will solve this problem, putting forth a strong
proposal for new material and device applications. Thus, further efforts must
be made into developing highly compatible superconducting materials as qubit
density requirements become more stringent.

Superconductivity in group IV semiconductors has been presented in diamond
Ekimov  _et al._ (2004), silicon Bustarret  _et al._ (2006); Sardashti  _et
al._ (2021a), and germanium Sardashti  _et al._ (2021b); Fiedler  _et al._
(2012); Herrmannsdörfer  _et al._ (2009) previously, though these studies use
high energy synthesis techniques (e.g. ion implantation) and many questions
remain to be investigated. Particularly, when hyperdoping Ge with high energy
Ga ion beams, the origin of superconductivity has remained controversial due
to high energies driving possible phase segregationSardashti  _et al._
(2021b); Skrotzki  _et al._ (2011); Fiedler  _et al._ (2012); Herrmannsdörfer
_et al._ (2009); Prucnal  _et al._ (2019). For example, typically a thick
silicon oxide layer is first deposited to protect the surface during the
implantation process. Sardashti  _et al._ (2021b, a); Fiedler  _et al._
(2012)After implantation and subsequent flash annealing at high temperature to
activate the Ga acceptors, superconductivity is attained. However, if the
protective oxide layer is etched away, the superconducting state
disappears.Sardashti  _et al._ (2021b) This has resulted in the hypothesis
that the Ga-metal is diffusing to the SiOx/Ge interface and coalescing into a
percolated network of metallic Ga, which is in itself an elemental BCS
superconductor even in a confined state Charnaya  _et al._ (1998, 2009).

Here we present the first report of molecular beam epitaxial (MBE) growth of
superconducting hyperdoped Ga:Ge thin films. We show that under the
appropriate growth conditions Ga-segregation is suppressed and a unique
superconducting phase is obtained. We find distinctive signatures between
superconductivity that is originated from Ga-metal as well as dispersive
hyperdoping, schematically shown in Figure 1, through a combined electrical
and structural analysis.

Figure 1: a.) Schematic of growth method A. We first co-deposit Ga and Ge
species followed by a Si cap that we oxidize in the load lock. We then re-
introduce the wafer into UHV and anneal at 700C. b.) Schematic of growth
method B. Here, we co-deposit Ga and Ge species in two steps, separated by a
thin layer of Si which we wish to alloy into the film. Lastly we cap with Si
before removing from vacuum. No annealing is done with method B.

The materials presented here were all grown in a custom Varian Gen II MBE
chamber on 2" un-doped Ge (001) wafers. Prior to growth, wafers are etched ex-
situ in DI-water at 90∘C and then immediately loaded into vacuum. The wafers
were then outgassed in-situ at 400∘C for 30 minutes and then finally flash
annealed at 650∘C for 5 minutes before cooling to the growth temperature. Pure
Germanium and silicon are deposited via Thermionics HM2 e-Gun operating with a
10 kV acceleration voltage. Gallium doping is done with a standard knudsen
cell source (MBE Komponenten) at a source temperature of 950∘C, or an
approximate flux at the source opening of $\sim$1.5$\times$1016/cm2s as
estimated by the kinetic theory of gases. The Ge source is operated at a 10kV
acceleration voltage at 0.05A emission current while Si is evaporated with
0.01A emission current. Substrates are mounted in indium-free bayonet style
holders and the reported substrate temperatures are measured with a
thermocouple in close proximity to the backside of the wafer. The substrate is
rotated at a constant 10 rpm throughout film growth to promote uniformity.

In this study, we discuss two different sample growths and processing methods
to help illuminate the Ga dopant atom behavior. The first method A seen in
Figure 1a uses room temperature co-deposition of Ga and Ge onto the substrate
followed by a capping layer of silicon under UHV. We oxidize the capping layer
in the chamber load lock. The sample is then re-introduced to the growth
chamber and flash annealed at 700∘ and cooled back down, emulating the
processing conditions of ion implantation samples Sardashti  _et al._ (2021b);
Herrmannsdörfer  _et al._ (2009); Fiedler  _et al._ (2012). The second method
B, schematically pictured in Figure 1b uses room temperature co-deposition of
Ga and Ge, however halfway through the growth of the superconducting layer,
growth is paused and a thin 1nm thick layer of silicon is included before
growth of Ga and Ge is continued. Deposition of Si at an intermediate stage of
growth introduces a small amount of Si that alloys into the hyperdoped region,
which is expected to increase the average phonon frequency, $\omega_{ln}$. As
follows from the McMillan equation Blase  _et al._ (2009); Carbotte (1990)
seen in Equation 1, we expect this to then give rise to an increased $T_{c}$,
or superconducting transition temperature. We then cap the sample with an
additional layer of silicon after completing the deposition of hyperdoped Ge.
No post-annealing is performed for samples grown using method B.

$T_{c}=\frac{h\omega_{ln}}{1.2k_{B}}Exp\left[\frac{-1.04(1+\lambda_{ep})}{\lambda_{ep}-\mu^{*}(1+0.62\lambda_{ep})}\right]$
(1)

Where $\lambda_{ep}$ is the electron-phonon coupling parameter and $\mu^{*}$
is the screened retarded Coulomb repulsion parameter. Growth method B attempts
to utilize the McMillan formula Blase  _et al._ (2009) in Eq. 1 which suggests
alloying with silicon to increase the average phonon frequency of the system
($\omega_{ln}$), thus enhancing the observed $T_{c}$ for a given density of
states. Additionally, an increase in $\omega_{ln}$ is further expected to also
increase the electron-phonon coupling potential, $V_{ep}$. By the relationship
$V_{ep}=\lambda_{ep}/N(E_{F})$, where $N(E_{F})$ refers to the density of
states at the Fermi level, we see the electron-phonon coupling parameter
($\lambda_{ep}$) increases with the density of states. Both of these factors
could contribute to an enhanced $T_{c}$ Carbotte (1990).

The crystallinity, compositional distribution, and film morphology of the
films grown in this study were examined with scanning transmission electron
microscopy (STEM) in a JEOL ARM200F, equipped with a spherical aberration
corrector for probe mode, and operated at 200 keV. The samples were prepared
with cross-sectional tripod polishing to 20 $\mu$m thickness, followed by
shallow angle Ar+ ion milling with low beam energies([1]3 keV), and LN2 stage
cooling in a PIPS II ion mill. Cross-sectional STEM images of a sample grown
using Method A are seen in Figure 2a-b. Rapid annealing films under vacuum
induces an extreme segregation of gallium metal out of the Ge matrix to form
amorphous droplets on the surface of the substrate/film. These droplets
selectively form in the near surface region and have a faint shadowing feature
at the Ga-Ge interface which we attribute to the heavy Ga-content in that
region. The droplets on average are hundreds of nm in diameter, with a similar
spacing separating the droplets from one another. The droplets are not
isotropic and extend further into the substrate than they protrude from the
surface.

Figure 2: Bright field (BF) cross-sectional STEM images of a Ga-segregated Ge
film. (a) Long-range scan showing significant segregation and agglomeration of
Ga-droplets. (b) Zoom-in on region indicated in (a) showing the triple
interface of epoxy(vacuum), Ga-droplet, and underlying Ge. (c) EDS map of
region denoted by blue box in (a) of the interface between a droplet and the
substrate. (d) EDS map of the region highlighted in the orange box in (a) of
the region between Ga-rich droplets.

Zooming into the interface between the droplet and the underlying Ge, we
confirm the amorphous nature of the Ga metal, as seen in Figure 2b. This
distinctly contrasts the crystalline nature of the underlying Ge. In this
scenario, the Ga-rich regions exhibit significant out-diffusion of Ga metal
which we speculate is likely due to the high temperature anneal used to
activate the Ga-dopants. We further confirm this compositional shift through
EDS maps, presented in Figure 2c-d, of both a droplet region (blue) and the
region between droplets (orange). Figure 2c is zoomed in on the interface at
the bottom of a droplet and the underlying Ge, depicted in Fig. 2a as the
small blue box. We see a $\sim$5 nm thick Ga-rich Ge region before the
composition becomes nearly pure Ga. Between droplets, as seen in the area
highlighted in the orange box in Fig. 2a, we instead see a very thin Ga-rich
region on the order of few nanometers thick that serve as metallic
interconnects between the Ga droplets.

To suppress this Ga metal segregation, in our method B we deposit the heavily
doped layers at room temperature, however this time we do not anneal at
elevated temperatures. Removing the anneal step allows for maintaining a
nominally consistent film composition and eliminate the observed Ga metal
precipitates. Such behavior is pictured in Figure 3 in which cross-sectional
STEM images for a sample grown using Method B present a fully connected film
with an abrupt film/substrate interface.

Figure 3: (a) High resolution TEM image to show the general film structure of
a crystalline, hyperdoped Ge film. (b) Zoom-in on region indicated in (a) to
present the epitaxial interface between the hyperdoped region and the
underlying Ge buffer. (c) EDS elemental mapping of a zoomed out region showing
elemental composition of the film and hillocks. Yellow pixels are Si, red is
Ge, and cyan is Ga.

In the second method in which we utilize no post-anneal for carrier
activation, Fig. 3 shows a nominally sharp interface between the heavily-doped
region and the undoped germanium. Fig. 3a shows a fully complete film coverage
with no obvious signatures of significant Ga segregation, albeit the film is
quite rough. We attribute film roughness to the difference in surface energy
between Si and Ge giving rise to a 3D island growth mechanism of Si on Ge
during the Si cap growth. This behavior of the Si-Ge system has been
previously observed and reported on Tsu  _et al._ (1994); Kawabata  _et al._
(1989); Copel  _et al._ (1989). Furthermore, as we zoom in on the interface,
we observe no obvious discontinuities in the atomic columns suggesting the Ge
lattice is maintained as Ga has been incorporated.

Fig. 3c shows the EDS elemental mapping of a zoomed out region showing
elemental composition of the film. The interface between Ge and doped Ge is
marked. The effective thickness of the doped Ge film is 5 nm before hillocks.
The map also shows that hilllocks are predominantly Ga-doped Si while in some
places Ge is present. The continuous atomic registry is highly promising for
continued development of coherent superconducting germanium thin films.

Electrical measurements are conducted in an Oxford Triton pulse-tube dilution
refrigerator with a base temperature of 15 mK and magnetic field capabilities
up to 14T. Measurements are collected using a standard Van der Pauw wiring
configuration on square pieces from near the center of each wafer. On-chip
contacts are annealed In-Sn eutectic at each of the four corners.

Transport measurements for both methods are presented in Figure 4. In our Ga-
segregated films, grown via method A, the observed critical temperature and of
0.89 K at zero field, and critical field values of $B_{c}^{\perp}\sim
0.05~{}T$, and $B_{c}^{\parallel}\sim 310mT$ at T = 15 mK, are highly
suggestive of a Ga-metal origin. The transitions are marked by the 10% value
of the normal resistance. Reported literature values Eisenstein (1954) for Ga
of 1.1K and 0.05T, respectively, agree well with our measurements. The
observed reduction in critical temperature is most likely due to the Ga-
droplets behaving as a weakly connected superconductor such that discrete
“puddles” of superconducting Ga metal host the parent superconducting phase,
but due to the low density we do not observe exactly the same Tc as bulk Ga.
The enhanced in-plane field of 0.31 T we attribute to the thin film nature of
the superconducting film Tedrow and Meservey (1973).

Looking closer at the sheet resistance as a function of temperature, many
kinks are observed between the range of $\sim$7K and the total superconducting
transition at 0.7K. We attribute the high temperature kink to the formation of
Ga metal crystalline polymorphs as a result of the anneal, all of which have
been shown previously to exhibit a superconducting transition temperature of
6K or greater Charnaya  _et al._ (1998, 2009); Moura  _et al._ (2017); Teske
and Drumheller (1999). The other features that are presented at more moderate
temperatures $\sim$2-4K are more difficult to confidently assign to Ga-related
phases, but could be due to either percolated Ga metal networks or sparse
regions of hyperdoped Ge matrix.

Figure 4: a.) Sheet resistance vs. temperature (black) and magnetic field
(red, at 15mK) of a sample grown using Method A. Critical transition
temperature is near 0.7K, with a critical out-of-plane magnetic field of
roughly 50mT and critical in-plane magnetic field of $\sim$310mT. b.) Sheet
resistance vs temperature (black) and magnetic field (red, at 15mK) of a
sample grown using Method B. Sample B exhibits a transition temperature of
near 5.3K, with an enhanced critical out-of-plane field of 430mT and critial
in-plane field of $\sim$1.03T.

Fig. 4b shows a different trend for samples grown using Method B. Here, we
observe significantly enhanced superconducting properties compared to that of
the Ga-segregated films with $T_{C}=5.27K$, $B_{c}^{\perp}=0.43T$, and
$B_{c}^{\parallel}=1.03T$. Here, in the sheet resistance versus temperature we
observe a much sharper transition that starts $\sim$6.9K and reaches the zero
resistance state at $\sim$5K. Previous reports of superconductivity in Ga
metal polymorphs have observed superconductivity at 6.9K in $\epsilon$-Ge and
$\beta$-Ga at 6K Teske and Drumheller (1999); Moura  _et al._ (2017).
Unfortunately, magnetic field behavior for many of these crystal polymorphs of
Ga is not reported in literature, however reported values for nanoconfined
particles of $\beta$-Ga are on the order of $\sim$430 Oe Moura  _et al._
(2017). While longer-extent TEM images do show a single small $\beta$-Ga
nanoparticle present in the FIB slice, the observed out-of-plane critical
magnetic field exhibited by our hyperdoped Ge film is significantly larger
than any reported values for crystalline phases of Ga. Thus, we are confident
in ruling out competing Ga phases as the origin of the observed
superconductivity due to their sparsity in our film. If the superconductivity
observed here were a result of sparse interconnected grains of some
alternative phase of Ga, we would expect a much larger transitional region,
similar to what we observe for the sample grown via method A and as has been
reported previously in studies on Nb islands Eley  _et al._ (2011).

To help understand the superconducting phase in the sample grown using method
B, we conduct Hall measurements, above $B_{c}$, in the Van der Pauw geometry
and compare against the theorized quantities for an electronic origin. We
measure a 2D hole concentration of roughly 2.86$\times$1012/cm2 for the
specific sample presented in this work, or a density of
$\sim$1$\times$1019/$cm^{3}$ carriers. Comparing to previous reports of
covalent superconductivity in this and related systems Blase  _et al._ (2009);
Bustarret  _et al._ (2006); Ekimov  _et al._ (2004); Herrmannsdörfer  _et al._
(2009), we see that not only is the transition temperature observed in this
study significantly higher than is predicted for the Ge system ($\sim$100s mK)
Cohen (1964a, b), but we also measure drastically lower carrier densities.

In conclusion, we have presented here our report of superconductivity in Ga-
doped Germanium achieved via molecular beam epitaxy. The superconducting phase
is sensitive to processing conditions and we observe that high temperature
flash anneals strongly promote Ga-segregation in the matrix. Furthermore, we
find that the MBE deposition process enables carrier activation even without a
post-anneal in which superconductivity is attained. The hyperdoped film we
presented here exhibits a critical temperature of $T_{C}=5.27K$ and critical
field values of $B_{C}^{\perp}=0.43T,B_{C}^{\parallel}=1.03T$ at 15 mK. We
lastly found that the interface between un-doped Ge and the hyperdoped Ge
layer remains highly coherent after growth. Therefore the results shown here
present MBE-growth of superconducting group IV material as a highly promising
approach for the integration of superconducting Ge films, though much more
work is required to fully realize full group IV superconducting circuitry.

## Author Declarations

### Conflict of Interest

The authors have no conflicts to disclose.

## Author’s Contributions

Patrick J. Strohbeen:  Conceptualization (equal); Data Curation (lead); Formal
Analysis (lead); Investigation (lead); Methodology (equal); Project
Administration (supporting); Software (supporting); Supervision (equal);
Validation (equal); Visualization (lead); Writing - Original Draft (lead);
Writing - Review & Editing (equal); Aurelia M. Brook:  Formal Analysis
(supporting); Investigation (supporting); Software (lead); Visualization
(supporting); Writing - Review & Editing (equal); Wendy L. Sarney:
Investigation (supporting); Writing - Review & Editing (supporting); Javad
Shabani:  Conceptualization (equal); Funding Acquisition (lead); Methodology
(equal); Project Administration (lead); Supervision (equal); Validation
(equal); Writing - Review & Editing (equal);

###### Acknowledgements.

This project was supported by AFOSR award FA9550-21-1-0338.

## Data Availability Statement

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## References

  * Patra  _et al._ (2018) B. Patra, R. M. Incandela, J. P. G. van Dijk, H. A. R. Homulle, L. Song, M. Shahmohammadi, R. B. Staszewski, A. Vladimirescu, M. Babaie, F. Sebastiano, and E. Charbon, IEEE J. Solid-State Circuits 53, 309 (2018).
  * Shim and Tahan (2016) Y.-P. Shim and C. Tahan, “Semiconductor-inspired design principles for superconducting quantum computing,” (2016).
  * Zhao  _et al._ (2020) R. Zhao, S. Park, T. Zhao, M. Bal, C. McRae, J. Long, and D. P. Pappas, Phys. Rev. Applied 14, 064006 (2020).
  * Shim and Tahan (2014) Y.-P. Shim and C. Tahan, Nat. Commun. 5, 4225 (2014).
  * McRae  _et al._ (2021) C. R. H. McRae, A. McFadden, R. Zhao, H. Wang, J. L. Long, T. Zhao, S. Park, M. Bal, C. J. Palmstrøm, and D. P. Pappas, J. Appl. Phys. 129, 025109 (2021).
  * Nakamura  _et al._ (1999) Y. Nakamura, Y. A. Pashkin, and J. S. Tsai, Nature 398, 786 (1999).
  * Vion  _et al._ (2002) D. Vion, A. Aassime, A. Cottet, P. Joyez, H. Pothier, C. Urbina, D. Esteve, and M. H. Devoret, Science 296, 886 (2002).
  * Ekimov  _et al._ (2004) E. A. Ekimov, V. A. Sidorov, E. D. Bauer, N. N. Mel’nik, N. J. Curro, J. D. Thompson, and S. M. Stishov, Nature 428, 542 (2004).
  * Bustarret  _et al._ (2006) E. Bustarret, C. Marcenat, P. Achatz, J. Kačmarčik, F. Lévy, A. Huxley, L. Ortéga, E. Bourgeois, X. Blase, D. Débarre, and J. Boulmer, Nature 444, 465 (2006).
  * Sardashti  _et al._ (2021a) K. Sardashti, T. Nguyen, M. Hatefipour, W. L. Sarney, J. Yuan, W. Mayer, K. Kisslinger, and J. Shabani, Appl. Phys. Lett. 118, 073102 (2021a).
  * Sardashti  _et al._ (2021b) K. Sardashti, T. D. Nguyen, W. L. Sarney, A. C. Leff, M. Hatefipour, M. C. Dartiailh, J. Yuan, W. Mayer, and J. Shabani, Phys. Rev. Mater. 5, 064802 (2021b).
  * Fiedler  _et al._ (2012) J. Fiedler, V. Heera, R. Skrotzki, T. Herrmannsdörfer, M. Voelskow, A. Mücklich, S. Facsko, H. Reuther, M. Perego, K.-H. Heinig, B. Schmidt, W. Skorupa, G. Gobsch, and M. Helm, Phys. Rev. B 85, 134530 (2012).
  * Herrmannsdörfer  _et al._ (2009) T. Herrmannsdörfer, V. Heera, O. Ignatchik, M. Uhlarz, A. Mücklich, M. Posselt, H. Reuther, B. Schmidt, K.-H. Heinig, W. Skorupa, M. Voelskow, C. Wündisch, R. Skrotzki, M. Helm, and J. Wosnitza, Phys. Rev. Lett. 102, 217003 (2009).
  * Skrotzki  _et al._ (2011) R. Skrotzki, T. Herrmannsdörfer, V. Heera, J. Fiedler, A. Mücklich, M. Helm, and J. Wosnitza, Low Temp. Phys. 37, 877 (2011).
  * Prucnal  _et al._ (2019) S. Prucnal, V. Heera, R. Hübner, M. Wang, G. P. Mazur, M. J. Grzybowski, X. Qin, Y. Yuan, M. Voelskow, W. Skorupa, L. Rebohle, M. Helm, M. Sawicki, and S. Zhou, Phys. Rev. Materials 3, 054802 (2019).
  * Charnaya  _et al._ (1998) E. V. Charnaya, C. Tien, K. J. Lin, and C. S. Wur, Phys. Rev. B 58, 467 (1998).
  * Charnaya  _et al._ (2009) E. V. Charnaya, C. Tien, M. K. Lee, and Y. A. Kumzerov, J. Phys.: Condens. Matter 21, 455304 (2009).
  * Blase  _et al._ (2009) X. Blase, E. Bustarret, C. Chapelier, T. Klein, and C. Marcenat, Nat. Mater. 8, 375 (2009).
  * Carbotte (1990) J. P. Carbotte, Rev. Mod. Phys. 62, 1027 (1990).
  * Tsu  _et al._ (1994) R. Tsu, H. Z. Xiao, Y.-W. Kim, M.-A. Hasa, H. K. Birnbaum, J. E. Greene, D.-S. Lin, and T.-C. Chiang, J. Appl. Phys. 75, 240 (1994).
  * Kawabata  _et al._ (1989) H. Kawabata, H. Ueba, and C. Tatsuyama, J. Appl. Phys. 66, 634 (1989).
  * Copel  _et al._ (1989) M. Copel, M. C. Reuter, E. Kaxiras, and R. M. Tromp, Phys. Rev. Lett. 63, 632 (1989).
  * Eisenstein (1954) J. Eisenstein, Rev. Mod. Phys. 26, 277 (1954).
  * Tedrow and Meservey (1973) P. M. Tedrow and R. Meservey, Phys. Rev. B 8, 5098 (1973).
  * Moura  _et al._ (2017) K. O. Moura, K. R. Pirota, F. Béron, C. B. R. Jesus, P. F. S. Rosa, D. Tobia, P. G. Pagliuso, and O. F. de Lima, Sci. Rep. 7, 15306 (2017).
  * Teske and Drumheller (1999) D. Teske and J. E. Drumheller, J. Phys.: Condens. Matter 11, 4935 (1999).
  * Eley  _et al._ (2011) S. Eley, S. Gopalakrishnan, P. M. Goldbart, and N. Mason, Nat. Phys 8, 59 (2011).
  * Cohen (1964a) M. L. Cohen, Phys. Rev. 134, A511 (1964a).
  * Cohen (1964b) M. L. Cohen, Rev. Mod. Phys. 36, 240 (1964b).