# Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with
Pose Annotations

Adel Ahmadyan  Liangkai Zhang  Jianing Wei  Artsiom Ablavatski  Matthias
Grundmann
{ahmadyan, liangkai, jianingwei, artsiom<EMAIL_ADDRESS>
###### Abstract

3D object detection has recently become popular due to many applications in
robotics, augmented reality, autonomy, and image retrieval. We introduce the
Objectron dataset to advance the state of the art in 3D object detection and
foster new research and applications, such as 3D object tracking, view
synthesis, and improved 3D shape representation. The dataset contains object-
centric short videos with pose annotations for nine categories and includes 4
million annotated images in $14,819$ annotated videos. We also propose a new
evaluation metric, 3D Intersection over Union, for 3D object detection. We
demonstrate the usefulness of our dataset in 3D object detection tasks by
providing baseline models trained on this dataset. Our dataset and evaluation
source code are available online at http://www.objectron.dev.

Figure 1: Example videos in our dataset. Each sample is a video annotated with
the object’s 3D bounding box.

##  1 Introduction

Figure 2: Our dataset consists of object-centric videos, which capture
different views of the same objects from different angles.

The state of the art in machine learning has achieved exceptional accuracy on
many computer vision tasks solely by training models on images. Building upon
these successes and advancing 3D object understanding has great potential to
power a wider range of applications, such as augmented reality, robotics,
autonomy, and image retrieval. Yet, understanding objects in 3D remains a
challenging task due to the lack of large real-world datasets compared to 2D
tasks (e.g., ImageNet [7], COCO [19], and Open Images [17]). To empower the
research community for continued advancement in 3D object understanding, there
is a strong need for the release of object-centric video datasets, which
capture more of the 3D structure of an object, while matching the data format
used for many down-stream vision tasks (i.e., video or camera streams), to aid
in the training and benchmarking of machine learning models. The object-
centric approach is consistent with how our brains perceive new objects too.
For example, when a child wants to learn the shape of a chair, they’ll walk
around and look at the chair from different angles to pick up information. In
other words, "We must also move in order to perceive" [11][31].

We present the Objectron dataset, a collection of short, object-centric video
clips capturing a larger set of common objects from different angles. Each
video clip is accompanied by augmented reality (AR) session metadata that
includes camera poses, sparse point-clouds, and surface planes. The data also
contains manually annotated 3D bounding boxes for each object, which describe
the object’s position, orientation, and dimensions. The dataset consists of
$14,819$ annotated video clips and $4M$ annotated images collected from a geo-
diverse sample (covering ten countries across five continents). Figure 2 shows
an example video in our dataset.

There are several advantages of using Objectron dataset over existing works:

  * •

Videos contain multiple views of the same object, enabling many applications
well beyond 3D object detection. This includes multi-view geometric
understanding, view synthesis, 3D shape reconstruction, etc.

  * •

The 3D bounding box is present in the entire video and is temporally
consistent, thus enabling 3D tracking applications.

  * •

Our dataset is collected in the wild to provide better generalization for
real-world scenarios in contrast to datasets that are collected in a
controlled environment [12][4].

  * •

Each instance’s translation and size are stored in metric scale, thanks to
accurate on-device AR tracking and provides sparse point clouds in 3D,
enabling sparse depth estimation techniques. The images are calibrated and the
camera parameters are provided, enabling the recovery of the object’s true
scale.

  * •

Our annotations are dense and continuous, unlike some of the previous work
[24] where viewpoints have been discretized to fit into bins.

  * •

Each object category contains hundreds of instances, collected from different
locations across different countries in different lighting conditions.

We have conducted experiments for 3D object detection tasks using our dataset.
We proposed a novel method to compute the precise 3D IoU of oriented 3D
bounding boxes. These experiments can be used as baselines for future
research.

##  2 Previous Work

In this section, we review several commonly used datasets for 3D object
detection and compare them with Objectron.

BOP challenge [14] consists of a set of benchmark for 3D object detection and
combines many of these smaller datasets into a larger one. Most of the images
in the dataset are taken in a very controlled environment and feature clutter
and heavy occlusion that apppear in industrialized setups. T-LESS [13]
features industrialized objects that lack texture or color. Rutgers APC [22]
contains 14 textured objects from the Amazon picking challenge. LineMOD [12]
is the most commonly used dataset for object pose estimation. Similar to
LineMOD, IC-BIN dataset [8] adds a few more categories. YCB [4] dataset
contains videos of objects and their poses in a controlled environment. In
comparison to these datasets, Objectron has a larger scale and contains high-
resolution videos of common objects in the wild.

ObjectNet3D [32] is dataset that contains 3D object poses from images.
Similarly, Pascal3D+ [33] adds 3D pose annotations to the Pascal VOC and a few
images from the ImageNet dataset. In comparison to Objectron, these datasets
contain more categories and instances but they only include images.
Furthermore, the objects are annotated with a 2D-to-3D alignment process, so
their pose is annotated up to 6-DoF (instead of nine) and camera intrinsics
are not available. So it is not possible to recover the object’s scale from
these datasets. Pix3D [26] contains pixel-level 2D-3D pose alignment.
Similarly, 3DObject[24] provides discretized viewpoint annotations for 10
everyday object categories.

Another type of dataset used for 3D object detection tasks is scene datasets.
In these datasets, a video of a scene is recorded with an RGBD camera or
LIDAR. ScanNet [6] is a large scale video dataset of indoor scenes with
semantic annotations. The dataset contains over 1500 scenes recorded in RGBD
videos. The dataset does not contain 3D pose information, however, Scan2CAD
[3] annotates the original scans with ShapeNetCore [5] models to label each
object’s pose. Rio [29] is another dataset that contains indoor scans
annotated with an object’s 3D pose. In comparison Objectron videos are object-
centric, and we have an order of magnitude more samples in our dataset.

Another approach is to use synthetic data. ShapeNet [5] includes CAD models
for many objects and has been widely used [28][25], However, the sample’s
visual quality is limited and it does not generalize well to real-world
applications. Synthetic datasets offer valuable data for training and
benchmarking, but the ability to generalize to the real-world is unknown.
HyperSim[23] generates photo-realistic scenes with object pose annotation, but
reproducibility is challenging since they have not released any of the
dataset’s assets.

##  3 Data Collection and Annotation

###  3.1 Object Categories

In Objectron dataset, the aim was to select meaningful categories of common
objects that form a representative set of all categories that are practically
relevant and technically challenging. Our goal was to capture these objects in
their common environment, and in relative context whether it would be in
store, indoor or outdoor environment. We also included objects of various
sizes, ranging from a few centimeters (e.g. cups) to as large as chairs and
bikes.

The object categories in the dataset contain both rigid, and non-rigid
objects. We included non-rigid categories such as bikes and laptops
specifically since we expect techniques using CAD models or strong priors will
face challenges estimating the pose of these object categories. We should
mention non-rigid objects remain stationary during the period of each video.

Many 3D object detection models are known to exhibit difficulties in
estimating rotations of symmetric objects [18]. Symmetric objects have
ambiguity in their one, two, or even three degrees of rotation. Therefore we
added categories like "cups" and "bottles" specifically to test it.

It has been shown that vision models pay special attention to texts in the
images [10]. Re-producing texts and labels correctly are important in
generative models too. Therefore we added categories of objects with very
distinct texts in their labels such as "books" and "cereal boxes". Since our
dataset is collected from a geo-diverse set of countries, multiple different
languages are present in the videos as well. We should report these
categories, despite having relatively simple box-shaped geometry, have very
different texture patterns. So our baseline experiments have difficulty
estimating their poses accurately.

Since we strive for real-time perception we included a few categories (shoes
and chairs) that enable exciting applications, such as augmented reality and
image retrieval.

###  3.2 Data Collection

Our data collection consists of the video recording when the camera moves
around the object and looks at it from different angles. We also capture
camera poses, point-clouds, and surface planes through an AR session (e.g.
ARKit [2] or ARCore [1]). AR solutions track a set of features through the
video and estimate their 3D coordinates on-device in real-time. Meanwhile,
they estimate camera poses using bundle-adjustment and filtering. We emphasize
all the translation and scale reported in our dataset are on the metric scale.
We released both of these files (the video recording and the AR session
metadata) in the dataset. We assume the standard pinhole camera model, and
provide calibration, extrinsics and intrinsics matrix for every frame in the
dataset.

All the videos are recorded in $1440\times 1920$ resolution at $30fps$ using
the back-camera of high-end phones. We only enabled data collection on a very
few phone models (<5) to make sure the output quality remains consistent
across our dataset. To avoid drift in the AR session, we kept the video length
short at around $10sec$. The collectors are instructed to avoid rapid
movements to avoid blurry images. Finally, the object remains stationary in
all of our videos.

By using mobile phones for data collection we were able to quickly launch a
data collection campaign across many different countries. The dataset is geo-
diverse and covers 10 countries over five continents. Figure 3 shows which
countries are represented in the dataset. The samples are distributed
uniformly in each region.

Figure 3: Countries where we collected data from.

###  3.3 Data Annotation

Efficient and accurate data annotation is the key to building large-scale
datasets. Annotating 3D bounding boxes for each image is time-consuming and
expensive. Instead, we annotate 3D objects in a video clip and populate them
to all frames in the clip, scaling up the annotation process, and reducing the
per image annotation cost. The user interfaces for the annotation tool is
shown in Figure 4(a). The input to our annotation tool is a video sequence of
stationary objects. It covers different viewing angles of objects.

Next, we show the 3D world map to the annotator side-by-side with the images
from the video sequence (Figure 4(a)). The annotator draws a 3D bounding box
in the 3D world map, and our tool projects the 3D bounding box over all the
frames given pre-computed camera poses from the AR sessions (such as ARKit or
ARCore). The annotator looks at the projected bounding box and makes necessary
adjustments (position, orientation, and the scale of a 3D bounding box) so the
projected bounding box looks consistent across different frames. At the end of
the process, the user saves the 3D bounding box and the annotation.

(a) Our annotation tool

Figure 4: Data annotation. The annotated 3D box is verified at multiple views,
and then populated to all images in the sequence.

The benefits of our approach are 1) by annotating a video once, we get
annotated images for all frames in the video sequence; 2) by using AR, we can
get accurate metric sizes for bounding boxes.

###  3.4 Annotation Variance

The accuracy of our annotation hinges on two factors: 1) the amount of drift
in the estimated camera pose throughout the captured video, and 2) the
accuracy of the raters annotating the 3D bounding box. We empirically observed
our videos drift $<2\%$ in length. To further reduce the drift, we usually
capture shorter sequences below $10sec$. Figure 5 shows the distribution of
the length of the videos in our dataset. To evaluate the accuracy of the
rater, we asked eight annotators to re-annotate same sequences. A few samples
are shown in Figure 6. Overall for the chairs, the standard deviation for the
chair orientation, translation, and scale was $4.6\degree$, $1cm$, and $4cm$,
respectively which demonstrates insignificant variance of the annotation
results between different raters.

Figure 5: Distribution of the video length in our dataset. Majority of the
videos are 10 seconds long (300 frames) and the longest video is 2022 frames
long.

Figure 6: The overlay of the 3D bounding boxes annotated by different
annotators shows the annotations from different raters are very close.

##  4 Objectron Dataset

Figure 7: View-point distribution of samples per object category. The top row shows the azimuth distribution in polar graph, and the bottom row denotes the elevation distribution. class | bike | book | bottle | camera | cereal_box | chair | cup | laptop | shoe
---|---|---|---|---|---|---|---|---|---
No. videos | 476 | 2024 | 1928 | 815 | 1609 | 1943 | 2204 | 1473 | 2116
No. frames | 150k | 576k | 476k | 233k | 396k | 488k | 546k | 485k | 557k
No. instances | 487 | 2084 | 2014 | 931 | 1638 | 2362 | 2378 | 1710 | 3500
Avg instance per video | 1.03 | 1.02 | 1.04 | 1.14 | 1.01 | 1.21 | 1.07 | 1.16 | 1.65
Table 1: Per-category statistics of Objectron dataset.

In this section, we describe the details of our Objectron dataset and provide
some statistics. The dataset contains videos in 9 categories: bikes, books,
bottles, cameras, cereal boxes, chairs, cups, laptops, and shoes. Some of
these objects are non-rigid (e.g. bikes or laptops) and all of them remain
stationary during the video recording. In each video, the camera moves around
the object, capturing it from different angles. In total there are $17,095$
object instances that appear in $4M$ annotated images from $14819$ annotated
videos (not counting the unreleased evaluation set for future competitions).
The dataset is collected from ten different countries. This is important for
categories that contain texts and labels, such as books, cereal boxes and
bottles. So our samples contain different languages as well as different local
environments. Each category is divided to train and test sets. Table 1 shows
detailed per-category statistics of our dataset.

Each sample contains the high-resolution image, along with the camera pose,
point-cloud (from tracking), and planar surfaces in the environment. The data
also contains manually annotated 3D bounding boxes for each object, which
describe the object’s orientation, translation, and size relative to the
camera pose. Our coordinate system follows a left-hand rule, where $+y$ axis
is up. From the pose, we also compute the object’s bounding box 3D keypoints,
projected 2D keypoints, as well as azimuth and elevation. Furthermore, each
frame in the video contains the camera pose, and the camera’s projection and
view matrices.

The annotation of the object contains its rotation, translation w.r.t. the
camera center as well as the object’s scale. To get a better understanding of
the viewpoint distribution, we computed the azimuth of each object instance
w.r.t the camera center. Here azimuth 0 degree indicates the object is being
viewed from the front. Figure 7-top shows the azimuth distribution for
different object categories in our dataset. For some categories, there is a
specific bias toward the front and top views.

##  5 Baseline Experiments and Evaluations

Our dataset is released with an evaluation code to assess the performance of
3D object detection algorithms using various metrics. The evaluation code
computes the average precision for multiple 2D and 3D metrics such as 3D IoU,
2D projection error, view-point error, polar and azimuth error, and rotation
error.

Except for the 3D IoU, the other metrics are fairly standard and we refer the
readers to our code as well as other references for further details. In this
section, we explain our implementation of 3D IoU metric, which to the best of
our knowledge, is novel.

###  5.1 3D Intersection Over Union

Intersection over Union (IoU), also known as Jaccard index, is an evaluation
metric for computer vision tasks such as object detection, segmentation, and
tracking. It measures how close the prediction bounding boxes are to the
ground truth. The IoU takes two boxes as input: the predicted box (by the
model) and the ground truth box and computes the intersection volume of the
two boxes. The IoU metric is invariant under change in the scale, as well as
any rigid transformations that belong to the SE3 group. Therefore if we change
the coordinate system, or rotate, translate, and scale both boxes by the same
transformation, their IoU does not change. This property becomes useful later.
The IoU is a normalized metric and the it’s value ranges between 0 to 1, where
1 is considered the perfect score.

For evaluation, although 3D IoU has been used in previous work, its
computation is overly simplified based on some assumptions. In [30], 3D boxes
are assumed to be axis-aligned. Another approach, used by autonomous datasets
(e.g. in [21] and others), is to project the 3D bounding boxes to the ground
plane and then compute the intersection of the 2D projected polygons. Then the
intersection volume is estimated by multiplying the area of the 2D
intersecting polygon with the height of the 3D bounding boxes. Although this
approach works for vehicles on the road, it has two limitations: 1) The object
should sit on the same ground plane, which limits the degrees of freedom of
the box from 9 to 7. The box only has freedom in yaw, and the roll and pitch
are set to 0. 2) it assumes the boxes have the same height. For the Objectron
datasets, these assumptions do not hold.

We propose an algorithm for computing accurate 3D IoU values for general
3D-oriented boxes. First, we compute the intersection points between the faces
of the two boxes using the Sutherland-Hodgman Polygon clipping algorithm [9].
Let $x$ denote the predicted box and $y$ denote the annotation label. To
compute the intersecting points between the boxes $x$ and $y$, first transform
both boxes using the inverse transformation of the box $x$. The transformed
box $x$ will be axis-aligned and centered around the origin while the box $y$
is brought to the coordinate system of the box $x$ and remains oriented.
Volume remains invariant under rigid-body transformation. We can compute the
intersecting points in the new coordinate system and estimate the volume from
the transformed intersection points. Using this coordinate system allows for
more efficient and simpler polygon clipping against boxes since each surface
is perpendicular to one of the coordinate axes.

(a) Clip a polygon from two boxes using the Sutherland-Hodgman algorithm.

(b) Compute intersection volume (green) using the convex hull algorithm.

Figure 8: Accurate computation of 3D IoU using polygon-clipping algorithm.

Next, we clip each face in box $y$, which is a convex polygon, against the
axis-aligned box $x$. There is a well-known polygon clipping algorithm in
computer graphics [9], where each world polygon is clipped against the camera
frustum to determine the rendered environment. We use a robust Sutherland-
Hodgman algorithm to perform the clipping. To clip a polygon against a plane,
the edges of the plane are hypothetically extended to infinity. We iterate
over each edge in the polygon in clockwise order and determine whether that
edge intersects with any faces in the axis-aligned box $x$. For each vertex in
the box $y$, we check whether any of them are inside the box $x$. We add those
vertices to the intersecting vertices as well. We repeat the whole process
swapping the box $x$ and $y$. We refer the readers to [9] for the details of
the polygon clipping algorithm. Figure 8(a) shows an example of a polygon
clipping.

The volume of the intersection is computed by the convex hull of all the
clipped polygons, as shown in Figure 8(b). Finally, the IoU is computed from
the volume of the intersection and volume of the union of two boxes. We are
releasing the evaluation metrics source code along with the dataset.

Figure 9: 3D IoU computation for symmetric objects: Rotating the bounding box
along the $Y$ axis of symmetry to maximize 3D IoU.

For symmetric objects, such as bottles or cups, the 3D IoU metric is not well-
defined. In those instances, we rotate the estimated bounding box along the
symmetry axis and evaluate each rotated instance, then pick the bounding box
that maximizes the IoU. Figure 9 shows an example for asymmetric cup.

###  5.2 Baselines for 3D object detection

Figure 10: Evaluation of MobilePose network[15] on the Objectron dataset

Figure 11: Evaluation of two-stage network on the Objectron dataset

Model | bike | book | bottle | camera | cereal_box | chair | cup | laptop | shoe
---|---|---|---|---|---|---|---|---|---
MobilePose [15] | 0.3486 | 0.1818 | 0.5449 | 0.4762 | 0.5496 | 0.7112 | 0.3722 | 0.5548 | 0.4230
Two-stage | 0.6127 | 0.5218 | 0.5744 | 0.8016 | 0.6272 | 0.8505 | 0.5388 | 0.6735 | 0.6606
(a) Average precision at 0.5 3D IoU metric for different categories.

Model | bike | book | bottle | camera | cereal_box | chair | cup | laptop | shoe
---|---|---|---|---|---|---|---|---|---
MobilePose [15] | 0.1581 | 0.0840 | 0.0818 | 0.0773 | 0.0454 | 0.0892 | 0.2263 | 0.0736 | 0.0655
Two-stage | 0.0828 | 0.0477 | 0.0405 | 0.0449 | 0.0337 | 0.0488 | 0.0541 | 0.0291 | 0.0391
(b) Mean pixel error of 2D projection of box vertices for different
categories.

Model | bike | book | bottle | camera | cereal_box | chair | cup | laptop | shoe
---|---|---|---|---|---|---|---|---|---
MobilePose [15] | 0.4907 | 0.4159 | 0.4426 | 0.5634 | 0.8905 | 0.6437 | 0.0907 | 0.6432 | 0.3991
Two-stage | 0.8234 | 0.7222 | 0.8003 | 0.8030 | 0.9404 | 0.8840 | 0.6444 | 0.8561 | 0.5860
(c) Average precision at $15\degree$ Azimuth error for different categories.

Model | bike | book | bottle | camera | cereal_box | chair | cup | laptop | shoe
---|---|---|---|---|---|---|---|---|---
MobilePose [15] | 0.7995 | 0.6363 | 0.7020 | 0.5573 | 0.8144 | 0.7329 | 0.6735 | 0.5750 | 0.5017
Two-stage | 0.9390 | 0.8616 | 0.8567 | 0.8437 | 0.9476 | 0.9272 | 0.8365 | 0.7593 | 0.7544
(d) Average precision at $10\degree$ Elevation error for different categories.

Table 2: Evaluation of different baseline models for the Objectron dataset.

We provide baseline results for 3D object detection and viewpoint estimation.
We trained a state-of-the-art model [15] over our dataset for detecting 3D
bounding boxes. MobilePose is a lightweight network that is designed to
perform in real-time on mobile devices. We evaluate the network’s output using
our evaluation code and report various metrics, such as average precision for
3D IoU, 2D pixel projection error, azimuth, and elevation111The trained models
and their evaluation report can be downloaded from objectron.dev. For each
category, we trained the network separately without any pre-training or
hyperparameter optimization. Table 2 shows the evaluation results. We
empirically observed pre-training and hyperparameter optimization can
significantly improve the baseline results. Each model was trained for 100
epochs ($\sim$12 hours) on eight V100 GPUs.

The original MobilePose network also uses the shape information obtained from
synthetic data. However, we showed it also works by training purely on real
data without any shape information. In our implementation, we used MobileNetV2
as backend, and added two heads to the network: 1) An attention head that
creates an attention mask at the center keypoint of the 3D bounding box, and
2) a regression head, that predicts the x-y adjustment of the eight other
keypoint from the center keypoint. The network predicts the nine 2D projected
keypoints, which are later lifted to 3D using EPNP algorithm [15].

We also designed a new two-stage architecture for 3D object detection. The
first stage estimates a 2D crop of the object of the size $224\times 224$
using SSD model[20][16], followed by a second stage model using EfficientNet-
Lite [27] architecture which uses the 2D crop to regress the keypoints of the
3D bounding box. We use a similar EPnP algorithm as in [15] to lift the 2D
predicted keypoints to 3D. This network is very lightweight (5.2MB size) and
runs at 83fps on Samsung S20 mobile GPU. Figure 11 shows the evaluation
results, including the average precision plot for 3D IoU, azimuth, and
elevation for the two-stage model.

The two-stage network first uses a 2D object detector (SSD network in our
implementation) to detect a $224\times 224$ crop of the object. Then the
network (as shown in Figure 12) uses an EfficientNet-lite network to encode
the input image to an $7\times 7\times 1152$ embedding vector, followed by a
fully connected layer to regress the 9 2D keypoints. The network uses a
similar EPnP algorithm as in [15] to lift the 2D predicted keypoints to 3D.
The hyper-parameters of the training jobs are provided in Table 3.

For the average precision, first, the detector has to detect the 3D bounding
box using the center of the box, then proceed to compute the other metrics.
The experimental result shows the model is more accurate in estimating
elevation than azimuth because the distribution of the elevation in our
dataset (Figure 7) is biased toward $45\degree$, but azimuth is uniformly
distributed. In other words, in our videos, the data collector is looking down
and walking around the object to capture the video. Data augmentation
techniques, such as affine transformation or cropping, can change the
distribution of viewpoints in the dataset and help generalization. We should
also highlight how badly the network performs on estimating the rotation of
the ’cup’, as evident by the average precision of azimuth for the cups (Figure
10 and Table 2(c)).

Figure 12 shows the overview architecture of the models that we used as
baselines. Both models are capable of achieving real-time performance on
mobile devices.

| MobilePose | Two-stage
---|---|---
Epoch | 100 | 250
Learning rate | 1e-2, decayed to 1e-3 in 30 epoch | 1e-2 to 1e-6 exponential annealing
Batch-size | 128 | 64
Optimizer | adam | adam
Input-size | $480\times 640\times 3$ | $224\times 224\times 3$
Input-size | $120\times 160$ | $9\times 2$
Loss | MSE on attention + $L_{1}$ on regression | Per vertex MSE normalized on diagonal edge length
Backend | MobileNetV2 | EfficientNet-Lite
Table 3: Hyper parameters for the baseline models.

(a) The architecture of the MobilePose[15] model.

(b) Architecture of the two-stage model. The red blocks are $1\times 1$
convolutional layers, green blocks are depthwise convolutional layers, and
blue blocks are addition layers for skip connection. The black block at the
end is a fully connected layer.

Figure 12: The baseline models used for 3D Object Detection task.

##  6 Details of the Objectron data format

The data is stored in Google storage objectron bucket for public access. The
supplementary code for parsing and evaluation is available at
https://github.com/google-research-datasets/Objectron. For each sample, the
dataset provides the raw video file (in MOV file format, at 30fps, and
$1440\times 1920$ resolution, the AR Metadata, and the annotation result. The
AR metadata contains the camera transformation, view, projection, and
intrinsic matrix. The camera transformation contains the camera transformation
from the first frame in the sequence. Furthermore, the sparse point-cloud in
the world-coordinate and surface planes (including the normal and extend,
boundary points, and plane alignment w.r.t. gravity vector) are provided.

For each object instance, the annotation data includes the bounding box’s
orientation, translation, and scale, as well as the 3D vertices in the world
and camera coordinate system and their 2D projection (with depth) in the image
plane. Each instance has a label string that corresponds to the object’s
category. The bounding box transformation transforms an axis-aligned unit
bounding box to the annotated bounding box in the world-coordinate system. For
each frame in the video, we also compute the transformed bounding box in the
camera coordinate system as well.

Besides raw data, we also provided a pre-processed dataset that can be easily
connected to existing input pipelines for model training. We converted our
entire dataset to Tensorflow tf.Example and tf.SequenceExample for image and
video models, respectively. Both formats are stored as Tensorflow records. We
implementing example pipelines to feed this data to PyTorch, Tensorflow, and
Jax training pipelines efficiently.

##  7 Conclusion

This paper introduces the Objectron dataset: a large scale object-centric
dataset of $14,819$ short videos in the wild with object pose annotation. We
developed an efficient and scalable data collection and annotation framework
based on on-device AR libraries. We also presented results of our proposed
two-stage 3D object detection model trained on the Objectron dataset as the
baseline. By releasing this dataset, we hope to enable the research community
to push the limits of 3D object geometry understanding and foster new research
and applications in 3D understanding, video models, object retrieval, view
synthetics, and 3D reconstruction.

## References

  * [1] ARcore.  https://developers.google.com/ar.  Accessed: 2020-11-16.
  * [2] ARkit.  https://developer.apple.com/augmented-reality/.  Accessed: 2020-11-16.
  * [3] Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X Chang, and Matthias Niessner.  Scan2CAD: Learning CAD Model Alignment in RGB-D Scans.  Computer Vision and Pattern Recognition, Nov. 2018.
  * [4] Berk Calli, Arjun Singh, James Bruce, Aaron Walsman, Kurt Konolige, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar.  Yale-CMU-Berkeley dataset for robotic manipulation research.  The International Journal of Robotics Research, 36(3):261–268, Apr. 2017.
  * [5] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository.  arXiv. cs.GR, Dec. 2015.
  * [6] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Niessner.  ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes.  Proc. Computer Vision and Pattern Recognition, pages 5828–5839, 2017.
  * [7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. IEEE, Apr. 2009.
  * [8] Andreas Doumanoglou, Rigas Kouskouridas, Sotiris Malassiotis, and Tae-Kyun Kim.  Recovering 6D Object Pose and Predicting Next-Best-View in the Crowd.  IEEE Conference on Computer Vision and Pattern Recognition, pages 3583–3592, 2016.
  * [9] Christer Ericson.  Real-Time Collision Detection.  CRC Press, 2004.
  * [10] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel.  ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.  International Conference on Learned Representation, Nov. 2019.
  * [11] James J Gibson.  The Ecological Approach to Visual Perception.  1979\.
  * [12] Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab.  Model Based Training, Detection and Pose Estimation of Texture-Less 3D Objects in Heavily Cluttered Scenes.  In Asian Conference in Computer Vision (ACCV), pages 548–562. Springer, Berlin, Heidelberg, Berlin, Heidelberg, Nov. 2012.
  * [13] Tomas Hodan, Pavel Haluza, Stepan Obdrzalek, Jiri Matas, Manolis Lourakis, and Xenophon Zabulis.  T-LESS: An RGB-D dataset for 6D pose estimation of texture-less objects.  IEEE Robotics and Automation Latters.
  * [14] Tomas Hodan, Rigas Kouskouridas, Tae-Kyun Kim, Federico Tombari, Kostas Bekris, Bertram Drost, Thibault Groueix, Krzysztof Walas, Vincent Lepetit, Ales Leonardis, Carsten Steger, Frank Michel, Caner Sahin, Carsten Rother, and Jiri Matas.  BOP: Benchmark for 6D Object Pose Estimation.  Computer Vision and Pattern Recognition, (Chapter 36):589–600, Oct. 2018.
  * [15] Tingbo Hou, Adel Ahmadyan, Liangkai Zhang, and Jianing Wei.  MobilePose: Real-Time Pose Estimation for Unseen Objects with Weak Shape Supervision.  arXiv 2003.03522, 2020.
  * [16] Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, and Kevin Murphy.  Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors.  In Computer Vision and Pattern Recognition, pages 7310–7311, 2017\.
  * [17] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari.  The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale.  International Journal of Computer Vision, (7):1956–1981, Nov. 2020\.
  * [18] Yann Labbe, Justin Carpentier, Mathieu Aubry, and Josef Sivic.  CosyPose: Consistent multi-view multi-object 6D pose estimation.  In Proceedings of the European Conference on Computer Vision, Aug. 2020.
  * [19] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C Lawrence Zitnick, and Piotr Dollar.  Microsoft COCO: Common Objects in Context.  Computer Vision and Pattern Recognition, May 2014.
  * [20] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  SSD: Single Shot MultiBox Detector.  Computer Vision and Pattern Recognition, (Chapter 2):21–37, Dec. 2015.
  * [21] Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka.  3D Bounding Box Estimation Using Deep Learning and and Geometry.  Computer Vision and Pattern Recognition, Dec. 2016.
  * [22] Colin Rennie, Rahul Shome, Kostas E Bekris, and Alberto F De Souza.  A Dataset for Improved RGBD -based Object Detection and Pose Estimation for Warehouse Pick-and-Place.  IEEE Robotics and Automation Latters.
  * [23] Mike Roberts and Nathan Paczan.  Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding.  arXiv 2003.03522, 2020.
  * [24] Silvio Savarese and Fei-Fei Li.  3D generic object categorization, localization and pose estimation.  In IEEE Workshop on Applications of Computer Vision, 2007.
  * [25] Srinath Sridhar, Davis Rempe, Julien Valentin, Bouaziz Sofien, and Leonidas J Guibas.  Multiview Aggregation for Learning Category-Specific Shape Reconstruction.  Conference on Neural Information Processing Systems, pages 2348–2359, 2019.
  * [26] Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B Tenenbaum, and William T Freeman.  Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling.  Computer Vision and Pattern Recognition, Apr. 2018.
  * [27] Mingxing Tan and Quoc V Le.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks.  Computer Vision and Pattern Recognition, May 2019.
  * [28] Shubham Tulsiani, Or Litany, Charles R Qi, He Wang, and Leonidas J Guibas.  Object-Centric Multi-View Aggregation.  July 2020.
  * [29] Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, and Matthias Niessner.  RIO: 3D Object Instance Re-Localization in Changing Indoor Environments.  Proceedings IEEE International Conference on Computer Vision, pages 7658–7667, 2019.
  * [30] H Wang, S Sridhar, J Huang, J Valentin, and S Song.  Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation.  In The IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [31] Fei Xia, Amir R Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese.  Gibson Env: Real-world perception for embodied agents.  In Computer Vision and Pattern Recognition, 2018.
  * [32] Yu Xiang, Wonhui Kim, Wei Chen, Jingwei Ji, Christopher Choy, Hao Su, Roozbeh Mottaghi, Leonidas Guibas, and Silvio Savarese.  ObjectNet3D: A Large Scale Database for 3D Object Recognition.  In European Conference on Computer Vision, pages 160–176, Cham, Oct. 2016. Springer, Cham.
  * [33] Y Xiang, R Mottaghi, and S Savarese.  Beyond PASCAL: A benchmark for 3D object detection in the wild.  In IEEE Workshop on Applications of Computer Vision, Mar. 2014.


# Estimating Semantic Similarity between In-Domain and Out-of-Domain Samples

Rhitabrat Pokharel  Ameeta Agrawal
Department of Computer Science
Portland State University
<EMAIL_ADDRESS>

###### Abstract

Prior work typically describes out-of-domain (OOD) or out-of-distribution
(OODist) samples as those that originate from dataset(s) or source(s)
different from the training set but for the same task. When compared to in-
domain (ID) samples, the models have been known to usually perform poorer on
OOD samples, although this observation is not consistent. Another thread of
research has focused on OOD detection, albeit mostly using supervised
approaches. In this work, we first consolidate and present a systematic
analysis of multiple definitions of OOD and OODist as discussed in prior
literature. Then, we analyze the performance of a model under ID and
OOD/OODist settings in a principled way. Finally, we seek to identify an
unsupervised method for reliably identifying OOD/OODist samples without using
a trained model. The results of our extensive evaluation using 12 datasets
from 4 different tasks suggest the promising potential of unsupervised metrics
in this task.

##  1 Introduction

What happens when you train a machine learning model on a dataset and use it
to predict a sample whose source is unknown? Would you fully rely on the
model’s prediction on the test sample? Basically, this situation is
encountered in most real-world scenarios where the test sample may differ
considerably from the training samples. Recent works show that models perform
poorer on the samples that come from a different distribution (Gokhale et al.,
2022). In many real-world scenarios, such as health and law, false predictions
or misclassified results could have significant consequences, and as such
identifying out-of-domain or out-of-distribution data beforehand is critical.

Paper | Setup | Term |  Metrics |  Task
---|---|---|---|---
Chrysostomou and Aletras (2022) | A | OOD |  - |  Sentiment classification
Le Berre et al. (2022) | A | OOD |  Accuracy |  MCQ
Lin et al. (2022) | A | OODist |  - |  Extractive QA
Nejadgholi et al. (2022) | A | OOD |  AUC, F1 |  Sentiment classification
Chiang and Lee (2022) | A | OODist |  Cosine similarity, Confidence score, Probability distribution |  Sentiment classification
Mishra and Arunkumar (2022) | A | OODist |  NLI diagnostics |  NLI
Varshney et al. (2022) | A | OOD |  Accuracy |  NLI, Duplicate detection, Sentiment analysis, MCQ, Commonsense Reasoning
Omar et al. (2022) | A | OODist |  Accuracy, Success rate, Error rate, Diversity, Fairness, IBP tightness, Robustness |  Classification, Paraphrasing, NLI
Adila and Kang (2022) | A | OODist |  Confidence, Variability |  NLI
Singhal et al. (2022) | A | OOD |  Accuracy |  NLI, Phrase identification
Agrawal et al. (2022) | A | OOD |  Accuracy |  Visual QA
Aghazadeh et al. (2022) | A, B | OODist |  Accuracy |  Metaphorical knowledge
Chen et al. (2023) | A, B | OODist |  Accuracy |  Sentiment analysis, Toxicity detection, News Classification, Dialogue Intent Classification
Mai et al. (2022) | B | OODist |  - |  Anomaly detection
Garg et al. (2022) | B | OOD |  Accuracy |  Rating generation, Toxicity classification
Jin et al. (2021) | B | OOD |  False Positive Ratio, AUROC, AUPR |  Text Classification
Atwell et al. (2022) | C | OOD |  h-discrepancy |  Discourse parsing
Gokhale et al. (2022) | C | OOD |  Accuracy, EM |  NLI, QA, Image classification
Table 1: A survey of recent works using various setups to study OODist or OOD
settings. Here, A describes the cases where the train set is from one dataset,
and the test set from another dataset; B describes the scenario where the
train and test sets are two subsets of the same dataset; and C is a
combination of both A and B. The “Metrics" column represents the metrics,
while the “Task" column lists the tasks studied in these papers. Note that
several papers whose setup can be described as A use different terms.

Previous works have defined OOD and OODist data in different ways or used them
interchangeably. Early works define data that comes from a related but
different domain as OOD (Dai et al., 2007), whereas OODist data has been
defined as the data that might have been collected at a different time (Ovadia
et al., 2019). In recent studies, (Chrysostomou and Aletras, 2022) use the
term OOD to describe different datasets for the same task (e.g., SST, IMDb,
and Yelp for sentiment classification), whereas (Lin et al., 2022) use OODist
to describe the datasets that are not in the training set, including those
that are subsets of the same dataset (e.g., PDTB 2.0 (Carlson et al., 2002)).
In this paper, we first present a focused analysis of all the various
terminologies used in this context in recent works.

Another thread of research has focused on identifying OOD/OODist samples,
mostly through supervised methods (Varshney et al., 2022; Chiang and Lee,
2022; Gokhale et al., 2022). However, considering that trained models may not
always be available, we take a complementary approach in this work to identify
metric(s) that may be able to support OOD detection in an unsupervised manner.

The first part of our methodology focuses on establishing to what extent
performance (e.g., accuracy) can inform the detection of OOD samples111As
formally distinguishing between the two terms remains beyond the scope of this
paper, in this work we use the terms OOD and OODist interchangeably.. Our
results indicate that indeed performance can serve as a reliable metric for
estimating OODness, however, this requires a supervised model. To address this
limitation, in the second part of this work, we explore several unsupervised
metrics for estimating semantic similarity between the training and test
samples. We hypothesize that an unsupervised metric which sufficiently
correlates with performance, may be considered as a feasible alternative for
detecting OOD samples.

The major contributions of this paper are:

  * •

an in-depth exploration of the usage of the terms OOD and OODist in recent
works;

  * •

a systematic assessment of the effectiveness of performance in estimating
OODness, and an investigation of unsupervised approaches for identifying
OODness;

  * •

an extensive evaluation across four different tasks using a total of twelve
datasets; we will also make our code available for facilitating
reproducibility.

Task |  Datasets |  train/ val/ test
---|---|---
Sentiment |  IMDb, SST2, Yelp |  3310/ 428/ 909
MCQ |  SCIQ, CS, QASC |  8134/ 926/ 920
Extractive QA |  SQUAD, News, Trivia |  61688/ -/ 4212
NLI |  MNLI, WNLI, QNLI |  635/ 71/ 146
Table 2: Task and dataset details

##  2 Related Work

Prior research has often used the terms OOD and OODist interchangeably. In
some works, dataset $X$ is described to be OODist to dataset $Y$ if they are
different datasets, but support the same task (Lin et al., 2022; Aghazadeh et
al., 2022; Chiang and Lee, 2022; Mishra and Arunkumar, 2022; Omar et al.,
2022; Adila and Kang, 2022), while in other works, the term OOD is used to
describe the similar setting Chrysostomou and Aletras (2022); Le Berre et al.
(2022); Nejadgholi et al. (2022); Varshney et al. (2022). Beyond that, while
some consider different subsets of the same dataset to be OODist Mai et al.
(2022); Garg et al. (2022); Jin et al. (2021), others refer to these as OOD to
describe distributionally different datasets Atwell et al. (2022); Gokhale et
al. (2022).

When it comes to detecting OOD or OODist samples, using the model’s accuracy
(Le Berre et al., 2022; Aghazadeh et al., 2022; Gokhale et al., 2022; Omar et
al., 2022), input features, hidden features representations, and output
probability distribution of the network layers Chiang and Lee (2022), or AUC
and F1 score Nejadgholi et al. (2022) have been well-studied. Table 1 presents
a brief summary of some recent works.

##  3 Method

###  3.1 Problem Definition

Given two datasets, $\mathcal{X}=\\{x_{1},...,x_{m}\\}$ and
$\mathcal{Y}=\\{y_{1},...,y_{m}\\}$, the goal is to assess the correlation
between the performance of the two datasets under ID/OOD settings and their
(semantic) similarity. The performance is measured by training a model on one
of the datasets, say, $\mathcal{X}_{train}$ and testing it on the test set
$\mathcal{X}_{test}$ which represents the ID setting, and $\mathcal{Y}_{test}$
representing the OOD setting. The ID similarity is computed by averaging the
similarity between the instances of $\mathcal{X}_{train}$ and
$\mathcal{X}_{test}$, while OOD similarity is measured between
$\mathcal{X}_{train}$ and $\mathcal{Y}_{test}$.

###  3.2 Datasets

We study four different tasks using a total of 12 datasets (3 datasets for per
task). We include the most common tasks that have been used in prior work.

(i) Sentiment Analysis: given a text, classify its sentiment as negative or
positive.

(ii) Multiple Choice Question Answering (MCQ): given a question and a context,
select the correct answer from a pool of possible answers.

(iii) Extractive Question Answering (QA): given a question and a context, find
the answer to the question from the context.

(iv) Natural Language Inference (NLI): given a premise and a hypothesis,
determine whether the hypothesis contradicts, entails, or is neutral with
respect to the premise.

Table 2 presents the details of the datasets and the tasks. For sentiment
classification, we use IMDb (Maas et al., 2011), SST2 (Socher et al., 2013),
and Yelp (Zhang et al., 2015) datasets. We experiment with SCIQ Welbl et al.
(2017), CommonsenseQA (CS) (Talmor et al., 2019), and QASC (Khot et al., 2020)
for the MCQ task. For the Extractive QA task, SQUAD, News, and Trivia (Fisch
et al., 2019) datasets are selected from the MRQA dataset (note that since
these datasets do not have a separate test set, we use the validation data as
the test set). The NLI datasets include MNLI, QNLI, and WNLI from the GLUE
benchmark Wang et al. (2018). All the other datasets were accessed from the
HuggingFace repository222https://huggingface.co/datasets/.

Data preparation: Prior work has largely overlooked the effect of an important
aspect – dataset size – in such studies. As such, we control the dataset size
as a variable in our study by maintaining the size of all train, validation
(when available), and test splits for all three datasets per task by
downsampling them to match the size of the smallest dataset in each set. For
instance, all the splits of all three sentiment analysis datasets are
downsampled to be of equal size. Additionally, we balance the number of
instances for each class when possible (e.g., in the sentiment datasets).

###  3.3 Metrics

We use three categories of metrics, one for measuring the performance of the
model, another for estimating the similarity between the two datasets, and the
third for computing the correlation between performance and similarity.

Performance Metrics. We report accuracy for the classification tasks, i.e.,
sentiment analysis, MCQ, and NLI tasks, and F1 score for extractive Question
Answering task to measure the correctness of model predictions.

Trained on | Tested on | Performance
---|---|---
IMDb-train | IMDb-test | 0.90
Yelp-test | 0.87
SST2-test | 0.17
SST2-train | SST2-test | 0.89
IMDb-test | 0.21
Yelp-test | 0.16
Yelp-train | Yelp-test | 0.93
IMDb-test | 0.86
SST2-test | 0.19
SCIQ-train | SCIQ-test | 0.64
QASC-test | 0.18
CS-test | 0.34
CS-train | CS-test | 0.49
SCIQ-test | 0.58
QASC-test | 0.84
QASC-train | QASC-test | 0.92
SCIQ-test | 0.51
CS-test | 0.48
SQUAD-train | SQUAD-test | 0.86
News-test | 0.51
Trivia-test | 0.55
News-train | News-test | 0.66
SQUAD-test | 0.77
Trivia-test | 0.56
Trivia-train | Trivia-test | 0.66
SQUAD-test | 0.52
News-test | 0.31
MNLI-train | MNLI-test | 0.57
WNLI-test | 0.56
QNLI-test | 0.54
WNLI-train | WNLI-test | 0.42
MNLI-test | 0.26
QNLI-test | 0.47
QNLI-train | QNLI-test | 0.83
MNLI-test | 0.43
WNLI-test | 0.56
Table 3: Performance results under different ID/OOD settings. Instances where
ID performance is better than OOD performance are indicated in blue.

Similarity Metrics. To estimate the closeness among the ID and OOD datasets,
we use metrics related to semantic similarity (higher value means the samples
are from nearby distributions) and semantic distance (higher value indicates
less similarity). These include: (i) Cosine Similarity: measures the distance
between the samples from two sources333We estimate this using word2vec
embeddings.. (ii) Mauve Score: measures the similarity between two texts444We
use the default embeddings (GPT-2) https://pypi.org/project/mauve-text/.
(Pillutla et al., 2021). (iii) Wasserstein Distance (Wstn): measures the
distance between the two distributions and if the distributions overlap
enough, then they are close to each other555We use the universal sentence
encoder for estimating this. (Weng, 2019). (iv) Jensen Shannon Distance (JSD):
quantifies the similarity between two probability distributions, where the
smaller the value, the closer the distributions666We used word2vec embeddings.
(Manning and Schutze, 1999).

Correlation Metrics. Lastly, we use two commonly used correlation metrics –
Kendall Tau and
Pearson777https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.corr.html
(we also experimented with Spearman which gave similar results), with the goal
of understanding the relationship between performance and similarity of
datasets under ID/OOD settings.

###  3.4 Measuring Performance and Similarity

For measuring the performance, we fine-tune a BERT base uncased model for 2
epochs on each $\mathcal{X}_{train}$ and test it on $\mathcal{X}_{test}$ (ID)
and $\mathcal{Y}_{test}$ (OOD). For estimating the similarity between the ID
and OOD datasets, we randomly sample two sets of 20 instances,
$\mathcal{X}_{train20}$ and $\mathcal{Y}_{test20}$, and estimate pairwise
similarity between all of these samples, obtaining a total of 400 similarity
scores which are then averaged to compute the similarity.

##  4 Results and Discussion

Performance analysis: Table 3 presents the results of the performance
experiments, where we observe that the model performance under ID settings is
generally better than under OOD settings, except for three exceptions,
suggesting that performance can indeed serve as a reasonably dependable metric
for detecting OOD. However, this requires a supervised model, which motivates
us to explore unsupervised approaches for estimating OODness. It is worth
noting that while Garg et al. (2022) found that OOD accuracy is less than the
ID accuracy, this observation does not always hold true according to our
analysis.

Correlation between performance and similarity: Figure 1 presents the heatmap
visualizing the correlation (Kendall and Pearson) between performance and
similarity metrics, across all 12 datasets for the four tasks (the full set of
results is included in Appendix A). In looking at the results, we observe that
according to Kendall Tau correlation analysis, Wasserstein distance (Wstn)
shows the most consistent correlation (in 10 out of 12 cases), whereas
according to Pearson correlation, both Wasserstein and Cosine are acceptable
metrics (in 9 out of 12 cases). In all the scenarios, however, JSD is clearly
the least correlated metric. This suggests the potential of unsupervised
approaches in estimating OOD samples.

Figure 1: (a) Kendall and (b) Pearson correlation between performance and
dataset similarity, evaluated over 12 datasets with each serving as an ID
dataset once. For Cosine and Mauve, darker shades are desirable, whereas for
Wstn and JSD, lighter shades indicate better correlation.

##  5 Conclusion

In this work, we aim to identify unsupervised approaches for identifying OOD
samples. We conducted an in-depth analysis of different unsupervised
similarity metrics and estimated their correlation with performance of a model
under ID/OOD settings. Our findings indicate that Wasserstein distance
presents a promising metric for determining OOD samples. The natural question
of how to determine the appropriate threshold, however, remains to be explored
in future work. Another direction worth exploring is to verify the robustness
of these similarity metrics when estimated using different embeddings.

## Limitations

While our analysis suggests some promising results, we acknowledge some
limitations of this work such as:

  * •

on some datasets, the ID performance was observed to be less than the OOD
performance, and further investigation is needed to study this observation in
detail and bring additional insights.

  * •

all the analysis in this study focuses on datasets in English language, and it
will be interesting to investigate whether our findings will generalize to
other languages.

## Acknowledgements

We are thankful to the anonymous reviewers for their valuable feedback.

## References

  * Adila and Kang (2022) Dyah Adila and Dongyeop Kang. 2022.  Understanding out-of-distribution: A perspective of data dynamics.  In _Proceedings on "I (Still) Can’t Believe It’s Not Better!" at NeurIPS 2021 Workshops_ , volume 163 of _Proceedings of Machine Learning Research_ , pages 1–8. PMLR.
  * Aghazadeh et al. (2022) Ehsan Aghazadeh, Mohsen Fayyaz, and Yadollah Yaghoobzadeh. 2022.  Metaphors in pre-trained language models: Probing and generalization across datasets and languages.
  * Agrawal et al. (2022) Aishwarya Agrawal, Ivana Kajić, Emanuele Bugliarello, Elnaz Davoodi, Anita Gergely, Phil Blunsom, and Aida Nematzadeh. 2022.  Rethinking evaluation practices in visual question answering: A case study on out-of-distribution generalization.  _arXiv preprint arXiv:2205.12191_.
  * Atwell et al. (2022) Katherine Atwell, Anthony Sicilia, Seong Jae Hwang, and Malihe Alikhani. 2022.  The change that matters in discourse parsing: Estimating the impact of domain shift on parser error.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 824–845, Dublin, Ireland. Association for Computational Linguistics.
  * Carlson et al. (2002) Lynn Carlson, Mary Ellen Okurowski, and Daniel Marcu. 2002.  _RST discourse treebank_.  Linguistic Data Consortium, University of Pennsylvania.
  * Chen et al. (2023) Sishuo Chen, Wenkai Yang, Xiaohan Bi, and Xu Sun. 2023.  Fine-tuning deteriorates general textual out-of-distribution detection by distorting task-agnostic features.  _arXiv preprint arXiv:2301.12715_.
  * Chiang and Lee (2022) David Cheng-Han Chiang and Hung-Yi Lee. 2022.  Understanding, detecting, and separating out-of-distribution samples and adversarial samples in text classification.  _CoRR_ , abs/2204.04458.
  * Chrysostomou and Aletras (2022) George Chrysostomou and Nikolaos Aletras. 2022.  An empirical study on explanations in out-of-domain settings.
  * Dai et al. (2007) Wenyuan Dai, Gui-Rong Xue, Qiang Yang, and Yong Yu. 2007.  Co-clustering based classification for out-of-domain documents.  In _Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 210–219.
  * Fisch et al. (2019) Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019\.  MRQA 2019 shared task: Evaluating generalization in reading comprehension.  In _Proceedings of 2nd Machine Reading for Reading Comprehension (MRQA) Workshop at EMNLP_.
  * Garg et al. (2022) Saurabh Garg, Sivaraman Balakrishnan, Zachary C Lipton, Behnam Neyshabur, and Hanie Sedghi. 2022.  Leveraging unlabeled data to predict out-of-distribution performance.  _arXiv preprint arXiv:2201.04234_.
  * Gokhale et al. (2022) Tejas Gokhale, Swaroop Mishra, Man Luo, Bhavdeep Sachdeva, and Chitta Baral. 2022\.  Generalized but not Robust? comparing the effects of data modification methods on out-of-domain generalization and adversarial robustness.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2705–2718, Dublin, Ireland. Association for Computational Linguistics.
  * Jin et al. (2021) Di Jin, Shuyang Gao, Seokhwan Kim, Yang Liu, and Dilek Hakkani-Tür. 2021.  Towards textual out-of-domain detection without in-domain labels.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_.
  * Khot et al. (2020) Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020\.  Qasc: A dataset for question answering via sentence composition.  _arXiv:1910.11473v2_.
  * Le Berre et al. (2022) Guillaume Le Berre, Christophe Cerisara, Philippe Langlais, and Guy Lapalme. 2022\.  Unsupervised multiple-choice question generation for out-of-domain Q&A fine-tuning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 732–738, Dublin, Ireland. Association for Computational Linguistics.
  * Lin et al. (2022) Bill Yuchen Lin, Sida Wang, Xi Lin, Robin Jia, Lin Xiao, Xiang Ren, and Scott Yih. 2022.  On continual model refinement in out-of-distribution data streams.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3128–3139, Dublin, Ireland. Association for Computational Linguistics.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Mai et al. (2022) Kimberly T. Mai, Toby Davies, and Lewis D. Griffin. 2022.  Self-supervised losses for one-class textual anomaly detection.
  * Manning and Schutze (1999) Christopher Manning and Hinrich Schutze. 1999.  _Foundations of statistical natural language processing_.  MIT press.
  * Mishra and Arunkumar (2022) Swaroop Mishra and Anjana Arunkumar. 2022.  A proposal to study "is high quality data all we need?".
  * Nejadgholi et al. (2022) Isar Nejadgholi, Kathleen Fraser, and Svetlana Kiritchenko. 2022.  Improving generalizability in implicitly abusive language detection with concept activation vectors.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5517–5529, Dublin, Ireland. Association for Computational Linguistics.
  * Omar et al. (2022) Marwan Omar, Soohyeon Choi, DaeHun Nyang, and David Mohaisen. 2022.  Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions.  _arXiv e-prints_ , page arXiv:2201.00768.
  * Ovadia et al. (2019) Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. 2019.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  _Advances in neural information processing systems_ , 32.
  * Pillutla et al. (2021) Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021.  MAUVE: Measuring the gap between neural text and human text using divergence frontiers.  In _Advances in Neural Information Processing Systems_.
  * Singhal et al. (2022) Prasann Singhal, Jarad Forristal, Xi Ye, and Greg Durrett. 2022.  Assessing out-of-domain language model performance from few examples.  _arXiv preprint arXiv:2210.06725_.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Varshney et al. (2022) Neeraj Varshney, Swaroop Mishra, and Chitta Baral. 2022.  ILDAE: Instance-level difficulty analysis of evaluation data.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3412–3425, Dublin, Ireland. Association for Computational Linguistics.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_.
  * Welbl et al. (2017) Johannes Welbl, Nelson F Liu, and Matt Gardner. 2017.  Crowdsourcing multiple choice science questions.  _arXiv preprint arXiv:1707.06209_.
  * Weng (2019) Lilian Weng. 2019.  From gan to wgan.  _ArXiv_ , abs/1904.08994.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level Convolutional Networks for Text Classification.  _arXiv:1509.01626 [cs]_.

##  Appendix A Experimental Results

Trained | Tested |  Model Accuracy | Cosine | Mauve |  Wstn | JSD
---|---|---|---|---|---|---
IMDb | IMDb |  0.90 | 0.92 | 1 |  0.004 | 0.21
IMDb | Yelp |  0.87 | 0.87 | 0.91 |  0.0039 | 0.26
IMDb | SST2 |  0.17 | 0.78 | 0.42 |  0.0052 | 0.36
SST2 | SST2 |  0.89 | 0.66 | 0.99 |  0.0032 | 0.46
SST2 | IMDb |  0.21 | 0.77 | 0.22 |  0.0051 | 0.38
SST2 | Yelp |  0.16 | 0.72 | 0.004 |  0.0046 | 0.41
Yelp | Yelp |  0.93 | 0.86 | 0.98 |  0.0036 | 0.26
Yelp | IMDb |  0.86 | 0.87 | 0.76 |  0.0041 | 0.27
Yelp | SST2 |  0.19 | 0.73 | 0.94 |  0.0038 | 0.4
SCIQ | SCIQ |  0.64 | 0.82 | 1 |  0.004 | 0.33
SCIQ | QASC |  0.18 | 0.66 | 0.01 |  0.008 | 0.46
SCIQ | CS |  0.34 | 0.78 | 1 |  0.004 | 0.37
CS | CS |  0.49 | 0.71 | 0.94 |  0.003 | 0.45
CS | SCIQ |  0.58 | 0.62 | 0.01 |  0.007 | 0.48
CS | QASC |  0.84 | 0.61 | 0.004 |  0.005 | 0.49
QASC | QASC |  0.92 | 0.75 | 1 |  0.003 | 0.4
QASC | SCIQ |  0.51 | 0.78 | 0.99 |  0.004 | 0.38
QASC | CS |  0.48 | 0.66 | 0.004 |  0.006 | 0.48
SQUAD | SQUAD |  0.86 | 0.84 | 0.99 |  0.0037 | 0.34
SQUAD | NEWS |  0.51 | 0.82 | 0.32 |  0.0041 | 0.33
SQUAD | TRIVIA |  0.55 | 0.81 | 0.04 |  0.0059 | 0.33
NEWS | NEWS |  0.66 | 0.89 | 0.91 |  0.0036 | 0.23
NEWS | SQUAD |  0.77 | 0.86 | 0.11 |  0.0046 | 0.31
NEWS | TRIVIA |  0.56 | 0.84 | 0.89 |  0.0039 | 0.27
TRIVIA | TRIVIA |  0.66 | 0.88 | 0.99 |  0.0031 | 0.23
TRIVIA | SQUAD |  0.52 | 0.82 | 0.04 |  0.0062 | 0.34
TRIVIA | NEWS |  0.31 | 0.82 | 0.99 |  0.0042 | 0.29
MNLI | MNLI |  0.57 | 0.72 | 0.97 |  0.0035 | 0.43
MNLI | WNLI |  0.56 | 0.71 | 0.27 |  0.0032 | 0.43
MNLI | QNLI |  0.54 | 0.73 | 0.99 |  0.0037 | 0.42
WNLI | WNLI |  0.42 | 0.74 | 0.79 |  0.0032 | 0.41
WNLI | MNLI |  0.26 | 0.68 | 0.66 |  0.0036 | 0.46
WNLI | QNLI |  0.47 | 0.67 | 0.004 |  0.0035 | 0.46
QNLI | QNLI |  0.83 | 0.75 | 0.97 |  0.0036 | 0.41
QNLI | MNLI |  0.43 | 0.64 | 0.66 |  0.0039 | 0.45
QNLI | WNLI |  0.56 | 0.58 | 0.01 |  0.0034 | 0.48
Table 4: The results for the sentiment, MCQ, extractive QA, and NLI datasets.