0 Research

# ASDF: Assembly State Detection Utilizing Late Fusion by Integrating 6D Pose
Estimation

Hannah Schieber
e-mail<EMAIL_ADDRESS>Shiyu Li${}^{\texttt{\tiny{\textdagger}}}$
Niklas Corell${}^{\texttt{\tiny{\textdaggerdbl}}}$ Philipp
Beckerle${}^{\texttt{\tiny{+}}}$ Julian
Kreimeier${}^{\texttt{\small{\textparagraph}}}$ Daniel
Roth${}^{\texttt{\small{\textbardbl}}}$
Technical University of Munich
School of Medicine and Health
Department Clinical Medicine
Machine Intelligence in Orthopedics
Clinic for Orthopedics and Sports
Orthopedics${}^{\textit{*,\textdagger,\textparagraph,\textbardbl}}$ Friedrich-
Alexander-Universität Erlangen-Nürnberg (FAU)
Erlangen, Germany${}^{\textit{*,\textdagger,\textdaggerdbl,+}}$

###### Abstract

In medical and industrial domains, providing guidance for assembly processes
is critical to ensure efficiency and safety. Errors in assembly can lead to
significant consequences such as extended surgery times, and prolonged
manufacturing or maintenance times in industry. Assembly scenarios can benefit
from in-situ ar visualization to provide guidance, reduce assembly times and
minimize errors. To enable in-situ visualization 6D pose estimation can be
leveraged. Existing 6D pose estimation techniques primarily focus on
individual objects and static captures. However, assembly scenarios have
various dynamics including occlusion during assembly and dynamics in the
assembly objects appearance. Existing work, combining object detection/6D pose
estimation and assembly state detection focuses either on pure deep learning-
based approaches, or limit the assembly state detection to building blocks. To
address the challenges of 6D pose estimation in combination with assembly
state detection, our approach asdf builds upon the strengths of yolov8, a
real-time capable object detection framework. We extend this framework, refine
the object pose and fuse pose knowledge with network-detected pose
information. Utilizing our late fusion in our Pose2State module results in
refined 6D pose estimation and assembly state detection. By combining both
pose and state information, our Pose2State module predicts the final assembly
state with precision. Our evaluation on our asdf dataset shows that our
Pose2State module leads to an improved assembly state detection and that the
improvement of the assembly state further leads to a more robust 6D pose
estimation. Moreover, on the gbot dataset, we outperform the pure deep
learning-based network, and even outperform the hybrid and pure tracking-based
approaches.

###### keywords:

Synthetic data, assembly state detection, 6D pose estimation.

We present asdf. Our approach fuses RGB and depth information as well as 6D
pose estimation and state predictions. In our Pose2State module we calculate
the final state in a late fusion step by combining 6D pose estimation and
assembly state detection.

AR

    Augmented Reality
ASDF

    Assembly State Detection utilizing late Fusion
ADD

    Average Distance Error
ATE

    absolute trajectory error
BVIP

    blind or visually impaired people
CNN

    convolutional neural network
FoV

    field of view
FPS

    Farthest Point Sampling
GAN

    generative adversarial network
GCN

    graph convolutional Network
GNN

    Graph Neural Network
GBOT

    Graph-based Object Tracking
HMI

    Human-Machine-Interaction
HMD

    head-mounted display
DoF

    Degrees of Freedom
MR

    mixed reality
IoT

    internet of things
ICP

    Iterative Closest Point
ICG

    Iterative Correspondence Geometry
LLFF

    Local Light Field Fusion
BLEFF

    Blender Forward Facing
LPIPS

    learned perceptual image patch similarity
NeRF

    neural radiance fields
NVS

    novel view synthesis
NFOV

    narrow field-of-view
MLP

    multilayer perceptron
MRS

    Mixed Region Sampling
OR

    Operating Room
PBR

    physically based rendering
PSNR

    peak signal-to-noise ratio
PnP

    Perspective-n-Point
SUS

    system usability scale
SSIM

    similarity index measure
SfM

    structure from motion
SLAM

    simultaneous localization and mapping
TP

    True Positive
TN

    True Negative
thor

    The House Of inteRactions
UEQ

    User Experience Questionnaire
VR

    virtual reality
WHO

    World Health Organization
YCB

    Yale-CMU-Berkeley
YOLO

    you only look once

Introduction

Augmented Reality (AR) enables the dynamic visualization of object assembly
guidance, presenting an innovative paradigm to support the assembly task
execution. This can be realized through the utilization of instruction
animations that seamlessly track 3D markers or leverage colliders to navigate
through assembly steps [4, 8, 18]. An alternative approach to markers is the
utilization of 6D pose estimation of assembly objects. These can combine
assembly state detection and estimating the object poses to enable in-situ
visualizations. By discerning assembly states, AR-driven in-situ visualization
can provide guidance precisely tailored to the current assembly context.

In order to provide in-situ guidance the assembly state is a relevant
information and the knowledge of each objects position in space, namely its 6D
object poses. Retrieving this knowledge is especially challenging in dynamic
assembly scenarios [47, 37]. However, existing 6D pose estimation approaches
[41, 25] are often limited to static scenes [16, 43] as the benchmark on which
they are evaluated are static. Alternatively, object tracking is more often
applied in dynamic scenarios with moving objects [35, 34, 19], but occlusion
can be challenging for pure tracking-based approaches [19]. Dynamic scenarios
with individual object assembly can occur in the medical or industry context
[47, 18, 24]. There errors in assembly can lead to either wrong manufacturing,
extended manufacturing times or in the medical context can even be challenging
for ones life.

In that regard, AR guidance has tremendous potential. AR assembly guidance
combined with 6D pose estimation and state-detection can enable in-situ
guidance, and error detection during the assembly. Marker-based tracking,
however, limits real-world applicability. To enable marker-less 6D pose
estimation and state detection, using a convolutional neural network
(CNN)-based approach can be beneficial. Existing 6D pose estimation approaches
often focus on a single object [26, 41, 15, 35, 28] instead of assembly
groups.

Moreover, many existing deep learning-based assembly state detection
approaches focus on 2D [31, 22, 48] or are limited in their evaluation to one
or two objects [22, 24]. Moreover, current mulit-object 6D pose estimation
approaches [41, 42] or tracking approaches [32, 35, 19] only provide the 6D
object pose information.

###  0.1 Contribution

To address these existing limitations we present Assembly State Detection
utilizing late Fusion (ASDF) a deep learning-based assembly state and 6D pose
estimation detection algorithm. We leverage deep learning building upon the
real-time capable you only look once (YOLO) architecture fusing 6D pose
estimation and assembly state detection for more precise object poses. For
training we leverage purely synthetic data using online available 3D printing
parts. We evaluate several aspects, such as network size and pose refinement
on our ASDF dataset. Moreover, we compare our approach in 6D pose estimation
performance on the Graph-based Object Tracking (GBOT) dataset. By evaluating
the 6D pose estimation performance on this dataset, we can show first, the
transfer ability from our domain-randomized training images to the GBOT
evaluation images with different background, occlusion, blur and light
conditions, and second, that the assembly state detection can lift the 6D pose
performance.

In summary, we contribute:

  * •

ASDF integrates late fusion in our Pose2State module, leading to refined
assembly state detection and object pose estimation

  * •

A synthetic dataset encompassing 6D object poses and assembly states utilizing
3D printable part of reproducibility

  * •

Evaluation on two datasets, demonstrating the advantage of our approach
compared to pure deep learning-based approaches

##  1 Related Work

Enabling in-situ guidance of assembly objects, requires the knowledge of an
objects position in space, namely its 6D object pose, the knowledge of its
assembly state and leveraging assembly data for training and evaluating deep
learning-based approaches.

###  1.1 6D Object Pose Estimation and Tracking

Early approaches for instance-level 6D pose estimation applied template
matching [11]. With the progress of deep learning based approaches (CNN,
Transformer) one-stager [10, 46] and two-stager [41] approaches have been
developed.

One-stager approaches are end-to-end trainable [10, 46] They extract features
from a segmentation or object detection backbone. These can be regressed
directly [46] or other output like keypoints can be feed to Perspective-n-
Point (PnP) [38]/Least Squares Fitting [10]. Amini et al. [2] introduce
YoloPose which directly regresses keypoints in an image and presents a
learnable module to replace PnP.

Two-stager approaches, apply a state-of-the-art object detection algorithm,
for example Faster R-CNN and build the 6D pose estimation on top of these
predictions. Wang et al. [41] utilize geometric feature regression on top of
the object detection algorithm. This results in 2D-3D correspondences and
Surface Region Attention which is leveraged in Patch-PnP. Similarly, Pix2Pose
[25] leverages 2D bounding boxes followed by mask prediction and bounding box
refinement. The final result is predicted using RANSAC PnP.

In addition to deep learning-based 6D pose estimation, another field is object
tracking. Object tracking starts with an initialization and then assumes the
6D pose in ongoing frames. Stoiber et al. [35] proposes Iterative
Correspondence Geometry (ICG) combines visual, regions and depth information.
Their improvement ICG$+$ [32] additionally considers SIFT and ORB features.
Additionally, Mb-ICG [34] tackles kinematic structures. Object tracking can
get lost in highly occluded scenes. There, combining deep learning-based
approaches with tracking can be promising promising [20, 19]. For example,
DeepIM [20] combines the re-initialization by PoseCNN [46]. Li et al. [19]
propose a combination of an extension of YOLOv8 and ICG. While their pose
initialization relies on the CNN, they utilize an assembly state-graph for
tracking assembled objects. Additionally, they provide the GBOT dataset for
tracking assembled parts. However, the focus lies on 6D pose estimation
instead of assembly detection.

###  1.2 Assembly State Detection for Augmented Reality

While the object pose is on essential aspect for assembly guidance another on
is the assembly state itself. Zauner et al. [47] utilized markers and build an
assembly graph for AR-based assembly instruction. Duplotrack [9] utilizes
point cloud alignment whenever the assembly building blocks change.

In terms of 2D assembly state detection, Kleinbeck et al. [18] combine YOLOv5
and synthetic data for building block assembly guidance. The guidance steps
are rendered in a Hololens. Similarly, Stane et al. [31] provide AR glass
based guidance. For their state detection they introduce a convolution block
in YOLO. Overall, their state detection improved the YOLO-based object
prediction.

Liu et al. [22] integrate a two-fold attention mechanism in a 2D object
detection architecture. Zhou et al. [48] address mobile AR guidance utilizing
regions-of-interest for state identification. The method is two-fold, in a
first step the regions-of-interest are extracted, in a second step, the state
recognition is trained.

Other approaches considering 3D/6D build upon the relative pose between
objects [45, 23, 24]. Wu et al. [45] follow a tree like graph structure to
determine the assembly state. Similarly, Murray et al. [24] address 6D pose
estimation and assembly prediction in industrial settings defining an assembly
state graph. Their network build upon Pix2Pose [25]. To address multi-view
input they utilized depth estimation and project pixels to 3D. Su et al. [37]
utilize a TridentNet as backbone and add a pose prediction head. They predict
the pose and assembly state for one object with five assembly states and
provide AR guidance for utilizing the network.

###  1.3 Real-world and Synthetic Datasets

For 6D pose estimation/object tracking multiple benchmarks exist, for example,
one common benchmark is the YCB-V benchmark [46]. However, this and other
common ones considers single objects without state changes [5, 46, 16, 14].
Moreover, providing 6D pose datasets with complex scenes is challenging. To
address this one common approach can the utilization of synthetic data [29].
In the 6D pose domain one common approach for synthetic data generation [7,
27, 3, 29].

####  1.3.1 Assembly State Datasets

One common area for assembly datasets is IKEA furniture. The IKEA-Manual
dataset [44] provides assembly data including the 3D pose of each part,
specifically detailing the rotation of the 3D components. However, it does not
explore the translation of individual parts. Addressing this limitation, the
IKEA assembly dataset introduced by Su et al. [36] fills the gap, although it
is not publicly accessible.

Other works build upon individual objects [24, 37]. However, this makes
reproduciblity more challenging. Schoonbeek et al. [30] present a hybrid
dataset focusing on action recognition and assembly state detection. They use
multiple modalities (RGB, Depth, Gaze, Stereo) and record their dataset with a
Hololens 2. Su et al. [37] reconstruct a real-world coffee machine and sample
the parts on images to generate a synthetic assembly state dataset. The
dataset features five states for the coffee machine assembly. Li et al. [19]
propose the use of 3D printable objects and present a synthetic dataset
featuring these objects. The ground truth contains the 6D object poses at
several assembly states. However, the dataset focuses purely on the 6D pose.

Figure 1: Architecture of ASDF. We highlight our contributions to the
YOLOv8Pose architecture in blue. ASDF utilizes RGB and depth data. The RGB
images are fed into the image backbone and the depth data is used to refine
the object poses ($[Rt]$). The image backbone predicts the state ($s$) based
on the RGB image. In the Translation Refinement module, the translation offset
is calculated. Using the relative pose between the assemblies in the assembly
group, we predict a second state assumption in our Pose-based Assembly
Detection module. In our final Pose2State module, we weight the individual
state predictions to predict the one with the highest probability.

####  1.3.2 Synthetic Data

The use of synthetic data for either pre-training a 6D pose estimation or
purely training on synthetic data can be usefully as recording a real-world
marker-less dataset with only a low annotation error can be challenging [16,
29]. For generating synthetic data in the context of 6D pose
estimation/assembly state detection or in general object detection one
challenge is the so-called sim-to-real gap [29, 39]. This gap denotes the
domain gap between synthetic images and real world captures. To address this
gap one common approach is randomizing several parameters like light,
background and object textures [29, 39]. Moreover, to focus on the objects of
interest but not purely over-fit on them adding distracting objects to the
training set can be beneficial [29, 40, 1]. Tremblay et al. [40] investigated
this by combining realistic and randomized images and added geometric
distracting objects leading to, i. e. occlusion of the actual object of
interest. This lifted the object detection performance. Alghonhaim et al. [1]
benchmarked domain randomization considering background, textures and
distractors. They proved as well that distractors are benefical for
generalization.

While the constant progress in 6D pose estimation is promising for deep
learning-based approaches for single objects, the combination of 6D pose
estimation and assembly state detection is even more promising for AR guidance
approaches. However, for this tasks the used objects hugely vary from building
blocks up to engines with over a hundred assembly states. Therefore, we
contribute a comparable approach to existing 6D pose estimation/object
tracking approaches and a dataset building upon these re-usable assembly
assets to improve comparability in this domain.

##  2 Method

###  2.1 Assembly State Detection utilizing late Fusion

ASDF combines 6D pose estimation and assembly state detection, building upon
YOLOv8Pose [19]. RGB-D images serve as input, with the backbone processing the
pure RGB image while leveraging the depth image and resulting point cloud for
pose refinement. The system encompasses both 6D pose estimation and assembly
state detection, which we elaborate on separately. These distinct components
are fused within our Pose2State module, as depicted in Figure 1.

####  2.1.1 6D Pose Estimation

For 6D pose estimation, we utilize YOLOv8Pose [19], which extends the output
of YOLOv8 using RANSAC PnP. To establish our keypoints in 3D space, we utilize
the 2D keypoints predicted by the backbone. Employing Farthest Point Sampling
(FPS) [19, 26], we distribute the points as widely apart from each other as
possible. For the final number of keypoints (17), we strike a balance between
computational cost (as more keypoints lead to higher computation cost) and
performance.

##### Assembly Pose Translation Refinement

Our assembly pose refinement step further refines the estimated translation by
determining the boundary values of each assembly in each assembly group, as
illustrated in Figure 1. To achieve this, we determine the required movement
along one coordinate axis (z-axis) and scale the other two axes accordingly.
For calculating this movement perpendicular to the camera plane, we transform
the 3D surface points $P_{3D}$ of the component using the transformation
matrix obtained from the initial pose estimation by our CNN
($T_{\small{DL}}$).

The transformed points $P^{\prime}_{3D}$ of the 3D surface points $P_{3D}$
using $T_{\small{DL}}$ can be expressed as:

$P^{\prime}_{3D}=\cdot P_{3D}\cdot T_{\small{DL}}$ (1)

Next, we back-project these points onto the 2D image plane ($P_{2D}$) and
filter them using the predicted bounding box, resulting in visible points
($P^{\prime}_{2D}$). Subsequently, through a second filtering process, we
reduce the number of keypoints to those closest to the camera plane. Based on
the 2D points, we approximate the depth information using the depth input.
This results in approximated 3D points.

Finally, based on the selected points in $P^{\prime}_{2D}$ and the
approximated depth, we determine the required movement along one coordinate
axis (i. e., $z$) and scale the other two axes (i. e., $y$ and $z$)
accordingly to refine the translation part. Using the approximated 3D points,
we calculate the difference to the corresponding 3D surface points. This
allows estimating the shift for all points using:

$E=\sum_{i=1}^{n}W(d_{i})\cdot d_{i}$ (2)

$d_{i}$ denotes the estimated shift distance for point $i$ and $n$ represents
the total number of points. $E$ is the the overall estimate of the required
shift for all points.

To estimate the shift distance for each point, we use $d_{i}$. Then, to obtain
an overall estimate, we combine these individual estimations using a weighting
function $W(d_{i})$. This function assigns higher weights to points with
smaller differences. This weighting prohibits that occluded objects retrieve a
false calculation during the refinement step.

The resulting translation along the axis perpendicular to the image plane can
now be used to determine the vector along the camera view axis that represents
the final translation.

####  2.1.2 Assembly State Detection

The final assembly state detection combines three key components: deep
learning-based state detection, relative pose-based state detection, and
consideration of the previous state, employing a weighted combination
approach.

##### Deep learning-based Assembly State Detection

The relationship between assembly state and 6D pose estimation is
interdependent. Each pose of an assembly part within a group contributes
valuable information to the overall assembly state, and conversely, the
current assembly state informs the relative poses of the objects to each
other.

The assembly state detection system utilizes both relative pose information
and deep learning-driven state detection. To seamlessly integrate state
detection into our framework, we represent each state as a distinct class for
the network to learn. Additionally, to enhance prediction accuracy, we
maintain a assembly state recording for the whole assembly. For each part of
the assembly we keep a record if pose matrix exists.

##### Pose-based Assembly State Detection

The relative pose between objects offers valuable information regarding the
current state of each assembly. These relative poses are inherently known, as
they are crucial for generating ground truth training and evaluation data.

To determine the assembly state based on the relative pose from one assembly,
we adopt a strategy of selecting the base part of each assembly. This base
part serves as a reference point from which the relative poses of the other
parts are considered. By anchoring the analysis to these base parts, we can
effectively interpret the relative poses of the entire assembly, allowing for
accurate assessment and monitoring of its overall configuration and state.

####  2.1.3 Pose2State Module

Given the inherent co-dependency between pose and state, we harness this
relationship within our Pose2State module. This module is designed to forecast
the probability $(SP)$ of the assembly state $s_{x}$ at time $t$. Our
Pose2State module seamlessly integrates two key components: the deep learning-
based prediction $(SP_{\small{DL}})$ and the prediction derived from the pose-
based assembly $(SP_{\small{P}})$. By combining these predictions, our module
effectively determines the final assembly state, leveraging both the insights
from deep learning techniques and the pose-based assembly analysis.

$SP_{\small{DL+P}}(s_{x})_{t}=w_{1}\cdot P_{\small{DL}}(s_{x})_{t}+w_{2}\cdot
P_{\small{P}}(s_{x})_{t}$ (3)

For the final assembly state $SP_{f}$ we consider the time component and
leverage the previously stored assembly state $f_{t-1}$. The combination of
the individual predictions is then weighted:

$SP_{\small{f+f_{t-1}}}(s_{x})_{t}=w_{3}\cdot
P_{\small{f}}(s_{x})_{t-1}+w_{4}\cdot P_{\small{f}}(s_{x\pm 1})_{t-1}$ (4)

This results in the final state calculation:

$P_{f}(s_{x})_{t}=\frac{SP_{\small{DL+P}}(s_{x})_{t}+SP_{\small{f+f_{t-1}}}(s_{x})_{t}}{w_{1}+w_{2}+w_{3}+w_{4}}$
(5)

###  2.2 Assembly State Dataset

Figure 2: Example images of our ASDF dataset. For training we combine
assembled and unassembled data (top) and additionally provide hand occlusion
(top-left), varying background and light conditions as well as distracting
objects. For evaluation we capture all images in a table-top manner (bottom),
add hand occlusion and distracting objects. We include the state information
in our labels. An example of the state complexity can be seen on the bottom
right.

To train ASDF, we created a synthetic dataset using 3D printable parts to
facilitate real-world testing. We utilized the 3D printable parts from the
GBOT dataset [19] as a foundation. In addition to that existing reproducible
setup [19], we introduced assembly state information and incorporated hand
occlusion during training to enhance the model’s robustness. Our dataset
adheres to the camera specifications of the Azure Kinect DK, with a resolution
set to $1280\times 720$ pixels. The camera is positioned in a top-down view to
simulate a capturing setup commonly found in medical or industrial scenarios.

####  2.2.1 Synthetic Dataset

Table 1: Number of states and synthetic evaluation images for each assembly part. Assembly | No. States | Number of images
---|---|---
NanoVise | 8 | 191
ScrewClamp | 10 | 231
GearedCaliper | 5 | 111
CornerClamp | 3 | 66

##### Training Dataset

Our synthetic dataset features 6D pose estimation and assembly state detection
ground truth data. It is generated using BlenderProc [7] and incorporates
several domain randomization techniques, including distractors, randomized
noise, lighting variations, occlusions, and background changes [1, 29]. Domain
randomization plays a vital role in synthetic data generation [29, 1].
Therefore, we introduce distracting objects [12, 17] and simulate hand
occlusion to mimic real-world scenarios, as illustrated in Figure 2 and Figure
3.

##### Test Dataset

We adhere to the defined assembly process for each assembly object by
generating a continuous set of assembly images for each object. We maintain
the presence of distracting objects and incorporate hands at realistic
assembly positions. The objects are randomly distributed within the camera’s
field of view. We consistently introduce hand occlusion in the evaluation set
and provide pose and state information for each image.

A key aspect of our test set is the inclusion of incorrectly assembled states.
In a real-world assembly scenario mounting together parts with missing pieces
in between can happen especially if the parts are not necessarily connecting
parts. Therefore, we added incorrect assembly states. Since each object
contains various assembly states, the number of evaluation images per assembly
object varies, as illustrated in Table 1. This variability ensures
comprehensive testing of the model’s ability to detect and handle different
assembly scenarios.

##  3 Evaluation

###  3.1 Implementation and Specifications

We trained our network for 300 epochs using early stopping. The training for
all comparisons is executed on one machine with an Intel Core i9-10980XE CPU,
128 GB RAM and one NVIDIA GeForce RTX 3090 graphics card.

As loss term, we follow YOLOv8 combining the class-wise loss ($L_{cls}$),
bounding box loss ($L_{box}$), the task-specific loss ($L_{tsk}$) all weighted
by $\lambda$.

$L_{tsk}$ is comprised of the pose loss (${L}_{pose}$) which is defined as
${L}_{pose}=||\mathbf{K}_{pred}-\mathbf{K}_{gt}||_{2}$. This is the L2 loss
using the predicted keypoints (${L}_{pose}$) from the ground truth keypoints
($\mathbf{K}_{gt}$) and the Cross-Entropy (CE) Loss using the keypoints.

$L_{total}=\sum_{i,j,k}(\lambda_{cls}L_{cls}+\lambda_{box}L_{box}+\lambda_{tsk}L_{tsk})$
(6)

###  3.2 Metrics

We evaluate our system based on two primary aspects: 6D pose estimation and
state detection. For the pose prediction we report the absolute translation
($e_{t}$), rotation error $e_{r}$ and Average Distance Error (ADD)(S) [11].
ADD(S) describes the average distance error for unsymmetric (ADD) and
symmetric (ADD(S)):

$e_{\acs{add}}=\frac{1}{n_{v}}\sum_{i=1}^{n_{v}}||(\mathbf{R}_{pred}\mathbf{x}+\mathbf{T}_{pred})-(\mathbf{R}_{gt}\mathbf{x}+\mathbf{T}_{gt})||,$
(7)

$e_{\acs{add}\textnormal{-}S}=\frac{1}{n_{\textrm{v}}}\sum_{i=1}^{n_{\textrm{v}}}\min_{j\in[n_{\textrm{v}}]}||(\mathbf{R}_{pred}\mathbf{x}_{i}+\mathbf{T}_{pred})-(\mathbf{R}_{gt}\mathbf{x}_{j}+\mathbf{T}_{gt})||$
(8)

$\mathbf{x}$ is a vertex point from the 3D object mesh, $n_{\textrm{v}}$
represent the number of vertices of the object mesh.

Following the definition of Hodan et al. [13], we calculate the translation
error according to:

$e_{t}=||\mathbf{T}_{\small{gt}}-\mathbf{T}_{\small{pred}}||_{2}$ (9)

Where $e_{t}$ represents the error between the ground truth translation
${t}_{\small{gt}}$ and the predicted translation ${t}_{\small{pred}}$.

The rotation error ($e_{R}$) is calculated according to the following formula:

$e_{R}=\frac{1}{n_{f}}\sum_{i=1}^{n_{f}}||arccos(\frac{tr(\mathbf{R}_{pred}\mathbf{R}_{gt}^{T}))-1}{2})||$
(10)

$\mathbf{R}_{gt}$ denotes the ground truth rotation and $\mathbf{T}_{gt}$ the
ground truth translation. $\mathbf{R}_{pred}$ describes the predicted rotation
and $\mathbf{T}_{pred}$ the predicted rotation.

For the sate detection we calculate the F1 score. The F1 score consist of
calculation Precision and Recall for each state ($s$) and each assembly part
($c$).

$F1=\frac{2\times\small{Precision}_{s/c}\times\small{Recall}_{s/c}}{\small{Precision}_{s/c}+\small{Recall}_{s/c}}$
(11)

Since we target AR to utilize our state and pose detection, we additionally
evaluate the runtime. For the runtime we calculated the mean over all assembly
objects.

Figure 3: Evaluation images of synthetic images (left) and real-world images (right). The ground truth is visualized with a pink overlay. In the synthetic dataset, the hand occlusion mimics similar occlusion compared to the real-world capturing. Figure 4: Example renderings including predictions on the ASDF test set. The ground truth (left, green), the pure YOLOv8-based pose and state prediction (center, yellow) and our prediction using ASDF (right, blue). The current predicted state is denoted in every top-left corner and the pose is shown with a pink overlay. Table 2: Results on the ASDF dataset: The ADD(S)($\uparrow$) is calculated with $10cm$ threshold and with XX, the translation error is in milimeters (mm) ($e_{ave\\_{trans}}$ denoted as $e_{trans}$, $\downarrow$) and the rotation error ($e_{ave\\_rot}$ denoted as $e_{rot}$, $\downarrow$) in degrees. The best results among all methods are labeled in bold. We compare YOLOv8Pose using deep learning-based state detection with ASDF. To report the state performance we report the F1 score.  |  YOLOv8Pose + Assembly Classes | ASDF
---|---|---
Assembly | F1 $\uparrow$ |  $\acs{add}(S)$ $\uparrow$ |  $e_{trans}$ [mm] $\downarrow$ |  $e_{rot}[^{\circ}]$ $\downarrow$ | F1 $\uparrow$ |  $\acs{add}(S)$ $\uparrow$ |  $e_{trans}$ [mm] $\downarrow$ |  $e_{rot}[^{\circ}]$ $\downarrow$
Corner Clamp | 87.88 | 89.22 | 15.74 | 8.51 | 93.85 | 95.69 | 5.39 | 8.76
Screw Clamp | 73.28 | 85.53 | 23.07 | 22.48 | 84.35 | 95.02 | 9.12 | 20.68
Geared Caliper | 58.77 | 81.11 | 22.92 | 16.69 | 60.91 | 96.33 | 4.65 | 17.03
Nano Vise | 75.45 | 83.82 | 21.88 | 5.04 | 78.95 | 95.92 | 7.63 | 8.22
Mean | 73.85 | 84.92 | 20.90 | 13.18 | 79.52 | 95.74 | 6.70 | 13.67

###  3.3 Datasets

Table 3: Ablation study on the impact of pose refinement steps and runtime.
The best performing approach per category is bold. We compare our keypoint-
based ASDF with and additional segmentation network and the commonly used ICP.
The performance is evaluated on the ASDF dataset.

Method | Network | Runtime [ms] | etrans [mm] | erot [°]
---|---|---|---|---
No refinement |  |  |
YOLOv8Pose + state | n | 24.83 | 29.13 | 18.76
YOLOv8Pose + state | xl-p6 | 50.83 | 19.11 | 11.14
Translation refinement |  |  |
ASDF-mini | n | 32.45 | 8.13 | 18.76
ASDF \+ Seg | n | 64.46 | 8.52 | 18.76
ASDF \+ Seg | xl-p6 | 85.03 | 6.53 | 11.14
ICP refinment |  |  |  |
ASDF \+ ICP | n | 151.77 | 12.27 | 18.96
ASDF +ICP | xl-p6 | 145.76 | 9.24 | 14.35
Ours |  |  |
ASDF | xl-p6 | 55.70 | 5.98 | 11.14

Table 4: Ablation study on the impact the underlying network size. We compare the individual network sizes considering their runtime and pose translation and rotation error for the ASDF dataset. Network | Runtime [ms] | etrans [mm] | erot [°]
---|---|---|---
n | 24.83 | 29.13 | 18.76
m | 33.72 | 27.85 | 15.21
l | 35.55 | 23.69 | 13.09
xl-p6 | 50.83 | 19.11 | 11.14

##### GBOT Dataset:

The GBOT dataset [19] is a exclusively a synthetic dataset designed for 6D
object pose/tracking tasks. It comprises 3D printable assembly parts and
contains five evaluation sequences, each presenting different levels of
difficulty for object tracking (normal, dynamic, hand occlusion, and blur).
Utilizing the GBOT dataset, we conduct a comprehensive comparison of our
approach against state-of-the-art methods in 6D pose estimation performance,
particularly focusing on assemblies with dynamically changing states.

##### ASDF Dataset:

The ASDF dataset provides ground truth annotations for 6D object poses and
assembly states of 3D printing parts. It includes four distinct assembly parts
and varying numbers of images as outlined in Table 1. Within the context of
the ASDF dataset, we assess the performance trade-offs, evaluate the
effectiveness of our pose refinement module, and analyze the accuracy of our
assembly state detection approach.

###  3.4 ASDF Results

####  3.4.1 Performance Tradeoff

Since ASDF leverages object pose information to refine the state, our initial
focus lies on evaluating the pose performance, specifically concentrating on
rotation and translation errors. As depicted in Table 4, it is evident that
the fastest network performance can be attained using the smallest network
configuration, which is a rational expectation. Notably, networks m and l
exhibit similar performance, whereas the xl-p6 architecture, serving as the
underlying framework, demonstrates the most favorable results in terms of
rotation and translation errors.

####  3.4.2 Pose Refinement

As n shows the best Runtime and xl-p6 the best performance in Table 4, we
compare these two networks using different refinement methods (refinement w/o
seg, w seg and ICP).

The improvement of our translation refinement compared to the pure network-
based output can be seen in Figure 6. In terms of numbers, the result can be
seen in in Table 4.

ASDF builds upon the use of keypoints for the 6D pose estimation. However, we,
considered other refinement methods as well to lift the pose performance.
Since semantic segmentation provides clearer object boundaries, we compared
our keypoint-based approach using a refinement step with a second YOLOv8
semantic segmentation network.

A common refinement step in 6D pose estimation is the use of ICP. As shown in
Table 3 neither the additional segmentation network nor ICP-based pose
refinement can outperform ASDF.

####  3.4.3 Assembly State Detection

As shown in Table 4 and Table 3, ASDF shows promising results in terms of 6D
pose estimation. However, the main focus of ASDF is the 6D pose estimation and
assembly state detection. Therefore, we compare pure deep learning-based 6D
pose estimation and ASDF on the ASDF dataset.

Comparing the assembly state detection using the F1 score, see Table 2, shows
that ASDF can predict the assembly state better (79.52) compared to a pure
deep learning-based approach (73.85). Moreover, as denoted in Table 2, the 6D
pose performance shows an improved performance by outperforming the deep
learning-based approach in ADD(S) (95.74 (ours) compare to 84.92). Moreover,
for the translation error we can reduce the average error by 14.2 mm.

For qualitative analysis we selected images from our test set, see Figure 4.
It becomes apparent that, the state detection of ASDF is more precise compared
to the deep learning-based approach. As shown in Figure 4 on different
assemblies in different states the predicted poses are more accurate as well
for assembled (Screw Clamp) and unassembled (Nano Vise, Caliper) parts.

Moreover, we 3D printed the real-world assembly and manually annotated one
sequence. As shown in Figure 5, our approach allows the detect the correct
state during a real-world state transition.

Figure 5: Real-world capturing. Success case of our ASDF (right, blue arrow)
on a real-world capturing for the state transition. For comparing the real-
world performance we annotated the sequences using depth input from multiple
cameras. We can show that our approach (right) receives the state correct
after the transition step. The adapted YOLOv8Pose (center, yellow arrow)
struggles with the state transition compared to the ground truth (left, green
arrow). Figure 6: Example comparison of YOLOv8Pose and ASDF. YOLOv8Pose shows
an offset in 3D (left, yellow) while our translation refinement (right, blue)
can address this shift. Figure 7: Qualitative comparison of YOLOv8Pose [19],
GBOT [19] and ASDF (ours) on the GBOT dataset: We found that during assembly
the results of ASDF are better compared to GBOT (Screw Clamp). Due to the
improvement of pose refinement, ASDF performs better than YOLOv8Pose (Screw
Clamp, Nano Vise).  Table 5: Results on the GBOT dataset: We compare our
approach on the GBOT Benchmark with a pure deep learning-based approach, the
combined approach of deep learning and object tracking (GBOT \+ re-init) and
pure tracking-based approaches. Rotational errors are only evaluated for
unsymmetrical objects. The ADD(S)($\uparrow$) is calculated with $10cm$
threshold, the translation error is in centimeters (cm) ($e_{ave\\_{trans}}$
denoted as $e_{trans}$, $\downarrow$) and the rotation error ($e_{ave\\_rot}$
denoted as $e_{rot}$, $\downarrow$) in degrees. The best results among all
methods are labeled in bold. In this evaluation, tracking is initialized with
ground truth pose only for the first frame and the pose is not reinitialized
afterwards. In the last column, the tracking results of tracking re-
initialization by pose estimation are shown.

Approach | 6D pose estimation | Tracking | 6D pose estimation + tracking
---|---|---|---
Asset | Condition |  |  |  |  |  |  |
ASDF (ours) |  YOLOv8Pose [19] | SRT3D [33] | ICG [35] | ICG+SRT3D [21] | GBOT [19] | GBOT + re-init [19]
$\acs{add}(S)$ | $e_{trans}$ | $e_{rot}$ | $\acs{add}(S)$ | $e_{trans}$ | $e_{rot}$ | $\acs{add}(S)$ | $e_{trans}$ | $e_{rot}$ | $\acs{add}(S)$ | $e_{trans}$ | $e_{rot}$ | $\acs{add}(S)$ | $e_{trans}$ | $e_{rot}$ | $\acs{add}(S)$ | $e_{trans}$ | $e_{rot}$ | $\acs{add}(S)$ | $e_{trans}$ | $e_{rot}$
| Normal | 100.0 | 1.8 | 3.0 | 91.5 | 9.3 | 3.8 | 89.8 | 4.7 | 25.8 | 100.0 | 0.1 | 38.7 | 100.0 | 1.3 | 38.7 | 100.0 | 0.6 | 2.1 | 99.5 | 0.7 | 2.7
Corner | Dynamic | 100.0 | 1.8 | 2.7 | 99.0 | 2.5 | 4.8 | 88.4 | 3.7 | 25.0 | 100.0 | 1.5 | 46.8 | 100.0 | 2.0 | 47.8 | 100.0 | 0.6 | 23.2 | 100.0 | 0.5 | 3.5
Clamp | Hand | 68.6 | 8.3 | 38.4 | 45.4 | 54.1 | 97.1 | 66.6 | 8.8 | 37.1 | 68.4 | 7.4 | 59.4 | 68.4 | 7.5 | 58.3 | 81.9 | 5.7 | 90.4 | 90.6 | 4.4 | 84.7
| Blur | 100.0 | 1.9 | 3.4 | 97.3 | 4.0 | 4.3 | 88.9 | 5.0 | 30.4 | 100.0 | 0.5 | 2.2 | 100.0 | 1.0 | 3.0 | 100.0 | 0.6 | 2.1 | 99.9 | 0.6 | 2.1
| Normal | 100.0 | 0.4 | 4.7 | 99.6 | 1.4 | 10.1 | 90.6 | 1.8 | 5.4 | 100.0 | 0.2 | 2.6 | 100.0 | 0.3 | 3.0 | 100.0 | 0.2 | 2.1 | 100.0 | 0.5 | 3.3
Geared | Dynamic | 100.0 | 0.3 | 4.1 | 99.9 | 1.3 | 9.3 | 92.7 | 1.4 | 9.9 | 100.0 | 0.2 | 2.5 | 100.0 | 0.4 | 3.6 | 100.0 | 0.2 | 2.3 | 100.0 | 0.5 | 3.6
Caliper | Hand | 100.0 | 0.8 | 6.9 | 99.2 | 2.4 | 12.5 | 96.5 | 0.9 | 4.4 | 85.4 | 3.0 | 30.0 | 85.5 | 3.1 | 30.0 | 85.4 | 3.0 | 30.0 | 99.6 | 0.8 | 7.0
| Blur | 100.0 | 0.4 | 4.9 | 99.6 | 1.4 | 9.9 | 98.9 | 0.9 | 8.7 | 100.0 | 0.2 | 2.5 | 100.0 | 0.3 | 2.8 | 100.0 | 0.2 | 2.2 | 100.0 | 0.5 | 3.6
| Normal | 100.0 | 1.4 | 4.8 | 71.1 | 8.8 | 4.8 | 74.1 | 6.6 | 13.8 | 89.8 | 2.1 | 16.7 | 89.4 | 2.3 | 16.5 | 99.8 | 0.6 | 7.2 | 93.8 | 2.4 | 3.6
Nano | Dynamic | 90.0 | 1.6 | 5.0 | 72.6 | 7.6 | 4.6 | 63.4 | 10.3 | 15.6 | 87.3 | 3.9 | 15.3 | 87.3 | 4.1 | 15.8 | 96.0 | 2.4 | 20.0 | 92.9 | 2.5 | 3.7
Vise | Hand | 98.6 | 1.9 | 5.3 | 65.9 | 11.0 | 4.7 | 61.4 | 13.6 | 15.3 | 76.5 | 6.0 | 18.3 | 75.8 | 6.1 | 18.1 | 72.9 | 8.7 | 14.5 | 87.8 | 3.1 | 7.3
| Blur | 100.0 | 1.5 | 5.4 | 70.9 | 9.9 | 5.2 | 61.9 | 11.6 | 15.2 | 91.6 | 1.9 | 11.4 | 91.5 | 2.1 | 11.1 | 95.7 | 0.7 | 25.1 | 92.7 | 3.0 | 4.7
| Normal | 93.5 | 1.1 | 8.6 | 73.5 | 7.7 | 17.6 | 86.5 | 4.7 | 8.9 | 96.0 | 1.2 | 1.1 | 95.9 | 1.4 | 2.3 | 98.8 | 0.4 | 0.9 | 83.7 | 3.0 | 4.7
Screw | Dynamic | 94.5 | 1.1 | 9.4 | 82.0 | 6.7 | 18.4 | 86.4 | 5.6 | 27.0 | 95.9 | 1.7 | 2.1 | 95.9 | 2.2 | 3.4 | 98.8 | 0.6 | 1.7 | 91.6 | 2.1 | 6.1
Clamp | Hand | 92.1 | 1.8 | 13.5 | 67.2 | 14.2 | 39.8 | 60.1 | 14.3 | 37.7 | 73.4 | 6.9 | 53.4 | 73.1 | 7.1 | 53.4 | 68.7 | 7.9 | 61.9 | 83.9 | 4.9 | 27.2
| Blur | 96.1 | 0.9 | 8.7 | 85.6 | 6.5 | 18.6 | 86.5 | 5.6 | 34.6 | 95.7 | 3.0 | 6.6 | 95.7 | 3.2 | 7.7 | 98.6 | 1.2 | 1.1 | 91.1 | 3.0 | 4.8
Mean | Overall | 95.8 | 1.7 | 8.1 | 82.52 | 9.3 | 16.59 | 80.8 | 6.2 | 19.7 | 91.3 | 2.5 | 19.4 | 91.2 | 2.8 | 19.7 | 93.5 | 2.1 | 17.9 | 94.2 | 2.0 | 10.8

###  3.5 GBOT Results

Since our system addresses 6D object pose estimation and assembly-state
detection existing approaches face the limitation of being either used for
state detection combined with 2D object detection [31] or are limited to non
accessible datasets [24, 37]. However, in terms of 6D pose estimation and 6D
object tracking some approaches can handle 6D pose estimation for assembled
parts.

Given that we utilize the same objects as Li et al. [19], we assess our
trained models from the ASDF dataset on the GBOT dataset without the need for
retraining on their training set. As illustrated in Table 5, our approach
demonstrates superior performance compared to GBOT and their fusion of GBOT
with deep learning-based re-initialization of object tracking. Moreover, in
comparison to state-of-the-art tracking methods, Li et al. [19] introduce a
deep learning-based approach for 6D pose estimation, which shares the same
base architecture as our method. The outcomes presented in Table 5 shows that
our approach, with integrated state detection during both training and
inference, achieves better results than a pure deep learning-based approach.

Furthermore, we conduct a visual comparison between our approach and GBOT,
including their initialization network YOLOv8Pose, as depicted in Figure 7.
The qualitative and quantitative analysis reveals that ASDF outperforms their
deep learning-based and hybrid approach. Particularly, in the assembly
process, ASDF exhibits notable improvements compared to GBOT.

Figure 8: Fail case during state transition. Example of a potential fail case
during state transition. The frames during the step between two states
(transition) are prone to state prediction errors.

##  4 Discussion

ASDF uses deep learning-based 6D position estimation and assembly state
detection. Our approach shows more robust results compared to a pure deep
learning-based approach on our ASDF and the GBOT dataset. Robust 6D pose
estimation and assembly state detection allow adding visual overlays, e.g. see
Figure 7 or even highlighting the current assembly state , see Figure 4. The
more robust and correct the prediction is, the more reliably an AR interface
can display this information.

Our evaluation has shown that combining object pose prediction with assembly
state detection can lead to better results in 6D position estimation. However,
the best performing approach is not always the most runtime friendly.

The assembly processes are complicated by occlusions and state changes.
Previous work aimed at constant 6D position estimation of mounted objects
using object tracking. However, they lack the additional state information
[34, 19]. In addition, these approaches would have to reinitialize the
tracking process in the event of occlusion. GBOT proposes the use of re-
initialization with a real-time capable architecture. As the comparison with
GBOT shows, we can even outperform their re-initialization approach in terms
of 6D position estimation performance. This makes our work valuable in real-
world scenarios.

The state transition, i.e. the change from state A to B, is a frequent
challenge not only in the detection of assembly states. However, it is
generally a challenge in phase detection or action detection [6]. A common
approach in this regard, which is not considered in our evaluation, is to
define a time frame between transitions, which is not considered in the
evaluation. We did not do this as we were aiming for a real-world scenario
where constant knowledge of the assembly state and object position is crucial.
Therefore, we included these potentially error-prone parts in our dataset. As
shown in Figure 8, ASDF is challenged by this transition, but can retrieve
more robust results with our Pose2State module compared to the pure deep
learning-based approach.

###  4.1 Limitations

In terms of performance we aimed for the highest accuracy. However, for the
runtime this adds some additional overhead.

The current approach focuses on reliable pose estimations and assembly state
detection using a single camera. To address occlusion a multi-camera approach
or even fusing static and dynamic camera input could provide additional
information for the estimation.

##  5 Conclusion

ASDF is a deep learning-based approach that utilizes late fusion of assembly
state and 6D pose estimation to enhance its prediction. Our approach enables
smart guidance as it can continuously track the 6D object pose and state of
several assembly objects. This can be utilized in the medical or industry
context. ASDF can track objects from assembly state zero (unassembled) until
the full assembly is completed. Moreover, it can better detect state changes
compared to a pure deep learning-based approach using the Pose2State module.

In addition to our assembly dataset, where we outperform the pure deep
learning-based approach, ASDF has demonstrated that awareness of the assembly
state leads to an improved performance compared to the state-of-the-art on the
GBOT dataset. The scenes in these datasets present various challenges, and
ASDF has shown significant improvement compared to YOLOv8Pose, even
outperforming GBOT.

In conclusion, our approach and dataset represent a promising step towards
developing comparable backends for smart AR guidance in assembly processes.

## References

  * [1] R. Alghonaim and E. Johns.  Benchmarking domain randomisation for visual sim-to-real transfer.  In IEEE International Conference on Robotics and Automation (ICRA), 2021.
  * [2] A. Amini, A. Selvam Periyasamy, and S. Behnke.  YOLOPose: Transformer-based multi-object 6d pose estimation using keypoint regression.  In Intelligent Autonomous Systems 17: Proceedings of the 17th International Conference IAS-17, pp. 392–406. Springer, 2023.
  * [3] S. Borkman, A. Crespi, S. Dhakad, S. Ganguly, J. Hogins, Y.-C. Jhang, M. Kamalzadeh, B. Li, S. Leal, P. Parisi, C. Romero, W. Smith, A. Thaman, S. Warren, and N. Yadav.  Unity perception: Generate synthetic data for computer vision, 2021.
  * [4] E. Bottani and G. Vignali.  Augmented reality technology in the manufacturing industry: A review of the last decade.  IISE Transactions, 51(3):284–310, 2019. doi: 10 . 1080/24725854 . 2018 . 1493244
  * [5] Y.-W. Chao, W. Yang, Y. Xiang, P. Molchanov, A. Handa, J. Tremblay, Y. S. Narang, K. Van Wyk, U. Iqbal, S. Birchfield, and others.  DexYCB: A benchmark for capturing hand grasping of objects.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9044–9053, 2021.
  * [6] K. C. Demir, H. Schieber, T. Weise, D. Roth, M. May, A. Maier, and S. H. Yang.  Deep learning in surgical workflow analysis: a review of phase and step recognition.  IEEE Journal of Biomedical and Health Informatics, 2023.
  * [7] M. Denninger, M. Sundermeyer, D. Winkelbauer, D. Olefir, T. Hodan, Y. Zidan, M. Elbadrawy, M. Knauer, H. Katam, and A. Lodhi.  BlenderProc: Reducing the reality gap with photorealistic rendering.  In Robotics: Science and Systems (RSS) Workshops, 2020.
  * [8] M. Eswaran, A. K. Gulivindala, A. K. Inkulu, and M. V. A. R. Bahubalendruni.  Augmented reality-based guidance in product assembly and maintenance/repair perspective: A state of the art review on challenges and opportunities.  Expert Systems with Applications, 2023.
  * [9] A. Gupta, D. Fox, B. Curless, and M. Cohen.  DuploTrack: a real-time system for authoring and guiding duplo block assembly.  In Proceedings of the 25th annual ACM symposium on User interface software and technology, UIST ’12, pp. 389–402. Association for Computing Machinery, 2012. doi: 10 . 1145/2380116 . 2380167
  * [10] Y. He, H. Haibin, F. Haoqiang, C. Qifeng, and S. Jian.  Ffb6d: A full flow bidirectional fusion network for 6d pose estimation.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
  * [11] S. Hinterstoisser, V. Lepetit, S. Ilic, S. Holzer, G. Bradski, K. Konolige, and N. Navab.  Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes.  In Computer Vision–ACCV 2012: 11th Asian Conference on Computer Vision, Daejeon, Korea, November 5-9, 2012, Revised Selected Papers, Part I 11, pp. 548–562. Springer, 2013.
  * [12] T. Hodan, P. Haluza, S. Obdrzalek, J. Matas, M. Lourakis, and X. Zabulis.  T-LESS: An RGB-d dataset for 6d pose estimation of texture-less objects.  IEEE Winter Conference on Applications of Computer Vision (WACV), 2017.
  * [13] T. Hodan, J. Matas, and S. Obdrzalek.  On evaluation of 6d object pose estimation.  vol. 9915, pp. 606–619. Springer International Publishing, 2016. doi: 10 . 1007/978-3-319-49409-8_52
  * [14] T. Hodan, M. Sundermeyer, B. Drost, Y. Labbe, E. Brachmann, F. Michel, C. Rother, and J. Matas.  BOP challenge 2020 on 6d object localization.  European Conference on Computer Vision Workshops (ECCVW), 2020\.
  * [15] T. Jantos, M. Hamdad, W. Granig, S. Weiss, and J. Steinbrener.  PoET: Pose estimation transformer for single-view, multi-object 6d pose estimation.  In 6th Annual Conference on Robot Learning (CoRL 2022), 2022.
  * [16] H. Jung, S.-C. Wu, P. Ruhkamp, G. Zhai, H. Schieber, G. Rizzoli, P. Wang, H. Zhao, L. Garattoni, S. Meier, D. Roth, N. Navab, and B. Busam.  HouseCat6d – a large-scale multi-modal category level 6d object pose dataset with household objects in realistic scenarios, 2023.
  * [17] R. Kaskman, S. Zakharov, I. Shugurov, and S. Ilic.  HomebrewedDB: RGB-d dataset for 6d pose estimation of 3d objects, 2019\.
  * [18] C. Kleinbeck, H. Schieber, S. Andress, C. Krautz, and D. Roth.  ARTFM: Augmented reality visualization of tool functionality manuals in operating rooms.  In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 736–737. IEEE, 2022.
  * [19] S. Li, H. Schieber, N. Corell, B. Egger, J. Kreimeier, and D. Roth.  GBOT: Graph-based 3d object tracking for augmented reality-assisted assembly guidance.
  * [20] Y. Li, G. Wang, X. Ji, Y. Xiang, and D. Fox.  Deepim: Deep iterative matching for 6d pose estimation.  In Proceedings of the European Conference on Computer Vision (ECCV), pp. 683–698, 2018.
  * [21] Y. Li, F. Zhong, X. Wang, S. Song, J. Li, X. Qin, and C. Tu.  For a more comprehensive evaluation of 6dof object pose tracking.  2023\.
  * [22] H. Liu, Y. Su, J. Rambach, A. Pagani, and D. Stricker.  Tga: Two-level group attention for assembly state detection.  In 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pp. 258–263. IEEE, 2020.
  * [23] F. Manuri, A. Pizzigalli, and A. Sanna.  A state validation system for augmented reality based maintenance procedures.  Applied Sciences, 9(10):2115, 2019. doi: 10 . 3390/app9102115
  * [24] K. Murray, J. Schierl, K. Foley, and Z. Duric.  Equipment assembly recognition for augmented reality guidance.  pp. 109–118, 2024.
  * [25] K. Park, T. Patten, and M. Vincze.  Pix2pose: Pixel-wise coordinate regression of objects for 6d pose estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7668–7677, 2019.
  * [26] S. Peng, Y. Liu, Q. Huang, X. Zhou, and H. Bao.  Pvnet: Pixel-wise voting network for 6dof pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4561–4570, 2019.
  * [27] W. Qiu and A. Yuille.  UnrealCV: Connecting computer vision to unreal engine.  In European Conference on Computer Vision Workshops (ECCVW), pp. 909–916, 2016.
  * [28] J. Rambach, A. Pagani, M. Schneider, O. Artemenko, and D. Stricker.  6dof object tracking based on 3d scans for augmented reality remote live support.  Computers, 7(1):6, 2018.
  * [29] H. Schieber, K. C. Demir, C. Kleinbeck, S. H. Yang, and D. Roth.  Indoor synthetic data generation: A systematic review.  Computer Vision and Image Understanding, p. 103907, 2024. doi: 10 . 1016/j . cviu . 2023 . 103907
  * [30] T. J. Schoonbeek, T. Houben, H. Onvlee, F. van der Sommen, et al.  Industreal: A dataset for procedure step recognition handling execution errors in egocentric videos in an industrial-like setting.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4365–4374, 2024.
  * [31] A. Stanescu, P. Mohr, M. Kozinski, S. Mori, D. Schmalstieg, and D. Kalkofen.  State-aware configuration detection for augmented reality step-by-step tutorials.  In 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 157–166. IEEE, 2023.
  * [32] M. Stoiber, M. Elsayed, A. E. Reichert, F. Steidle, D. Lee, and R. Triebel.  Fusing visual appearance and geometry for multi-modality 6dof object tracking, 2023.  Publication Title: arXiv preprint arXiv:2302.11458.
  * [33] M. Stoiber, M. Pfanne, K. H. Strobl, R. Triebel, and A. Albu-Schaeffer.  SRT3d: A sparse region-based 3d object tracking approach for the real world.  International Journal of Computer Vision, 130(4):1008–1030, 2022\.
  * [34] M. Stoiber, M. Sundermeyer, W. Boerdijk, and R. Triebel.  A multi-body tracking framework–from rigid objects to kinematic structures, 2022.  Publication Title: arXiv preprint arXiv:2208.01502.
  * [35] M. Stoiber, M. Sundermeyer, and R. Triebel.  Iterative corresponding geometry: Fusing region and depth for highly efficient 3d tracking of textureless objects.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [36] Y. Su, M. Liu, J. Rambach, A. Pehrson, A. Berg, and D. Stricker.  IKEA object state dataset: A 6dof object pose estimation dataset and benchmark for multi-state assembly objects, 2021.  Publication Title: arXiv preprint arXiv:2111.08614.
  * [37] Y. Su, J. Rambach, N. Minaskan, P. Lesur, A. Pagani, and D. Stricker.  Deep multi-state object pose estimation for augmented reality assembly.  In 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pp. 222–227, 2019. doi: 10 . 1109/ISMAR-Adjunct . 2019 . 00-42
  * [38] B. Tekin, S. N. Sinha, and P. Fua.  Real-time seamless single shot 6d object pose prediction.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [39] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel.  Domain randomization for transferring deep neural networks from simulation to the real world.  In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23–30, 2017. doi: 10 . 1109/IROS . 2017 . 8202133
  * [40] J. Tremblay, A. Prakash, D. Acuna, M. Brophy, V. Jampani, C. Anil, T. To, E. Cameracci, S. Boochoon, and S. Birchfield.  Training deep networks with synthetic data: Bridging the reality gap by domain randomization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2018.
  * [41] G. Wang, F. Manhardt, F. Tombari, and X. Ji.  Gdr-net: Geometry-guided direct regression network for monocular 6d object pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16611–16621, 2021.
  * [42] H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and L. J. Guibas.  Normalized object coordinate space for category-level 6d object pose and size estimation.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [43] P. Wang, H. Jung, Y. Li, S. Shen, R. P. Srikanth, L. Garattoni, S. Meier, N. Navab, and B. Busam.  PhoCaL: A multi-modal dataset for category-level object pose estimation with photometrically challenging objects.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21222–21231, 2022.
  * [44] R. Wang, Y. Zhang, J. Mao, R. Zhang, C.-Y. Cheng, and J. Wu.  IKEA-manual: Seeing shape assembly step by step.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, eds., Advances in Neural Information Processing Systems, vol. 35, pp. 28428–28440. Curran Associates, Inc., 2022.
  * [45] L.-C. Wu, I.-C. Lin, and M.-H. Tsai.  Augmented reality instruction for object assembly based on markerless tracking.  In Proceedings of the 20th ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, I3D ’16, pp. 95–102. Association for Computing Machinery, 2016. doi: 10 . 1145/2856400 . 2856416
  * [46] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox.  Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes, 2017.  Publication Title: arXiv preprint arXiv:1711.00199.
  * [47] J. Zauner, M. Haller, A. Brandl, and W. Hartman.  Authoring of a mixed reality assembly instructor for hierarchical structures.  In The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings., pp. 237–246, 2003. doi: 10 . 1109/ISMAR . 2003 . 1240707
  * [48] B. Zhou and S. Güven.  Fine-grained visual recognition in mobile augmented reality for technical support.  IEEE Transactions on Visualization and Computer Graphics, 26(12):3514–3523, 2020. doi: 10 . 1109/TVCG . 2020 . 3023635

  *[AR]: Augmented Reality
  *[CNN]: convolutional neural network
  *[ASDF]: Assembly State Detection utilizing late Fusion
  *[YOLO]: you only look once
  *[GBOT]: Graph-based Object Tracking
  *[PnP]: Perspective-n-Point
  *[ICG]: Iterative Correspondence Geometry
  *[FPS]: Farthest Point Sampling
  *[ADD]: Average Distance Error
  *[ICP]: Iterative Closest Point