# Generative Category-Level Shape and Pose Estimation with Semantic Primitives

Guanglin Li1,2 Yifeng Li2 Zhichao Ye1 Qihang Zhang3 Tao Kong2 Zhaopeng Cui1
Guofeng Zhang1
1State Key Lab of CAD&CG, Zhejiang University 2ByteDance AI Lab 3Multimedia
Laboratory, The Chinese University of Hong Kong

###### Abstract

Empowering autonomous agents with 3D understanding for daily objects is a
grand challenge in robotics applications. When exploring in an unknown
environment, existing methods for object pose estimation are still not
satisfactory due to the diversity of object shapes. In this paper, we propose
a novel framework for category-level object shape and pose estimation from a
single RGB-D image. To handle the intra-category variation, we adopt a
semantic primitive representation that encodes diverse shapes into a unified
latent space, which is the key to establish reliable correspondences between
observed point clouds and estimated shapes. Then, by using a SIM(3)-invariant
shape descriptor, we gracefully decouple the shape and pose of an object, thus
supporting latent shape optimization of target objects in arbitrary poses.
Extensive experiments show that the proposed method achieves SOTA pose
estimation performance and better generalization in the real-world dataset.
Code and video are available at https://zju3dv.github.io/gCasp.

††footnotetext:  Work done during Guanglin Li’s internship at ByteDance.

> Keywords: Category-level Pose Estimation, Shape Estimation

##  1 Introduction

Estimating the shape and pose for daily objects is a fundamental function
which has various applications, including 3D scene understanding, robot
manipulation and autonomous warehousing [1, 2, 3, 4, 5, 6]. Early works for
this task are focused on instance-level pose estimation [7, 8, 9, 10, 11],
which aligns the observed object with the given CAD model. However, such a
setting is limited in real-world scenarios since it is hard to obtain the
exact model of a casual object in advance. To generalize to these unseen but
semantically familiar objects, category-level pose estimation is raising more
and more research attention [12, 13, 14, 15, 16, 17] since it could
potentially handle various instances of the same category in real scenes.

Existing works on category-level pose estimation usually try to predict pixel-
wise normalized coordinates for instances within one class [12] or adopt a
canonical prior model with shape deformations to estimate object poses [14,
15]. Although great advances have been made, these one-pass prediction methods
are still faced with difficulties when large shape differences exist within
the same category. To handle the variety of intra-class objects, some works
[18, 16] leverage neural implicit representation [19] to fit the shape of the
target object by iteratively optimizing the pose and shape in a latent space,
and achieve better performance. However, in such methods, the pose and shape
estimation are coupled together and rely on each other to get reliable
results. Thus, their performance is unsatisfactory in real scene (see Sec.
4.2). So we can see that the huge intra-class shape differences and coupled
estimation of shapes and poses are two main challenges for the existing
category-level pose estimation methods.

To tackle these challenges, we propose to estimate the object poses and shapes
with semantic primitives from a generative perspective. The insight behind
this is that the objects of a category are often composed of components with
the same semantics although their shapes are various, _e.g._ , a cup is
usually composed of a semicircular handle and a cylindrical body (see Fig.
1(a)). This property inspired us to handle huge differences in intra-class
shapes by fusing this category-level semantic information. Specifically, we
propose to encode object shapes into semantic primitives, which construct a
unified semantic representation among objects in the same category (see Fig.
1(b)). Following Hao et al. [20], we map diverse shapes and the corresponding
semantic primitives into the same shared latent space of a generative model.
In this way, we can obtain the correspondence between the observed point cloud
and the latent space by dividing the observed point cloud into several
semantic primitives with a part segmentation network.

In order to disentangle the shape and pose estimation, we further extract a
novel SIM(3)-invariant descriptor from the geometric distribution of the point
cloud’s semantic primitives. By leveraging such SIM(3)-invariant descriptors,
we can perform latent optimization of object shapes in arbitrary poses and
obtain the resulting shape consistent with the observation. After obtaining
the estimated shape, we use a correspondence-based algorithm to recover the
size and pose by aligning semantic primitives of the target object with that
of the estimated shape.

In summary, our contributions are as follows: 1) We propose a novel category-
level pose estimation method, which uses semantic primitives to model diverse
shapes and bridges the observed point clouds with implicitly generated shapes.
2) We propose an online shape estimation method, which enables object shape
optimization in arbitrary poses with a novel SIM(3)-invariant descriptor. 3)
Even when training on the synthetic dataset only, our method shows good
generalization and achieves state-of-the-art category-level pose estimation
performance on the real dataset.

Figure 1: (a) An example of different instances in the same categories, we
highlight the same semantic parts (_e.g._ , lens of cameras) between different
instances. (b) The detailed shapes (top row) are decomposed into semantic
primitives (bottom row). Different colors on primitives indicate different
semantic labels, which are consistent among different shapes. (c) An overview
of the generative model consisting of two auto-decoders (_i.e._ , $g_{f}$ and
$g_{c}$) sharing the same latent space. $g_{f}$ captures fine details, and
$g_{c}$ represents the shapes by simple and semantically consistent shape
primitives.

##  2 Related Works

Instance-Level Object Pose Estimation. Instance-level pose estimation assumes
the CAD model of test objects is known in the training phase, and previous
methods are mainly divided into three categories. The first category of
methods [21, 8, 7, 22] directly regress the 6D object pose from RGB or RGB-D
images. PoseCNN [7] extends 2D detection architecture to regress 6D object
pose from RGB images, and DenseFusion [8] combines color and depth information
from the RGB-D input and makes the regressed pose more accurate. The second
are correspondence-based methods [9, 10, 11, 23]. They predict the
correspondences between 2D images and 3D models, and then recover the pose by
Perspective-n-Point algorithm [24]. For example, PVNet [9] predicts the 3D
keypoints on the RGB image by a voting scheme. CDPN [11] predicts the dense
correspondences between image pixels and 3D object surface. The third category
is rendering-based methods [25, 26], which recover the pose by minimizing the
re-projection error between the posed 3D model and 2D image through a
differentiable render. Compared with these instance-level methods, our
category-level method does not need the known CAD model as prior.

Category-Level Object Pose Estimation. Category-level methods [12, 27, 13] aim
at estimating the pose for the arbitrary shape of the same category. To
overcome the intra-category variations, NOCS [12] introduces normalized object
canonical space, which establishes a unified coordinate space among instances
of the same category. CASS [27] learns a variational auto-encoder to recover
object shape and predicts the pose in an end-to-end neural network. These
methods directly regress the pose or coordinates, and thus struggle to form an
intra-category representation. Other methods utilize a prior model of each
category for the category-level representation. For example, SPD [14] and SGPA
[15] predict deformation on canonical prior models and recover the 6D pose
through correspondence between the deformed model and the observed point
cloud. However, when facing a significant difference between the prior model
and the target object, the deformation prediction tend to fail. DualPose [28]
takes the consistency between an implicit pose decoder and an explicit one
into account and refines the predicted pose during testing. Besides, FS-Net
[13] proposes a novel data augmentation mechanism to overcome intra-category
variations. SAR-Net [17] recovers the shape by modeling symmetric
correspondence of objects, which makes pose and size prediction easier.
Rencently, CenterSnap [29] jointly optimizes detection, reconstruction and
pose when training and achieves real-time inference performance. To utilize
the diverse and continuous shapes in generative models, iCaps[16] and
ELLIPSDF[18] introduce the generative models [19, 20] to their shape and pose
estimation. Shapes and poses are jointly optimized in these works. Unlike
these previous works, we establish a semantically consistent representation
and decouple the shape and pose estimation, which makes the optimization more
robust.

##  3 Method

Figure 2: Overview of the proposed method. The input of our method is the
point cloud observed from a single-view depth image. (a) Our method first
extracts semantic primitives $\tilde{\mathcal{C}}$ from object point cloud
$P_{0}$ by a part segmentation network $\phi$. (b) We calculate a
SIM(3)-invariant shape descriptor $f(\tilde{\mathcal{C}})$ from
$\tilde{\mathcal{C}}$ and optimize a shape embedding $\hat{z}$ in the latent
space by the difference between $f(\tilde{\mathcal{C}})$ and $f(g_{c}(z))$,
where $g_{c}$ and $g_{f}$ are the coarse and fine branches of the generative
model $g$, detailed in Sec. 3.1. (c) The similarity transformation
$\\{\tilde{s},\tilde{\bm{R}},\tilde{\bm{t}}\\}$ is recovered through the
semantic correspondences between $\tilde{\mathcal{C}}$ and optimized shape
$g_{c}(\hat{z})$. (d) Further, we can simply apply the transformation
$\\{\tilde{s},\tilde{\bm{R}},\tilde{\bm{t}}\\}$ on the fine geometry generated
through $g_{f}(\hat{z})$, and reconstruct the object-level scene as a by-
product besides our main results, which is detailed in the Sec. B in the supp.
material.

Problem Formulation. Given observed point cloud
$P_{0}=\\{\bm{p}_{i}|i=1,...,N_{0}\\}$ of an object with known category, our
goal is to estimate the complete shape, scale and 6D pose for this object. The
recovered shape is represented by a shape embedding $\hat{z}$ in a latent
space. The estimated scale and 6D pose are represented as a similarity
transformation $\\{s,\bm{R},\bm{t}\\}\in\text{SIM(3)}$, where scale
$s\in\mathbb{R}$, 3D rotation $\bm{R}\in\text{SO(3)}$ and 3D translation
$\bm{t}\in\mathbb{R}^{3}$. SIM(3) and SO(3) indicate the Lie group of 3D
similarity transformations and 3D rotation, respectively.

Overview. As illustrated in Fig 2, our method takes the RGB-D image as input.
A Mask-RCNN[30] network is used to obtain the mask and category of each object
instance. In the following stages, our method only processes the observed
instance point cloud back-projected from the masked depth image. To overcome
the huge intra-category variations, we utilize a unified intra-category
representation of semantic primitives (Sec 3.1). Then, we process the observed
point cloud with a part segmentation network to extract semantic primitives
which establish a connection between the observed object and the latent space
(Sec 3.2). To estimate the shape of the target with its pose unknown, we
design a SIM(3)-invariant descriptor to optimize the generative shape in the
latent space (Sec 3.3). Finally, we estimate the 6D pose and scale of the
object through the semantic correspondence between the observed point cloud
and our recovered generative shape (Sec 3.4).

###  3.1 Semantic Primitive Representation

The huge intra-category differences make it challenging to represent objects
with a standard model. To overcome this problem, we utilize a semantic
primitive representation. The insight behind the semantic primitive
representation is that although different instances in the same category have
various shapes, they tend to have similar semantic parts, _e.g._ , each camera
instance has a lens, and each laptop has four corners (see Fig 1(a)). In
semantic primitive representation, each instance is decomposed into several
parts, and each part always corresponds to the same semantic part in different
instances (see Fig 1(b)). To generate this representation, we learn a
generative model $g$ following the network structure in [20] and map instances
of the same category into a latent space.

Specifically, Fig 1(c) gives an overview of the generative model. The
generative model $g$ expresses shapes at two granular levels, one capturing
detailed shapes and the other representing an abstracted shape using
semantically consistent shape primitives. We note the fine-scale model as
$g_{f}$ and the coarse-scale one as $g_{c}$. $g_{f}$ and $g_{c}$ are two auto-
decoders sharing the same latent code $z$. $g_{f}(z,\bm{x})=SDF(\bm{x})$,
where $\bm{x}$ is a 3D position in canonical space and $SDF$ denotes the
signed distance function. $g_{c}(z)=\\{\bm{\alpha}_{i}|i=1,...,N_{c}\\}$,
where $N_{c}$ is the number of primitives and $\bm{\alpha}_{i}$ is the
parameters of a primitive. We use sphere primitives here and
$\bm{\alpha}=(\bm{c},r)$, where $\bm{c}$ is the 3D center of a sphere and $r$
is the radius. Please refer to [20] and Sec. A.3 in the supp. material for
more details.

###  3.2 Extract Semantic Primitives from Point Cloud

We use a part segmentation network to obtain the semantic primitives of the
observation point cloud in two stages. First, we predict a semantic label
$l_{i}$ for each point $\bm{p}_{i}$ in the observed point cloud $P_{0}$, and
then a centralization operation is performed on the points to extract the
center of each primitive.

Semantic Label Prediction. For each point $\bm{p}_{i}\in P_{0}$, its semantic
label $l_{i}$ is defined in terms of closest primitive center $\bm{c}_{j}$:

$\displaystyle l_{i}=\underset{j=1...N_{c}}{\rm
argmin}{||\bm{p}_{i}-\bm{c}_{j}||_{2}}.$ (1)

We treat point label prediction as a classification problem and follow the
part segmentation version of 3D-GCN [31] to perform point-wise classification.
Standard cross entropy is used as the loss function. In addition, because of
imperfect segmentation masks produced by Mask R-CNN, the point cloud back-
projected from the masked depth image may contain extra background points. For
these outlier points, we add another “dust” label in the point-wise
classification to filter out them.

Symmetric Loss. Many common objects in robot manipulation have symmetrical
structures, and most objects (_e.g., bottles_) are symmetrical about an axis.
The unique pre-labeled ground truth cause ambiguity when the object rotates
around the axis. In order to eliminate the influence of symmetric structure on
our classification task, we introduce the symmetric loss as the same as [12].
For each symmetric object in the dataset, we define a symmetric axis and make
the loss identical when the object rotates around the axis. Specifically, we
generate a separate ground truth for each possible angle of rotational
symmetry. Given the rotation angle $\theta$, the rotated semantic label
$l_{\theta,i}$ is defined as:

$l_{\theta,i}=\underset{j=1...N_{c}}{\rm
argmin}{||\bm{p}_{i}-\bm{c}_{\theta,j}||_{2}},$ (2)

where $\bm{c}_{\theta,j}$ is the position of $\bm{c}_{j}$ after $\theta$
degrees of rotation about the axis. Then we define the symmetric loss
$L_{sym}$ as:

$L_{sym}=\underset{\theta\in\Theta}{\min}~{}{L(\tilde{l}_{i},l_{\theta,i})}.$
(3)

where $L$ is the standard cross entropy, $\tilde{l}_{i}$ is the predicted
label by the network, and we set $\Theta=\\{0^{\circ}\\}$ for non-symmetric
objects and $\Theta=\\{i\cdot 60^{\circ}|i=0,...,5\\}$ for symmetric objects
in our experiments.

Primitive centralization. After predicting the semantic label of each 3D
point, we count the semantic labels
$\tilde{\mathcal{L}}=\\{l_{i}|i=1,...,\tilde{N}_{c}\\}$ which appear in the
partially observed point cloud. Then, we calculate the primitive centers
$\tilde{\mathcal{C}}=\\{\bm{c}_{l}|l=l_{1},...,l_{\tilde{N}_{c}}\\}$ from the
labeled points by averaging the points with the same label. Specifically,

$\displaystyle\bm{c}_{l}=\frac{1}{N_{l}}\sum_{i=1}^{N_{0}}\bm{p}_{i}\mathbb{I}(\tilde{l}_{i}=l).$
(4)

where $N_{l}$ is the total number of points with predicted label $l$,
$\mathbb{I}$ is an indication function indicating whether the predicted label
$\tilde{l}_{i}$ equals $l$.

Figure 3: The proposed atomic SIM(3)-invariant descriptor. The angle
$\theta_{1}$ and $\theta_{2}$, which we called atomic descriptors are
invariant with different object poses. And our shape descriptor concatenated
by multiple atomic descriptors is also invariant.

###  3.3 Shape Optimization with SIM(3)-Invariant Descriptor

We decouple the shape estimation from the pose estimation to perform shape
estimation for the observed object when the pose is unknown. In order to
achieve this pose-invariant shape optimization, we propose a novel
SIM(3)-invariant descriptor extracted from semantic primitive representation.

SIM(3)-invariant Descriptor. Given a set of 3D points
$\mathcal{C}=\\{\bm{c}_{i}|i=1,...,N\\}$ and a SIM(3)-transformation
$\\{s,\bm{R},\bm{t}\\}$, a SIM(3)-invariant descriptor $f_{0}$ can be
expressed as:

$\displaystyle f_{0}(\mathcal{C})=f_{0}(s\bm{R}\mathcal{C}+\bm{t}).$ (5)

To make our descriptor SIM(3)-invariant, first, we obtain a translation-
invariant descriptor by calculating the vectors $\bm{v}$ composed of every two
points:

$\displaystyle\bm{v}_{ij}=(\bm{c}_{i}+\bm{t})-(\bm{c}_{j}+\bm{t})=\bm{c}_{i}-\bm{c}_{j}.$
(6)

After that, we normalize the vector $\bm{v}$ to eliminate the scale factor
$s$:

$\displaystyle\overline{\bm{v}}_{ij}=\frac{s\bm{v}_{ij}}{||s\bm{v}_{ij}||_{2}}=\frac{\bm{v}_{ij}}{||\bm{v}_{ij}||_{2}}.$
(7)

Finally, we eliminate the effect of rotation transformation by calculating the
dot product between two vectors (see Fig. 3):

$\displaystyle\theta(\bm{c}_{i},\bm{c}_{j},\bm{c}_{k},\bm{c}_{o})=\bm{R}\overline{\bm{v}}_{ij}\cdot\bm{R}\overline{\bm{v}}_{ko}=\overline{\bm{v}}_{ij}^{T}\bm{R}^{T}\bm{R}\overline{\bm{v}}_{ko}=\overline{\bm{v}}_{ij}^{T}\overline{\bm{v}}_{ko},$
(8)

where $\cdot$ denotes the dot product of two vectors, and $\theta$ denotes a
function which takes 4 points as input and a SIM(3)-invariant scalar as output
(as the angles shown in Fig. 3)

We call $\theta$ the atomic descriptor. However, only a single atomic
descriptor calculated from 4 points (primitives centers) is not enough to
describe an object’s shape. Thus, we design the SIM(3)-invariant shape
descriptor through a function $f$, which concatenates multiple atomic
descriptors to form a SIM(3)-invariant shape descriptor. With our estimated
primitives $\tilde{\mathcal{C}}$ and their semantic labels
$\tilde{\mathcal{L}}$, the set $\tilde{\mathcal{L}}^{4}$ is denoted as:

$\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\tilde{\mathcal{L}}^{4}=\\{(i,j,k,o)\mid
i,j,k,o\in\tilde{\mathcal{L}},i\neq j,k\neq o\\},}$ (9)

and the SIM(3)-invariant shape descriptor $f$ is denoted as:

$\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}f(\tilde{\mathcal{C}},\tilde{\mathcal{L}})=\underset{(i,j,k,o)\in\tilde{\mathcal{L}}^{4}_{sub}}{\oplus}\theta(\bm{c}_{i},\bm{c}_{j},\bm{c}_{k},\bm{c}_{o}),}$
(10)

where $\oplus$ denotes the concatenate operation which concatenates multiple
scalars into a vector in order, and $\tilde{\mathcal{L}}^{4}_{sub}$ is a
subset randomly chosen from $\tilde{\mathcal{L}}^{4}$ for each optimization.
We choose the subset of $\tilde{\mathcal{L}}^{4}$ because
$|\tilde{\mathcal{L}}^{4}|$ would be large, which is computational expensive.
In Sec. 4.3, an ablation is conducted to analyze the effect of
$|\tilde{\mathcal{L}}^{4}_{sub}|$, which indicates the number of atomic
descriptors in a shape descriptor.

Online Shape Optimization. Given the shape embedding $\hat{z}$ in the latent
space and estimated primitives $\tilde{\mathcal{C}}$ with semantic labels
$\tilde{\mathcal{L}}$, we can measure the shape difference between the
generative shape $g_{c}(\hat{z})$ and the observation $P_{0}$ through the
SIM(3)-invariant descriptor. We denote this difference as $\tilde{e}$:

$\displaystyle\tilde{e}=||f(\tilde{\mathcal{C}},\tilde{\mathcal{L}})-f(\hat{\mathcal{C}}(g_{c}(\hat{z})),\tilde{\mathcal{L}})||_{2}+\eta||\hat{z}||_{2},$
(11)

where $\hat{\mathcal{C}}(g_{c}(\hat{z}))=\\{\bm{\hat{c}}_{i}|i=1,...,N_{c}\\}$
is the primitive centers of $g_{c}(\hat{z})$. Obviously, $f$ is a
differentiable function and $g_{c}$ is a differentiable neural network. We
optimize the shape embedding $\hat{z}$ via gradient descent in an online
iterative way and set $\eta=0.0001$ to weight the regularization term.

###  3.4 6D Pose and Size Estimation

With the primitive centers $\hat{\mathcal{C}}(g_{c}(\hat{z}))$ of the
optimized shape and the observed point cloud, we use Umeyama algorithm [32] to
compute the SIM(3) transformation by minimizing following error function:

$\displaystyle\tilde{s},\tilde{\bm{R}},\tilde{\bm{t}}=\underset{s,\bm{R},\bm{t}}{\rm
argmin}\frac{1}{N_{0}}\sum_{i=1}^{N_{0}}||\bm{p}_{i}-s\bm{R}(\bm{\hat{c}}_{l_{i}}+\bm{t})||^{2}.$
(12)

##  4 Experiments

Table 1: Comparison of our method with other five SOTA methods [12, 14, 15,
17, 16] on REAL275 and CAMERA25. ‘S’ is synthetic data and ‘R’ is real data.

Dataset |  Methods |  Training Data |  Canonical Model |  $\text{IoU}50$ |  $\text{IoU}75$ |  $5^{\circ}2cm$ |  $5^{\circ}5cm$ |  $10^{\circ}2cm$ |  $10^{\circ}5cm$ |  Learnable Parameters(M)
---|---|---|---|---|---|---|---|---|---|---
REAL275 (real) | NOCS [12] | RGB(S+R) | Regress | 78.0 | 30.1 | 7.2 | 10.0 | 13.8 | 25.2 | -
SPD [14] | RGBD(S+R) | Deform | 77.3 | 53.2 | 19.3 | 21.4 | 43.2 | 54.1 | 18.3
SGPA [15] | RGBD(S+R) | Deform | 80.1 | 61.9 | 35.9 | 39.6 | 61.3 | 70.7 | 23.3
SAR-Net [17] | D(S) | Deform | 79.3 | 62.4 | 31.6 | 42.3 | 50.3 | 68.3 | 6.3
iCaps [16] | D(S) | Generate | - | - | - | 22.3 | - | - | 80.9
Ours | D(S) | Generate | 79.0 | 65.3 | 46.9 | 54.7 | 64.2 | 76.3 | 2.3
CAMERA25 (synthetic) | NOCS [12] | RGB(S+R) | Regress | 83.9 | 69.5 | 32.3 | 40.9 | 48.2 | 64.6 | -
SGPA [15] | RGBD(S+R) | Deform | 93.2 | 88.1 | 70.7 | 74.5 | 82.7 | 88.4 | 23.3
SAR-Net [17] | D(S) | Deform | 86.8 | 79.0 | 66.7 | 70.9 | 75.3 | 80.3 | 6.3
Ours | D(S) | Generate | 95.7 | 89.3 | 71.7 | 77.0 | 80.1 | 86.9 | 2.3

Figure 4: Qualitative comparisons between our method and SGPA [15] on REAL275
dataset [12]. The red arrows and dotted box point out the inaccurate results
of SGPA compared with our method.

###  4.1 Datasets and Evaluation Metrics

We evaluate our method on NOCS [12] dataset which consists of six objects
categories including bottle, bowl, camera, can, laptop and mug. The NOCS
dataset contains a virtual dataset called CAMERA25 and a real-world dataset
called REAL275. The CAMERA25 dataset contains 300K synthetic images and there
are 25K images for evaluation among them. The REAL275 contains 4.3K real-scene
images for training and 2.75K real-scene images for evaluation.

Two aspects of metrics are adopted for evaluation: 3D IoU and pose error. 3D
IoU measures the overlap of two posed 3D bounding boxes, and we compute the
average precision at threshold values of $50\%$ and $75\%$. The pose error
measures the rotation and translation errors between the predicted pose and
ground truth pose, and we compute the average precision at threshold values
($5^{\circ},2cm$), ($5^{\circ},5cm$), ($10^{\circ},2cm$) and
($10^{\circ},5cm$).

###  4.2 Comparison with State-of-the-Art Methods

We compared our proposed method with several SOTA methods [12, 14, 15, 17,
16]. NOCS [12] directly regress pixel-level canonical coordinate. SPD [14] and
SGPA [15] are the methods which recover canonical shape by deforming a prior
shape. iCaps [16] and our methods utilize a generative model to refine and
recover the canonical model respectively. Table 1 shows the quantitative
results. Overall, our method needs the minimum parameters learned for the
shape and pose estimation task and shows the results of the smallest domain
gap between the synthetic dataset and the real-world dataset. In detail, our
method outperforms other existing methods on all metrics but $\text{IoU}50$ on
the real-world REAL275 dataset. Fig. 4 shows qualitative comparisons with SGPA
[15]. On the synthetic CAMERA25 dataset, our method also shows improvement on
most metrics compared with other methods, except for a small gap with SGPA
[15] on two metrics, ($10^{\circ},2cm$) and ($10^{\circ},5cm$). Besides, Fig.
5 shows the average precision curves of each category under different
thresholds on IoU, rotation and translation on REAL275 and CAMERA25 dataset.
Compared to the other generative method iCaps [16], our method shows far
better results, indicating the advantage of our decoupled shape and pose
estimation. It should be noted here that our trained generative model (Sec.
3.1) consists of about 4.3M parameters for each category. The generative
models are only pre-trained on ShapeNetCore [33] and need no more fine-tuning
for the specific shape and pose estimation task. Thus, we compare the
learnable parameters with other methods. Specifically, our method needs only
additionally train a part segmentation network (2.3M) for all categories on
the synthetic training data provided by CAMERA25 [12], which leads to our
better generalization in real scenes.

###  4.3 Ablation Study

To justify the design choices of our method, we explore the following issues
through experiments:

  * •

Effectiveness of the SIM(3)-invariant descriptor. Does the descriptor-based
shape optimization make the optimized shape more accurate and benefit the pose
estimation?

  * •

Effect of different numbers of primitives. Semantic primitives are the basis
of our method. How many primitives are better for pose estimation?

  * •

Effect of different numbers of atomic descriptors
$|\tilde{\mathcal{L}}^{4}_{sub}|$. Our shape descriptor is concatenated by
multiple atomic descriptors. How many atomic descriptors are more effective
for shape optimization?

Table 2: Pose estimation results with recovered shape by mean, random and
optimized shape embedding on REAL275.

Embedding | $\text{IoU}50$ | $\text{IoU}75$ | $5^{\circ}2cm$ | $5^{\circ}5cm$ | $10^{\circ}2cm$ | $10^{\circ}5cm$
---|---|---|---|---|---|---
Mean | 77.7 | 54.5 | 31.7 | 36.0 | 51.4 | 61.0
Random | 76.8 | 51.6 | 28.7 | 34.4 | 47.6 | 60.3
Optimized | 79.0 | 65.3 | 46.9 | 54.7 | 64.2 | 76.3

Table 3: Results of our method with different numbers of primitives on REAL275

Number | $\text{IoU}50$ | $\text{IoU}75$ | $5^{\circ}2cm$ | $5^{\circ}5cm$ | $10^{\circ}2cm$ | $10^{\circ}5cm$
---|---|---|---|---|---|---
64 | 79.2 | 63.7 | 41.2 | 51.4 | 58.9 | 74.9
128 | 78.9 | 64.5 | 43.5 | 51.2 | 61.2 | 74.6
256 | 79.0 | 65.3 | 46.9 | 54.7 | 64.2 | 76.3
512 | 77.2 | 63.3 | 45.7 | 53.6 | 61.7 | 74.4

Figure 5: Curves of each category under different thresholds on IoU, rotation
and translation on REAL275 (left) and CAMERA (right) dataset.

Effectiveness of SIM(3)-invariant descriptors. We verify the effectiveness of
our SIM(3)-invariant descriptors in two aspects. First, we verify that
descriptor-based shape optimization improves the accuracy of shape
reconstruction. We compare the reconstruction error of our optimized shape
with the shape when the shape embedding $z=\bm{0}$ (denoted as “Mean” shape)
and a random shape when the shape embedding $z\sim\mathcal{N}(\bm{0},\bm{1})$
(denoted as ”Random” shape). Table 5 shows that our optimized shapes are more
accurate. In Table 5, we also compare our method with SGPA [15], even without
shape loss to supervise our method, our coarse-level shape reconstruction
shows comparable results with SGPA. Second, we verify that descriptor-based
shape optimization improves the pose estimation results. We compare the
results of the pose estimation under the three canonical shapes (mean, random
and optimized) in the Table 3. The results show that we get far better pose
estimation results based on the optimized shape. Besides, Fig. 6 visualizes
cases where shapes become more consistent with observations as optimization
progresses.

Effect of different numbers of primitives. We conduct an ablation study on the
number of semantic primitives. We experiment the number with 64, 128, 256, and
512. As shown in Table 3, the results show that the accuracy of pose
estimation is improved with the increase of the number of primitives, which
indicates that more primitives can represent more complex geometry. The
results show a decline when the number is set to 512 because 512 primitives
are relatively redundant for our 1024 input points, and more primitive classes
also decline the accuracy of part segmentation network (Sec. 3.2). Based on
these observations, we use 256 primitives as the default setting in other
experiments.

Table 4: Comparison of shape reconstruction accuracy of our methods and SGPA
in CD metric ($\times 10^{-3}$) on REAL275. Mean and Random refer to the shape
when shape embedding $z=\bm{0}$ and $z\sim\mathcal{N}(\bm{0},\bm{1})$.

Methods | bottle | bowl | camera | can | laptop | mug | all
---|---|---|---|---|---|---|---
SGPA [15] | 2.93 | 0.89 | 5.51 | 1.75 | 1.62 | 1.12 | 2.44
Mean | 9.55 | 4.39 | 12.4 | 3.03 | 6.59 | 2.26 | 6.05
Random | 11.6 | 5.49 | 16.1 | 3.74 | 7.12 | 3.63 | 7.56
Optimized | 2.05 | 1.55 | 10.1 | 1.63 | 2.12 | 2.93 | 3.46

Table 5: Comparison of shape reconstruction accuracy when different number of
atomic primitives are randomly chosen to optimize the shape in CD ($\times
10^{-3}$) metric.

$|\tilde{\mathcal{L}}^{4}_{sub}|$ | w/o. RANSAC | w. RANSAC
---|---|---
$10^{1}$ | 6.79 $\pm$ 6.50 | 4.38 $\pm$ 5.41
$10^{2}$ | 4.86 $\pm$ 5.38 | 3.97 $\pm$ 5.15
$10^{4}$ | 3.68 $\pm$ 4.34 | 3.59 $\pm$ 4.54
$10^{6}$ | 3.46 $\pm$ 4.05 | 3.44 $\pm$ 4.13

Figure 6: Visualization of the shape optimization process. As optimization
progresses, our optimized shapes are more consist with observations.

Effect of different numbers of atomic descriptors. As illustrated in Sec. 3.3,
a single atomic descriptor is insufficient to describe an object’s shape, and
concatenating all atomic descriptors to form a shape descriptor would be
computationally expensive and unnecessary. Thus, we conduct experiments and
analyze the influence of the number of atomic descriptors on the shape
optimization results. Specifically, we randomly choose
$\\{10^{1},10^{2},10^{4},10^{6}\\}$ atomic descriptors to form the shape
descriptor and use the chamfer distance to evaluate the quality of the
optimized shape. As shown in Table. 5, the reconstruction error of the
optimized shape decreases as the number of atomic descriptors increases, and
applying RANSAC in the random selection can further reduce this error. In the
main experiments, we choose $10^{6}$ atomic descriptors depending on the
computational resources and do not use RANSAC for efficiency.

##  5 Conclusion and Limitations

We propose a novel category-level shape and pose estimation framework by
utilizing the semantic primitive representation. Leveraging a novel
SIM(3)-invariant descriptor, we decouple shape and pose estimation and
optimize the implicitly generated shape in an online iterative way.
Experiments results demonstrate the advantages of our method on the real-scene
data compared with other SOTA methods.

Limitations. As other methods [14, 15, 17], our method also relies on the pre-
processed masks of object instances by Mask R-CNN and extra annotation of
symmetry axes when training. Furthermore, although we can reconstruct the
object mesh (Sec. B in the supp. material), this by-product has no influence
or contribution to our work. Future work could consider the mesh
reconstruction or implicit texture reconstruction [34] to get better shape and
pose estimation results Besides, our method does not consider the occlusion
and collision between objects, and may cause ambiguity from single-view
observation. Feasible future work could fuse color information into our input,
or expand the proposed method to multi-view observation to address these
problems.

#### Acknowledgments

We would like to thank Bangbang Yang for the useful discussion on this paper.

## References

  * Wang et al. [2022] T. Wang, Z. Xinge, J. Pang, and D. Lin.  Probabilistic and geometric depth: Detecting objects in perspective.  In _Conference on Robot Learning_ , pages 1475–1485. PMLR, 2022.
  * Yuan et al. [2022] W. Yuan, C. Paxton, K. Desingh, and D. Fox.  Sornet: Spatial object-centric representations for sequential manipulation.  In _Conference on Robot Learning_ , pages 148–157. PMLR, 2022.
  * Chen et al. [2022] T. Chen, J. Xu, and P. Agrawal.  A system for general in-hand object re-orientation.  In _Conference on Robot Learning_ , pages 297–307. PMLR, 2022.
  * Cheng et al. [2022] S. Cheng, K. Mo, and L. Shao.  Learning to regrasp by learning to place.  In _Conference on Robot Learning_ , pages 277–286. PMLR, 2022.
  * Chen et al. [2022] T. Chen, J. Xu, and P. Agrawal.  A system for general in-hand object re-orientation.  In _Conference on Robot Learning_ , pages 297–307. PMLR, 2022.
  * Sucar et al. [2020] E. Sucar, K. Wada, and A. Davison.  NodeSLAM: Neural object descriptors for multi-view shape reconstruction.  In _2020 International Conference on 3D Vision (3DV)_ , pages 949–958. IEEE, 2020.
  * Xiang et al. [2017] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox.  Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes.  _arXiv preprint arXiv:1711.00199_ , 2017.
  * Wang et al. [2019] C. Wang, D. Xu, Y. Zhu, R. Martín-Martín, C. Lu, L. Fei-Fei, and S. Savarese.  Densefusion: 6d object pose estimation by iterative dense fusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3343–3352, 2019.
  * Peng et al. [2019] S. Peng, Y. Liu, Q. Huang, X. Zhou, and H. Bao.  Pvnet: Pixel-wise voting network for 6dof pose estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4561–4570, 2019.
  * He et al. [2020] Y. He, W. Sun, H. Huang, J. Liu, H. Fan, and J. Sun.  Pvn3d: A deep point-wise 3d keypoints voting network for 6dof pose estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11632–11641, 2020.
  * Li et al. [2019] Z. Li, G. Wang, and X. Ji.  Cdpn: Coordinates-based disentangled pose network for real-time rgb-based 6-dof object pose estimation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7678–7687, 2019.
  * Wang et al. [2019] H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and L. J. Guibas.  Normalized object coordinate space for category-level 6d object pose and size estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2642–2651, 2019.
  * Chen et al. [2021] W. Chen, X. Jia, H. J. Chang, J. Duan, L. Shen, and A. Leonardis.  Fs-net: Fast shape-based network for category-level 6d object pose estimation with decoupled rotation mechanism.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1581–1590, 2021.
  * Tian et al. [2020] M. Tian, M. H. Ang, and G. H. Lee.  Shape prior deformation for categorical 6d object pose and size estimation.  In _European Conference on Computer Vision_ , pages 530–546. Springer, 2020.
  * Chen and Dou [2021] K. Chen and Q. Dou.  Sgpa: Structure-guided prior adaptation for category-level 6d object pose estimation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2773–2782, 2021.
  * Deng et al. [2022] X. Deng, J. Geng, T. Bretl, Y. Xiang, and D. Fox.  iCaps: Iterative category-level object pose and shape estimation.  _IEEE Robotics and Automation Letters_ , 7(2):1784–1791, 2022.
  * Lin et al. [2022] H. Lin, Z. Liu, C. Cheang, Y. Fu, G. Guo, and X. Xue.  SAR-Net: Shape alignment and recovery network for category-level 6d object pose and size estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6707–6717, 2022.
  * Shan et al. [2021] M. Shan, Q. Feng, Y.-Y. Jau, and N. Atanasov.  ELLIPSDF: Joint object pose and shape optimization with a bi-level ellipsoid and signed distance function description.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5946–5955, 2021.
  * Park et al. [2019] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove.  DeepSDF: Learning continuous signed distance functions for shape representation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 165–174, 2019.
  * Hao et al. [2020] Z. Hao, H. Averbuch-Elor, N. Snavely, and S. Belongie.  Dualsdf: Semantic shape manipulation using a two-level representation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7631–7641, 2020.
  * Schwarz et al. [2015] M. Schwarz, H. Schulz, and S. Behnke.  RGB-D object recognition and pose estimation based on pre-trained convolutional neural network features.  In _2015 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 1329–1335. IEEE, 2015.
  * Kehl et al. [2017] W. Kehl, F. Manhardt, F. Tombari, S. Ilic, and N. Navab.  Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 1521–1529, 2017.
  * Oberweger et al. [2018] M. Oberweger, M. Rad, and V. Lepetit.  Making deep heatmaps robust to partial occlusions for 3d object pose estimation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 119–134, 2018.
  * Lepetit et al. [2009] V. Lepetit, F. Moreno-Noguer, and P. Fua.  Epnp: An accurate o (n) solution to the pnp problem.  _International Journal of Computer Vision_ , 81(2):155–166, 2009.
  * Park et al. [2020] K. Park, A. Mousavian, Y. Xiang, and D. Fox.  Latentfusion: End-to-end differentiable reconstruction and rendering for unseen object pose estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10710–10719, 2020.
  * Yen-Chen et al. [2021] L. Yen-Chen, P. Florence, J. T. Barron, A. Rodriguez, P. Isola, and T.-Y. Lin.  inerf: Inverting neural radiance fields for pose estimation.  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 1323–1330. IEEE, 2021.
  * Chen et al. [2020] D. Chen, J. Li, Z. Wang, and K. Xu.  Learning canonical shape space for category-level 6d object pose and size estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11973–11982, 2020.
  * Lin et al. [2021] J. Lin, Z. Wei, Z. Li, S. Xu, K. Jia, and Y. Li.  Dualposenet: Category-level 6d object pose and size estimation using dual pose network with refined learning of pose consistency.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3560–3569, 2021.
  * Irshad et al. [2022] M. Z. Irshad, T. Kollar, M. Laskey, K. Stone, and Z. Kira.  Centersnap: Single-shot multi-object 3d shape reconstruction and categorical 6d pose and size estimation.  In _2022 International Conference on Robotics and Automation (ICRA)_ , pages 10632–10640. IEEE, 2022.
  * He et al. [2017] K. He, G. Gkioxari, P. Dollár, and R. Girshick.  Mask r-cnn.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 2961–2969, 2017.
  * Lin et al. [2020] Z.-H. Lin, S.-Y. Huang, and Y.-C. F. Wang.  Convolution in the cloud: Learning deformable kernels in 3d graph convolution networks for point cloud analysis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1800–1809, 2020.
  * Umeyama [1991] S. Umeyama.  Least-squares estimation of transformation parameters between two point patterns.  _IEEE Transactions on Pattern Analysis & Machine Intelligence_, 13(04):376–380, 1991.
  * Chang et al. [2015] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al.  Shapenet: An information-rich 3d model repository.  _arXiv preprint arXiv:1512.03012_ , 2015.
  * Irshad et al. [2022] M. Z. Irshad, S. Zakharov, R. Ambrus, T. Kollar, Z. Kira, and A. Gaidon.  Shapo: Implicit representations for multi-object shape, appearance, and pose optimization.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part II_ , pages 275–292. Springer, 2022.
  * Wang et al. [2020] C. Wang, R. Martín-Martín, D. Xu, J. Lv, C. Lu, L. Fei-Fei, S. Savarese, and Y. Zhu.  6-pack: Category-level 6d pose tracker with anchor-based keypoints.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 10059–10066. IEEE, 2020.
  * Lorensen and Cline [1987] W. E. Lorensen and H. E. Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  _ACM Siggraph Computer Graphics_ , 21(4):163–169, 1987.
  * Zhou et al. [2022] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra.  Detecting twenty-thousand classes using image-level supervision.  _arXiv preprint arXiv:2201.02605_ , 2022.
  * Chitta et al. [2012] S. Chitta, I. Sucan, and S. Cousins.  Moveit![ros topics].  _IEEE Robotics & Automation Magazine_, 19(1):18–19, 2012.

##  A Implementation Details

###  A.1 Training Details

The object point cloud is back-projected from the depth image with the Mask-
RCNN segmentation results provided by [14]. We randomly sample $1024$ points
from the object point cloud as our input. The network architecture and
training protocol of our generative model $g$ keep the same as [20]. We train
a single generative model for each category with the CAD models provided by
ShapeNetCore [33]. Our part segmentation network architecture follows the part
segmentation version of 3D-GCN [31], and we set $256$ primitives in our main
experiments. The segmentation network is trained on CAMERA dataset (about 600K
instances) with a batch size of 32 on a single NVIDIA Tesla V100 graphics
card. Moreover, we set the initial learning rate as $0.0005$ and multiply it
by a factor 0.2 every 10 epochs. We use the ground truth pose provided by [35]
and apply the pose on generated primitives to annotate the ground truth of
part segmentation.

###  A.2 Training Data Augmentation

To obtain the object point cloud which is the input of our method from given
images, we utilize Mask R-CNN [30] to segment the object masks from images.
Because the results provided by Mask R-CNN are often imperfect, the object
point cloud back-projected from the masked depth image would contain
background points. To filter out these outlier points, we train our part
segmentation network (Sec. 3.2) by making augmentation on the ground truth
segmentation masks. As shown in Fig. 7, we adopt the following two strategies
[17] on the ground truth mask : (a) random 0 to 5 dilations on the 2D mask.
(b) random 0 to 15-pixel crop on the mask’s bounding box.

Figure 7: Augmentation on synthetic training data. (a) Dilation on 2D mask.
(b) Random crop on mask’s bounding box. The black points in the object point
cloud are noise points.

###  A.3 Training a DualSDF Model

In order to generate the semantic primitive representation for each shape, we
pre-trained a generative model for each category following the network
structure of DualSDF [20]. The generative model represents an object shape in
two granularities. One coarse-level branch represents the coarse structure by
a certain number of simple shape primitives, and the other fine-level branch
represents the fine structure of the object by SDF.

As we mentioned in Sec. 3.1, given a shape embedding $z$, the coarse-level
branch could be expressed as $g_{c}(z)=\\{\bm{\alpha}_{i}|i=1,...,N_{c}\\}$,
and the fine-level branch could be expressed as $g_{f}(z,\bm{x})=SDF(\bm{x})$.
A signed distance field (SDF) refers to the closest distance from a point
$\bm{x}$ to the surface of the object model. $SDF(\bm{x})<0$ indicates
$\bm{x}$ is inside the model and $SDF(\bm{x})>0$ indicates outside. The two
auto-decoders $g_{f}$ and $g_{c}$ are supervised by a set of pairs of 3D
points $\bm{x}$ and their corresponding ground truth signed distance values
$s=SDF(\bm{x})$. The primitive set
$\bm{\alpha}=\\{\bm{\alpha}_{i}|i=1,...,N_{c}\\}$ can be learned by minimizing
the difference between predicted and ground truth signed distance values:

$\displaystyle\hat{\bm{\alpha}}=\underset{\bm{\alpha}}{\rm
argmin}{~{}L_{SDF}(d(\bm{x},\bm{\alpha}),s)},$ (13)

where $L_{SDF}$ is a truncated L1 loss and $d$ indicates the distance from
$\bm{x}$ to the nearest surface of $\bm{\alpha}$. We use sphere as primitive
here, thus $\bm{\alpha}_{i}=(\bm{c}_{i},r_{i})$, where $\bm{c}_{i}$ is the
sphere center and $r_{i}$ is the sphere radius. $d$ is expressed as:

$\displaystyle d(\bm{x},\bm{\alpha})=\underset{1\leq i\leq
N_{c}}{\min}{||\bm{p}-\bm{c}_{i}||_{2}-r_{i}},$ (14)

In our implementation, we add a truncated value to the ground truth $s$ to
make these primitives evenly distributed on the object’s surface.
Specifically, the Eq. 13 can be rewritten as:

$\displaystyle\hat{\bm{\alpha}}=\underset{\bm{\alpha}}{\rm
argmin}{~{}L_{SDF}(d(\bm{x},\bm{\alpha}),\overline{s})},$ (15)

where

$\displaystyle\overline{s}=\left\\{\begin{array}[]{ll}s,&s\geq-\frac{t}{2}\\\
\vspace{-0.2cm}\\\ -s-t,&s<-\frac{t}{2}\end{array}\right..$ (16)

and $t=0.02$ is a truncated value in our experiments.

The generated primitive examples of six categories are shown in Fig. 8, and we
can see that fewer primitives lack geometric details but will make it easier
for our part segmentation network (Sec. 3.2) to learn. Fig. 8 also shows that
512 primitives are redundant to represent a shape, which drops all metrics on
pose estimation. Sec. 4.3 shows the ablation study on the different number of
primitives.

Figure 8: Visualization of the generated primitive representations of six
categories (_bottle, bowl, camera, can, laptop_ and _mug_) with different
numbers (64, 128, 256 and 512) of primitives. We can see that 64 primitives
(top row) lack of geometric details and 512 primitives (bottom row) are
redundant to represent a shape.

##  B By-Product: Object Mesh Reconstruction

Figure 9: Visualization of our mesh reconstruction experiment. The mesh of
each object is recovered by the optimized shape embedding $\hat{z}$ and the
fine-level decoder $g_{f}$. We transform the object model to the world
coordinate by our estimated pose. The top row is the original scene and the
bottom row is our object-level scene reconstruction.

Although the fine-level decoder $g_{f}$ provides a fine-level structural
representation of the objects, we do not exploit it in our main experiment
(shape and pose estimation). To explore the potential of the shared latent
space shared by $g_{c}$ and $g_{f}$, we use our optimized shape embedding
$\hat{z}$ to generate a mesh model for each object through the fine-level
decoder $g_{f}$.

Table 6: Comparison of shape reconstruction accuracy of our reconstructed
primitives and mesh in CD metric ($\times 10^{-3}$) on REAL275.

Shape | bottle | bowl | camera | can | laptop | mug | all
---|---|---|---|---|---|---|---
Semantic Primitives | 2.05 | 1.55 | 10.1 | 1.63 | 2.12 | 2.93 | 3.45
Mesh | 3.51 | 3.27 | 8.55 | 3.55 | 2.68 | 2.89 | 4.08

Specifically, for each object, we grid the canonical frame to $48^{3}$ points
and get the SDF value of each point by the optimized shape embedding $\hat{z}$
and the trained fine-level generative model $g_{f}$. After that, we utilize
the Marching Cubes algorithm [36] to recover the object mesh. Table 6 shows a
comparison of shape reconstruction of the primitives and mesh. With the mesh
of each object, we can reconstruct the object-level scene by transforming
these meshes to the world coordinate using our estimated pose. Fig. 9 shows
qualitative results of our object mesh reconstruction. Without further
optimization, the different reconstructed meshes keep consistent with their
observations. For example, the reconstructed camera lens in the first scene
(first column) is longer, while the one in the second scene (second column) is
shorter.

##  C Shape Ambiguity of Partially Observed Point Cloud

Fig. 10 shows two cases of shape ambiguity when the objects are highly
occluded, and these cases lead to some failure cases of our method on CAMERA
[12] dataset. The top row shows a bottle with the top half occluded. Although
our estimated shape is consistent with the partially observed point cloud, it
still leads to translation errors when evaluating (about $10cm$ translation
error in this case). The bottom row shows a laptop with its keyboard occluded.
In this case, the screen of our estimated laptop is consistent with the
observation, but there is still a rotation error when evaluating (about
$20^{\circ}$ rotation error in this case).

Figure 10: Shape ambiguity cases. When an object is occluded causing shape
ambiguity, although our estimated shape and GT shape are both consistent with
the observation, this still causes relatively large translation errors (top
row) and rotation errors (bottom row) in the evaluation.

##  D More Results on NOCS-REAL275

Fig. 11 shows more qualitative results of our method compared with SGPA [15].
Compared with our method, the results of SGPA are relatively poor in some
cases of laptops and cameras.

##  E More Visualization of Shape Optimization

Fig. 12 gives more visualization of the process of our shape optimization. As
the optimization proceeds, the structure of the object is optimized to be
consistent with the observation. For example, the lens of the camera is
gradually stretched, the angle between the screen and keyboard of the laptop
gradually increases, and the handle of the mug becomes rectangular. Throughout
our framework, shape optimization accounts for the bulk of the time, taking
about 0.005s per iteration.

##  F Real World Experiments

Applicability to unseen scenarios and settings is critical for robotic tasks.
To verify the effectiveness of our model in real scenarios other than the
dataset provided by NOCS [12], we test our method on several different real-
world scenarios. These scenarios contain different kinds of household objects
of different shapes, and the photos of these scenarios are taken from the view
of a robotic arm. We use Detic [37] to pre-process the masks of each object
and treat the cups with handles as mugs and those without handles as cans.
Fig. 13 shows visualization results, and tight orientated bounding boxes mark
out objects. The red, green and blue axes represent the x, y, z axes of the
canonical coordinate.

Besides, we conduct robotic grasping experiments to demonstrate the
application of our category-level pose estimation. Specifically, we first
estimate pose of each object from the view of a robotic arm. And then, we use
the MoveIt![38] to plan a feasible path for the robot arm to grasp the top
part of objects. Refer to our video for details.

Figure 11: More qualitative comparisons between our method and SGPA [15] on
REAL275 dataset. The red dotted boxes indicate the results of SGPA with worse
accuracy than ours. Figure 12: More visualization of our shape optimization
process. The objects in the figure are optimized by 20 iterations (from left
to right). As the optimization proceeds, the shape of the objects changes
significantly. For example, the lens of the camera is gradually stretched, the
angle between the screen and keyboard of the laptop gradually increases, and
the handle of the mug becomes rectangular. Figure 13: Results of real scenes.
These scenarios contain different kinds of household objects of different
shapes. The objects are marked out by tight orientated bounding boxes. The
red, green and blue axes represent the x, y, and z axes of the canonical
coordinate, respectively.


# Biomedical and Clinical Language Models for Spanish: On the Benefits of
Domain-Specific Pretraining in a Mid-Resource Scenario

Casimiro Pio Carrino  Text Mining Unit
Barcelona Supercomputing Center  Jordi Armengol-Estapé  Text Mining Unit
Barcelona Supercomputing Center  Asier Gutiérrez-Fandiño  Text Mining Unit
Barcelona Supercomputing Center  Joan Llop-Palao  Text Mining Unit
Barcelona Supercomputing Center  Marc Pàmies  Text Mining Unit
Barcelona Supercomputing Center  Aitor Gonzalez-Agirre  Text Mining Unit
Barcelona Supercomputing Center  Marta Villegas  Text Mining Unit
Barcelona Supercomputing Center

###### Abstract

This work presents biomedical and clinical language models for Spanish by
experimenting with different pretraining choices, such as masking at word and
subword level, varying the vocabulary size and testing with domain data,
looking for better language representations. Interestingly, in the absence of
enough clinical data to train a model from scratch, we applied mixed-domain
pretraining and cross-domain transfer approaches to generate a performant bio-
clinical model suitable for real-world clinical data. We evaluated our models
on Named Entity Recognition (NER) tasks for biomedical documents and
challenging hospital discharge reports. When compared against the competitive
mBERT and BETO models, we outperform them in all NER tasks by a significant
margin. Finally, we studied the impact of the model’s vocabulary on the NER
performances by offering an interesting vocabulary-centric analysis. The
results confirm that domain-specific pretraining is fundamental to achieving
higher performances in downstream NER tasks, even within a mid-resource
scenario. To the best of our knowledge, we provide the first biomedical and
clinical transformer-based pretrained language models for Spanish, intending
to boost native Spanish NLP applications in biomedicine. Our best models are
freely available in the HuggingFace hub: https://huggingface.co/BSC-TeMU.

##  1 Introduction

The success of transformer-based models in the general domain [11] has
encouraged the development of language models for domain-specific scenarios
[9, 30, 5, 17]. Specifically, in the biomedical domain, there has been a
proliferation of models [24, 6, 4, 13] since the first BioBERT [18] model was
published. Unfortunately, there is still a significant lack of biomedical and
clinical models in languages other than English, despite the increasing
efforts of the NLP community [21, 27]. Consequently, general-domain pretrained
language models supporting Spanish, such as mBERT [11] and BETO [8], are often
used as a proxy in the absence of a genuine domain-specialized model. To fill
this gap, we trained several biomedical language models experimenting with a
wide range of pretraining choices, namely, masking with word and subword
units, varying the vocabulary size and experimenting with the data domain.
Furthermore, we applied cross-domain transfer and mixed-domain pretraining
using biomedical and clinical data to train bio-clinical models intended for
clinical applications. To evaluate our models, we choose the fundamental Named
Entity Recognition (NER) task in information retrieval defined for biomedical
and clinical scenarios, the former based on biomedical documents and the
latter on real hospital documents. As the main result, our models obtained a
significant gain over both mBERT and BETO models in all tasks. Additionally,
we studied the impact of the model’s vocabulary on several downstream tasks by
performing a vocabulary-centric analysis. The evaluation results highlights
the importance of domain-specific pretraining over continual pretraining from
general domain data in a mid-resource scenario. Our main contributions are:

  1. 1.

We release the first Spanish biomedical and clinical transformer-based
pretrained language models, trained with the largest biomedical corpus known
to date.

  2. 2.

We assess the effectiveness of these models in different settings, two
biomedical scenarios and a demanding clinical scenario with real hospital
reports.

  3. 3.

We show that biomedical models exhibit a remarkable cross-domain transfer
ability on the clinical domain.

  4. 4.

We perform an in-depth vocabulary and segmentation analysis, which offers
insights on domain-specific pretraining and raises interesting open questions.

##  2 Related work

In the last years, several language models for both the biomedical and
clinical domain have been trained through unsupervised pretraining of
transformer-based architectures [14]. The first trained model was BioBERT
[18], where the authors adapted the BERT model [11], trained with general-
domain data, to the biomedical domain by continual pretraining. Similarly,
other works followed the continual pretraining approach to train the BlueBERT
[24] and ClinicalBERT [4] models. When enough in-domain data is available,
training from scratch has been used as an alternative method to continual
pretraining, leading to the SciBERT [6] and PubMedBERT [13] models. However,
the SciBERT model uses mixed-domain data from the biomedical and computer
science domain, while PubMedBERT leverages only data belonging to the
biomedical domain. Interestingly, in [13], the authors call into question the
benefits of mixed-domain pretraining, estimating its negative impact on a set
of downstream tasks belonging to an extensive biomedical benchmark (named
BLURB) that they provide.

Our pretraining approach employed training from scratch with mixed-domain data
by combining biomedical and clinical resources. However, in contrast to [6,
13] i) we used a corpus 3-5 times smaller, ii) we performed pretraining from
scratch with mixed-domain data from biomedical documents and clinical notes
and iii) we assess the suitability of cross-domain transfer from the
biomedical to the clinical domain.

##  3 Corpora

This work considers two corpora with very different sizes and domains, namely,
a clinical corpus and a biomedical one. The clinical corpus contains 91M
tokens from more than 278K clinical documents (including discharge reports,
clinical course notes and X-ray reports). For the biomedical corpus we
gathered data from a variety of sources, namely:

  * •

Medical crawler333https://zenodo.org/record/4561970: Crawler of more than
3,000 URLs belonging to Spanish biomedical and health domains introduced in
[7].

  * •

Clinical cases misc.: A miscellany of medical content, essentially clinical
case. Note that a clinical case report is different from a scientific
publication where medical practitioners share patient cases and it is
different from a clinical note or document.

  * •

Scielo444https://github.com/PlanTL-SANIDAD/SciELO-Spain-Crawler: Scientific
publications written in Spanish crawled from the Spanish SciELO server in
2017.

  * •

PubMed: Open-access articles from the PubMed repository crawled in 2017.

  * •

BARR2_background555https://temu.bsc.es/BARR2/downloads/background_set.raw_text.tar.bz2:
Biomedical Abbreviation Recognition and Resolution (BARR2) containing Spanish
clinical case study sections from a variety of clinical disciplines.

  * •

Wikipedia_life_sciences: Wikipedia articles crawled on 04/01/2021 with the
Wikipedia API python library111https://pypi.org/project/Wikipedia-API/
starting from the "Ciencias_de_la_vida" category up to a maximum of 5
subcategories. Multiple links to the same articles are then discarded to avoid
repeating content.

  * •

Patents: Google Patent in Medical Domain for Spain (Spanish). The accepted
codes (Medical Domain) for Json files of patents are: "A61B", "A61C","A61F",
"A61H", "A61K", "A61L","A61M", "A61B", "A61P".

  * •

EMEA666http://opus.nlpl.eu/download.php?f=EMEA/v3/moses/en-es.txt.zip:
Spanish-side documents extracted from parallel corpora made out of PDF
documents from the European Medicines Agency.

  * •

mespen_Medline777https://zenodo.org/record/3562536: Spanish-side articles
extracted from a collection of Spanish-English parallel corpus consisting of
biomedical scientific literature. The collection of parallel resources are
aggregated from the MedlinePlus source.

Notice that, although the biomedical documents undoubtedly share a significant
percentage of medical terms with clinical notes, the syntax and vocabulary may
change radically due to the specific contexts and the idiosyncrasies of the
user-generated content in clinical texts.

We cleaned each biomedical corpus to get the final corpus, and left the
clinical corpus uncleaned. For each biomedical resource, we applied a cleaning
pipeline with customized operations designed to read data in different
formats, split it into sentences, detect the language, remove noisy and ill-
formed sentences, deduplicate and eventually output the data with their
original document boundaries. Finally, to remove repetitive content, we
concatenated the entire corpus and deduplicate again, obtaining a total of
968M words. Table 1 shows detailed statistics of each dataset.

Corpus name | No. tokens | Documents | Sentences
---|---|---|---
Medical crawler | 745,705,946 | 1,580,577 | 34,347,553
Clinical cases misc. | 102,855,267 | 41,592 | 5,565,854
Scielo | 60,007,289 | 16,981 | 2,668,231
BARR2_background | 24,516,442 | 127,635 | 1,029,600
Wikipedia_life_sciences | 13,890,501 | 15,791 | 832,027
Patents | 13,463,387 | 99,611 | 253,924
EMEA | 5,377,448 | 284,119 | 284,575
mespen_Medline | 4,166,077 | 5,293 | 322,619
PubMed | 1,858,966 | 892 | 103,674
Table 1: List of individual sources constituting the biomedical corpus. The
number of tokens refers to _white-spaced_ tokens calculated on cleaned
untokenized text.

##  4 Models pretraining

The models trained in this work employed a RoBERTa [19] base model with 12
self-attention layers. Following the original training, we prescind the
auxiliary Next Sentence Prediction task used in BERT, and used masked language
modelling as the pretraining objective. We used Subword Masking (SWM) as in
[19], and the Whole Word Masking (WWM) 222https://github.com/google-
research/bert technique that masks all sub-words belonging to the same word
[10]..

We tokenized with the Byte-Level BPE algorithm introduced in [25] and employed
in the original RoBERTa [19], unlike previous biomedical language models [6,
13, 18] that use WordPiece [11] or SentencePiece [16] segmentations. We
learned a cased vocabulary of 52k and 30k tokens to perform a comparative
analysis.

We run the training for 48 hours with 16 NVIDIA V100 GPUs of 16GB DDRAM, using
Adam optimizer [15] with peak learning rate of 0.0005 and an effective batch
size of 2,048 sentences888Through gradient accumulation as implemented in
Fairseq [22].. We left the other hyperparameters in their default values as
the original RoBERTa trainings. We then selected the model with the lowest
perplexity in a holdout subset as the best model. Moreover, training was
performed at the document level, preserving document boundaries. Document-
level training may be crucial to enforce long-range dependencies and push the
model towards the comprehension of entire documents, fostering the modelling
of long-range dependencies.

We applied the pretraining method described above to train a variety of
models, which can be divided into two groups, the Biomedical language models
and the Bio-clinical language models.

##### Biomedical language models:

We used the biomedical corpora described in section 3 of about 968M tokens to
train biomedical language models. To study the impact of the masking mechanism
and the vocabulary size, we experimented with the SWM and WWM techniques and
with vocabularies of 52k and 30k tokens. We refer to the four variants as
bio-52k-SWM, bio-52k-WWM, bio-30k-SWM and bio-30k-WWM.

##### Bio-clinical language models:

Due to the lack of a large-scale clinical corpus of comparable size to the
biomedical one, we combined the biomedical and clinical corpora described in
section 3 to train bio-clinical language models suitable for clinical
settings. Furthermore, we also trained a bio-clinical model variant that
leverages a vocabulary learned only from the clinical corpus with a vocabulary
size of 52k. We refer to these two models as bio-cli-52k and bio-
cli-52k-vocab-cli.

##  5 Downstream NER tasks

We performed Named Entity Recognition (NER) tasks as a testbed for the our
models since they are essential building blocks for many biomedical Text
Mining and NLP applications. We employed a standard linear layer as a token
classification head and the BIO tagging schema [26] to fine-tune the
pretrained models for the NER tasks. During fine-tuning, both the pretrained
model and the classification layers parameters are learned with stochastic
gradient descent.

Fine-tuning pretrained transformer-based language models for NER tasks by
adding a linear layer on top of them is a usual practice in the literature
(both in general-purpose models [11, 19] and domain-specific ones [18]).
Remarkably, this method obtained impressive performances compared to more
sophisticated classification layers such as Conditional Random Field layers on
top of Bidirectional Long Short-Term Memory Recurrent Networks [23].
Furthermore, its simplicity allows a head-to-head comparison of different
pretraining strategies and baseline models, emphasising the ability of the
pretrained representations.

We applied the fine-tuning method described above to three different NER
datasets. The first two are data from two shared tasks and use annotations on
curated medical data (clinical cases extracted from medical literature). The
last one uses medical records from the ICTUSnet project999https://ictusnet-
sudoe.eu/es/.

PharmaCoNER [12] is a track on chemical and drug mention recognition from
Spanish medical texts. The authors collected a manually classified collection
of clinical case report sections derived from open access Spanish medical
publications, named the Spanish Clinical Case Corpus (SPACCC). The corpus
contained a total of 1,000 clinical cases and 396,988 words and was manually
annotated, with a total of 7,624 entity mentions, corresponding to four
different mention types101010For a detailed description, see
https://temu.bsc.es/pharmaconer/. The track received several system
submissions from the NLP community [29, 2, 3, 33]

CANTEMIST [20] is a shared task specifically focusing on named entity
recognition of tumor morphology, in Spanish. The CANTEMIST
corpus111111CANTEMIST corpus: https://doi.org/10.5281/zenodo.3878178 is a
collection of 1,301 oncological case reports written in Spanish, with a total
of 63,016 sentences and 1,093,501 tokens. Several systems employing different
strategies have been proposed to tackle the task [1, 32, 31]

The ICTUSnet data set consists of 1,006 hospital discharge reports of patients
admitted for stroke from 18 different Spanish hospitals. It contains more than
79,000 annotations for 51 different kinds of variables. The dataset is part of
the ICTUSnet project, whose main objective was the development of an
information extraction system to support domain experts when identifying
relevant information in discharge reports.

##  6 Evaluation and Main Results

We evaluated our models on two biomedical benchmarks, PharmaCoNER and
CANTEMIST, as well as the clinical ICTUSnet dataset, described in section 5.
For each NER dataset and model, we fine-tune for 10 epochs with a batch size
of 32 and a maximum sentence length of 512 tokens. After training, we select
as the best model the one with the highest F1 score on the development set.
Finally, we computed the evaluation scores by feeding the test set to the best
model. Tables 2 and 3 show the results for the biomedical models and bio-
clinical models, respectively. We would like to remark that the ICTUSnet
dataset is a challenging task since its consists of real hospital discharge
reports. Moreover, for the biomedical models, the ICTUSnet evaluation
represents a cross-domain transfer experiment from the biomedical to clinical
domain.

Overall, our models achieved the best scores, beating both mBERT and BETO
significantly. The biomedical models showed a remarkable cross-domain transfer
ability compared to the bio-clinical models, achieving competitive
performances on the ICTUSnet task. Nonetheless, the bio-clinical models
obtained the best performances, indicating that the mixed-domain pretraining
is practical approach to mitigate the lack of large-scale real-world clinical
data. These results confirm the suitability of pretraining from scratch in a
scenario with medium-size resources. We finally point out that, although our
primary objective aside from achieving the best task performance, the results
obtained are promising, evincing that a more sophisticated classification
layer could further improve the tasks performances.

| Bio | mBERT | BETO
---|---|---|---
vocab size | 52k | 30k | 120k | 31k
masking | SWM | WWM | SWM | WWM | SWM | WWM
PharmaCoNER |  |  |  |  |  |
\- F1 | 89.48$\pm$ 0.60 | 89.62$\pm$0.57 | 89.47$\pm$0.33 | 89.85$\pm$0.47 | 87.46$\pm$0.23 | 88.18$\pm$0.41
\- Precision | 87.85$\pm$ 1.15 | 88.18$\pm$0.99 | 88.33$\pm$0.42 | 88.41$\pm$0.41 | 86.50$\pm$0.95 | 87.12$\pm$0.52
\- Recall | 91.18$\pm$ 0.74 | 91.11$\pm$0.40 | 90.64$\pm$0.76 | 91.35$\pm$0.58 | 88.46$\pm$0.57 | 89.28$\pm$0.69
CANTEMIST |  |  |  |  |  |
\- F1 | 83.87$\pm$ 0.41 | 83.00$\pm$0.17 | 82.85$\pm$0.36 | 83.23$\pm$0.34 | 82.61$\pm$0.67 | 82.42$\pm$0.06
\- Precision | 81.70$\pm$0.52 | 80.81$\pm$0.27 | 80.93$\pm$0.48 | 81.27$\pm$0.72 | 81.12$\pm$0.65 | 80.91$\pm$0.41
\- Recall | 86.17$\pm$0.46 | 85.32$\pm$0.40 | 84.87$\pm$0.27 | 85.29$\pm$0.52 | 84.15$\pm$0.78 | 84.00$\pm$0.45
ICTUSnet |  |  |  |  |  |
\- F1 | 88.12$\pm$0.25 | 87.94$\pm$0.43 | 87.77$\pm$0.35 | 88.21$\pm$0.26 | 86.75$\pm$0.21 | 85.95$\pm$0.25
\- Precision | 85.56$\pm$0.26 | 84.88$\pm$0.80 | 84.33$\pm$0.28 | 85.20$\pm$0.57 | 83.53$\pm$0.35 | 83.10$\pm$0.57
\- Recall | 90.83$\pm$0.47 | 91.22$\pm$0.49 | 91.50$\pm$0.66 | 91.45$\pm$0.44 | 90.23$\pm$0.46 | 89.02$\pm$0.61
Table 2: Evaluation scores (F1, Precision and Recall) for the PharmaCoNER, CANTEMIST and ICTUSnet NER tasks. We compared our biomedical models with two general-domain baseline models, namely, multilingual BERT and BETO. Rows in light gray stress that the Biomedical model is performing cross-domain transfer of the ICTUSnet task since it belongs to the clinical domain. The scores are averaged across 5 random runs with standard deviation as error bar. | Bio-cli | mBERT | BETO
---|---|---|---
vocab size-domain | 52k | 52k-vocab-cli | 120k-general | 31k-general
PharmaCoNER |  |  |  |
\- F1 | 90.04$\pm$0.13 | 89.29$\pm$0.52 | 87.46$\pm$0.23 | 88.18$\pm$0.41
\- Precision | 88.92$\pm$0.36 | 87.67$\pm$0.96 | 86.50$\pm$0.95 | 87.12$\pm$0.52
\- Recall | 91.18$\pm$0.29 | 90.98$\pm$0.47 | 88.46$\pm$0.57 | 89.28$\pm$0.69
CANTEMIST |  |  |  |
\- F1 | 83.34$\pm$0.39 | 84.41$\pm$0.33 | 82.61$\pm$0.67 | 82.42$\pm$0.06
\- Precision | 81.48$\pm$0.66 | 82.75$\pm$0.25 | 81.12$\pm$0.65 | 80.91$\pm$0.41
\- Recall | 85.30$\pm$0.45 | 86.15$\pm$0.53 | 84.15$\pm$0.78 | 84.00$\pm$0.45
ICTUSnet |  |  |  |
\- F1 | 88.08$\pm$0.12 | 88.45$\pm$0.63 | 86.75$\pm$0.21 | 85.95$\pm$0.25
\- Precision | 84.92$\pm$0.29 | 84.92$\pm$0.86 | 83.53$\pm$0.35 | 83.10$\pm$0.57
\- Recall | 91.50$\pm$0.29 | 92.3$\pm$0.52 | 90.23$\pm$0.46 | 89.02$\pm$0.61
Table 3: Evaluation scores (F1, Precision and Recall) for the PharmaCoNER,
CANTEMIST and ICTUSnet NER tasks. We compared our Bio-clinical models with two
baselines obtained with non domain-specific models in Spanish, namely,
multilingual BERT and BETO. The scores are averaged across 5 random runs with
standard deviation as error bar.

##  7 Discussion and Analysis

This section attempts to shed light on the evaluation results by conducting a
vocabulary analysis and segmentation experiments.

###  7.1 Vocabulary overlap

We look at the evaluation results through the lens of the model’s vocabulary.
Undoubtedly, the vocabulary plays a crucial role during the downstream
transfer since it is responsible for encoding the task-specific data.
Intuitively, we expect that the more the overlap between vocabulary and task’s
tokens, the better. A high overlap leverages more pretrained representations
that could be beneficial for fine-tuning. Specifically, we hypothesize that
the performance on each downstream task may be related to the number of
vocabulary tokens used to encode the task’s data. Therefore, we first tokenize
the three NER tasks with each model’s vocabulary and then calculate the
vocabulary overlap with each task, expressed as the number of tokens. The
results shown in Table 4 seem to support our hypothesis by showing that the
best evaluation scores for each task are obtained by the model with the
maximum number of tokens overlap. Accordingly, the lowest overlap exhibited by
the baselines models may explain their lowest evaluation scores.

Model | PharmaCoNER | CANTEMIST | ICTUSnet
---|---|---|---
Bio-cli-52k | 20,620 (40%) | 22,001 (42%) | 23,360 (45%)
Bio-cli-vocab-cli-52k | 20,335 (39%) | 23,095 (44%) | 30,467 (59%)
Bio-52k | 19,978 (38%) | 20,951 (40%) | 21,449 (41%)
Bio-30k | 15,792 (53%) | 16,302 (54%) | 16,266 (54%)
BETO | 12,829 (41%) | 13,044 (42%) | 13,388 (43%)
mBERT | 11,084 (9%) | 11,434 (9%) | 13,187 (11%)
Table 4: The vocabulary overlap is expressed as the intersection of each
model’s vocabulary and segmented tokens for each NER task. We also show the
percentage of overlapped tokens to the total vocabulary size. Note that the
SWM and WWM variants of the biomedical models are merged under the same names
since they are equivalent for the vocabulary analysis.

###  7.2 Impact of segmentation

Intuitively, it is reasonable to assume that a proper domain-specific
segmentation should preserve the integrity of biomedical terms, minimizing the
number of subword units required to encode them. As pointed out in [13],
models employing out-of-domain vocabulary "are forced to divert
parametrization capacity and training bandwidth to model biomedical terms
using fragmented subwords". Therefore, from one side, over-segmenting may have
a negative influence on the downstream performance. On the other side, under-
segmentation that employs term-specific units could dramatically increase the
vocabulary size. Moreover, under-segmenting could prevent the model from
detecting relatedness between terms based on the shared subwords, especially
in morphologically rich terminology. Therefore, we conducted a meticulous
analysis of domain-specific terms segmentation to study the trade-off between
under and over-segmentation conditions and shed light on its relation with the
NER downstream performances.

####  7.2.1 Splitting terms

We analyzed, both qualitatively and quantitatively, how the different models
under study segment biomedical terms. We retrieved all the biomedical terms
used as annotations in the CANTEMIST, PharmaCoNER and ICTUSnet tasks and
segmented them applying each model’s tokenizer. Table 7 shows the quality of
segmentation on a random set of NER annotations, making a comparison between
mBERT, BETO and our best model bio-cli-52k-vocab-cli. Table 5 shows the
average number of subwords generated by each model computed over all the NER
annotations. As expected, mBERT and BETO split terms into many more pieces
than our models, on average. However, the variations across biomedical and
clinical models also point out that the vocabulary size and mixed-domain
pretraining are relevant factors determining the final segmentation. Finally,
Table 6 illustrates, for each model, how many tokens are segmented with a
given number of subwords (up to more than 4 subwords). Again, both mBERT and
BETO tend to over-segment when compared to our models. As an example, our best
model (bio-cli-vocab-cli-52k) has roughly 50% of PharmaCoNER annotations
segmented in either one or two tokens. In comparison, mBERT and BETO have less
than 20% of the annotations segmented in that same amount of subwords.

####  7.2.2 Dissecting the F1 score

Finally, we seek a more precise relationship between the evaluation scores and
the model’s over-segmentation, expressed as the average number of subwords per
term. Specifically, for each model and NER task, we group the annotations in
the test split by the number of subwords they are shattered into and then
recalculate the performance scores for each group of annotations. From the
results presented in Figure 1, it can be observed that the F1 score decreases
as the number of subwords increases. On average, the decreasing trend is
exhibited across the three tasks, with a more considerable variation starting
from 7 subwords. The analysis suggests that over-segmentation should be
avoided in order to obtain higher F1 scores, confirming the intuition that a
helpful segmentation preserves terms integrity. Note that, rather than
criticising subword segmentation, we believe that further experiments
unravelling the relationship between the segmentation and the downstream task
performances could guide the design of an optimal biomedical segmentation.

Figure 1: F1 score computed on the subset of annotations from the CANTEMIST, PharmaCoNer and ICTUSnet test sets that are shuttered in a given number of subword units. Model | Mean | Median
---|---|---
bio-cli-vocab-cli-52k | 6.01 | 4
bio-cli-52k | 6.34 | 4
bio-52k | 6.63 | 5
bio-30k | 7.12 | 5
BETO | 8.60 | 6
mBERT | 8.93 | 7
Table 5: Comparison of mean and median number of tokens generated by each model on the PharmaCoNER, CANTEMIST and ICTUSnet annotations. Note that the SWM and WWM variants of the biomedical models are merged under the same names since they are equivalent for the vocabulary analysis. | mBERT | BETO | bio-30k | bio-52k | bio-cli-52k | bio-cli-vocab-cli-52k
---|---|---|---|---|---|---
PharmaCoNER | 1 tok | 2.79% | 3.04% | 7.47% | 15.76% | 18.22% | 24.99%
2 tok | 15.35% | 14.73% | 22.16% | 25.93% | 26.59% | 24.46%
3 tok | 23.18% | 22.36% | 26.55% | 26.55% | 25.44% | 19.82%
| 4 tok | 26.51% | 27.94% | 21.95% | 17.32% | 15.51% | 15.39%
| 5+ tok | 32.17% | 31.93% | 21.87% | 14.44% | 14.24% | 15.34%
CANTEMIST | 1 tok | 0.41% | 0.32% | 1.02% | 1.82% | 1.89% | 2.84%
2 tok | 2.04% | 2.88% | 8.20% | 10.56% | 11.88% | 14.29%
3 tok | 4.18% | 5.04% | 10.70% | 12.79% | 12.40% | 11.63%
| 4 tok | 6.0% | 7.61% | 13.08% | 14.31% | 14.99% | 15.33%
| 5+ tok | 87.37% | 84.15% | 67.0% | 60.52% | 58.84% | 55.91%
ICTUSnet | 1 tok | 1.12% | 1.37% | 1.70% | 2.29% | 3.18% | 6.11%
2 tok | 2.37% | 2.82% | 4.85% | 5.94% | 8.12% | 11.43%
3 tok | 11.36% | 10.21% | 15.37% | 16.38% | 20.90% | 22.43%
| 4 tok | 10.04% | 9.93% | 13.36% | 12.81% | 11.08% | 9.65%
| 5+ tok | 75.11% | 75.67% | 64.72% | 62.58% | 56.72% | 50.4%
Table 6: Percentage of biomedical terms from the PharmaCoNER, CANTEMIST and ICTUSnet annotations that are by each model into one subword up to more than five subwords. Note that the SWM and WWM variants of the biomedical models are merged under the same names since they are equivalent for the vocabulary analysis. Original term | mBERT | BETO | bio-cli-vocab-cli-52k
---|---|---|---
ADN | ADN | ADN | ADN
HBsAg | HB-s-A-g | H-B-s-A-g | HBsAg
antitransglutaminasa | anti-tra-ns-gl-uta-minas-a | anti-trans-gl-uta-minas-a | anti-trans-glutaminasa
clonidina | c-loni-dina | clon-idi-na | clon-idina
diclofenaco | di-clo-fen-aco | dic-lo-fen-aco | diclofenaco
glucagonoma | g-luca-gono-ma | glu-ca-gono-ma | gluca-gono-ma
hemangioblastomas | hem-ang-io-blast-omas | hem-ang-io-blas-tomas | hema-ngio-blas-tomas
hidrocortisona | hi-dro-cor-tis-ona | hidro-cor-tis-ona | hidrocortisona
inmunoglobulina | in-mun-og-lob-ulin-a | inmun-o-glo-bul-ina | inmun-oglobulina
insulina | ins-ulin-a | insulina | insulina
intratumorales | intrat-umo-rales | intra-tum-ora-les | intra-tum-orales
leucemia | le-uce-mia | le-ucemia | leucemia
lidocaína | lid-oca-ína | li-doc-a-ína | lidocaína
linfoma | li-nfo-ma | linf-oma | linfoma
metastasis | meta-stas-is | metas-tasis | metastasis
metilfenidato | met-il-fen-idat-o | metil-fen-ida-to | metil-fenidato
orofaringe | oro-fari-nge | oro-far-ing-e | orofaringe
transaminasas | trans-ami-nas-as | trans-aminas-as | transaminasas
Table 7: Segmentation comparison between mBERT, BETO and our best model bio-
cli-vocab-cli-52k. The terms are extracted randomly from the NER annotations
and are presented in alphabetic order.

##  8 Open Questions

In this section, we open up interesting questions motivated by the evaluation
results and analysis presented in previous sections.

##### Is WWM better than SWM?

The comparison between SWM and WWM pretraining techniques shows, in the case
of the biomedical evaluation (see Table 2), that the impact provided by the
latter is affected by the vocabulary size. In particular, the WWM technique
shows consistent superiority only with a vocab size of 30k. This evidence is
in contrast to the finding, pointed out in [13], that WWM is in general
beneficial. We believe that further ablation studies are necessary to
elucidate the interplay between the vocabulary size and the masking mechanism.

##### Is mixed-domain pretraining beneficial?

The evaluation scores show that the bio-clinical models obtained the best
performances across all tasks. Surprisingly, the bio-clinical model with
clinical vocabulary obtains the highest performance on the CANTEMIST and
ICTUSnet test sets. These results suggest that mixed-domain data might not
always degrade performance, questioning the finding presented in [13], where
they show the negative impact of mixed-domain pretraining with biomedical and
computer science text, as applied in SciBERT [6]. In our case, since we deal
with two distinct but close domains, the biomedical and the clinical ones, we
somehow expect that mixed-domain pretraining could profit by adding more
training data. On the other side, the results of the bio-cli-52k-vocab-cli
model also highlight that, under the same training size conditions, the
vocabulary plays an important role. In general, we believe further experiments
are needed to understand the limitations of mixed-domain pretraining. However,
in our evaluation scenario, we hypothesize the reason behind the encouraging
performances may be related to how much overlap the specific NER data has with
each model’s vocabulary, as partially supported by the results obtained in
Table 4.

##  9 Conclusions and Future Work

In this work, we trained the first biomedical and clinical transformer-based
pretrained language models for Spanish. We then evaluated them on a set of NER
tasks, including a demanding one based on real hospital discharge reports. Our
models overcome two competitive baselines, namely mBERT and BETO, representing
superior solutions for biomedical NLP applications in Spanish. Finally, we
analyzed the results by performing an in-depth analysis involving the model
vocabularies and segmentations. Throughout the work, we outline some
underexplored aspects of language model pretraining, such as the feasibility
of the mixed-domain approach, the effectiveness of cross-domain transfer for
clinical settings and the interplay between the vocabulary size and the token
masking mechanism. Moreover, we showed the impact of different terms
segmentation on the evaluation score, suggesting that over-segmentation can be
detrimental for downstream tasks such as NER. Overall, we experimentally show
that domain-specific pretraining has a positive impact than general-domain
pretraining in a mid-resource scenario.

As future work, we suggest extending the evaluation to other tasks apart from
NER, arguably the most studied task in the biomedical and clinical NLP
literature but not the only relevant one. In addition, according to open
questions raised in Section 8, we will perform more in-depth experiments to
elucidate under which conditions mixed-domain pretraining is advantageous, and
we will investigate the relationship between the vocabulary size and the
masking mechanism.

##  10 Accessibility

We released our best models under the Apache License 2.0 to encourage the
development of Spanish NLP applications in the biomedical and clinical
domains. We uploaded the model in the HuggingFace models hub under the
following links:

  * •

bio-52k-SWM: https://huggingface.co/BSC-TeMU/roberta-base-biomedical-es

  * •

bio-cli-52k: https://huggingface.co/BSC-TeMU/roberta-base-biomedical-clinical-
es

## Acknowledgements

This work was partially funded by the Spanish State Secretariat for
Digitalization and Artificial Intelligence (SEDIA) within the framework of the
Plan-TL and by Fundació La Marató de TV3 under the project 201712.31.

## References

  * [1] M. Cuadros A. García-Pablos, N. Perez.  Vicomtech at cantemist 2020.  In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020), CEUR Workshop Proceedings, 2020.
  * [2] L. Akhtyamova, Paloma Martínez, K. Verspoor, and J. Cardiff.  Testing contextualized word embeddings to improve ner in spanish clinical case narratives.  IEEE Access, 8:164717–164726, 2020.
  * [3] Liliya Akhtyamova.  Named entity recognition in spanish biomedical literature: Short review and bert model.  In 2020 26th Conference of Open Innovations Association (FRUCT), pages 1–7, 2020.
  * [4] Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott.  Publicly available clinical BERT embeddings.  In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72–78, Minneapolis, Minnesota, USA, June 2019. Association for Computational Linguistics.
  * [5] Dogu Araci.  Finbert: Financial sentiment analysis with pre-trained language models.  CoRR, abs/1908.10063, 2019.
  * [6] Iz Beltagy, Kyle Lo, and Arman Cohan.  SciBERT: A pretrained language model for scientific text.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615–3620, Hong Kong, China, November 2019. Association for Computational Linguistics.
  * [7] Casimiro Pio Carrino, Jordi Armengol-Estapé, Ona de Gibert Bonet, Asier Gutiérrez-Fandiño, Aitor Gonzalez-Agirre, Martin Krallinger, and Marta Villegas.  Spanish biomedical crawled corpus: A large, diverse dataset for spanish biomedical language models, 2021.
  * [8] José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge Pérez.  Spanish pre-trained bert model and evaluation data.  In PML4DC at ICLR 2020, 2020.
  * [9] Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos.  LEGAL-BERT: The muppets straight out of law school.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898–2904, Online, November 2020. Association for Computational Linguistics.
  * [10] Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu.  Pre-training with whole word masking for chinese bert.  ArXiv, abs/1906.08101, 2019.
  * [11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [12] Aitor Gonzalez-Agirre, Montserrat Marimon, Ander Intxaurrondo, Obdulia Rabal, Marta Villegas, and Martin Krallinger.  PharmaCoNER: Pharmacological substances, compounds and proteins named entity recognition track.  In Proceedings of The 5th Workshop on BioNLP Open Shared Tasks, pages 1–10, Hong Kong, China, November 2019. Association for Computational Linguistics.
  * [13] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon.  Domain-specific language model pretraining for biomedical natural language processing, 2020.
  * [14] Katikapalli Subramanyam Kalyan, A. Rajasekharan, and S. Sangeetha.  Ammu - a survey of transformer-based biomedical pretrained language models.  ArXiv, abs/2105.00827, 2021.
  * [15] Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  International Conference on Learning Representations, 12 2014.
  * [16] Taku Kudo.  Subword regularization: Improving neural network translation models with multiple subword candidates.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia, July 2018. Association for Computational Linguistics.
  * [17] Jieh-Sheng Lee and Jieh Hsiang.  Patentbert: Patent classification with fine-tuning a pre-trained BERT model.  CoRR, abs/1906.02124, 2019.
  * [18] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  Bioinformatics, 36(4):1234–1240, 09 2019.
  * [19] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized BERT pretraining approach.  CoRR, abs/1907.11692, 2019.
  * [20] A Miranda-Escalada, E Farré, and M Krallinger.  Named entity recognition, concept normalization and clinical coding: Overview of the cantemist track for cancer text mining in spanish, corpus, guidelines, methods and results.  In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020), CEUR Workshop Proceedings, 2020.
  * [21] Aurélie Névéol, H. Dalianis, G. Savova, and Pierre Zweigenbaum.  Clinical natural language processing in languages other than english: opportunities and challenges.  Journal of Biomedical Semantics, 9, 2014.
  * [22] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli.  fairseq: A fast, extensible toolkit for sequence modeling.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48–53, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [23] Rrubaa Panchendrarajan and Aravindh Amaresan.  Bidirectional LSTM-CRF for named entity recognition.  In Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation, Hong Kong, 1–3 December 2018. Association for Computational Linguistics.
  * [24] Yifan Peng, Shankai Yan, and Zhiyong Lu.  Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets.  In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 58–65, Florence, Italy, August 2019. Association for Computational Linguistics.
  * [25] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [26] Erik F Sang and Sabine Buchholz.  Introduction to the conll-2000 shared task: Chunking.  arXiv preprint cs/0009008, 2000.
  * [27] Elisa Terumi Rubel Schneider, João Vitor Andrioli de Souza, Julien Knafou, Lucas Emanuel Silva e Oliveira, Jenny Copara, Yohan Bonescki Gumiel, Lucas Ferro Antunes de Oliveira, Emerson Cabrera Paraiso, Douglas Teodoro, and Cláudia Maria Cabral Moro Barra.  BioBERTpt - a Portuguese neural language model for clinical named entity recognition.  In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pages 65–72, Online, November 2020. Association for Computational Linguistics.
  * [28] Felipe Soares, Marta Villegas, Aitor Gonzalez-Agirre, Martin Krallinger, and Jordi Armengol-Estapé.  Medical word embeddings for Spanish: Development and evaluation.  In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 124–133, Minneapolis, Minnesota, USA, June 2019. Association for Computational Linguistics.
  * [29] Manuel Stoeckel, Wahed Hemati, and Alexander Mehler.  When specialization helps: Using pooled contextualized embeddings to detect chemical and biomedical entities in Spanish.  In Proceedings of The 5th Workshop on BioNLP Open Shared Tasks, pages 11–15, Hong Kong, China, November 2019. Association for Computational Linguistics.
  * [30] Wen Tai, H. T. Kung, Xin Dong, Marcus Comiter, and Chang-Fu Kuo.  exBERT: Extending pre-trained models with domain-specific vocabulary under constrained training resources.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1433–1439, Online, November 2020. Association for Computational Linguistics.
  * [31] Ramya Vunikili, HN Supriya, Vasile George Marica, and Oladimeji Farri.  Clinical ner using spanish bert embeddings.  In IberLEF@ SEPLN, pages 505–511, 2020.
  * [32] Ying Xiong, Yuanhang Huang, Qingcai Chen, Xiaolong Wang, Yuan Nic, and Buzhou Tang.  A joint model for medical named entity recognition and normalization.  In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020), CEUR Workshop Proceedings, 2020.
  * [33] Ying Xiong, Yedan Shen, Yuanhang Huang, Shuai Chen, Buzhou Tang, Xiaolong Wang, Qingcai Chen, Jun Yan, and Yi Zhou.  A deep learning-based system for PharmaCoNER.  In Proceedings of The 5th Workshop on BioNLP Open Shared Tasks, pages 33–37, Hong Kong, China, November 2019. Association for Computational Linguistics.