# Active Coarse-to-Fine Segmentation of Moveable Parts from Real Images

Ruiqi Wang1, Akshay Gadi Patil1, Fenggen Yu1, Hao Zhang1
1Simon Fraser University

###### Abstract

We introduce the first active learning (AL) framework for high-accuracy
instance segmentation of moveable parts from RGB images of real indoor scenes.
As with most human-in-the-loop approaches, the key criterion for success in AL
is to minimize human effort while still attaining high performance. To this
end, we employ a transformer that utilizes a masked-attention mechanism to
supervise the active segmentation. To enhance the network tailored to moveable
parts, we introduce a coarse-to-fine AL approach which first uses an object-
aware masked attention and then a pose-aware one, leveraging the hierarchical
nature of the problem and a correlation between moveable parts and object
poses and interaction directions. Our method achieves close to fully accurate
(96% and higher) segmentation results, with semantic labels, on real images,
with 82% time saving over manual effort, where the training data consists of
only 11.45% annotated real photographs. At last, we contribute a dataset of
2,550 real photographs with annotated moveable parts, demonstrating its
superior quality and diversity over the current best alternatives.

Figure 1: Instance segmentation of moveable parts, with semantic labels, on
photos of real-world scenes. Left shows comparison with OPDFormer, the current
state of the art, where the small red $\times$s indicate erroneous or missed
labels. As an application of accurate moveable part segmentations (lamps on
top right), we can manipulate 3D reconstructions of articulated objects
(Section 7).

##  1 Introduction

Most objects we interact with in our daily lives have dynamic movable parts,
where the part movements reflect how the objects function. Perceptually,
acquiring a visual and actionable understanding of object functionality is a
fundamental task. In recent years, motion perception and functional
understanding of articulated objects have received increasing attention in
computer vision, robotics, and VR/AR applications. Aside from per-pixel or
per-point motion prediction, the segmentation of moveable parts serves as the
basis for many downstream tasks, including robot manipulation, action
planning, and part-based 3D reconstruction.

Figure 2: Overview of our pose-aware masked attention network for moveable
part segmentation of articulated objects in real scene images. Utilizing a
two-stage framework, we first derive a _coarse_ segmentation by predicting the
object mask, its 6 DoF pose, and the interaction direction, subsequently
isolating the interaction surface of the objects. In the _fine_ segmentation
stage, we combine the object mask and interaction surface to form a refined
mask, enabling the extraction of fine-grained instance segmentation of
moveable parts.

In this paper, we tackle the problem of instance segmentation of moveable
parts in one or more articulated objects from RGB images of real indoor
scenes, as shown in Figure 1. Note that we use the term articulated objects in
a somewhat loose sense to refer to all objects whose parts can undergo
motions; such motions can include opening a cabinet door, pulling a
drawer111Strictly speaking, articulated motions are realized by “two or more
sections connected by a flexible joint,” which would not include the drawer
opening motion. However, as has been done in other works in vision and
robotics, we use the term loosely to encompass more general part motions., and
the movements of a lamp arm. Most prior works on motion-related segmentations
[38, 17, 11] operate on point clouds, which are more expensive to capture than
images while having lower resolution, noise, and outliers. Latest advances on
large language models (LLMs) have led to the development of powerful generic
models such as SAM [16], which can excel at generating quality object masks
and exhibit robust zero-shot performance across diverse tasks owing to their
extensive training data. However, these methods remain limited in
comprehensive understanding of moveable object parts.

To our knowledge, OPD [14], for “openable part detection”, and its follow-up,
OPDMulti [28], for “openable part detection for multiple objects”, represent
the state of the art in moveable part segmentation from images. However,
despite the fact that both methods were trained on real object/scene images,
there still remains a large gap between synthetic and real test performances:
roughly 75% vs. 30% in segmentation accuracy [28]. The main reason is that
manual instance segmentation on real images to form ground-truth training data
is too costly. Hence, as a remedy, OPD and OPDMulti both opted to manually
annotate 3D mesh or RGB-D reconstructions from real-world articulated object
scans and project the obtained segmentation masks onto the real images.
Clearly, such an indirect annotation still leaves a gap between rendered
images of digitally reconstructed 3D models and real photographs, with both
reconstruction errors and re-projection errors due to view discrepancies
hindering the annotation quality on images.

To close the aforementioned gap by addressing the annotation challenge, we
present an active learning (AL) [2, 24, 40] approach to obtain high-accuracy
instance segmentation of moveable parts, with semantic labels, directly on
real scene images containing one or more articulated objects. AL is a semi-
supervised learning paradigm, relying on human feedback to continually improve
the performance of a neural segmentation model. As with most human-in-the-loop
approaches, the key criterion for success in AL is to minimize human effort.
To this end, we employ a transformer-based [7] segmentation network that
utilizes a masked-attention mechanism [6]. To enhance the network for moveable
part segmentation, we introduce a coarse-to-fine AL model which first uses an
object-aware masked attention and then a pose-aware one, leveraging the
hierarchical nature of the problem and a correlation between moveable parts
and object poses and interaction directions.

As shown in Figure 2, in the coarse annotation stage, our AL model with
object-aware attention predicts object masks, poses, and interaction
directions, so as to help isolate interaction surfaces on the articulated
objects. In the fine annotation stage, we combine the object masks and
interaction surfaces to predict refined segmentation masks for moveable object
parts, also with human-in-the-loop. Unlike prior works on active segmentation
[36, 29] which mainly focused on the efficiency of human annotations, our
network learns the regions-of-interests (ROIs) from the pose-aware masked-
attention decoder for better segmentation sampling in AL iterations in the
second stage.

In summary, our main contributions include:

  * •

We introduce the first AL framework for instance segmentation, with semantic
labels, of moveable parts from RGB images of real indoor scenes. Our method
achieves close to fully accurate (96% and higher) segmentations, with 82% time
saving over manual effort, where the training data contains only 11.45%
annotated real images.

  * •

Our coarse-to-fine active segmentation model, with both object- and pose-aware
masked-attention mechanisms, lead to reduced human effort and improved
accuracy in moveable part segmentation over state-of-the-art methods,
including OPD [14] and OPDMulti [28].

  * •

Our scalable AL model allows us to accurately annotate a dataset of 2,550 real
photos of articulated objects in indoor scenes. We show the superior quality
and diversity of our new dataset over current alternatives [14, 28], and the
resulting improvements in segmentation accuracy.

##  2 Related Works

#### Articulated objects dataset.

The last few years have seen the development of articulation datasets on 3D
shapes. Of the many, ICON [10] build a dataset of 368 moving joints
corresponding to various parts of 3D shapes from the ShapeNet dataset [5]. The
Shape2Motion dataset [32] provides kinematic motions for 2,240 3D objects
across 45 categories sourced from ShapeNet and 3D Warehouse [1]. The PartNet-
Mobility dataset [35] consists of 2,374 3D objects across 47 categories from
the PartNet dataset [21], providing motion annotations and part segmentation
in 3D.

All these datasets are obtained via manual annotations and are _synthetic_ in
nature. Since sufficient training data is made available by these synthetic
datasets, models trained on them can be used for fine-tuning on _real-world_
3D articulated object datasets with limited annotations.

Recently, OPD [14] and its follow-up work OPDMulti [28], provide two 2D image
datasets of real-world articulated objects: OPDReal and OPDMulti. In OPDReal,
images are obtained from frames of RGB-D scans of indoor scenes containing a
single object. OPDMulti, on the other hand, captures multiple objects. Both
datasets come with 2D segmentation labels on all _openable_ parts along with
their motion parameters. However, due to the nature of annotation process, the
2D part segmentation masks obtained via 3D-to-2D projection do not fully cover
all openable parts in the image. Also, in OPDReal, objects are scanned from
within a limited distance range. Practical scenarios and use cases are likely
going to have large camera pose and distance variations. OPDMulti, on the
other hand, although incorporates such viewpoint variations, a large portion
of this dataset contains frames without any articulated objects [28], which
directly affects model training on OPDMulti.

To overcome these limitations, we contribute a 2D image dataset of moveable
objects present in the real world (furniture stores, offices, homes), captured
using iPhone 12 Pro and 14. We then use our _coarse-to-fine_ AL framework
(Figure 3 and Section 4) to learn generalized 2D segmentations for moveable
object parts.

#### Part segmentation in images.

Early approaches [31, 30, 34] to 2D semantic part segmentation developed
probabilistic models on human and animal images. While not addressing the 2D
semantic part segmentation problem as such, [12, 20, 3, 15, 22] tackled the
problem of estimating 3D articulations from human images, which requires an
understanding of articulated regions in the input image.

Recently, the development of large visual models, such as SAM [16], has
addressed classical 2D vision tasks, such as object segmentation, surpassing
all existing models. Such large pre-trained models can be directly employed
for _zero-shot_ segmentation on new datasets. Follow-up works [43, 8] to SAM
aim at multi-modal learning by generalizing to natural language prompts. For
the task of moveable part segmentation in real scene images, we observe an
unsatisfactory performance using such models. This is expected since they were
never trained on any moveable parts datasets, and therefore, lack an
understanding of articulated objects. To our knowledge, OPDFormer [28] is the
state-of-the-art that can segment moveable parts in an input image, and is
built on the Mask2Former architecture [6]. In our work, we use a transformer
architecture in a _coarse-to-fine_ manner to obtain moveable part segmentation
(Section 4).

#### Active learning for image segmentation.

Active learning (AL) is a well-known technique for improving model performance
with limited labeled data. This, in turn, allows the expansion of labeled
datasets for downstream tasks. Prior works [25, 27, 4, 37, 26] have presented
different AL frameworks to acquire labels with minimum cost for 2D
segmentation tasks. There exist AL algorithms for such tasks [23, 33] that are
specifically designed to reduce the domain gap by aligning two data
distributions. We cannot borrow such methods to reduce the domain gap between
synthetic and real scene images of moveable objects because of the large
feature differences (our synthetic images contain no background, unlike real
scene images).

More recently, [29, 36] employed AL to refine initial 2D segmentation masks
through key point or region selection, requiring little human guidance. Due to
potentially multiple moveable parts, such point/region selection is ambiguous
for articulated objects. As such, we design an AL framework that reduces
manual effort by focusing on: (a) using an improved part segmentation model
(Section 4.1), and (b) employing a _coarse-to-fine_ strategy (Section 4.2).

##  3 Problem Statement

Figure 3: Our coarse-to-fine Active Learning (AL) training pipeline. The
_coarse_ AL applys on interaction directions and retains high-quality
predictions while manually rectifying the rest. These rectified predictions
form a constructive prior for refined mask prediction. Subsequently, the
_fine_ AL stage utilizes these refined masks, employing an iterative training
method with continuous human intervention for accurate part mask annotation.

Given a set of images $D$ captured from the real-world scene, our input is a
single RGB image $I\in D$ containing one or more articulated objects $o_{i}$
from one or more categories
$c_{i}\in\\{\text{cabinet},\text{dishwasher},\text{fridge},\text{microwave},\text{oven},\text{washer}\\}$.
We assume that each object $o_{i}$ has more than one moveable parts
$P=\\{p_{1},\dots,p_{k}\\}$ according to its functionality. Our first goal is
to predict the 2D bounding box $b_{i}$, the segmentation mask $m_{i}$
represented by a 2D polygon and the semantic label
$l_{i}\in\\{\text{door},\text{drawer}\\}$ for each moveable part. Extending
the above goal, we also aim to build a labeled image dataset that provides
accurate 2D segmentation masks and labels for all $p_{i}$’s, for all $I\in D$.

##  4 Method

To address the above problem, we propose an active learning setup that
consists of a transformer-based learning framework coupled with a human-in-
the-loop feedback process. To this end, we present an end-to-end pose-aware
masked-attention network (Fig 2) that works in a _coarse-to-fine_ manner for
part segmentation and label prediction. By making use of _coarse_ and _fine_
features from the network, segmentation masks are further refined by humans in
the AL setup (Fig 3), resulting in precise moveable part masks.

###  4.1 Pose-aware masked-attention network

Fig 2 provides a comprehensive depiction of our network architecture,
encompassing two distinct stages. In the _coarse_ stage, the network processes
a single RGB image and computes a refined mask based on outputs from multiple
heads, which accurately pinpoints the region containing moveable parts. This
stage filters out noise predictions on background and extraneous portions of
the object. Subsequently, _fine_ stage takes the refined mask and image
features to generate part masks, bounding boxes, and semantic labels.

####  _Coarse_ stage.

There are three steps in _coarse_ stage. First, the input image is passed
through a backbone object detector network, producing multi-scale feature maps
$f$ and 2D object bounding boxes $b^{o}$. A pixel decoder [42] upsamples $f$
for subsequent processing in the _fine_ stage. Second, we use a modified
version of the multi-head attention-based encoder and decoder [42] to process
$f$. Inspired by [13], we replace the learned object query embedding with
normalized centre coordinates $(c_{x},c_{y})$, width and height $(w,h)$ from
the detected 2D bounding box, enabling the decoder to generate new object
query embeddings and estimate 6DoF pose from the 2D bounding box. Third, the
decoded queries are passed into multiple MLP heads for (a) object class
prediction, (b) 6DoF object pose estimation, (c) object interaction direction
and (d) object mask prediction.

Table 1: Dataset statistics across six articulated object categories for OPDReal, OPDMulti and our datasets. Microwave and Oven categories are merged due to their co-occurrence in real scenes. Compared to OPDReal, our dataset is relatively more balanced in terms of sample distribution of different categories, allowing segmentation models to generalize better. OPDMulti does not provide category-wise information, and only 19K out of 64K total images are valid with target and annotation. Parts/img shows the average parts annotated for each image. Our dataset exhibits the most object and part diversity among the three datasets.  |  | Category | Total | Parts/img
---|---|---|---|---
|  |  Storage |  Fridge |  Dishwasher |  Micro.&Oven |  Washer |  |
OPDReal[14] |  Objects |  231 |  12 |  3 |  12 |  3 |  284 |  2.22
Images |  27,394 |  1,321 |  186 |  823 |  159 |  30K
image % |  91.67% |  3.93% |  0.62% |  2.75% |  0.53% |  100%
Parts |  787 |  27 |  3 |  13 |  3 |  875
OPDMulti[28] |  Objects |  - |  - |  - |  - |  - |  217 |  1.71
Images |  - |  - |  - |  - |  - |  19K/64K
Parts |  - |  - |  - |  - |  - |  688
Ours |  Objects |  176 |  51 |  31 |  62 |  13 |  333 |  4.33
Images |  925 |  370 |  315 |  775 |  175 |  2550
image % |  36.27% |  14.51% |  12.35% |  30.39% |  6.8% |  100%
Parts |  896 |  159 |  31 |  62 |  13 |  1161

We obtain the object class with fully connected layers. For 6DoF pose
estimation, we use two identical MLP heads with different output dimensions –
one for estimating camera translation
$\tilde{t}=\left(\tilde{t}_{x},\tilde{t}_{y},\tilde{t}_{z}\right)$, and the
other for estimating the camera rotation matrix $\tilde{R}\in SO(3)$ as
described in [41].

The MLP head for interaction direction prediction outputs a set of possible
interaction directions $d\in\\{\pm x,\pm y,\pm z\\}$ corresponding to the 6DoF
coordinates. Using $b^{o}$ and the estimated 6DoF object pose, we can obtain
the corresponding 3D _oriented_ bounding box $B^{o}$, which tightly fits the
$b^{o}$. From among the eight vertices in $B^{o}$, we select vertices of the
face along the interaction direction as the representative 2D box for the
interaction surface, and use it to crop the input image. This cropped image is
further multiplied with the object 2D binary mask to filter out background
pixels, obtaining the refined binary object mask $m_{r}$, which guides the
subsequent _fine_ stage to focus exclusively on the relevant features of the
articulated object.

####  _Fine_ stage.

There is just one component to this stage, which is the masked-attention
decoder from Mask _2_ Former [6] (see Figure 2). It is made up of a cascade of
three identical layers, $L_{i}$’s. $L_{1}$ takes as input image features
$f_{pd}$ and the refined mask, $m_{r}$, and outputs a binary mask which is fed
to the next layer. Eventually, the binary mask at the output of $L_{3}$ is
multiplied with $f_{pd}$ resulting in moveable part segmentation in the RGB
space. We call this our _pose-aware masked-attention decoder_.

#### Loss functions.

We formulate the training loss as below

$L=L_{class}+L_{dir}+L_{om}+L_{pos}+L_{fine}$ (1)

where $L_{class}$ is the binary-cross entropy for object class prediction,
$L_{dir}$ is the cross-entropy loss for interaction direction prediction,
$L_{om}$ is the binary mask loss for object mask prediction. We define the
loss for pose estimation as $L_{pos}=\lambda_{t}L_{t}+\lambda_{rot}L_{rot}$,
where $L_{t}$ is the L2-loss of the translation head and $L_{rot}$ is the
geodesic loss [19] of the rotation head. We set $\lambda_{t}$ and
$\lambda_{rot}$ to 2 and 1 respectively. We use a pixel-wise cross-entropy
loss for the _fine_ stage.
When pre-training, we jointly train our two-stage network in an end-to-end
fashion (see Section 6). During fine-tuning on real images with part
annotations, we fix MLP weights since ground truth poses and object masks are
not available.

###  4.2 Coarse-to-fine active learning strategy

Our active learning setup, consisting of human-in-the-loop feedback, unfolds
in a coarse-to-fine manner (see Figure 3). We independently run AL workflow on
outputs of both _coarse_ and _fine_ stages from Section 4.1.

In _coarse_ AL part, _Coarse_ stage generates predictions for the _test set_.
During this phase, users validate interaction direction predictions and
rectify inaccuracies. With ground-truth interaction directions established,
refined masks $m_{r}$ are computed and input into the _fine_ stage.

In _fine_ AL part, part segmentation mask and label outcome from the _Fine_
stage are subject to user evaluation, categorized as perfect, missed, or fair.
Specifically: i) A perfect prediction implies coverage of all moveable parts
in the final segmentation masks, without any gaps, as well as accurate class
labels for each segmented part; ii) A missed prediction effectively refers to
a null segmentation mask, and/or erroneous class labels; iii) A fair
prediction denotes an output segmentation mask that may exhibit imperfections
such as gaps or rough edges, and/or may have inaccuracies in some part class
labels. We provide extensive examples of these scenarios in our supplements.
During the AL process, perfect predictions are directly incorporated into the
next-iteration training set. For all wrong predictions, we employ the labelme
[39] annotation interface to manually annotate the part mask polygons, and
include such images in the next-iteration training set. Fair predictions, on
the other hand, remain in the _test set_ for re-evaluation.

The AL workflow on the _fine_ stage continues iteratively until all images
within the _test set_ transition to the training set, becoming well-labeled
and eventually leaving the test set vacant. Benefiting from the verified
ground-truth interaction direction established in the _coarse_ AL part, the
_Fine_ stage hones in on features of the target surface, omitting noisy object
parts. This streamlined focus notably expedites the annotation process.
Further insights into the human verification and annotation procedures will be
provided in our supplementary materials.

##  5 Datasets and Metrics

#### Datasets.

We use three kinds of real image datasets in our experiments: (1) OPDReal
[14], (2) OPDMulti [28], and (3) our dataset. Our dataset images are obtained
from the real world by taking photographs of articulated objects in indoor
scenes from furniture stores, offices, and homes, captured using iPhone 11Pro,
12, 12Pro and 14. Images are captured with varying camera poses and distances
from the objects, and an image can contain more than one object, with multiple
moveable parts per object. Differences to OPD and OPDMulti datasets are
explained in Section 2.

We consider six object categories – Storage, Fridge, Dishwasher, Microwave,
Washer, and Oven. A comparison of dataset statistics is presented in Table 1.
OPDReal comprises of $\sim$30K images, with each image depicting a single
articulated object. OPDMulti contains $\sim$64K images. Among these, only 19K
images are considered “valid”, containing at least one articulated object. Our
dataset has a total of 2,550 images, with each image showcasing objects from
several categories. We organize our dataset according to the primary object
depicted in each image. Our dataset stands out by offering the highest
diversity of objects and parts among the compared datasets, including 333
different articulated objects and 1,161 distinct parts.

In terms of the moveable part annotation, both OPDReal and OPDMulti generate
annotations on a 3D mesh reconstructed from the RGB-D scans, and project these
3D annotations back to the 2D image space to get 2D part masks. This process
is prone to reconstruction and projection errors. We, on the other hand,
create annotations on the captured images directly using our _coarse-to-fine_
active learning framework. See the supplementary material for annotation
quality comparisons.

From Table 1, we observe that the majority of data samples in OPDReal belong
to the Storage category (91.67%), with the rest ($<$10%) distributed among the
remaining categories. In contrast, our dataset offers a more uniform data
distribution across all six categories.

#### Metrics.

To evaluate model performance and active learning efficiency, we use the
following metrics:

  * •

Mean Average Precision (mAP): Following OPD [14], we report<EMAIL_ADDRESS>for
correctly predicting the part label and 2D mask segmentation with IoU $\geq$
0.5.

  * •

Annotation time: To evaluate the AL setup, we measure manual annotation _and_
verification times. For methods without AL, we record the time for labeling
only the _test set_ , which is the manual annotation time on network
segmentation predictions.

##  6 Experiments

We start our experiments by rendering synthetic images from PartNet-Mobility
dataset [35] with diverse articulation states, enabling us to obtain
sufficient annotations for training 2D segmentation networks and support
transfer learning applications. The synthetic dataset contains $\sim$32K
images, evenly distributed across categories, and randomly partitioned into
training (90%) and test sets (10%).

We implement our network in PyTorch on two NVIDIA Titan RTX GPUs. All images
are resized to 256$\times$256 for training. During pre-training on the
PartNet-Mobility dataset, we use the Adam optimizer with an initial learning
rate (_lr_) of 2.5$e$-4, reducing it by a factor of $\gamma=$0.1 at 1K and
1.5K epochs separately over a total of 2K epochs. When fine-tuning on real
images, we use the same _lr_ and $\gamma$ at 3.5K and 4K epochs over a total
of 4.5K epochs.

###  6.1 Competing methods

We compare our active coarse-to-fine part segmentation model with three 2D
segmentation methods and also analyze two variants of our proposed approach.

  * •

Grounded-SAM [8], which combines Grounding-DINO [18] and Segment Anything
[16], is a large visual model that can be used for zero-shot 2D object
detection and segmentation, and supports text prompts.

  * •

OPD-C [14], which is the first work for detecting openable parts in images
based on MaskRCNN [9].

  * •

OPDFormer-C [28], a follow-up of OPD-C based on Mask2Former [6], is the state-
of-the-art for openable part segmentation of multiple articulated objects in
images.

  * •

$\text{Ours}_{w/oAL}$ is a variant that does not use human feedback – it
infers part segmentation results based only on the transformer-based model.

  * •

$\text{Ours}_{f-AL}$ is variant of our approach that uses only the _fine_
stage of the AL framework. That is, verification and annotation of just the
part masks is done.

Table 2: Quantitative comparison against competing segmentation methods and variants of our method. In the table, “AL” indicates whether the method uses active learning. “Time” column represents the annotation time metric described in Section 5.  Method | AL |  segm mAP $\uparrow$ |  Time (hr) $\downarrow$
---|---|---|---
Grounded-SAM[8] | - | 22.9 | 35.5
OPD-C[14] | - | 46.4 | 28.3
OPDFormer-C[28] | - | 71.8 | 16.3
$\text{Ours}_{w/oAL}$ | - | 82.1 | 13.4
$\text{Ours}_{f-AL}$ | ✓ | 96.6 | 7.5
Ours | ✓ | 96.7 | 6.5
Table 3: Ablation study on our key components.

Row ID | Obj. | Pos. | Dir. | AL |  segm mAP $\uparrow$ |  Time (hr) $\downarrow$
---|---|---|---|---|---|---
1 | - | - | - | - | 71.8 | 16.3
2 | ✓ | - | - | - | 74.6 | 15.1
3 | ✓ | ✓ | ✓ | - | 82.1 | 13.4
4 | ✓ | - | - | ✓ | 92.4 | 8.1
5 | - | ✓ | ✓ | ✓ | 96.2 | 7.3
6 | ✓ | ✓ | ✓ | ✓ | 96.7 | 6.5

###  6.2 Evaluation on Our Dataset

We split our dataset into 550/2000 as train/test sets and evaluate the
performance of all competing methods on our _test set_. Note that all methods,
except Grounded-SAM, undergo fine-tuning on our _train set_. Figure 6 shows
qualitative results of different methods on our _test set_.

#### Quantitative comparison.

Table 2 compares four non-AL methods. Among these, Grounded-SAM has the lowest
performance due to its direct evaluation on the _test set_ without fine-
tuning. Its poor performance also shows that current large AI models are still
limited in understanding object parts without adequate training on well-
labeled data. Conversely, $\text{Ours}_{w/oAL}$ model, fine-tuned on our
_train set_ , outperforms competing methods, achieving $>$80% segmentation
mAP. The architectural designs of OPD-C and OPDFormer, based respectively on
vanilla MaskRCNN and Mask2Former, are tailored for universal segmentation
tasks. Therefore, they inadequately address the unique characteristics of
articulated objects, whose movable parts are closely tied to object pose and
interaction directions. In contrast, our approach effectively leverages the
hierarchical structure of the scene, object, and part present in the input
image, resulting in a much better performance. Beyond accuracy metrics, we
observe that all non-AL methods require significant human effort for labeling
the _test set_. This is mainly due to imperfect predictions from these models.

As seen in the last two rows of Table 2, using the AL framework, our models
show improved performance, reaching over 95% accuracy, with 81.7% time saving
in terms of manual effort. The segmentation performance of the two AL methods
is close since they share identical network architecture. But they differ in
training strategies, impacting labeling efficiency. Specifically, our method
is 1 hour faster than $\text{Ours}_{f-AL}$, demonstrating the importance of
our _coarse-to-fine_ AL strategy in minimizing human effort.

#### Ablation study.

Table 3 highlights the need and contributions of key components of our method
on improving prediction accuracy and minimizing human efforts. Columns 2-5
respectively indicate the presence of: Obj. object mask head; Pos. pose
estimation head; Dir. interaction direction prediction head; AL active
learning. Row 4 uses the _fine_ AL stage on part mask alone due to the absence
of pose and interaction direction prediction module. Row 5 and 6 use our
_coarse-to-fine_ AL strategy. Results in Table 3 clearly justify the _coarse-
to-fine_ design in our method, which gives the best performance (see row 6).

Table 4: Quantitative comparison against competing segmentation methods and
our model variant on OPDReal, OPDMulti test set.

|  segm mAP ($\uparrow$) |
---|---|---
| Grounded-SAM | OPD-C | OPDFormer-C | $\text{Ours}_{w/oAL}$
OPDReal | 16.5 | 44.5 | 46.3 | 51.6
OPDMulti | 8.0 | 25.6 | 27.6 | 31.5

Figure 4: Qualitative results on OPDReal and OPDMulti test set.
$\text{Ours}_{w/oAL}$ outperforms others on noisy GT and multiple objects.

###  6.3 Evaluation on OPDReal and OPDMulti

In addition, we assess the performance of different models on the OPDReal and
OPDMulti dataset, using their respective train and test splits. As shown in
Table 4, $\text{Ours}_{w/oAL}$ method outperforms the rest. However, with more
than 70% training data, all methods still fail to achieve $>$55% accuracy on
OPDReal. This limitation primarily stems from data skewness towards the
Storage category in OPDReal, which constitutes more than 90% of total samples,
and results in poor generalization across other object categories. Detailed
category-wise results are provided in the supplementary material. Performance
on OPDMulti is further compromised by an abundance of noisy data in its test
set[28]. From the qualitative results in Figure 4, we observe that some
openable parts are cluttered or missed in the GT annotation, while our method
accurately segments these parts. This discrepancy also contributes to the low
accuracy.

Figure 5: Reconstruct and manipulate bottle and dishwasher Figure 6:
Qualitative results on test set from our dataset. We visualize predictions
results on different object categories using 3 competing methods and our final
model. Our method outputs better segmentation masks over moveable parts across
multiple objects in the image with clear separation of parts and small parts
segmentation (Row 1, 4, 5). Our results also show that the _coarse-to-fine_
segmentation framework can effectively reduce segmentation errors from
unwanted objects (Row 2) and object side surfaces (Row 2, 3, 6, 8). More
results in the supplementary materials.

##  7 Application

Our work demonstrate practical applications in part based reconstruction and
manipulation of articulated objects from images. Given a set of multi-view RGB
images of an articulated object, our model predicts precise segmentation masks
of moveable parts in each image. This enables part based 3D reconstruction
using masked images for both moveable parts and the main body of the object.
The resulting 3D models of parts allow for easy manipulation of moveable parts
to unseen states in 3D as shown in Figure 5.

##  8 Conclusion

We present the first active segmentation framework for high-accuracy instance
segmentation of moveable parts in real-world RGB images. Our active learning
framework, integrating human feedback, iteratively refines predictions in a
_coarse-to-fine_ manner, and achieves close-to-error-free performance on the
test set. By leveraging correlations between the scene, objects, and parts, we
demonstrate that our method can achieve state-of-the-art performance on
challenging scenes with multiple cross-categories objects, and significantly
reduce human efforts for dataset preparation.

Additionally, we contribute a high-quality and diverse dataset of articulated
objects in real-world scene, complete with precise moveable part annotations.
We will expand it further to support the vision community for understanding
scene from images. We also hope our work catalyzes future motion- or
functionality-aware vision tasks.

## References

  * tri [2023] Trimble Inc. 3D Warehouse.  https://3dwarehouse.sketchup.com/, 2023.  Accessed: 2023-3-4.
  * Aggarwal et al. [2014] Charu C. Aggarwal, Xiangnan Kong, Quanquan Gu, Jiawei Han, and Philip S. Yu.  Active learning: A survey.  In _Data Classification: Algorithms and Applications_ , pages 571–597. 2014.
  * Ballan et al. [2012] Luca Ballan, Aparna Taneja, Jürgen Gall, Luc Van Gool, and Marc Pollefeys.  Motion capture of hands in action using discriminative salient points.  In _Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VI 12_ , pages 640–653. Springer, 2012.
  * Casanova et al. [2020] Arantxa Casanova, Pedro O Pinheiro, Negar Rostamzadeh, and Christopher J Pal.  Reinforced active learning for image segmentation.  In _International Conference on Learning Representations_ , 2020.
  * Chang et al. [2015] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  _arXiv preprint arXiv:1512.03012_ , 2015.
  * Cheng et al. [2022] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar.  Masked-attention mask transformer for universal image segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1290–1299, 2022.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _ICLR_ , 2021.
  * Grounded-SAM Contributors [2023] Grounded-SAM Contributors.  Grounded-Segment-Anything, 2023.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2961–2969, 2017.
  * Hu et al. [2017] Ruizhen Hu, Wenchao Li, Oliver Van Kaick, Ariel Shamir, Hao Zhang, and Hui Huang.  Learning to predict part mobility from a single static snapshot.  _ACM Transactions on Graphics (TOG)_ , 36(6):1–13, 2017.
  * Huang et al. [2021] Jiahui Huang, He Wang, Tolga Birdal, Minhyuk Sung, Federica Arrigoni, Shi-Min Hu, and Leonidas J Guibas.  Multibodysync: Multi-body segmentation and motion estimation via 3d scan synchronization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7108–7118, 2021.
  * Huang et al. [2020] Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung.  Arch: Animatable reconstruction of clothed humans.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3093–3102, 2020.
  * Jantos et al. [2022] Thomas Jantos, Mohamed Hamdad, Wolfgang Granig, Stephan Weiss, and Jan Steinbrener.  PoET: Pose Estimation Transformer for Single-View, Multi-Object 6D Pose Estimation.  In _6th Annual Conference on Robot Learning (CoRL 2022)_ , 2022.
  * Jiang et al. [2022] Hanxiao Jiang, Yongsen Mao, Manolis Savva, and Angel X Chang.  Opd: Single-view 3d openable part detection.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIX_ , pages 410–426. Springer, 2022.
  * Kanazawa et al. [2018] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7122–7131, 2018.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick.  Segment anything.  _arXiv:2304.02643_ , 2023.
  * Li et al. [2020] Xiaolong Li, He Wang, Li Yi, Leonidas J Guibas, A Lynn Abbott, and Shuran Song.  Category-level articulated object pose estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3706–3715, 2020.
  * Liu et al. [2023] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  _arXiv preprint arXiv:2303.05499_ , 2023.
  * Mahendran et al. [2017] Siddharth Mahendran, Haider Ali, and René Vidal.  3d pose regression using convolutional neural networks.  In _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , pages 2174–2182, 2017.
  * Mehta et al. [2017] Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt.  Vnect: Real-time 3d human pose estimation with a single rgb camera.  _Acm transactions on graphics (tog)_ , 36(4):1–14, 2017.
  * Mo et al. [2019] Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su.  PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Mueller et al. [2018] Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and Christian Theobalt.  GANerated hands for real-time 3d hand tracking from monocular rgb.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 49–59, 2018.
  * Ning et al. [2021] Munan Ning, Donghuan Lu, Dong Wei, Cheng Bian, Chenglang Yuan, Shuang Yu, Kai Ma, and Yefeng Zheng.  Multi-anchor active domain adaptation for semantic segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9112–9122, 2021.
  * Ren et al. [2020] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B. Gupta, Xiaojiang Chen, and Xin Wang.  A survey of deep active learning, 2020.
  * Sener and Savarese [2018] Ozan Sener and Silvio Savarese.  Active learning for convolutional neural networks: A core-set approach.  In _International Conference on Learning Representations_ , 2018.
  * Shin et al. [2021] Gyungin Shin, Weidi Xie, and Samuel Albanie.  All you need are a few pixels: Semantic segmentation with pixelpick.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops_ , pages 1687–1697, 2021.
  * Sinha et al. [2019] Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell.  Variational adversarial active learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019.
  * Sun et al. [2024] Xiaohao Sun, Hanxiao Jiang, Manolis Savva, and Angel X. Chang.  OPDMulti: Openable part detection for multiple objects.  In _Proc. of 3D Vision_ , 2024.
  * Tang et al. [2022] Chufeng Tang, Lingxi Xie, Gang Zhang, Xiaopeng Zhang, Qi Tian, and Xiaolin Hu.  Active pointly-supervised instance segmentation.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVIII_ , pages 606–623. Springer, 2022.
  * Wang and Yuille [2015] Jianyu Wang and Alan L Yuille.  Semantic part segmentation using compositional model combining shape and appearance.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1788–1797, 2015.
  * Wang et al. [2015] Peng Wang, Xiaohui Shen, Zhe Lin, Scott Cohen, Brian Price, and Alan L Yuille.  Joint object and part segmentation using deep learned potentials.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 1573–1581, 2015.
  * Wang et al. [2019] Xiaogang Wang, Bin Zhou, Yahao Shi, Xiaowu Chen, Qinping Zhao, and Kai Xu.  Shape2motion: Joint analysis of motion parts and attributes from 3d shapes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8876–8884, 2019.
  * Wu et al. [2022] Tsung-Han Wu, Yi-Syuan Liou, Shao-Ji Yuan, Hsin-Ying Lee, Tung-I Chen, Kuan-Chih Huang, and Winston H Hsu.  D 2 ada: Dynamic density-aware active domain adaptation for semantic segmentation.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIX_ , pages 449–467. Springer, 2022.
  * Xia et al. [2017] Fangting Xia, Peng Wang, Xianjie Chen, and Alan L Yuille.  Joint multi-person pose estimation and semantic part segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6769–6778, 2017.
  * Xiang et al. [2020] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su.  SAPIEN: A simulated part-based interactive environment.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Xie et al. [2022] Binhui Xie, Longhui Yuan, Shuang Li, Chi Harold Liu, and Xinjing Cheng.  Towards fewer annotations: Active learning via region impurity and prediction uncertainty for domain adaptive semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8068–8078, 2022.
  * Xie et al. [2020] Shuai Xie, Zunlei Feng, Ying Chen, Songtao Sun, Chao Ma, and Mingli Song.  Deal: Difficulty-aware active learning for semantic segmentation.  In _Proceedings of the Asian Conference on Computer Vision (ACCV)_ , 2020.
  * Yan et al. [2019] Zihao Yan, Ruizhen Hu, Xingguang Yan, Luanmin Chen, Oliver Van Kaick, Hao Zhang, and Hui Huang.  Rpm-net: recurrent prediction of motion and parts from point cloud.  _ACM Transactions on Graphics (TOG)_ , 38(6), 2019.
  * Yi et al. [2016] Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas.  A scalable active framework for region annotation in 3d shape collections.  _ACM Transactions on Graphics (ToG)_ , 35(6):1–12, 2016.
  * Zhan et al. [2022] Xueying Zhan, Qingzhong Wang, Kuan-hao Huang, Haoyi Xiong, Dejing Dou, and Antoni B. Chan.  A comparative survey of deep active learning, 2022.
  * Zhou et al. [2019] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li.  On the continuity of rotation representations in neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5745–5753, 2019.
  * Zhu et al. [2020] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  In _International Conference on Learning Representations_ , 2020.
  * Zou et al. [2023] Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee.  Segment everything everywhere all at once.  _arXiv preprint arXiv:2304.06718_ , 2023.


* [7] Brini, A.: The local Gromov-Witten theory of $\mathbb{CP}^{1}$ and integrable hierarchies. Comm. Math. Phys. 313, 571-605 (2012)
  * [8] Brini, A., Carlet, G., Rossi, P.: Integrable hierarchies and the mirror model of local $\mathbb{CP}^{1}$. Phys. D 241, 2156–2167 (2012)
  * [9] Buryak, A., Posthuma, H., Shadrin, S.: A polynomial bracket for the Dubrovin-Zhang hierarchies. J. Differ. Geom. 92, 153-185 (2012)
  * [10] Buryak, A.: Double ramification cycles and integrable hierarchies. Comm. Math. Phys. 336, 1085-1107 (2015)
  * [11] Buryak, A., Guéré, J., Rossi, P.: DR/DZ equivalence conjecture and tautological relations. Geom. Topol. 23, 3537-3600 (2019)
  * [12] Buryak, A., Rossi, P., Shadrin, S.: Towards a bihamiltonian structure for the double ramification hierarchy. Lett. Math. Phys. 111, 13 (2021)
  * [13] Carlet, G.: The extended bigraded Toda hierarchy. J. Phys. A 39, 9411-9435 (2006)
  * [14] Carlet, G., Dubrovin, B., Zhang, Y.: The extended Toda hierarchy. Mosc. Math. J. 4, 313-332 (2004)
  * [15] Carlet, G., Posthuma, H., Shadrin, S.: Deformations of semisimple Poisson pencils of hydrodynamic type are unobstructed. J. Differential Geom. 108, 63–89 (2018)
  * [16] Dubrovin, B.: Integrable systems in topological field theory. Nuclear Phys. B 379, 627-689 (1992)
  * [17] Dubrovin, B.: Integrable systems and classification of 2-dimensional topological field theories. Progr. Math., 115, pp. 313-359. Birkhäuser Boston, Boston, MA (1993)
  * [18] Dubrovin, B.: Geometry of 2D topological field theories. In: Integrable systems and quantum groups (Montecatini Terme, 1993), pp. 120-348, Lecture Notes in Math., 1620, Springer, Berlin (1996)
  * [19] Dubrovin, B.: Painlevé transcendents in two-dimensional topological field theory. In: The Painlevé property, pp. 287-412, CRM Ser. Math. Phys., Springer, New York (1999)
  * [20] Dubrovin, B., Zhang, Y.: Bi-Hamiltonian hierarchies in 2D topological field theory at one-loop approximation. Comm. Math. Phys. 198, 311-361 (1998)
  * [21] Dubrovin, B., Zhang, Y.: Virasoro symmetries of the extended Toda hierarchy. Comm. Math. Phys. 250, 161-193 (2004)
  * [22] Dubrovin, B., Zhang Y.: Normal forms of hierarchies of integrable PDEs, Frobenius manifolds and Gromov-Witten invariants. eprint arXiv: math/0108160 (2001)
  * [23] Dubrovin, B., Liu, S.-Q., Zhang, Y.: On Hamiltonian perturbations of hyperbolic systems of conservation laws. I. Quasi-triviality of bi-Hamiltonian perturbations. Comm. Pure Appl. Math. 59, 559-615 (2006)
  * [24] Dubrovin, B., Liu, S.-Q., Zhang, Y.: Frobenius manifolds and central invariants for the Drinfeld-Sokolov biHamiltonian structures. Adv. Math. 219, 780-837 (2008)
  * [25] Dubrovin, B., Liu, S.-Q., Yang, D., Zhang, Y.: Hodge integrals and tau-symmetric integrable hierarchies of Hamiltonian evolutionary PDEs. Adv. Math. 293, 382-435 (2016)
  * [26] Dubrovin, B., Liu, S.-Q., Yang, D., Zhang, Y.: Hodge-GUE correspondence and the discrete KdV equation. Comm. Math. Phys. 379, 461-490 (2020)
  * [27] Dubrovin, B., Liu, S.-Q., Zhang, Y.: Bihamiltonian cohomologies and integrable hierarchies II: the tau structures. Commun. Math. Phys. 361, 467-524 (2018)
  * [28] Faddeev L., Takhtajan L.: Hamiltonian Methods in the Theory of Solitons, Springer, Berlin (1986)
  * [29] Frenkel, E.: Deformations of the KdV hierarchy and related soliton equations. Internat. Math. Res. Notices 1996, 55-76 (1996)
  * [30] Fan, H., Jarvis, T., Ruan, Y.: The Witten equation, mirror symmetry, and quantum singularity theory. Ann. of Math. (2) 178, 1-106 (2013)
  * [31] Getzler, E.: The Toda conjecture. In: Symplectic geometry and mirror symmetry (Seoul, 2000), pp. 51-79, World Sci. Publ., River Edge, NJ (2001)
  * [32] Getzler, E.: The jet-space of a Frobenius manifold and higher-genus Gromov–Witten invariants. Aspects Math., E36, pp. 45-89, Friedr. Vieweg, Wiesbaden (2004)
  * [33] Givental, A., Milanov, T.: Simple singularities and integrable hierarchies. Progr. Math., 232, pp. 173-201, Birkhäuser Boston, Boston, MA (2005)
  * [34] Hernández Iglesias, F., Shadrin, S.: Bi-Hamiltonian recursion, Liu-Pandharipande relations, and vanishing terms of the second Dubrovin-Zhang bracket. Comm. Math. Phys. 392, 55-87 (2022)
  * [35] Hall, B.: Lie Groups, Lie Algebras, and Representations. An Elementary Introduction. Grad. Texts in Math., 222, 2nd edition, Springer, Cham (2015)
  * [36] Kontsevich, M.: Intersection theory on the moduli space of curves and the matrix Airy function. Comm. Math. Phys. 147, 1-23 (1992)
  * [37] Kuperschmidt, B. A.: Discrete Lax equations and differential-difference calculus. Astérisque, 123 (1985)
  * [38] Liu, Q. P., Wang, Y. Q.: A note on a discrete Schrödinger spectral problem and associated evolution equations. Acta Math. Sci. Ser. A (Chinese Ed.) 26, 773–777 (2006)
  * [39] Liu, S.-Q.: Lecture notes on bihamiltonian structures and their central invariants. In: B-model Gromov-Witten theory, pp. 573-625. Trends Math., Birkhäuser, Cham (2018)
  * [40] Liu, S.-Q., Zhang, Y.: Bihamiltonian cohomologies and integrable hierarchies I: a special case. Commun. Math. Phys. 324, 897–935 (2013)
  * [41] Liu, S.-Q., Qu, H., Zhang, Y.: Generalized Frobenius manifolds with non-flat unity and integrable hierarchies. eprint arXiv:2209.00483 (2022)
  * [42] Liu, S.-Q., Yang, D., Zhang, Y., Zhou, C.: The Hodge-FVH correspondence. J. Reine Angew. Math. 775, 259-300 (2021)
  * [43] Liu, S.-Q., Yang, D., Zhang, Y., Zhou, C.: The loop equation for special cubic Hodge integrals. J. Differ. Geom. 121, 341-368 (2022)
  * [44] Liu, S.-Q., Wang, Z., Zhang, Y.: Linearization of Virasoro symmetries associated with semisimple Frobenius manifolds. eprint arXiv: 2109.01846 (2021)
  * [45] Liu, S.-Q., Zhang, Y., Zhou, C., Fractional Volterra hierarchy. Lett. Math. Phys. 108, 261-283 (2018)
  * [46] Okounkov, A., Pandharipande, R.: Virasoro constraints for target curves. Invent. Math. 163, 47-108 (2006)
  * [47] Shabat, A.: Dressing chains and lattices. In: Proceedings of the workshop on nonlinearity, integrability and all that: twenty years after NEEDS, pp. 331–342. World Sci. Publ., River Edge, NJ (2000)
  * [48] Witten, E.: Two dimensional gravity and intersection theory on moduli space. Surveys in differential geometry (Cambridge, MA, 1990), 243-310, Lehigh Univ., Bethlehem, PA (1991)
  * [49] Zhang, Y.: On the $CP^{1}$ topological sigma model and the Toda lattice hierarchy. J. Geom. Phys. 40, 215-232 (2002)


utf8

# Is it Great or Terrible? Preserving Sentiment in Neural Machine Translation
of Arabic Reviews

Hadeel Saadany
Centre for Translation Studies
University of Surrey, UK
<EMAIL_ADDRESS>
&Constantin Orăsan
Centre for Translation Studies
University of Surrey, UK
<EMAIL_ADDRESS>

###### Abstract

Since the advent of Neural Machine Translation (NMT) approaches there has been
a tremendous improvement in the quality of automatic translation. However, NMT
output still lacks accuracy in some low-resource languages and sometimes makes
major errors that need extensive post-editing. This is particularly noticeable
with texts that do not follow common lexico-grammatical standards, such as
user generated content (UGC). In this paper we investigate the challenges
involved in translating book reviews from Arabic into English, with particular
focus on the errors that lead to incorrect translation of sentiment polarity.
Our study points to the special characteristics of Arabic UGC, examines the
sentiment transfer errors made by Google Translate of Arabic UGC to English,
analyzes why the problem occurs, and proposes an error typology specific of
the translation of Arabic UGC. Our analysis shows that the output of online
translation tools of Arabic UGC can either fail to transfer the sentiment at
all by producing a neutral target text, or completely flips the sentiment
polarity of the target word or phrase and hence delivers a wrong affect
message. We address this problem by fine-tuning an NMT model with respect to
sentiment polarity showing that this approach can significantly help with
correcting sentiment errors detected in the online translation of Arabic UGC.

##  1 Introduction

00footnotetext:  This work is licensed under a Creative Commons Attribution
4.0 International Licence. Licence details:
http://creativecommons.org/licenses/by/4.0/.

Translation of user generated content (UGC) such as user reviews is becoming
common on multilingual websites which sell products and services such as
amazon.com or booking.com. In this context, sentiment preservation in
automatic machine translation (these days usually neural machine translation
(NMT) output) is of great importance because many decisions about purchasing a
product or service are based on the comments made by others. There have been
different studies which explored the transfer of sentiment in MT, but most of
these studies assess how far automatic sentiment classification systems can
capture sentiment information from the translations [Afli et al., 2017, Araujo
et al., 2016, Shalunts et al., 2016]. The objective of most research in this
area is from a sentiment classification perspective rather than a translation
accuracy perspective. Hence, it measures how far automatic translation of a
language into English can help with the sentiment classification of that
language by applying the available English sentiment resources on the target
text [Demirtas and Pechenizkiy, 2013, Barhoumi et al., 2018, Mohammad et al.,
2016, Abdalla and Hirst, 2017].

This study is concerned with NMT accuracy of sentiment transfer at the
word/phrase level and shows that inaccurate translation can transfer a
completely opposite affect message. Moreover, the translation of UGC such as
product reviews constitutes a significant challenge for NMT online tools in
general and for Arabic UGC in particular. The reason is that Arabic UGC is
usually a mix of Dialectical Arabic (DA) and Modern Standard Arabic (MSA)
which differ significantly on the lexico-grammatical level. The same word or
phrase can have opposite sentiment polarities in the two versions of the
Arabic language, which often leads to a mistranslation of the sentiment
message. If the NMT engine is robust enough to handle this type of code-
switching, it can become more reliable not only in downstream NLP tasks such
as cross-lingual information retrieval, but also in real-life scenarios when
Internet users resort to online translation tools to check the reviews of a
particular product of interest. In this study, we assess the degree to which
the NMT online tools transfer sentiment accurately at the word/phrase level
and suggest methods for improving the accuracy of the translation of sentiment
in Arabic UGC. We aim to answer the following questions:

  1. 1.

What type of errors in the output of NMT of Arabic UGC cause problems in
sentiment preservation?

  2. 2.

How can a sentiment sensitive input for an NMT model help with a more accurate
sentiment polarity transfer of Arabic UGC?

  3. 3.

How can sentiment preservation in the target language be measured and whether
the BLEU score is the most appropriate metric for evaluating translation of
sentiment?

To answer the above research questions, this paper is divided as follows:
section 2 presents related work on sentiment transfer in MT. Section 3
analyzes sentiment translation errors of NMT online tools of Arabic reviews
and provides a qualitative typology of most frequent error types. In section
4, we present different methodological approaches for correcting the NMT
online sentiment transfer errors. Section 5 provides task-specific evaluation
metrics for assessing the sentiment accuracy improvement by the proposed
methods. Section 6 presents a conclusion on the different experiments as well
as limitations of the present study.

##  2 Related Work

Research on the translation of sentiment in MT has focused on the idea that
despite significant errors in sentiment transfer, automatic sentiment
classification systems are still able to capture sentiment information from
the translations [Demirtas and Pechenizkiy, 2013, Shalunts et al., 2016,
Mohammad et al., 2016, Barhoumi et al., 2018]. Salameh et al. [Salameh et al.,
2015] showed that although certain attributes of automatically translated text
‘may mislead humans’ with regards to the true sentiment of the source text,
they do not seem to affect the automatic sentiment analysis systems [Salameh
et al., 2015]. The rationale behind these studies is that if we have a good
machine translation model, it will eliminate the necessity to develop
sentiment analysis resources specific of the source language [Afli et al.,
2017]. Given the proliferation of English sentiment analysis tools, we can
always make use of them by conducting sentiment analysis on the English
translation of the source text, even if the translation is not of high quality
[Abdalla and Hirst, 2017, Araujo et al., 2016]. Studies also show that
developed MT models, as well as online translation tools such as Google
Translate and Microsoft Translate, can be relied upon to perform sentiment
classification of the target text despite any accuracy errors [Shalunts et
al., 2016]. This is because sentiment classification systems can learn an
appropriate model even from mistranslated text — especially when automatic
translation makes consistent errors [Salameh et al., 2015]. Moreover,
statistically, studies of sentiment translation have shown that automatic
translation leads to only about 60% match with manually annotated sentiment
labels. Yet, automatic sentiment classifiers can still perform well despite
these errors which can markedly impact human perception of sentiment in the
source tweet/review [Salameh et al., 2015, Mohammad et al., 2016].

Recently, MT studies started to tackle how sentiment can be preserved in the
translation of UGC from a translation accuracy perspective. Bérard et al.
[Bérard et al., 2019] show that back translation of restaurant reviews can
provide significant improvement over existing online systems particularly in
preserving sentiment of translated UGC. They translate a large corpus of
reviews from the target language into English and then use it in model
training. They use domain tags at the training stage to distinguish user-
generated source text. Their results prove that both synthetic data and domain
tags can achieve good results in preserving the affect polarity on the
sentence level. While their model is promising, they still point to serious
errors in the translation of UGC such as missing negations, hallucinations,
unrecognized named entities and insensitivity to context. They suggest that
this task is far from solved [Bérard et al., 2019].

Lohar et al. [Lohar et al., 2018] makes an attempt to improve the sentiment
transfer of translated tweets. They show that freely available translation
tools often cause the sentiment encoded in the original tweet to be altered.
As a consequence, they build separate negative, neutral and positive sentiment
SMT models to improve sentiment preservation in the target language. They show
that a translation model specific of each sentiment pole provides much better
results over a single baseline model trained on the whole twitter data,
regardless of the sentiment class. They attempt to strike a balance between
improving sentiment transfer and preserving translation accuracy as measured
by evaluative metrics such as BLEU and METEOR [Lohar et al., 2018]. A similar
technique is used by Si et al. [Si et al., 2019] as they build a valence
sensitive NMT model for the translation of ambiguous words that can have
different polarities in different contexts. Each input sentence is annotated
with a positive or negative label to indicate its polarity. They show that
adding this tag to the source sentence at the training time and creating dual
polarity embedding vectors for ambiguous words can improve sentiment transfer
at the word level [Si et al., 2019].

There has also been some research on finding alternative means for assessing
the transfer of sentiment in MT other than the typical accuracy metrics.
Bérard et al. [Bérard et al., 2019] show that automatic evaluation metrics
such as BLEU and METEOR tend to neglect sentiment discrepancies between source
and target output. They suggest assessing the accuracy of sentiment
preservation by targeted metrics that measure how well polysemous words are
translated, or how well sentiments expressed in the original text can be
recovered from its translation [Bérard et al., 2019]. To assess sentiment
preservation in MT, Lohar et al. [Lohar et al., 2017] use a sentiment lexicon-
based measure in combination with regular evaluation metrics such as the BLEU
score. Several studies also resort to human evaluation to measure how far a
model improves sentiment transfer at the word/phrase level [Si et al., 2019,
Mohammad et al., 2016].

In this study, we evaluate the preservation of sentiment in translation not as
a sentiment classification task, but from a translation accuracy perspective.
We show that translation inaccuracies at the word/phrase level can seriously
impact the transfer of sentiment in Arabic UGC, which can lead to problems for
users of the MT tools in real-life situations. Several commercial global
platforms rely on publicly available MT engines to translate product reviews
into the customers own language to facilitate communication between partners
and customers111For example, Booking.com uses Google API to translate reviews
on hotels for customers on the fly.. Inaccurate translation of reviewers’
sentiment would defeat the purpose of using such tools. Moreover, in
commercial situations, companies may want to find out what their users think
of particular products so the accuracy of each translation review counts.
Broadly speaking, online tools such as Google Translate, are commonly utilized
as an off-the-shelf solution for the translation of UGC in Arabic as well as
in other languages. Error-analysis of sentiment translation by online tools,
however, has proved that the true sentiment of Arabic reviews can be either
missed or flipped to its exact opposite pole.

##  3 Error Analysis

In order to measure how accurately NMT online tools transfer sentiment of
Arabic UGC, we chose a dataset of book reviews scraped from
Goodreads222https://www.goodreads.com [Aly and Atiya, 2013]. Each review has a
rating between 1-5 assigned by its author. The language of the reviews is a
mix of MSA and DA, with the largest majority of DA reviews in the Egyptian
dialect. Reviews in the dataset are of varying lengths, but a large number of
them have over than 100 tokens. Long reviews were split to a maximum of 20
tokens per review. After splitting, the data amounted to about 230,000
sentences. This dataset was translated into English using the Google Translate
API and was analysed using both manual and automatic error analysis, focusing
on mistranslation of sentiment. Automatic sentiment analysis tools were
utilized to detect sentiment errors in the dataset and subsequently select a
sample for manual error analysis.

Since the main objective is to assess the accuracy of sentiment translation at
the word level, we used an automated lexicon-based sentiment measure on the
Google Translate output. We applied the cloud-based Microsoft Azure Text
Analytics tools for sentiment analysis 333Microsoft Azure Text Analytics on
around 13,000 target sentences. The Azure’s Sentiment Analysis API generates
sentiment scores using classification features such as n-gram sentiment
scores, part-of-speech tags and word embeddings. It evaluates text and returns
a label (positive, neutral, negative) for each sentence as well as numeric
confidence scores that range from 0 to 1 for each sentiment category. Scores
closer to 1 indicate a higher confidence in the label’s classification, while
lower scores indicate lower confidence. For each sentence, the predicted
scores associated with the labels (positive, negative and neutral) add up to
1. Following traditional methods in sentiment classification [Pang et al.,
2002], we used the rating of the book review as indicative of its sentiment
polarity and compared it to the confidence scores generated by the Azure
Sentiment Analysis API. Accordingly, reviews were categorized based on
discrepancies between the ratings and the confidence scores. A positive review
that had a rating of 4 or above and an English negative sentiment score of 0.5
and above was extracted as an example of potential wrong negative polarity in
the target text. Similarly, reviews with negative ratings of 2 and below and a
positive English sentiment score of 0.5 and above were extracted as instances
of potential wrong positive polarity in the target text. This amounted to a
total of around 4,000 potentially negative sentiment errors and around 2,000
of potentially positive sentiment errors.

A sample of reviews of 1000 parallel sentences from the dataset that had
discrepancies between the automatic sentiment score and the review rating were
manually analyzed to detect reasons for these discrepancies. By analyzing the
causes of mistranslation of sentiment in this sample, the mistakes were
categorized into a five group typology. The typology of sentiment translation
errors are summarized in the following sections. One or two examples for each
type of errors will be mentioned in the following sections. The table in
appendix A gives more examples of each type.

###  3.1 Contronyms

Manual analysis of the data revealed that the first type of errors which
distorts the reviewer’s affect message is mistranslations of contronyms. These
are words used both in DA and MSA which can have the exact opposite sentiment
polarity in each of the two language varieties or in the same variety but in
different contexts. For example, the word ‘رهيبه ¿’ means ‘terrible’ in MSA,
but in DA it often means ‘great’. This word was frequently mistranslated as
‘terrible’ in the reviews dataset, causing a distortion of the sentiment of
the source text. For example, the review ‘ الروايه رهيبه عيبها الوحيد الجزء
الاخير ¿ ’ is translated as “The narration is terrible, its only flaw is the
last part”. The correct translation, however, is ‘The novel is great, its only
flaw is the last part’. Even when the infrequent positive use of the contronym
is used in Arabic MSA context it is flipped to a negative pole in the
translation. For example, in the review ‘ثم قال هذه الكلمه الرهيبه اقرأ ¿’
(then he said this magnificent word: Read) the word ‘رهيبه ¿’ is used
positively to mean ‘great’ or ‘magnificent’. The automatic translation,
however, flips it to the more common negative sense by translating the review
as ‘then he said this terrible word: Read’. Similarly, the negation of these
contronyms is often mistranslated and hence alters the sentiment message of
the source text. For example, the low-rated review ‘ ادب الكاتب مش الفظيع ¿’
(the writer’s literature is not that great) has the negated contronym ‘
الفظيع¿ ’ which can either mean ‘not terrible’ or ‘not great’ in MSA and DA
respectively. The review was mistranslated as ‘the writer’s literature is not
terrible’ which had a positive sentiment score whereas the original review had
a low rating.

Another example is the word ‘ جامد ¿’. In DA, it means ‘great’ or ‘awesome,’
whilst in MSA it refers to its literal meaning, i.e. ‘rigid’. Reviews stating
‘ كتاب جامد جدا¿ ’ (a very good book) were constantly mistranslated as ‘a very
rigid book’ which incorrectly reflected a negative sentiment score. A list of
contronyms that caused sentiment inconsistencies between source and target
text was identified by the manual analysis of the sample dataset and extracted
from the larger dataset of the Goodreads reviews (see appendix A for more
examples of this type of error).

###  3.2 Diacritic Errors

The vowels in the Arabic language are realized by diacritics which indicate
the pronunciation of the word. The same word can have different meanings based
on the diacritic marks assigned, since a change in a diacritic is a change of
a vowel sound. Arabic UGC is usually lacking diacritics since Arabic native
speakers can easily guess which diacritic mark is intended based on the
context of the word. Automatic translation, however, often fails to realize
the different meanings of words if diacritics are missing and this can lead to
a wrong sentiment polarity. For example, ‘ من اظرف ما قرات¿’ (One of the
nicest things I’ve read) is translated as ‘The envelope of what I read’. This
is because the word ‘ اظرف¿’ can either mean ‘the nicest’ or ‘most
entertaining’ if it has a ‘fatha’ (a short /a/) on the third letter or
‘envelopes’ if it has ‘Damma’ (a short /u/ as in “you”) on the same letter.
Moreover, absence of diacritics causes a confusion between the transitive and
intransitive use of sentiment adjectives. For example, the adjective ‘ متعبه
¿’ can either mean ‘tired’ if the diacritic ‘fatha’ (a short /a/) is on the
third letter or it can mean ‘tiring’ if the diacritic ‘kasrah’ (short /i/) is
on the same letter. Thus, for example, a book review with a positive rating
starting with ‘ متعبه هذه الروايه¿’ is mistranslated as ‘Tired of this
narration’. The correct translation of the adjective is ‘This novel is tiring’
where the reviewer is referring to the intellectual depth of the novel.
Diacritic errors as such cause a misinterpretation of reviewer’s sentiment
stance. More examples are given in Appendix A.

###  3.3 Idiomatic Expressions

Idiomatic expressions both in MSA and DA are consistently mistranslated in the
dataset which leads to a complete miss of the sentiment message in the review.
For example, the MSA phrase ‘خفيف الظل ¿’ is an idiom used to describe a
‘funny’ animate or inanimate noun. The idiom in the positive review ‘ كتاب
خفيف الظل ¿ ’ (a funny book) is mistranslated as ‘a light-shaded book’. The
target text incorrectly reflects a neutral sentiment rather than the correct
positive one. This idiom’s counterpart in DA ‘دمه خفيف ¿’ (funny) is also
constantly mistranslated in the dataset. The review ‘كتاب دمه خفيف جدا ¿’ (
the book is very funny) is mistranslated as ‘his blood book is very light’.
The manual analysis of the dataset set showed that, generally speaking,
idioms, either in MSA or DA, constituted a challenge to the online automatic
translation tool. A large number of idioms were literally translated which did
not only affect the sentiment preservation of the source text, but often
produced nonsensical target text. For example, the MSA phrase ‘ وهل يخفى
القمر¿’ is an idiom used to describe something that is unquestionably
commended by the speaker. If the idiom is used in reference to a book, a good
human translation would be: ‘It really shines through’. The Google Translate
gives a literal translation – ‘ Is the moon hidden?’– which flips the
sentiment polarity of the review from highly positive to neutral. (See
Appendix A for more examples).

###  3.4 Dialectical Expressions

Research studies have shown that dialectical Arabic presents several
challenges to MT in general [Zbib et al., 2012]. It was also observed from the
manual analysis of the sample data that dialectical expressions constituted a
special challenge for the preservation of sentiment in the source text. Arabic
UGC is acceptably written in DA or MSA or a mix of both in the same text. A
large number of DA sentiment expressions were either completely missed in the
translation or mistranslated. For example, positive adjectives such as ‘هايل
¿’ (great), or negative adjectives such as ‘ عبيط ¿’ (silly) were mostly
mistaken for proper nouns and transliterated into non-English words (Hayel,
Abit). In some instances, the translation was a complete opposite of the
intended affect message (e.g. ‘ من الجمل العبيطه المنتشره ¿’ (one of the
widespread silly sentences) was translated as ‘one of the popular sentences
spread ’ (see more example in Appendix A).

###  3.5 Negation

Another type of sentiment errors that is also associated with the use of DA in
Arabic UGC is the mistranslation of DA negation markers. Different Arabic
dialects often treat negative particles as clitics, and hence a letter is
added to the stem of the word to change it to negative [Mohamed et al., 2012].
The majority of DA in the dataset belongs to the Egyptian dialect where
negation is realized by the morpheme ‘مـِش ¿’ (mish) which is either placed in
front of the verb or preposition, or wrapped around it [Soltan, 2017]. From
the analysis, it was found out that the translation frequently either misses
the negation and hence flips the phrase to the opposite sentiment pole or
mistranslates the negated phrase all together. For example, in the review ‘
معجبنيش ان بطل الروايه ضعيف الشخصيه ¿’ (I didn’t like that the protagonist of
the novel has a weak character) the negation is missed and hence the online
translation output is ‘I admire that the protagonist of the novel is weak in
character’. There are several similar instances where the mistranslations of
the DA negative structure switches the sentiment to its opposite pole (see
more examples in appendix A).

##  4 Notes on Error Typology

In order to get an indication of the frequency of each type of errors in the
whole dataset, words/phrases belonging to each group were extracted based on
their frequency in the corpus then the frequency of their mistranslation in
the dataset was manually calculated. Figure 1 shows the frequency of the
mistranslation instances in the total reviews dataset of the extracted words
as representative of each type of errors. As can be seen almost all the
instances of the most frequent idiomatic and dialectical expressions are
mistranslated. Moreover, around 65% of the time the positive meaning of
contronyms was flipped to negative. Consequently, there were sentimentally
incongruous terms where a positive noun was described with a highly improbable
negative adjective (e.g. horrible achievement, terrible masterpiece, terrible
happiness, and so on).

Our error typology showed that more than one type of errors is due to code-
switching between DA and MSA in Arabic UGC. There have been several approaches
to tackle the challenges of translating Arabic DA such as paraphrasing source
text in MSA before translation [Salloum and Habash, 2013]. Other studies have
also proved that concatenating small amount of Arabic dialectical data can
significantly improve the translation quality [Zbib et al., 2012]. However,
our error analysis has shown that even in MSA context the sentiment of
words/phrases can be mistranslated. Pivoting on MSA can solve straightforward
problems such as dialectical phrases and dialectical idioms. However,
preserving the sentiment of the source text would require addressing the
polarity of words with opposite meanings such as contronyms. Since it is
beyond the scope of this paper to tackle all the error types, an attempt was
made to address the problem of contronyms in Arabic UGC. In this paper we
propose a sentiment-sensitive NMT model that is robust to the opposite
sentiment polarities of Arabic contronyms either due to code-switching between
DA and MSA or to contextual variations in Arabic UGC. Details of the
experiment are explained in the following sections.

Figure 1: Frequency of Error Types

##  5 Sentiment Oriented NMT System

In order to improve the translation of contronyms in Arabic UGC, we propose
two transformer (NMT) models infused with sentiment information at the
encoding stage. We show that training on a sentiment oriented small-sized data
can provide high performance results in preserving the sentiment of
challenging contronyms in Arabic UGC. Details of data preprocessing and model
architectures are explained in the following sections.

###  5.1 Parallel Data Preparation and Preprocessing

It is worth mentioning here that the available authentic parallel
English/Arabic data is mostly English to Arabic data (e.g. UN parallel
corpora, TEDx scripts, and Tatoeba project [Alotaibi, 2017, Ho and Simon,
2016]). The greatest part of this data is in Arabic MSA and is not sentiment-
oriented. Authentic Arabic(DA)-English parallel data in general and authentic
Arabic(UGC)-English parallel data in particular is very scarce. Recently, the
use of synthetic corpora in NMT led to promising results especially when
authentic parallel data is scarce [Chinea-Rios et al., 2017, Cheng et al.,
2020]. Moreover, infusing contextual cues in the input layer has proved
successful in improving the robustness of the NMT models for different
translation tasks even with relatively small-sized datasets [Johnson et al.,
2017, Pal et al., 2014, Si et al., 2019]. Accordingly, in order to identify
the correct sentiment polarity of contronyms in Arabic UGC, we opted for using
the synthetic parallel data of the Goodreads reviews dataset ($\approx
230,000$ sentences) for model training but with three main modifications.
First, all the mistranslation instances of the chosen list of contronyms were
manually post-edited (see appendix C for a list of most frequent contronyms
used in the dataset). Second, the Arabic script underwent a number of
preprocessing operations such as the normalization of orthographic letter
forms, deletion of elongation and extra spaces. This has significantly reduced
the number of out-of-vocabulary words. Third, we manually tagged all the
contronyms in the source text with the right sentiment polarity according to
its context. We experimented with both the tagged and the untagged post-edited
source text. Details of the model architectures are in the following section.

###  5.2 NMT System Setup

In order to explore how we can improve the translation quality, we constructed
three NMT models. The first is a baseline model that takes an untagged post-
edited source text as input. The baseline is a seq2seq model with an LSTM of
200 hidden states for the encoder and decoder models trained with global
attention. The other two models are sentiment sensitive models that take a
tagged source text as input. For the two sentiment sensitive models, we
mimicked the Google Translate setup [Vaswani et al., 2017] by using a
transformer for both the encoding and decoding layers with 8 heads of self-
attention and with an inner feed-forward layer of size 2048, but reduced the
number of training steps from 200k to 100k. We used the Adam optimizer with
$\beta 1=0.9$, $\beta 2=0.98$ and $\epsilon=10^{9}$ and the Google set up
special learning rate as described by Vaswani et al. [Vaswani et al., 2017].
The first of the two sentiment sensitive models was initialized with random
input vectors. For the second model, we created a vector space model (VSM) of
the tagged source dataset where each contronym was given two distinct vectors
according to its tagged sentiment polarity. A bag of words Word2Vec model was
used to create the pretrained vectors of the source text [Řehůřek and Sojka,
2010]. It was trained with a hierarchical softmax and a window size of 5
tokens. The pretrained word embeddings were used to initialize the second
transformer model with the same parameters used for the first. All experiments
were run using OpenNMT [Klein et al., 2017].

###  5.3 Evaluation results

The evaluation of the proposed models was conducted on two test sets. The
first was a held-out set from the Goodreads reviews ($\approx 47,000$ parallel
sentences). The second was a hand-crafted test set of 140 sentences where we
used the list of extracted contronyms with their positive and negative
sentiment connotations in an equal number of sentences and code-switched
between Arabic (MSA) and Arabic (DA) either in the same sentence or among
different sentences. A reference translation was created by manually
translating the hand-crafted set by a native speaker. In order to adequately
evaluate the performance of the models in preserving the polarity of
contronyms in the source text, we conducted two types of sentiment
evaluations, at the word level and at the sentence level on the held-out test
set and the hand-crafted test set respectively. We compared the quality
measures on both the sentence and word levels of the proposed models with the
Google Translate output for the test set and the hand-crafted test set. The
BLEU score was also used as a metric to assess how far the quality of the
translation is balanced with the preservation of sentiment by our proposed
models. Details of the experiment evaluations are explained in the next
sections.

####  5.3.1 Quality level

The first evaluation metric conducted on the two datasets was based on the
BLEU score. We used the metric-internal multi-detokenized BLEU [Sennrich et
al., 2015]. The BLEU score was used to check that the quality of the
translation is not distorted while fine-tuning the models for sentiment
preservation. Results in table 1 show that both transformer models with tagged
source text outperform the baseline on the two datasets. The first transformer
model with tagged input, but without pretrained vectors, achieves the highest
BLEU score on the test set and the negative hand-crafted test set with scores
38.77 and 44.83 respectively. The second transformer model trained on tagged
source and pre-trained sentiment-oriented vectors achieves the best BLEU score
on the positive hand-crafted test set (38.77). Results indicate that the
overall translation quality as measured by BLEU has not been impaired with the
sentiment-preservation approaches of the proposed transformer models.

| Sentence Level | Word Level | BLEU
---|---|---|---
| Hand-Crafted Set | Test Set | Test Set | Hand-Crafted Set
Model | Positive | Negative | Precision | Recall | F1 |  | Positive | Negative
Seq2seq (no tagging) | .24 | .44 | 0.60 | 0.52 | 0.55 | 33.9 | 31.48 | 36.94
Transformer 1 (tagging) | .14 | .21 | 0.74 | 0.65 | 0.69 | 38.77 | 37.56 | 44.83
Transformer 2 (tagging and pre-trained) | .06 | .14 | 0.85 | 0.79 | 0.81 | 37.14 | 38.82 | 42.06
Google Translate | .71 | .15 | 0.80 | .06 | .12 |  |  |

Table 1: Results of Three Evaluation Metrics for Assessing Sentiment
Preservation in Translation

####  5.3.2 Word-level Sentiment Evaluation

The BLEU score can reflect the translation quality of the NMT output, but for
the present study it would not be appropriate to capture how the opposite
sentiments of contronyms are correctly translated. This is because the BLEU
score does not give a penalty to a mistranslated sentiment lexicon that is
adequately proportional to the distortion of the sentiment message. The
translation of the right sentiment polarity of a contronym can be pivotal in
transferring the affect message of the source text. For example, the positive
use of the contronym ‘رهيبه ¿’ in the low-rated book review ‘ليست تحفه ابداعيه
رهيبه ¿’ (not a great creative masterpiece) is mistranslated as ‘not a
terrible creative masterpiece’ by the baseline model. The mistranslation of
the contronym completely distorts the sentiment message of the review,
however, the BLEU score for this mistranslation is around 76.

Accordingly, we measured the precision, recall and F1 score of the different
models to assess their ability to correctly predict the true positive and true
negative polarity of the contronyms in the test dataset. Table 1 shows that
the baseline model was not able to detect the correct sentiment orientation of
a contronym with high accuracy, as compared to the two transformer models,
despite the post-editing of the training dataset. Feeding in correct instances
was not sufficient to improve the sentiment preservation of Arabic contronyms.
Infusing linguistic information at the training stage, however, improved
sentiment accuracy. Moreover, the low F1 score of the Google Translate (.12)
was due to the fact that it was able to translate correctly instances of
contronyms when used with negative sentiment, but failed to translate those
where their positive meaning is used. Such positive cases constituted around
40% of the instances of contronyms in the dataset. This is because the
negative meaning of contronyms is more frequent in Arabic MSA, whereas the
positive is used more in Arabic DA context. As explained by the error
typology, Google Translate performs far better with MSA than DA. On the other
hand, the second transformer model which is trained on sentiment-sensitive
pretrained vectors and tagged source text achieved best performance in
depicting the true sentiment at the word level with an F1 score of .81 and a
precision score of .85. The sentiment-sensitive pretrained vectors of
contronyms and their polarity tagging with the second transformer model
significantly helped in translating the correct sentiment at the word-level.

####  5.3.3 Sentence-level Sentiment Evaluation

The second metric for evaluating the translation of sentiment in Arabic UGC
was carried out on the hand-crafted test set. In order to assess how the
correct or incorrect translation of contronyms affects the total sentiment
message of the source sentence, we propose a sentiment-score based metric. We
compute the distance between the sentiment score of the reference sentence and
the model output to measure not only how far the model preserves the sentiment
of a contronym, but also the effect of translation on the sentiment context.
We use the sentiment scoring methods used for error analysis, i.e. Microsoft
Azure Sentiment Analysis scoring. We measure a translation cost as the mean
square distance to the reference score:

$\displaystyle\mu_{C}=\frac{1}{N}\sum_{i=1}^{N}(s_{t}-s_{r})^{2}$ (1)

where $s_{t}$ is the score of the target sentence, $s_{r}$ is the score of the
reference translation, and $N$ is the number of sentences.

As seen from table 1 the second transformer model trained on tagged contronyms
and pretrained word vectors performed best (i.e. with the lowest cost) for
both the positive and negative reviews (.06, .14 respectively). It was not
only more sensitive to different polarities of contronyms due to the code-
switching between MSA and DA, but produced the lowest sentiment discrepancy
with the sentiment scores of the reference sentence. It is also worth noting
that Google Translate performed much better with the negative sense of
contronyms than the positive sense. This is in line with the findings
presented in the previous section. With negative contronyms, Google Translate
and the second transformer model had the lowest costs of .15 and .14,
respectively. However, Google Translate produced the highest discrepancy with
the positive instances (.71). Moreover, it was observed that the cost score
was highest with short sentences. For instance, the positive sense of the
contronym ‘ رهيبه ¿’ (awesome, great) in the short reference sentence ‘ رهيبه
بكل المقاييس¿’ (By all means awesome) is translated by Google Translate as
‘Terrible by all accounts’. In such cases, the sentiment cost was maximum. It
is evident that if a similar distortion of sentiment messages occurs in real-
life situations, it would have adverse effects on the reviewers judgement.
Examples of the output of the second transformer model (Trans2) as compared to
the reference translation (Ref) and Google Translate is given in Appendix B.

##  6 Conclusion

This study has shown that Arabic UGC has peculiar qualities which constitute a
challenge to automatic translation tools especially in its ability to preserve
the sentiment message. An error typology was derived after analysing the data.
This typology has highlighted how sentiment errors can impair the translation
of sentiment-oriented Arabic UGC such as product reviews. Since automatic
online translation tools are heavily relied upon by users and commercial
platforms to translate reviews, it is of essential importance to fine-tune NMT
models to the correct sentiment message in the source text. Moreover, it has
been common practice for NMT training to use big parallel data which involves
very high computational power and requires availability of large authentic
data. The proposed NMT models in this study, however, showed that infusing
contextual cues at the training stage of a relatively small data can improve
the translation of sentiment in Arabic UGC both on the word and sentence
level. This approach can help in providing greener training and make it
feasible to construct competitive NMT tools for low-resource domains such as
Arabic UGC. Moreover, we showed that translation quality metrics of sentiment-
oriented Arabic UGC needs to be supplemented with other metrics that assess
the preservation of sentiment. We proposed lexicon-based metrics that take
into account the sentiment score of single words as well as their context.
Finally, this study has tackled one of several challenges in the translation
of sentiment in Arabic UGC. Future research will address the whole spectrum of
challenges to improve the accuracy of sentiment preservation which is of vital
importance in the translation of Arabic UGC.

## References

  * [Abdalla and Hirst, 2017]  Mohamed Abdalla and Graeme Hirst.  2017\.  Cross-lingual sentiment analysis without (good) translation.  arXiv preprint arXiv:1707.01626.
  * [Afli et al., 2017]  Haithem Afli, Sorcha Maguire, and Andy Way.  2017\.  Sentiment translation for low resourced languages: Experiments on Irish general election tweets.  In 18th International Conference on Computational Linguistics and Intelligent Text Processing, Budapest, Hungry, pages 17–21.
  * [Alotaibi, 2017]  Hind M Alotaibi.  2017\.  Arabic-English parallel corpus: a new resource for translation training and language teaching.  Arab World English Journal (AWEJ) Volume, 8.
  * [Aly and Atiya, 2013]  Mohamed Aly and Amir Atiya.  2013\.  Labr: A large scale Arabic book reviews dataset.  In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 494–498.
  * [Araujo et al., 2016]  Matheus Araujo, Julio Reis, Adriano Pereira, and Fabricio Benevenuto.  2016\.  An evaluation of machine translation for multilingual sentence-level sentiment analysis.  In Proceedings of the 31st Annual ACM Symposium on Applied Computing, pages 1140–1145.
  * [Barhoumi et al., 2018]  Amira Barhoumi, Chafik Aloulou, Nathalie Camelin, Yannick Estève, and Lamia Belguith.  2018\.  Arabic sentiment analysis: an empirical study of machine translation’s impact.  In Proceedings of the second Conference on Language Processing and Knowledge Management Kerkennah (Sfax), Tunisia.
  * [Bérard et al., 2019]  Alexandre Bérard, Ioan Calapodescu, Marc Dymetman, Claude Roux, Jean-Luc Meunier, and Vassilina Nikoulina.  2019\.  Machine translation of restaurant reviews: New corpus for domain adaptation and robustness.  arXiv preprint arXiv:1910.14589.
  * [Cheng et al., 2020]  Shanbo Cheng, Shaohui Kuang, Rongxiang Weng, Heng Yu, Changfeng Zhu, and Weihua Luo.  2020\.  Auto-repair the synthetic data for neural machine translation.  arXiv preprint arXiv:2004.02196.
  * [Chinea-Rios et al., 2017]  Mara Chinea-Rios, Alvaro Peris, and Francisco Casacuberta.  2017\.  Adapting neural machine translation with parallel synthetic data.  In Proceedings of the Second Conference on Machine Translation, pages 138–147.
  * [Demirtas and Pechenizkiy, 2013]  Erkin Demirtas and Mykola Pechenizkiy.  2013\.  Cross-lingual polarity detection with machine translation.  In Proceedings of the Second International Workshop on Issues of Sentiment Discovery and Opinion Mining, pages 1–8.
  * [Ho and Simon, 2016]  Trang Ho and Allan Simon.  2016\.  Tatoeba: Collection of sentences and translations.  http://www.manythings.org/anki/.
  * [Johnson et al., 2017]  Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al.  2017\.  Google’s multilingual neural machine translation system: Enabling zero-shot translation.  Transactions of the Association for Computational Linguistics, 5:339–351.
  * [Klein et al., 2017]  Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush.  2017\.  OpenNMT: Open-Source Toolkit for Neural Machine Translation.  In Proc. ACL.
  * [Lohar et al., 2017]  Pintu Lohar, Haithem Afli, and Andy Way.  2017\.  Maintaining sentiment polarity in translation of user-generated content.  The Prague Bulletin of Mathematical Linguistics, 108(1):73–84.
  * [Lohar et al., 2018]  Pintu Lohar, Haithem Afli, and Andy Way.  2018\.  Balancing translation quality and sentiment preservation (non-archival extended abstract).  In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers), pages 81–88.
  * [Mohamed et al., 2012]  Emad Mohamed, Behrang Mohit, and Kemal Oflazer.  2012\.  Transforming standard Arabic to colloquial Arabic.  In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 176–180.
  * [Mohammad et al., 2016]  Saif M Mohammad, Mohammad Salameh, and Svetlana Kiritchenko.  2016\.  How translation alters sentiment.  Journal of Artificial Intelligence Research, 55:95–130.
  * [Pal et al., 2014]  Santanu Pal, Braja Gopal Patra, Dipankar Das, Sudip Kumar Naskar, Sivaji Bandyopadhyay, and Josef van Genabith.  2014\.  How sentiment analysis can help machine translation.  In Proceedings of the 11th International Conference on Natural Language Processing, pages 89–94.
  * [Pang et al., 2002]  Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan.  2002\.  Thumbs up? sentiment classification using machine learning techniques.  arXiv preprint cs/0205070.
  * [Řehůřek and Sojka, 2010]  Radim Řehůřek and Petr Sojka.  2010\.  Software Framework for Topic Modelling with Large Corpora.  In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45–50, Valletta, Malta, May. ELRA.  http://is.muni.cz/publication/884893/en.
  * [Salameh et al., 2015]  Mohammad Salameh, Saif Mohammad, and Svetlana Kiritchenko.  2015\.  Sentiment after translation: A case-study on Arabic social media posts.  In Proceedings of the 2015 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 767–777.
  * [Salloum and Habash, 2013]  Wael Salloum and Nizar Habash.  2013\.  Dialectal Arabic to English machine translation: Pivoting through modern standard Arabic.  In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 348–358.
  * [Sennrich et al., 2015]  Rico Sennrich, Barry Haddow, and Alexandra Birch.  2015\.  Neural machine translation of rare words with subword units.  arXiv preprint arXiv:1508.07909.
  * [Shalunts et al., 2016]  Gayane Shalunts, Gerhard Backfried, and Nicolas Commeignes.  2016\.  The impact of machine translation on sentiment analysis.  Data Analytics, 63:51–56.
  * [Si et al., 2019]  Chenglei Si, Kui Wu, Aiti Aw, and Min-Yen Kan.  2019\.  Sentiment aware neural machine translation.  In Proceedings of the 6th Workshop on Asian Translation, pages 200–206.
  * [Soltan, 2017]  Usama Soltan.  2017\.  The fine structure of the Neg-domain: evidence from Cairene Egyptian Arabic sentential negation.  Florida Linguistics Papers, 4(3).
  * [Vaswani et al., 2017]  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  2017\.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008.
  * [Zbib et al., 2012]  Rabih Zbib, Erika Malchiodi, Jacob Devlin, David Stallard, Spyros Matsoukas, Richard Schwartz, John Makhoul, Omar Zaidan, and Chris Callison-Burch.  2012\.  Machine translation of Arabic dialects.  In Proceedings of the 2012 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 49–59.

##  Appendix A Examples of the Error Typology for Sentiment Translation of
Arabic UGC

##  Appendix B Examples of Translation Models Output

##  Appendix C List of Most Frequent Contronyms